Caveats and Performance

What to know before relying on tether in a pipeline: the limits of the model, the limits of each backend, and what the commands cost.

Limits of the model

No cross-system atomicity

A commit records each object’s state within a capture window; point-consistency requires quiesced writers. The Neon quiescence check helps (Neon), but nothing coordinates two systems. promote lands system by system, and a failure part-way leaves the earlier ones landed (Landing it). Promotion is per system too: a system with no merge refuses a diverged base.

Concurrent writers on one branch

Two writers on one store branch (a --shared bookmark, two clones on one bookmark) are ordered by a compare-and-swap only where the backend has one: git, and Icechunk on s3://. Icechunk on local storage and Lance check the head just before moving it, which leaves a window (conditional moves). Keep such writers on one checkout, or give each its own bookmark.

Native handles do not follow new

Every open follows the bookmark the checkout is on now, but a handle already open keeps writing to the branch it was opened on. A notebook that opened a handle before another process ran tether new -b writes to the old branch. Open the object again after a new.

Icechunk never reuses a deleted tag name

A tether.* tag deleted by hand, or released by a gc you then regret, cannot be recreated under its name: Icechunk keeps a tombstone. tether keeps the pin id and moves the ref on instead. repair, or the next commit of that state, creates tether.<id>.2, then .3, and readers resolve a pin through its live generation. So nothing is lost while the store keeps the snapshot. There are a thousand generations per id.

Pin-then-commit ordering

Pins are created before the VCS commit. If anything after them fails (a manifest write, a listing, the VCS commit), commit releases the pins it created and puts the manifests back. It never releases a pin an earlier commit made and this one reused. If the process dies outright, its journal entry stays INCOMPLETE, and gc from any checkout keeps the pins that entry names; re-running the commit finishes it. One thing is never undone: a dataset commit that landed. If the adapter fails after writing it, history already names the pins, so the commit stands and the error is reported. Re-run tether verify after merging manifests across branches.

One writer per checkout, one per repository

Every writing command takes an advisory lock on the checkout (.tether/lock). commit, pull, new, restore, gc, drop, abandon, undo and repair, and any lazy fork a writable open performs, also take tether.lock in the VCS’s shared directory, which every checkout of the repository shares. So a gc here cannot race a commit in another workspace, and two checkouts cannot both find a branch absent and fork over each other. A second command waits, then gives up with a message naming the lock (30 s for the checkout, 60 s for the repository). Threads sharing one Repo take turns the same way. Separate clones share no lock.

Windows

Windows has no fcntl, so nothing can order two processes. Commands that write are refused there with one message, and a default open is read-only. status, verify, diff, log, ops and gc --dry-run work. Run the writing commands from a POSIX system, or WSL, until the lock has a Windows implementation. Nothing has run on Windows in CI yet.

Data history costs storage

Every pin holds bytes in its system until the commit that names it is dropped and gc releases it. --pin record skips the native ref and forks from the recorded state, at the price of retention-bound recoverability. Most store writes can be previewed as plans.

A clone is untrusted input

Nothing committed may choose an executable, an endpoint, SQL, or where a secret is read from. What the boundary does not cover: a manifest’s locator still names the store tether reads and writes, so a hostile clone can point an object at a bucket you can reach. Review manifests as you would review code (Security model).

Per backend

Neon

Experimental: tested against a fake of the Neon API, not a live project. This section is the one home for Neon’s caveats.

Lineage. Pins are child branches, so a working branch cannot be deleted while its pins exist. gc --prune-bookmarks plans it as keep-branch: cannot be deleted and names the pins, even under --force-prune. gc only drops leaf pins. new onto a different pin forks a sibling (<name>.2, .3, …) rather than reset a branch with children, while commit then new reuses the branch, since it sits at the pin. Neon cannot merge or promote a child into main: run production writes on the trunk bookmark.

A branch per commit. Each commit that changes a Neon object adds a branch. Neon plans cap branches per project (10 on Free) or bill those past the allowance. gc releases pins no commit references, and --pin record takes none at all. Pins are unprotected unless protected_pins = true in secrets.toml. Neon will not delete a protected branch, and only paid plans have a few, one per pinning commit. Branches are the storage (BRANCH_IS_STORAGE): gc --prune-bookmarks never deletes a Neon working branch holding unpinned data without --force-prune.

Control plane. Answers of 423 (another operation is running), 429 and 503 are retried with backoff for about a minute. commit refuses while pg_stat_activity shows active writers, unless --force. Objects that fingerprint one branch at once share one endpoint, created once.

Fingerprint cost. The LSN moves without user writes, so it is a volatile address key. Only commit_xid, the newest committed transaction id, and the lineage count as change; ids a restarted compute skipped never commit, so they do not count. A snapshot may wake a suspended compute, and creates the read-write endpoint on a branch with none. A plain status does not fingerprint.

Content is {commit_xid, lineage}. branch in a Neon state is the lineage: the object’s source branch, when the branch read descends from it. So a fork with no writes compares equal to the pin it was cut from. The price: two sibling branches of one lineage whose newest commits share a transaction id compare equal too. Neon exposes no content hash or history query to tell them apart, which is why gc never judges a Neon branch by its state alone, and promote is not offered.

Files on unversioned storage

Observed: detectably but not recoverably drifted. --file versioned on a versioned bucket makes a single remote object Addressable; local paths and prefixes stay Observed. A local path’s identity is its resolved form, so re-pointing a symlink makes a new store: its pins stay safe, but its old listings are not found.

Iceberg catalog variance

S3 Tables disables maintenance when user refs exist. Use --pin record there, so forks come from the snapshot id, valid until snapshot expiry.

Delta has no refs

Versions are recorded, not pinned. VACUUM (7-day default) and log retention (30 days) bound how long open -r works. A recreated table (new metadata id) verifies as drifted, not re-addressed.

Lance versions are branch-scoped

A branch deleted and created again restarts its version numbers, so a state from its earlier life verifies as missing rather than opening the new one. A pinned working branch outlives its bookmark until gc drops the tag, since Lance refuses to delete a branch a tag references; gc reports it as a failed delete-branch. A conditional fork onto an existing branch compares, then deletes and creates it again, so a peer’s branch created in that window can be replaced.

DuckLake attachments

A DuckDB-file metadata catalog can be attached only once per process. So tether attaches per operation, and a DuckLakeHandle owns its attachment: close() it (or use it as a context manager) before the next status, and do not hold your own writer connection open in the same process. Postgres, SQLite and MySQL catalogs have no such limit. Snapshots are catalog-wide, and table only scopes the handle. Put CREATE SECRET statements in [backends.ducklake] init_sql rather than credentials in the locator.

DuckLake and DuckDB versions

A DuckDB-file metadata catalog that a newer DuckDB has written can be unreadable to an older one. DuckDB 2.0 creates catalog files in a storage format 1.x cannot read, and its ducklake extension moves a 1.x file onto that format at its first write, even while the catalog stays on the DuckLake 1.0 spec. The extension sets the storage version itself, so DuckDB’s storage-compatibility settings do not prevent it. From then on tether, like every other reader on DuckDB 1.x, fails to attach the catalog, and its error says to upgrade duckdb. Upgrade readers, the environment tether runs in included, before writers. Postgres, SQLite and MySQL catalogs are not DuckDB files, and are unaffected.

Dolt working sets

Uncommitted changes on a branch are in no commit: a dirty branch is reported and refused at commit. Credentials come only from the server’s [uris."mysql://host:port"] entry in .tether/secrets.toml, and a server with none gets none. Dirty detection needs a Dolt with the dirty column on dolt_branches; older servers report clean.

git remotes

With remote set, a pin is a pin only once the tag push succeeds: a failed push fails the pin and removes the local tag, and unpin deletes on the remote first. A committed remote must name a configured remote; a URL goes in .tether/secrets.toml as [objects."<key>"] remote.

Performance

tether is an orchestrator: its cost is the network round trips it fans out, not Python.

Each command starts a few git or jj processes

On local objects, a command’s time is mostly git and jj. A status starts 6 git or 8 jj processes, at 4-6 ms per git call and 7-15 ms per jj call (jj loads the repository each time, and most calls snapshot the working copy), on top of about 75 ms to start Python. benchmarks/run.py in the repository measures each command.

Fan-out uses threads, not asyncio

Every backend SDK tether wraps (obstore, psycopg, pyiceberg, deltalake, lance, duckdb, pymysql, git) is synchronous, and the per-object Python work is microseconds. A thread pool gives the same wall-clock concurrency without forcing an async protocol on backend authors. snapshot, verify and the fork step of new all run concurrently. Credentials for one profile or role are resolved once, however many threads ask.

History walks are batched

gc, verify --all-history, repair --all-history and upgrade stream every manifest at every commit through one git cat-file --batch process (jj repositories are git-backed). Trees and blobs are cached by object id, and each distinct (system, state, pin) is verified once. On a repository with 200 commits and 5 objects this takes ~20-40 ms, against 6-12 s with one process per read. History rewrites (upgrade, abandon) read and transform only the commits that touch the dataset directory.

Opens and refreshes pay for what changed

Every open re-reads the workspace and the manifests, but a file whose size, mtime, ctime and inode are unchanged costs a stat, not a parse. A changed workspace.toml also costs one VCS call, to check it is still untracked.

Fingerprints are metadata-only, except the first local one

A remote object is one HEAD, a prefix one paged LIST, and the etag is the store’s content hash. Local files are sha256-hashed, so timestamps do not count as change. The hash is cached under .tether/cache/ by (size, mtime_ns, ctime_ns, inode); ctime, because an overwrite that restores the mtime cannot restore it. A file changed within two seconds of the last read is read again. The first fingerprint of a tree reads every byte (~1.5-3 GB/s per core, eight files at a time); every later one is an os.scandir walk (~4-20 us per file) plus the files whose stat changed.

Large local trees and the native scanner

With tether-vcs-native installed, a local directory’s walk, hash cache and hashing run in Rust, by the same rules and against the same cache file, so either scanner picks up where the other left off. A warm status --snapshot of 100,000 files takes 0.15 s instead of 1.8 s, and of a million 2.1 s instead of 21, in about a quarter of the memory; a first scan is 3-4× faster. It is built for Linux and macOS; elsewhere, and with TETHER_NATIVE=0, the Python scanner does the work. Until it is on PyPI, build it from the repository with uv sync --group native, which needs a Rust toolchain.

Content diffs are metadata-only too

Each backend reads manifests, logs or catalog tables, never data. So diff --content costs about one fingerprint per changed object: two manifest reads for Icechunk and Lance, one log walk for Delta, one catalog query for DuckLake, one prolly-tree diff for Dolt. Two stored file listings of 100k paths diff in about 0.2 s, or 40 ms with the native scanner.