Backends

One section per backend: its extra, tier and flags, the locator fields it reads, its [backends.<kind>] config, its state, how pins and forks map onto the system, its handle, and what diff --content reports. Credentials are never stored in manifests.

Kind	Extra	Maturity	Tier	Flags
file	– (local) / objectstore	stable	Observed; Addressable for a versioned remote object	CHEAP_FINGERPRINT, DIFF, CREATE (local directory)
icechunk	icechunk	stable	Forkable	ATOMIC_REF, DIFF, HISTORY, PROMOTE, CREATE; CONDITIONAL_REF on s3://
git	–	stable	Forkable	CHEAP_FINGERPRINT, ATOMIC_REF, CONDITIONAL_REF, DIFF, HISTORY, PROMOTE, MERGE, CREATE
lance	lance	stable	Forkable	ATOMIC_REF, DIFF, HISTORY
delta	delta	stable	Addressable	CHEAP_FINGERPRINT, RETENTION_BOUND, DIFF, HISTORY
neon	neon	experimental	Forkable	NEEDS_QUIESCENCE, RETENTION_BOUND, BRANCH_IS_STORAGE
iceberg	iceberg	experimental	Forkable	RETENTION_BOUND, DIFF, HISTORY, PROMOTE
ducklake	ducklake	experimental	Addressable	CHEAP_FINGERPRINT, RETENTION_BOUND, DIFF, HISTORY
dolt	dolt	experimental	Forkable	ATOMIC_REF, DIFF, HISTORY, PROMOTE, MERGE
memory	–	stable	Forkable	CHEAP_FINGERPRINT, ATOMIC_REF, CONDITIONAL_REF, DIFF, HISTORY, PROMOTE, MERGE, CREATE (tests only)

Maturity is what the backend declares (MATURITY), and tether backends prints it. Stable means the full lifecycle (register, commit, fork, write, commit, promote, gc) runs in CI against the real system: embedded stores and libraries, on local storage. The S3, GCS and Azure paths of file and icechunk are not yet cloud-tested; ROADMAP.md tracks that. Experimental means tested against a fake of a network service, not the service itself, and add says so. The tiers and flags are defined in Writing a Backend.

Conventions

• branch in a locator is the upstream branch (default main), which the trunk bookmark stands for. Other bookmarks fork tether.ws.<dataset8>.<bookmark> from the pin.
• Pins are named tether.<pin_id>, where pin_id is <dataset8>.<hash16>: the dataset’s id, then a content address of kind, identity and state. Backends treat the id as opaque.
• identity is the subset of the locator that names the system. Two objects of one dataset with the same identity and state share one pin, which commit cuts once; the second object is recorded at the state the pin was taken at.
• A local path’s identity is its resolved form: symlinks followed, ., .., doubled and trailing slashes gone, and file:// folded. So /p, /p/, file:///p and a path through a symlinked parent are one store to pin ids, listings and gc. Re-pointing a symlink therefore changes the store’s identity: its pins stay safe (gc never releases a pin any manifest names), but listings stored under the old identity are not found.
• Locators are committed and read from every directory and clone, so a relative local path is resolved against the directory add or import ran in, and stored absolute; a leading ~ is expanded. That covers file uri/path, git path, the icechunk, lance and delta uri, and DuckLake’s data_path and the path in metadata (after ducklake:, duckdb: or sqlite:). URLs and absolute paths are stored as given.
• Objects in one branch scope (two databases of a Neon project, two keys on one Icechunk store) share a branch per bookmark, and gc collects their references per scope.
• at is an object’s initial position (tether add --at / --pick), pinned by the first commit until a trunk pull moves the object onto the branch. What each system accepts, and what tether log lists:

Kind	at accepts	log lists
icechunk	snapshot id, tag, branch	Repository.ancestry from the ref: id, written_at, message
lance	version number (on the base branch) or tag	versions() of the branch, newest first, tags on each
iceberg	snapshot id or ref name	snapshot ancestry with operation and record counts
delta	version number	transaction log with operation and metrics
ducklake	snapshot id	ducklake_snapshots with each snapshot’s changes
dolt	commit hash, tag, or branch (HASHOF)	DOLT_LOG(ref)
git	any commit-ish (alias of ref)	git log with decorations
neon, file	not supported	–

What the flags mean in practice

PROMOTE and MERGE. PROMOTE fast-forwards the base branch to a fork (Icechunk reset_branch, Iceberg set-snapshot-ref, git update-ref or merge --ff-only, Dolt merges). MERGE three-way merges a diverged fork (git merge, DOLT_MERGE). Lance cannot move a branch head yet, and Neon cannot promote a child branch, so both are refused with instructions. Where Icechunk and Lance stand upstream on branch merge, and what blocks it, is a dated survey in docs/upstream-branch-merge.md.

CONDITIONAL_REF means a fork, promote or merge moves the ref only from the head the plan reviewed, in one atomic step. The others compare just before the move (conditional moves).

CREATE (experimental) is what tether add --create needs (Stores tether created). Lance, Delta and Iceberg tables need a schema, so they cannot be created empty. Neon, Dolt and DuckLake objects live inside a project, database or catalog tether does not own. Make those with their own tools and add them. --pin record removes PIN from any backend (Lever 1).

State keys: content and address

A state’s keys do two jobs:

• Content keys say what the data is: an Icechunk snapshot id, a Neon commit_xid, a git sha. Equality, pin ids and listing names use only these.
• Address keys say where to find it, and can move without the data changing: Neon’s lsn, and git’s change_id and dirty. Backends list them in VOLATILE_KEYS; open, pin and verify get the whole state.

Content diffs

tether diff A B compares manifests. --content also asks each changed object’s backend for its native diff, from metadata only, concurrently:

Kind	Unit	Source	Example detail
file (dir / prefix)	files	two stored listings (.tether/listings/)	modified a.tif +4.0 KiB
file (object)	objects	recorded size / etag / version	etag a -> b, version v1 -> v2
git	files	git diff --name-status + --numstat	modified main.py +1 -1
icechunk	nodes	Repository.diff(from, to)	modified /x 2 chunks
iceberg	snapshots	snapshot summaries along b’s ancestry to a	append: +5 rows, +1 files
delta	commits	transaction-log history() in (a, b]	WRITE: +2 rows, +1 files
lance	fragments	fragment ids / deletions / schema at both versions	added fragment 1 +1 rows
ducklake	tables	snapshot changes + ducklake_table_changes	modified main.t +5000 rows, -1 rows
dolt	tables	dolt_diff_summary + dolt_diff_stat	modified t +6 rows
neon	–	no native diff

A directory or prefix state is only a digest, so at commit time the file backend also hands the engine a per-file listing. It is stored content-addressed under .tether/listings/ and committed with the manifests, like DVC’s .dir files. diff --content compares two listings file by file, or falls back to a count and size summary. Entries are capped at 2000 per object; the counts are exact.

file: local paths and object stores

Plain artifacts: a local file or directory, or an object / prefix in S3 (s3://), GCS (gs://), or Azure Blob (az://, abfs://), through obstore.

Locator	uri (positional; path accepted). A trailing / (or empty key) means a prefix. region overrides the store region.
Config	storage_options = {...} passed to obstore.store.from_url. Committed: only region, allow_http, virtual_hosted_style_request and conditional_put; endpoints, account names and credentials only in .tether/secrets.toml. HTTP client options (allow_http, timeouts, proxies, …) work in any spelling there (AWS_ALLOW_HTTP, aws_allow_http, or inside client_options); a number is a duration in seconds, and an unknown key raises ConfigError. Otherwise credentials come from the environment or instance metadata.
Identity	{uri}, a local path in its resolved form (Conventions)
State	local file: type=file, size, sha256; local dir: type=dir, count, size, digest (digest over per-file sha256s); remote object: type=object, size, etag[, version_id]; prefix: type=prefix, count, size, digest (one paged LIST, etag+size per key). Local hashes are cached by (size, mtime_ns, ctime_ns, inode) in .tether/cache/file-hashes.json, so only files whose stat changed are re-read
Tier	Observed by default. --file versioned on a single remote object makes it Addressable via version_id (S3 version id / GCS generation / Azure version id; requires versioning on the bucket/container – a recorded state without a version id is refused by open). Local paths and prefixes are always Observed.
Pin / fork	none
Handle	FileHandle(uri, version_id) – always read-only
Listing	directories and prefixes hand the engine a per-file JSONL listing ({"p": path, "k": token, "s": size}) that is stored under .tether/listings/
Diff	files: per-path added/removed/modified with size deltas from two stored listings; falls back to a count/size summary when a listing is missing; single objects report changed size/etag/sha256/version
Create	a local directory only (mkdir plus a .tether-owner marker file the directory walk skips, so the directory fingerprints as empty until something is written). Ref-empty means nothing but the marker; delete_store is an rmdir that refuses anything else. Remote prefixes are not created or removed by tether.
Notes	--file immutable (default) turns any drift into an ImmutableObjectModified error at status; drift means the bytes changed, not the timestamp. A symlink to a file is read through. A symlink to a directory, or a dangling one, is recorded by its target: retargeting it is a change, but edits under a linked directory are not seen. Fingerprint cost: Performance.

icechunk: Icechunk repositories

Locator	uri (local path or s3://bucket/prefix), branch (default main), region (S3)
Config	none committed. .tether/secrets.toml may set profile, role_arn, literal keys, endpoint_url and region per URI prefix or object (two identities); otherwise S3 credentials come from the environment (from_env). A profile or role becomes a get_credentials callback that refreshes the keys, so a long-lived session does not expire mid-write; an icechunk without the callback gets static keys, rebuilt when they rotate. For an S3-compatible server (SeaweedFS, MinIO), allow_http = true and force_path_style = true in the same entry reach every Icechunk and obstore call, add --create and delete_store included.
Identity	{uri}, a local path in its resolved form
State	snapshot_id of the branch head
Pin	create_tag("tether.<id>", snapshot_id); tags are immutable, and the default expire_snapshots keeps a tagged snapshot. expire_snapshots(delete_expired_tags=True) deletes the tag and the snapshot with it; verify then reports the pin missing
Fork	create_branch(name, snapshot_id) off the tag, or reset_branch if it exists. A conditional move uses reset_branch(from_snapshot_id=) (icechunk 2.x), which is atomic on object stores only: on local filesystem storage racing processes can each win
Handle	IcechunkHandle(repository, session, branch, tag, snapshot_id); session is writable for a branch, read-only for a tag/snapshot
Diff	nodes: Repository.diff(from, to) – new/deleted groups and arrays, updated metadata, updated chunk counts per array, moves
Create	Repository.create, refused if anything exists at the path or under the prefix. The owner marker is the tether.owner key in the repository’s metadata; an icechunk without repository metadata (2.2.0 works) lacks CREATE. Ref-empty: no branch but the base, no tag beyond the pins the plan names, and the base still at the first snapshot. delete_store re-checks owner and emptiness, then removes only Icechunk’s own top-level names; anything else there is refused and kept.

neon: Neon Postgres

Experimental: tested against a fake of the Neon API, not a live project.

Locator	project_id, database, role (all required, since Neon builds no connection URI without them), branch (default main)
Config	api_key_env (default NEON_API_KEY), api_url (default Neon API v2), protected_pins (default false; see Caveats: Neon), in .tether/secrets.toml only
Identity	{project_id, branch}: a Neon branch at an LSN is a snapshot of the whole project, so database and role do not change pin ids. Two objects on two databases of one project share one pin per commit, and one working branch per bookmark
State	lsn, commit_xid, branch, timeline, probed over SQL. Content is commit_xid (the newest committed transaction id) and branch (the lineage); lsn and timeline (the branch the LSN was read on) are volatile. With no commit among the last 65536 ids, commit_xid reads <N, so a later commit still shows as a change. What this means for equality: Caveats: Neon
Pin	a child branch tether.<id> of the state’s timeline at its LSN (so a fork’s pin hangs off the fork, not main), protected only with protected_pins = true; RETENTION_BOUND by the project’s history retention
Fork	a child branch of the pin. Re-forking an existing working branch restores it onto the source. Once a pin hangs off the working branch Neon will not restore it in place, so the fork lands on a sibling (<name>.2, .3, …) while the old branch stays the pins’ parent, and the engine records the returned name. A branch tether reads or writes gets one read_write endpoint, created once however many objects ask at the same time; pins and time-travel reads get a read_only one, and each connection URI names its endpoint
Quiescence	commit refuses while pg_stat_activity shows active writers (--force skips)
Handle	NeonHandle(url, branch) – a Postgres connection URL
Diff	not supported (no native primitive)
Notes	pins are branches, and branches are the storage: what that means for gc, new, promote, cost and fingerprinting is in Caveats: Neon.

git: git and jj repositories

Pin the code (or any git-tracked data) that produced a dataset. Colocated jj repositories work through their .git.

Locator	path (local checkout, stored absolute, and outside the dataset’s own checkout; the positional locator works too: tether add code --kind git ../code), ref (default HEAD), remote (push pins there: the name of a remote the repository has configured; a URL or path only as [objects."<key>"] remote in .tether/secrets.toml), allow_dirty (pin despite uncommitted changes)
Config	git_path, jj_path – .tether/secrets.toml [vcs] or TETHER_GIT/TETHER_JJ only; a committed value is refused
Identity	{path} (resolved absolute)
State	sha; dirty (only meaningful – and only computed – for the checked-out ref); change_id when jj is available. Both dirty and change_id are volatile: the same sha is the same state – and pins identically – whether or not jj is installed or the checkout has stray files; pin still reads the full state and refuses a dirty tree
Pin	lightweight tag tether.<id> at the sha. With remote set the pin is pushed and is only a pin once the push succeeds: a failed push fails the pin and removes the local tag; unpin deletes on the remote first and fails if that fails, so local and remote never diverge silently
Fork	branch name at the tag’s sha. A conditional move is update-ref with the old value, atomic; a branch checked out in any worktree is refused
Handle	GitHandle(path, sha, worktree)
Diff	files: git diff --name-status -M with --numstat line counts (+a -d)
Create	git init -b <ref> plus one empty root commit with a fixed tree, author and dates, so every created store has the same root sha (GitBackend.EMPTY_ROOT_SHA); the owner marker is git config tether.owner. Ref-empty: no branch or tag beyond the base, the base at that root sha, and a clean working tree, since an untracked, staged or ignored file is data. delete_store re-checks both before it removes the directory.
Notes	remote-only (url) repositories are not supported yet. Every git call runs with core.fsmonitor, hooks and ext:: transports off and safe.bareRepository=explicit, so a repository’s config runs nothing; git 2.38 is required. promote and merge move a branch. With the default ref = HEAD that is the checked-out branch, and a detached HEAD (a colocated jj checkout always has one) is refused: set ref to the branch name. A merge, or a promote onto a checked-out base, compares the head and then merges in the checkout, so a commit landing in between is merged or refused, never discarded.

iceberg: Apache Iceberg tables

Experimental: driven through a fake of the pyiceberg table API; not yet run against a catalog service.

Locator	identifier (ns.table; table accepted), branch (default main), catalog_name (default default), catalog (dict of pyiceberg catalog properties, or set once in config)
Config	catalog = {...} pyiceberg properties. Committed (tether.toml or the locator): exactly type, warehouse and name; every other property – uri, token, credential, … – only in .tether/secrets.toml ([backends.iceberg] catalog or per identifier)
Identity	{identifier, branch, catalog_name}
State	snapshot_id (nothing else: metadata_location changes on every table commit on any branch). A new table with no snapshot yet has snapshot_id = -1, Iceberg’s “no current snapshot”
Pin	tag tether.<id>; with --pin record, the snapshot id alone, recoverable within snapshot retention (for S3 Tables and other catalogs where user refs disable maintenance). The empty state needs no tag: its pin verifies as held and opens at snapshot_id = -1, which pyiceberg’s scans refuse rather than read today’s data
Fork	create_branch(name, snapshot_id); an existing name is re-pointed (set-snapshot-ref). A table with no snapshot cannot be forked, so write it on the trunk first. Needs pyiceberg 0.11 or newer
Handle	IcebergHandle(table, ref, snapshot_id)
Diff	snapshots: one entry per snapshot on b’s ancestry back to a, from snapshot summaries (append: +5 rows, +1 files); when a is not an ancestor, total-* counters are compared instead

delta: Delta Lake tables

Locator	uri (positional), region (sets AWS_REGION)
Config	storage_options = {...} passed to DeltaTable; committed, the same four keys as file, and the rest in .tether/secrets.toml
Identity	{uri}, a local path in its resolved form
State	version, table_id (metadata id – a recreated table verifies as drifted)
Pin / fork	none: Delta has no tags or branches. Versions stay readable until VACUUM (7-day default) / log retention (30 days).
Handle	DeltaHandle(uri, version, table) – read-only DeltaTable at the version
Diff	commits: one entry per transaction-log commit in (a, b] with its operation and metrics (WRITE: +2 rows, +1 files)

lance: Lance datasets

Locator	uri (positional), branch (default main)
Config	storage_options = {...} passed to lance.dataset; committed, the same four keys as file, and the rest in .tether/secrets.toml
Identity	{uri}, a local path in its resolved form
State	branch, version, and on any branch but main its branch_id (Lance version numbers are branch-scoped and restart when a branch is deleted and re-created, so the id tells two lives of one name apart; --at N means version N of the locator’s branch – it is a base coordinate and is never consulted once a working branch exists; use a tag to name a version on another branch)
Pin	tags.create("tether.<id>", (branch, version)); tagged versions are exempt from cleanup_old_versions()
Fork	create_branch(name, tag). Lance refuses to delete a branch a tag references, so a pinned working branch outlives its workspace and a re-fork under the same name picks name.2, name.3, … A conditional fork onto an existing branch compares its head, then deletes and recreates it, which leaves a small window; one expecting “absent” is atomic
Handle	LanceHandle(uri, dataset, version, branch, tag); append with lance.write_dataset(table, handle.dataset, mode="append")
Diff	fragments: fragment ids added/removed, deletion-file changes, columns added/removed, from the two manifests
Notes	an untouched fork reports its parent’s (branch, version), so a no-op commit after tether new is a no-op.

ducklake: DuckLake catalogs

Experimental: tested against a local DuckDB-file catalog, not a Postgres or MySQL catalog with an object-store data_path.

Locator	metadata (positional uri accepted; the ducklake: prefix is added if missing, e.g. ducklake:postgres:dbname=lake), data_path, table (scopes the handle only; snapshots are catalog-wide). A DuckDB or SQLite file path in metadata (bare, duckdb: or sqlite:), and data_path, are stored absolute with ~ expanded; a pre-0.1.0b4 relative path is made absolute by tether upgrade
Config	init_sql = ["CREATE SECRET ...", ...] run on every fresh connection – .tether/secrets.toml only (it runs code); credentials there or in the environment, never in the locator
Identity	{metadata[, data_path]}
State	snapshot_id, snapshot_time_us (epoch microseconds; a recreated catalog verifies as drifted)
Pin / fork	none: DuckLake has no tags or branches. ducklake_expire_snapshots bounds how long a snapshot stays readable.
Handle	DuckLakeHandle(metadata, alias, connection, snapshot_id, attach_sql, data_path, table): a DuckDB connection with the catalog attached read-only (at SNAPSHOT_VERSION for a recorded state); table_ref() gives alias.table; close() / with handle: detaches
Diff	tables: per-table created/dropped/altered/inserted/deleted from the snapshots’ changes maps, with row counts from ducklake_table_changes for tables that still exist. A table is named by its path, a part that is not a plain identifier double-quoted: sales.europe.orders, "a.child".t
Notes	a DuckDB-file metadata catalog can be attached only once per process: operations attach/detach per call (serialized per catalog), and an open handle or your own writer connection in the same process blocks status. Postgres/SQLite/MySQL catalogs have no such limit.

dolt: Dolt databases

Experimental: tested against a fake of the Dolt SQL surface, not a running dolt sql-server.

Locator	host (default 127.0.0.1), port (default 3306), database (required), branch (default main), user; or url = mysql://user@host:port/database
Credentials	only the server’s [uris."mysql://host:port"] entry in .tether/secrets.toml (a mysql://host:port/database entry beats it): user and password, or user_env and password_env. A manifest chooses the host, so a server with no entry gets no password (and user root unless the locator names one)
Identity	{host, port, database}
State	commit (branch head hash), plus dirty = true when dolt_branches.dirty reports uncommitted working-set changes (refused at pin time)
Pin	CALL DOLT_TAG('tether.<id>', commit)
Fork	CALL DOLT_BRANCH(name, tag) (recreated with -D if it exists at another commit)
Promote / merge	DOLT_MERGE (--ff-only to promote) inside an explicit transaction, the only way Dolt reports which tables conflict or violate a constraint. Those come back as MergeConflict, and the merge is rolled back
Handle	DoltHandle(url, database, ref, commit): a revision-database URL mysql://user@host:port/db/<branch|tag|hash> (tags and hashes are read-only in Dolt); database_ref is db/ref for USE
Diff	tables: dolt_diff_summary (added/dropped/modified/renamed tables, schema changes) with dolt_diff_stat row counts
Notes	talks to dolt sql-server / Hosted Dolt via PyMySQL; the SQL surface sits behind the DoltClient protocol (tether.experimental.backends.dolt) so other transports can be substituted.

memory: in-process reference backend

The executable specification of the protocol, used by the conformance suite and the engine tests. Locator {system, branch}; state snapshot_id; tags and branches in a process-global MemoryStore (tether.backends.memory.default_store()). MemoryHandle.read() / .write(payload) operate on the current ref; diff reports changed payload keys.