Writing a Backend

For contributors adding a system tether does not know yet. A backend maps tether’s operations onto one class of system. One instance serves every object of its kind, and per-object addressing travels in the locator. Implement the ObjectBackend protocol (tether.backends.base), register it, and run the conformance suite. This page also defines the capability flags, and what experimental means for a backend.

The protocol

from tether.backends import Capability, ObjectBackend, ObjectDiff, VerifyReport, VerifyStatus, register_backend
from tether.handles import Handle
from tether.manifest import Locator, Pin, State, ref_for_pin


class MyBackend(ObjectBackend):
    kind = "mysys"
    capabilities = (
        Capability.FINGERPRINT | Capability.ADDRESSABLE | Capability.PIN | Capability.FORK
        | Capability.ATOMIC_REF | Capability.DIFF
    )

    def __init__(self, config: dict | None = None) -> None:
        self._config = config or {}          # [backends.mysys] from tether.toml

    def identity(self, locator: Locator) -> Locator: ...
    def fingerprint(self, locator: Locator, working_ref: str | None) -> State: ...
    def pin(self, locator: Locator, state: State, pin_id: str) -> Pin: ...
    def unpin(self, locator: Locator, pin: Pin) -> None: ...
    def list_pins(self, locator: Locator) -> set[str]: ...
    def verify(self, locator: Locator, state: State, pin: Pin | None, deep: bool) -> VerifyReport: ...
    def fork(self, locator: Locator, source: Pin | State, name: str, *, expected: State | None = None) -> str: ...
    def delete_working_ref(self, locator: Locator, ref: str) -> None: ...
    def open(self, locator: Locator, target: str | Pin | State | None, read_only: bool) -> Handle: ...


register_backend("mysys", lambda config: MyBackend(config))

Method by method:

Method	Contract
`identity(locator)`	The subset of the locator that names the system (not the branch, region, or credential references). It is hashed into the pin id, so two objects with the same identity and state share a pin. Pass a local path through `canonical_uri`, so every spelling of one directory is one identity.
`fingerprint(locator, working_ref)`	The current state at `working_ref` (a branch fork returned) or at the locator’s base ref when `None`. Must be a JSON-compatible dict whose content keys are stable across calls when nothing changed and different after any change. Keys that merely address the data (an LSN, a derived id) go in VOLATILE_KEYS so content_state excludes them from equality and pin ids; never include coordinates that move on unrelated activity (Iceberg’s `metadata_location`).
VOLATILE_KEYS (class attribute)	State keys that address but do not identify: excluded from drift detection, pin ids, listing names, and export hashes; still passed to pin / open / verify. Default: none.
`pin(locator, state, pin_id)`	Create the native ref `ref_for_pin(pin_id)` (`tether.<id>`) pointing at `state`. The id is opaque to the backend (`<dataset8>.<hash16>`; the engine namespaces it). Idempotent: if it exists and points at the same state, return it; if it points elsewhere, raise BackendError. Raise if the state is no longer reachable – `repair` relies on that.
`unpin(locator, pin)`	Delete the ref. A ref that is already gone is a no-op or a BackendError, never a library exception.
`list_pins(locator)`	Pin ids (without the `tether.` prefix) that exist natively, every dataset’s included. The engine keeps only its own namespace before `gc` subtracts the referenced ones.
`verify(locator, state, pin, deep)`	With a pin: OK if it resolves to `state`, DRIFTED if it points elsewhere, MISSING if gone. Without: UNKNOWN unless `deep`, then actually open the state.
`fork(locator, source, name, *, expected=None)`	Create writable branch `name` and return the working ref (usually `name`; a backend may pick a sibling if `name` is taken). `source` is a Pin, or a recorded `State` for `pin = "record"` objects (raise BackendError if it is no longer reachable). Reset contract: an existing `name` is moved back onto `source` (a reset, a ref update, a branch restore, delete-and-recreate), and left alone when it is already there; conformance checks it. Conditional move: `expected` is the head the caller reviewed, a `State` or ABSENT; anything else raises RefMovedError and moves nothing. Use the system’s compare-and-swap where it has one (git `update-ref <new> <old>`, Icechunk `reset_branch(from_snapshot_id=)`) and declare CONDITIONAL_REF; elsewhere call `check_expected(self, locator, expected, ref=name)` first. `None` moves unconditionally. A signature without the keyword is called without it, and the engine warns that its moves are unconditional.
`delete_working_ref(locator, ref)`	Delete a branch fork created; never the base branch. Ignore “already gone”.
`working_ref_blockers(locator, ref)`	Why ref cannot be deleted right now, or `None`. `gc --prune-bookmarks` / `forget-workspace` plan a blocked branch as kept with the reason instead of failing at apply. Neon: pin children. Default: none.
`rename_pin(locator, old, state, new_id)`	Give an existing pin a new id (`tether upgrade` when the naming scheme changes). Default: pin the state under the new id, then unpin the old – right wherever pins are tags. Override where a pin is a branch with children (Neon renames in place).
`rename_working_ref(locator, old, new)`	Rename a working branch; return the resulting ref. Default: fork new from `old`’s head state and delete `old`.
`list_working_refs(locator)`	Native branches named `tether.ws.*` (`WORKING_REF_PREFIX`) in this system, every dataset’s included; the engine filters to its own namespace. `gc --prune-bookmarks` fingerprints each stray one and deletes it only if its head is pinned or equals the base head (`--force-prune` overrides); `repair` uses it to notice a missing working branch. Default: none.
`promote(locator, source, *, expected=None)`	PROMOTE: fast-forward the base branch to `source` (a working ref, Pin, or `State`); refuse with BackendError unless the base head is an ancestor of it, and no-op when already there. Return the base’s new state. `expected` is the base head the plan reviewed: raise RefMovedError when the base holds another state, through the compare-and-swap where there is one, else `check_expected(self, locator, expected, what="promote")` first.
`merge(locator, source, message, *, expected=None)`	MERGE: three-way merge `source` into the base; raise `MergeConflict(conflicts=[...])` and leave the base untouched on conflicts. Return the base’s new state. The engine passes the state the plan reviewed, so merge that commit, not the branch head. Check `expected` with `check_expected(self, locator, expected, what="merge")` first: merges have no old-value form.
`ancestor_of(locator, ancestor, descendant)`	Whether `ancestor` (a state) is in `descendant`’s history; `None` when unknown. promote falls back to it when no fork point was recorded.
PROMOTE_HINT (class attribute)	The sentence promote shows when it refuses: what the user should do with the system’s own tools.
`open(locator, target, read_only)`	target is `None` (base ref), a `str` working ref, a Pin (read the pinned state), or a `State` (read an Addressable state). Return a Handle subclass carrying the native address and, where cheap, a native object.
`create(locator, *, owner)`	CREATE: make an empty store at the locator, write an owner marker in the store naming owner (a dataset id), and return the initial state. Raise BackendError if anything is already there – create never adopts.
`owner(locator)`	CREATE: the dataset id in the store’s owner marker; `None` when tether never wrote one or the store is gone.
`is_ref_empty(locator, *, ignoring=())`	CREATE: `True` when nothing remains but the base branch at its initial state, the marker, and the refs in `ignoring` (the ones a gc plan is about to delete) – and, for a store with a working area (a git checkout, a directory), no uncommitted content: files are data whether or not a ref names them; `False` otherwise; `None` when the backend cannot tell. `None` means keep.
`delete_store(locator)`	CREATE: remove the store. Called only after is_ref_empty returned `True` and owner named the caller’s dataset – and expected to check both again itself, right before the irreversible step, and to remove only what the store’s own layout accounts for.

The owner marker is the whole basis for a delete: manifests travel with clones and are untrusted, so a manifest field never authorizes one. Put the marker where the store keeps its own metadata (Icechunk repository metadata, git config, a .tether-owner file the directory walk skips) – never in a ref that list_pins or list_working_refs would report.

pin must be idempotent and say which it did: return Pin(..., created=True) when it made the ref, created=False when it found the ref already carrying the state. commit rolls back only the pins it created; a reused one belongs to the commit that made it. unpin and delete_working_ref must raise BackendError when the ref is still there afterwards – “already gone” is success, a refused deletion is not.

Optional members:

effective_capabilities(locator, policy) -> Capability when the tier varies per object (the file backend is Observed for a local directory and Addressable for a versioned remote object).
MATURITY (class attribute, default "stable"): "experimental" when the backend is tested against a fake of a network service rather than the service itself. tether backends prints it; add mentions it.
LOCAL_PATH_KEYS (class attribute): locator keys that may hold a local filesystem path, after any of LOCAL_PATH_PREFIXES (DuckLake’s ducklake:, sqlite:, …). At add/import the engine expands a leading ~ and resolves a relative path against the caller’s directory, so a committed locator means one path from every directory and clone.
branch_scope(locator) -> str and ref_namespace(locator) -> str when the native resource that owns branches, or the one whose pins list_pins returns, is wider than one object – a Neon project, an Iceberg table. The engine forks one branch per scope under a bookmark (members share it) and collects gc references per namespace. Both default to the canonical identity.
state_addressable(locator, state) -> bool (default True) when an Addressable backend can produce a state that nonetheless has no coordinate to reopen it by (a file object in a bucket without versioning has no version_id). commit records such a state recoverable = false.
check_quiescence(locator, working_ref) for NEEDS_QUIESCENCE backends; raise BackendError if writers are active.
listing(locator, state) -> str | None and diff(locator, a, b, *, listings) -> ObjectDiff for DIFF (below).
history(locator, ref, limit) -> list[HistoryEntry] for HISTORY: the system’s native history newest first (id, ISO when, message, refs). Each id must be accepted as the locator’s at field, and fingerprint must honor at (via base_at(locator)) when no working ref is given, so users can register an object at any listed state.

Raise BackendError (or CapabilityError for unsupported operations) with kind=self.kind; the engine aggregates per-object failures into MultiObjectError. Import optional libraries inside methods, so the module imports without them. Decorate the class with @wrap_library_errors and give it a _library_errors() staticmethod returning the library’s exception types, which every protocol method then re-raises as BackendError. The decorator wraps only methods defined on the class, so re-declare an inherited one. validate_locator runs the checks that are cheap at add time, so a refusal reaches the manifest’s author, not the first reader of a clone.

Capabilities decide behavior

Declare only what the system truly supports; the engine degrades explicitly.

Capability	The engine will…
FINGERPRINT	fingerprint (required)
ADDRESSABLE	let `open(state)` read a recorded state without a pin
PIN	call pin on commit, list_pins/unpin in `gc`, verify with the pin. Removed per object by `policy.pin = "record"`, in which case commit records the state instead
FORK	call fork in new (from the pin, or from the recorded state without one) and return writable handles from open; list_working_refs/delete_working_ref in `gc --prune-bookmarks`
CHEAP_FINGERPRINT	(informational) metadata-only reads
RETENTION_BOUND	(informational) surface that recoverability expires
NEEDS_QUIESCENCE	call `check_quiescence` before pinning
ATOMIC_REF	(informational) pins are create-if-absent
CONDITIONAL_REF	trust that fork, and promote / merge where declared, honour `expected` in one atomic step. Declare it only then; effective_capabilities can declare it per object (Icechunk: on `s3://` only). Conformance runs 8 racing forks and fails a backend that declares it and ignores `expected`.
DIFF	call diff for `diff --content`, listing at commit
HISTORY	call history for `tether log` and `add --pick`
BRANCH_IS_STORAGE	never delete a stray working branch in `gc --prune-bookmarks` without `--force-prune` (deleting reclaims the data, as in Neon)
PROMOTE	call promote in `tether promote` when the base is unchanged since the fork point
MERGE	call merge in `tether promote` when the base moved (or `--strategy merge`)
CREATE	(experimental) call create for `add --create` / `Repo.create`; owner, is_ref_empty, and delete_store in `gc --delete-stores` for a created store nothing references any more (tether.experimental.lifecycle)

Content diffs and listings

diff(locator, a, b, *, listings) returns an ObjectDiff: a unit naming what is counted (files, tables, arrays, …), exact added/removed/ modified counts, and entries (capped at 2000; use out.add(path, change, detail)). Read metadata only; never data.

If your state is a digest that cannot be diffed on its own, also implement listing(locator, state), returning a detailed text description (JSON lines work well). The engine commits it under .tether/listings/ and hands both sides back as listings=(text_a, text_b). Degrade to a summary line when either is None.

Experimental and graduation

Backends driven only through fakes of their service (Neon, Dolt, DuckLake, Iceberg) and the registry layer live under tether.experimental:

tether/experimental/backends/{neon,dolt,ducklake,iceberg}.py
tether/experimental/registry/{export,registry}.py

What is stable. Kind names (tether add --kind neon), extras (tether-vcs[neon]), the CLI commands, the Repo methods (repo.export(), plan_import, apply_import, import_objects), and the tether.<Symbol> re-exports (ExportBundle, ImportSpec, ImportReport, PublishReport, build_bundle, specs_from_rows). These do not change when something graduates. Module paths under tether.experimental are not API, and the alpha-era paths (tether.backends.neon, tether.export, tether.registry) are gone.

It is a real boundary. import tether and import tether.cli load nothing under tether.experimental.registry. The Repo methods import it on first call, the tether.<Symbol> names resolve lazily, and tether.experimental.cli.register(app) attaches the three CLI commands.

What “experimental” means. The backend passes the conformance suite and its engine lifecycle against a fake, not the service. tether backends reports experimental, and tether add prints a note. What only a live run can confirm (rate limits, eventual consistency, what a protected branch costs) remains to be seen.

Graduation is a file move. build_backend resolves a kind through _BUILTIN_MODULES, and the class’s MATURITY drives the label. Move the file to tether/backends/, point its _BUILTIN_MODULES entry there, set MATURITY = "stable", and update the table in Backends. Nothing users type or import changes. The registry graduates the same way, keeping the tether re-exports. ROADMAP.md has each backend’s call.

Conformance suite

tether.testing.run_conformance runs the tier-appropriate checks against any backend. Provide a harness that can create a fresh system and mutate it:

from tether.backends import ObjectBackend
from tether.manifest import Locator
from tether.testing import run_conformance


class MyHarness:
    def __init__(self) -> None:
        self.backend: ObjectBackend = MyBackend()
        self._n = 0

    def new_object(self) -> Locator:
        ...  # create an empty system with a `main` branch; return its locator

    def mutate(self, locator: Locator, working_ref: str | None) -> None:
        self._n += 1
        ...  # write something new at working_ref (or the base branch)

    def fresh_locator(self) -> Locator:  # CREATE backends only
        ...  # a locator where no store exists yet


def test_conformance() -> None:
    run_conformance(MyHarness())

What the suite checks, per capability:

Always: fingerprint stability and change, identity and pin-id determinism, and unpin.
DIFF: an empty self-diff, and a non-empty one after a mutation.
HISTORY: the newest entry is the current state and works as at, an older entry is another state, and limit bounds the result.
ADDRESSABLE: open by state, an older recorded state included.
PIN: create, list, verify shallow and deep, idempotency, and protection against later drift of the working ref.
FORK: a fresh fork equals the pinned state, and writes to it leave the base alone. A stale expected, or ABSENT onto an existing branch, raises RefMovedError and moves nothing.
CONDITIONAL_REF: of 8 threads forking with one expected, exactly one moves the branch; the backend must take expected.
PROMOTE and MERGE: the base follows the fork, a second promote is a no-op, a base that moved is refused, two diverged sides merge, and a stale expected is refused. ancestor_of answers in the right direction, or None.
CREATE: the store lifecycle at a fresh_locator(), from create through owner, is_ref_empty (with and without ignoring) and a second create refused, to delete_store and a create that works again.

A harness may set capabilities to exercise one tier of a multi-tier backend. tether.backends.memory is the executable specification the suite was written against. tether.backends.lance is a Forkable backend that had to bend the mapping (branch-scoped versions, undeletable tagged branches) without breaking the contract.

VCS adapters

Three members beyond reading and committing files. history_digest() digests every visible commit id, so a commit made in another workspace stales a gc plan. shared_dir() names the directory every checkout of the repository shares (git’s common dir, jj’s repo dir), where the repository lock and the pinned, created and touched indexes live. tracked(paths) says which of tether’s per-checkout files the VCS tracks, so find can refuse a clone that ships them.

Backends never touch the VCS. If you need to support another version-control system for the manifests, implement tether.vcs.VcsAdapter (resolve, read and list files at a revision, stream history, commit, new) and extend detect_vcs. The jj and git adapters read history through one git cat-file --batch process (GitObjectReader); a non-git store should aim for the same batching to keep gc and verify --all-history fast.