Pinning

Pinning is recording the exact state of every object in a commit so it can be read back later – and, where the system supports it, holding that state with a native reference that survives retention and cleanup. This page assumes Getting Started and moves to the objects a lab actually has: an Icechunk repository on S3, a Neon Postgres database (an experimental backend), and a directory of raw plate images on a shared filesystem.

Register the objects

tether init
tether add zarr/imaging --kind icechunk s3://bucket/imaging.icechunk --branch main
tether add db/metrics   --kind neon --project-id prj-9f2 --database metrics --role runner
tether add raw/plate1   --kind file /mnt/instruments/plate1/
tether status

dataset 0f2e9c7a11b3; on trunk bookmark main
       new  db/metrics  [neon/forkable]
       new  raw/plate1  [file/observed] unrecoverable
       new  zarr/imaging  [icechunk/forkable]

add writes manifests only; status is the first command to contact the systems. --branch main (the default) names the object’s upstream branch, and --at SNAPSHOT starts from a chosen state instead (Starting from an older snapshot). The tiers say what pinning will mean. Icechunk and Neon are Forkable, so a native ref will be created. The directory is Observed: its state can be recorded but not read back, hence unrecoverable.

Pin: the first commit

tether commit -m "Baseline: plate1 imaging + metrics"

  pinned zarr/imaging -> tether.0a1b2c3d.6674305d5e1fe5f6
  pinned db/metrics -> tether.0a1b2c3d.a91f0e2b77c3e5f6
  recorded raw/plate1 (not recoverable)
committed 8d5719841a09

Object	State recorded	Pin created
`zarr/imaging`	snapshot_id of `main`	Icechunk tag `tether.0a1b2c3d.6674305d5e1fe5f6` (immutable; the default `expire_snapshots` keeps tagged snapshots)
`db/metrics`	`lsn`, `commit_xid` of `main`	Neon child branch `tether.0a1b2c3d.a91f0e2b77c3e5f6` at that LSN (protected only with `protected_pins = true`)
`raw/plate1`	`count`, `size`, digest	none; the per-file listing is stored under `.tether/listings/`

Then the manifests (and the listing) were committed to your repository as 8d5719841a09:

jj diff -r 8d571984      # or: git show --stat 8d571984

Added regular file .tether/objects/zarr/imaging.toml:
        1: key = "zarr/imaging"
        2: kind = "icechunk"
        3: recoverable = true
        ...
       15: [state]
       16: snapshot_id = "K3F7Q2ZJ1XN4Y8W0"
       18: [pin]
       19: id = "0a1b2c3d.6674305d5e1fe5f6"
       20: ref = "tether.0a1b2c3d.6674305d5e1fe5f6"

Pin ids are content-addressed from (kind, locator identity, state). Commit again without changing anything and every object is unchanged; no new refs are created.

Pin again after the data moves

Later, the imaging repository gets a new commit on main and the metrics database has been written to. On the trunk those upstream branches are the working refs, so a fan-out shows them modified:

tether status --snapshot       # fingerprint now; plain `status` shows the last snapshot

dataset 0f2e9c7a11b3; on trunk bookmark main
  modified  db/metrics  [neon/forkable]
     clean  raw/plate1  [file/observed] unrecoverable
  modified  zarr/imaging  [icechunk/forkable]

commit pins the new heads (tether pull does the same and writes the message for you):

tether commit -m "Re-segmented plate1; metrics through 2026-09-04"

  pinned zarr/imaging -> tether.0a1b2c3d.deb4861eb388e5f6
  pinned db/metrics -> tether.0a1b2c3d.0c31e7aa5f92e5f6
committed cfc66fa79c0a

Two commits now describe two complete states of the whole dataset. Compare them:

tether diff 8d571984 cfc66fa7 --content

    changed  db/metrics
    changed  zarr/imaging  [+0 -0 ~3 nodes]
               modified  /labels  12 chunks
               modified  /masks  12 chunks
               modified  /  group metadata

Neon has no native diff, so db/metrics shows only that it changed; Icechunk reports the arrays whose chunks were rewritten.

Reproduce reads from a pinned commit

This is the payoff. Any process can resolve every object as it was at a commit:

tether open zarr/imaging -r 8d571984
tether open db/metrics -r 8d571984

s3://bucket/imaging.icechunk#tether.0a1b2c3d.6674305d5e1fe5f6
postgresql://runner@ep-quiet-lake-a1b2c3.us-west-2.aws.neon.tech/metrics?...

From Python the same handles come with native objects attached:

from tether import Repo
from tether.handles import IcechunkHandle, NeonHandle

repo = Repo.find(".")

imaging = repo.open("zarr/imaging", rev="8d571984")
assert isinstance(imaging, IcechunkHandle) and imaging.read_only
store = imaging.session.store          # read-only icechunk session at the tag

metrics = repo.open("db/metrics", rev="8d571984")
assert isinstance(metrics, NeonHandle)
metrics.url                            # connection URL for the pinned branch

For reproducible jobs, set TETHER_REV instead of passing rev everywhere; repo.open(key) then defaults to a read-only handle at that commit:

TETHER_REV=8d571984 python run_analysis.py

Record TETHER_REV (or repo.vcs.current_rev()) and each object’s pin.id next to your results and any output traces back to the exact state of every input:

provenance = {
    "tether_rev": os.environ.get("TETHER_REV") or repo.vcs.current_rev(),
    "pins": {k: (m.pin.id if m.pin else None) for k, m in repo.objects.items()},
}

raw/plate1 cannot be opened at a revision: it is Observed, and its state at 8d571984 is a digest, not an address. Directories that must be recoverable belong in an object store with versioning enabled (single objects with --file versioned).

Drift

status --snapshot fingerprints everything, so drift surfaces the next time you ask. The directory was registered with the default --file immutable, so if a file under /mnt/instruments/plate1/ changes – its content; a touch is not a change – status labels it error, with the reason on stderr, and exits 1:

tether status --snapshot

      error  raw/plate1  [file/observed] unrecoverable
error: raw/plate1: [raw/plate1] immutable object 'raw/plate1' changed since it
was committed; re-register with acceptance to record the new state

That is deliberate: an “immutable” input silently changing is the worst kind of drift. Either the change is legitimate (tether remove raw/plate1, add it again, commit), or something wrote where it should not have. Other drift is benign and exits 0: a moved branch shows modified and is pinned by the next commit, and on the trunk a branch-less object past its pin shows behind until pull.

Verify

Pins are references inside other systems, and those systems have their own operators. verify confirms they still resolve to what the manifests say:

tether verify                      # working-tree manifests, cheap checks
tether verify -r 8d571984 --deep   # actually open each pinned state
tether verify --all-history --deep # every commit in the repository

        ok  8d5719841a09:db/metrics
        ok  8d5719841a09:zarr/imaging
        ok  cfc66fa79c0a:db/metrics
        ok  cfc66fa79c0a:zarr/imaging

A report is ok, drifted (the ref points elsewhere), missing (the ref or snapshot is gone), or unknown (cannot tell without --deep). --all-history verifies each distinct pin in history once. verify exits 1 if anything is not ok, unknown included, so the CI form is verify --all-history --deep (A CI recipe).

A pin is never taken on trust. Before open --rev, new or promote --rev reads or forks from one, the engine checks it against the recorded state. A pin someone deleted falls back to the recorded state where the backend can address it, and tether repair recreates the ref. A pin someone moved is refused with PinDriftError, since it would hand back the wrong data under a commit’s name.

Garbage collection

Pins hold storage. When history no longer names a pin (an aborted, squashed or abandoned commit), tether gc releases it; tether gc alone is a dry run, and --no-dry-run applies. Run verify --all-history --deep first: a pin that is referenced but broken is a verify finding, not a gc one. Reclaiming Storage covers the rest.

Branching and Writing: change these objects on a bookmark, then land or discard the result.