Getting Started

Ten minutes, no cloud account: a local Icechunk repository and a directory of files, registered in a fresh repository, committed, changed on a branch, and landed. Every command below was run as shown; ids will differ on your machine.

Install

pip install "tether-vcs[cli,icechunk]"      # what this page needs
pip install "tether-vcs[cli,icechunk,neon]"  # add the backends you use
pip install "tether-vcs[all]"                # everything

The distribution is tether-vcs (PyPI reserves the bare name); the package you import and the command you run are both tether. Python 3.11 or newer.

Extras per backend: objectstore (S3/GCS/Azure for file; s3, gcs, azure are aliases), icechunk, neon, iceberg, delta, lance, ducklake, dolt.

NoteBefore you start
  • git 2.38+ or jj 0.43+ on PATH: tether keeps its history in your repository.
  • A POSIX system or WSL: on Windows, commands that write are refused.
  • Remote objects need credentials, which never go in the repository (Configuration).

Some data to track

Use anything you have, or make a small sample: an Icechunk repository with one array, and a directory of three files.

import pathlib, icechunk, zarr

data = pathlib.Path("data")
storage = icechunk.local_filesystem_storage(str(data / "imaging.icechunk"))
repo = icechunk.Repository.create(storage)
session = repo.writable_session("main")
root = zarr.create_group(store=session.store)
root.create_array("labels", shape=(4, 8, 8), chunks=(1, 8, 8), dtype="u1")[:] = 0
session.commit("plate1: empty labels")

raw = data / "raw" / "plate1"
raw.mkdir(parents=True)
for i in range(3):
    (raw / f"well_{i}.tif").write_bytes(bytes([i]) * 64)

Register and commit

tether lives inside a git or jj repository. Make one next to the data:

jj git init my-dataset && cd my-dataset      # or: git init my-dataset
tether init
tether add zarr/imaging --kind icechunk ../data/imaging.icechunk
tether add raw/plate1   --kind file     ../data/raw/plate1/
tether status
dataset 61d0206e8d81; on trunk bookmark main
        new  raw/plate1  [file/observed]
        new  zarr/imaging  [icechunk/forkable]

init and add wrote small TOML manifests under .tether/, and put the working copy on main, the dataset’s trunk. status was the first command to touch the data: it fingerprinted both objects. The header’s 61d0206e8d81 hashes the manifests as they are now, so it changes with every edit; the dataset’s permanent id is the [dataset] id in tether.toml. The bracket names each object’s tier. Icechunk is forkable: it can be pinned and branched. A directory is observed: tether can tell you it changed, nothing more.

tether commit -m "Baseline: plate1 imaging and raw files"
  pinned zarr/imaging -> tether.4e9503fc.73e341e966c83d38
  recorded raw/plate1 (not recoverable)
committed b55326ca55a0

One command did three things. It created an Icechunk tag (the pin) holding the snapshot, wrote both states into the manifests, and committed them with the main bookmark on the commit. b55326ca55a0 is now the handle for this exact combination of states.

Read it back

Any process can open an object as it was at a commit:

tether open zarr/imaging --rev b55326ca55a0
/path/to/data/imaging.icechunk#tether.4e9503fc.73e341e966c83d38
from tether import Repo

repo = Repo.find(".")
h = repo.open("zarr/imaging", rev="b55326ca55a0")   # read-only IcechunkHandle at the tag
zarr.open_group(h.session.store, mode="r")

For a whole job, set TETHER_REV=b55326ca55a0 and every repo.open(key) in it resolves read-only at that commit’s pins – no code change. The directory cannot be opened at a revision: its state is a digest, not an address, which is what not recoverable meant above.

Change it on a branch

A bookmark is a branch of the dataset and, with it, a branch in every system that has them. Start one:

tether new -b relabel
on bookmark relabel; working refs set up
  zarr/imaging -> tether.ws.4e9503fc.relabel  (created on first writable open)

Nothing was written yet. The branch appears in the Icechunk repository the first time you open the object writable, forked from the pin main holds:

import zarr
from tether import Repo

repo = Repo.find(".")
h = repo.open("zarr/imaging")             # writable: this forks tether.ws.4e9503fc.relabel
group = zarr.open_group(h.session.store, mode="a")
group["labels"][:] = 1
h.session.commit("Relabel plate1")        # an Icechunk commit on the branch

main – the bookmark and the Icechunk branch – is untouched. Record the result and compare it with the trunk:

tether commit -m "Relabel plate1"
tether diff main relabel --content
  pinned zarr/imaging -> tether.4e9503fc.89c7e63509153c4f
committed 7eb84262ce1b
    changed  zarr/imaging  [+0 -0 ~1 nodes]
               modified  /labels  4 chunks

commit pinned the branch head and moved the relabel bookmark onto the new commit. Both commits stay readable: open --rev b55326ca55a0 still gives the zeros.

Land it

promote makes each system’s upstream branch point at what the bookmark pinned, then moves the main bookmark to the commit. Like every planned command, it takes --dry-run, so look first:

tether promote --dry-run
plan: promote (1 write(s))
        fast-forward  zarr/imaging  [icechunk] main  main snapshot_id=SHS2X3EVBHMYV1BQ… -> snapshot_id=TY7E8S29CEP987JJ… (base unchanged since fork)
                      raw/plate1: not forkable; nothing to promote
tether promote
tether verify --all-history
  fast-forwarded zarr/imaging -> {'snapshot_id': 'TY7E8S29CEP987JJ1VX0'}
  skipped        raw/plate1
main -> 7eb84262ce1b
        ok  1609e5dbca9d:zarr/imaging
        ok  7eb84262ce1b:zarr/imaging
        ok  b55326ca55a0:zarr/imaging

Icechunk’s main was fast-forwarded, the main bookmark now names the relabel commit, and verify confirms every pin any commit made still resolves. The third commit is jj’s empty working-copy commit, with the same manifests. In CI, run verify --all-history --deep (the recipe).

What was written

Everything committed is small text, reviewable in jj diff or git diff: tether.toml and one manifest per object under .tether/objects/. The per-checkout files, .tether/workspace.toml (which bookmark you are on) and .tether/ops.jsonl (what tether did to the stores), are untracked. The full layout is in Configuration.

Next

Remote systems register the same way; only the locator changes. Backends marked experimental, like neon, have run against a fake of the service, not the service itself:

tether add zarr/imaging --kind icechunk s3://bucket/imaging.icechunk
tether add db/metrics   --kind neon --project-id prj-123 --database neondb --role runner
tether add features     --kind lance s3://bucket/features.lance
tether add code         --kind git ../analysis-code