# Configuration What a dataset is made of on disk, what `tether.toml` can say, where credentials and other per-checkout settings live, and what a cloned dataset can and cannot make tether do. # Layout / tether.toml # committed: format version, dataset id, snapshot/verify/new settings, # default policies, non-secret backend options .tether/ .gitignore # ignores workspace.toml, ops.jsonl, secrets.toml, cache/, lock secrets.toml # per checkout: credentials or references, endpoints, executables, SQL objects/.toml # committed, one per object: kind, locator, policy, state, pin, # recoverable, captured_at, origin (only when "created") listings/.jsonl # committed, content-addressed per-file listings of directory and # prefix states (for `diff --content`); pruned by gc workspace.toml # per checkout: workspace id, bookmark, working refs, pending forks, # base states, fork points, last snapshot ops.jsonl # per checkout: the operation log / # .jj/repo/ or .git/: one per clone, shared by its checkouts, never committed tether.lock # the repository-wide lock (see Caveats) tether-pinned.jsonl # pins this clone created: the only ones gc releases by default tether-created.jsonl # stores this clone made with `add --create` tether-touched.jsonl # stores this clone forked a branch in or pinned The three per-checkout files are refused while the VCS tracks them (the [security model](#security-model)). The indexes live with the repository, not in history or a checkout: an abandoned fork takes the manifest that named a store with it, and `forget-workspace` takes the checkout's files. They are memories, not authority. [Reclaiming Storage](reclaiming-storage.md) says how `gc` uses them, and why a re-clone starts without them. `pin_id = .`, and the native ref is `tether.`. An identical state yields the same pin within a dataset: re-committing an unchanged object is a no-op, and identical states dedupe to one pin. The identity is the backend's canonical form of the locator, such as a resolved local path ([Backends](backends.md#conventions)). # `tether.toml` A dataset's configuration is `tether.toml` at the dataset root, committed with the manifests and loaded into [RepoConfig](../reference/RepoConfig.md#tether.RepoConfig). Every key but `[dataset] id` has a default. A dataset whose version is older than this tether's is refused by every command except [`tether upgrade`](cli.md#upgrade). ``` toml [tether] version = 5 # the format this tether writes; older datasets need `tether upgrade` [dataset] id = "0a1b2c3d" # 8 hex chars; the namespace of this dataset's pins and branches in every store [snapshot] auto = false # `status` shows the last snapshot (--snapshot refreshes); true: fingerprint on every status [verify] on_status = false # also run a cheap verify of each committed object in status [new] fork = "lazy" # "lazy": create a working branch on first writable open; "eager": during `new` record = "fork" # pin=record objects: "fork" during `new` even when lazy; "defer": like the rest (needs fork = "lazy") [defaults] # policy for objects added without an explicit one file = "immutable" # immutable | versioned pin = "native" # native | record [vcs] prefer = "jj" # which adapter to use when both .jj and .git exist trunk = "main" # the bookmark that stands for every object's upstream branch [backends.file] # non-secret options only; one table per kind storage_options = { region = "us-west-2" } ``` # Dataset id `init` generates `[dataset] id`, and every clone shares it. Every native ref carries it (pins `tether..`, working branches `tether.ws..`), and `gc` stays inside that namespace. So datasets can share one Icechunk repository or Neon project without deleting each other's pins. Datasets from before the id existed get one from `tether upgrade`. `tether init --dataset-id ID` (`Repo.init(dataset_id=...)`) uses the id you give, which must be 8 lowercase hex characters. A dataset whose repository was lost gets its stores' pins and branches back this way, and `tether recover` finds the id when you do not know it ([Recovering a lost dataset repo](troubleshooting.md#recovering-a-lost-dataset-repo)). > **Warning: Do not change the id after committing** > > Before anything is pinned the id can still change; that is how `tether recover` takes back a lost dataset's refs. After that: > > - Pins and branches made under the old id stop being this dataset's: `gc` treats them as foreign and leaves them alone. > - `repair` recreates the pins the manifests name, under the new id. > - A fork meant to become an *independent* dataset must change the id *before* it pins anything, or the two share a namespace and can `gc` each other. # Policies A [Policy](../reference/Policy.md#tether.Policy) is stored per object (`[policy]` in its manifest). It is set with the `--file` and `--pin` options of `tether add` (or `Repo.add(policy=...)`) and changed with `tether set KEY... --file|--pin`. Whether writes fork or land upstream is not a policy: the trunk writes upstream, and any other bookmark forks. Manifests from before 0.1.0a9 carry a [write](../reference/handles.MemoryHandle.md#tether.handles.MemoryHandle.write) key, which is ignored and dropped by `tether upgrade`. | Field | Values | Meaning | |----|----|----| | `file` | `immutable` (default), `versioned` | `file` backend only. `immutable`: any drift is an error at [status](../reference/backends.VerifyReport.md#tether.backends.VerifyReport.status). `versioned`: a single object in a versioning-enabled bucket or container is Addressable through its version id. | | [pin](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.pin) | `native` (default), `record` | Any backend. `record` drops [PIN](../reference/testing.Capability.md#tether.testing.Capability.PIN): [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit) records the state without a native ref, and [new](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.new) forks from the recorded state. Cheaper, but recoverable only while the system keeps that state ([Lever 1](reclaiming-storage.md#lever-1-do-not-pin-what-you-will-not-need---pin-record)). | # Backend options `[backends.]` is handed to the backend when it is first used. The committed file may set only what a backend allowlists: `storage_options` for `file`, `delta` and `lance`, and `catalog` for `iceberg`. Inside those tables only the named keys pass (`region`, `allow_http`, `virtual_hosted_style_request`, `conditional_put`; `type`, `warehouse`, `name`). Any other key is refused by name, so an option tether has never heard of cannot reach a store from a clone. Everything else goes in `.tether/secrets.toml`, where the same tables may hold any key. A table set in both files is merged field by field, and `secrets.toml` wins a field both set. So a committed `catalog` keeps its `type` and `warehouse` when `secrets.toml` adds only the catalog's [uri](../reference/handles.FileHandle.md#tether.handles.FileHandle.uri). Per-backend keys: [Backends](backends.md). # Trust boundary: `.tether/secrets.toml` A cloned dataset is untrusted input. `tether.toml` and the manifests arrive with every clone, so nothing in them may choose an executable, an endpoint your credentials go to, SQL to run, or the variable a secret is read from. tether refuses a committed `tether.toml` that tries (`[vcs] git_path`, `[import] query`, `[backends.neon] api_url`, `[backends.ducklake] init_sql`, a `storage_options.endpoint`, …), and says where the key belongs: `.tether/secrets.toml`, per checkout and never committed, or the environment. ``` toml [vcs] git_path = "/usr/bin/git" # or TETHER_GIT / TETHER_JJ in the environment # jj_path = "/opt/jj/bin/jj" [backends.neon] # per-kind options the committed file may not set api_key_env = "NEON_API_KEY" [backends.ducklake] init_sql = ["CREATE SECRET (TYPE S3, PROVIDER credential_chain)"] [uris."s3://lab-a-bucket/"] # every object whose store URI starts here profile = "lab-a" # an AWS shared-config profile ... # role_arn = "arn:aws:iam::123456789012:role/lab-a" # ... or a role, assumed via STS ... # access_key_id = "..." # ... or, when nothing else will do, literal keys # secret_access_key = "..." # endpoint_url = "http://seaweedfs.local:8333" # allow_http = true # icechunk: plain HTTP to that endpoint # force_path_style = true # icechunk: bucket in the path, not the host [objects."zarr/imaging"] # one object; beats a URI prefix profile = "imaging" [uris."mysql://dolt.internal:3306"] # a Dolt server: the only one sent this password user = "svc" # (Dolt reads server entries only) password_env = "DOLT_PASSWORD" [import] # the default SQL for `tether import` (see Registries and SQL) query = "SELECT name AS key, 'file' AS kind, uri FROM catalog" ``` Resolution per object and per field, the most specific first: `[objects.""]`, then the longest matching `[uris.""]`, then `[backends.]`, then the environment and instance credential chain. A table given at two levels is merged, not replaced. Per-object entries also reach `add --create` and `gc --delete-stores`. `file`, `delta` and `lance` take the entries as storage options, and `icechunk` passes them to `s3_storage`. Dolt honours only the server's `[uris."mysql://host:port"]` entry (or one for `mysql://host:port/database`), because the manifest chooses the host a password is sent to. A `profile` or `role_arn` is resolved through `boto3` (install it when you use them) into explicit credentials. Concurrent callers share one resolution, cached until five minutes before the keys expire. Icechunk takes a profile or role as a refreshing credential source, so a session held for hours gets fresh keys instead of failing mid-write. The file may hold literal credentials, so tether warns when other users can read it (`chmod 600`) and never prints its contents: not in [status](../reference/backends.VerifyReport.md#tether.backends.VerifyReport.status), not in `--json`, not in an error. `init` lists it in `.tether/.gitignore`, and `Repo.find` adds the entry to an older dataset that lacks it. ## Advanced: two stores, two identities The case the per-URI and per-object entries exist for: a dataset tracks an Icechunk repository in the imaging lab's account and another in the sequencing core's, and one [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit) pins both. With `from_env` alone that is an `AWS_PROFILE` switch between commands, which one [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit) cannot do. ``` bash tether add zarr/imaging s3://imaging-lab-data/plate-01.icechunk --kind icechunk tether add zarr/reads s3://seqcore-archive/run-2026-09.icechunk --kind icechunk ``` ``` toml # .tether/secrets.toml (this checkout only; never committed) [uris."s3://imaging-lab-data/"] # every object in the imaging bucket profile = "imaging-lab" # a profile in ~/.aws/config [objects."zarr/reads"] # one object, whichever bucket it is in role_arn = "arn:aws:iam::210987654321:role/seqcore-reader" region = "us-east-2" ``` `tether commit` now opens `zarr/imaging` with the `imaging-lab` profile's keys and `zarr/reads` with a session assumed into the sequencing core's role, in one process. A third object with no entry keeps the ambient environment, and the manifests record only the two URIs. On CI the same layout works with literal keys the runner writes into the file before the first command. tether does not expand `${...}` itself, so write the values in, or use a role the runner's identity can assume ([A CI recipe](sharing-and-ci.md#a-ci-recipe)). # Security model What a cloned dataset (its `tether.toml`, manifests, history, and every other file in the checkout) can and cannot make tether do. It cannot: - **Run a program.** Executables come from `secrets.toml` or the environment. `secrets.toml`, `workspace.toml` and `ops.jsonl` are refused while the VCS tracks them, since a clone brings a tracked copy with it. `find` checks, and so does every later refresh (each [open](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.open), each writing command) that finds `workspace.toml` changed. The check runs before `secrets.toml` is read, so a jj reachable only through its `jj_path` needs `TETHER_JJ`. - **Run git code through a `git` object.** The `git` backend runs with `core.fsmonitor`, hooks and `ext::` transports off, and never takes a directory for a bare repository by itself (`safe.bareRepository`, which is why git 2.38 is required). A `git` object's [path](../reference/handles.GitHandle.md#tether.handles.GitHandle.path) must be absolute and outside the checkout, whose files came with the clone. - **Choose where writes go, or which pins `gc` releases.** Those are `workspace.toml` and the op log, refused when tracked. Another checkout's tracked copy is skipped with a warning naming the checkout. - **Send your credentials somewhere.** Endpoints, and Icechunk's `allow_http` and `force_path_style`, come from `secrets.toml` only. A Dolt password goes only to a server `secrets.toml` lists. - **Run SQL.** `[import] query` and `init_sql` come from `secrets.toml` or the command line. An import query runs as one statement in a read-only transaction. - **Push your code.** A git object's `remote` must name a remote the repository already has. - **Write outside `.tether/`.** Object keys are checked when a manifest is read, not only at [add](../reference/testing.ObjectDiff.md#tether.testing.ObjectDiff.add): no `..`, no `\`, no drive letters. It can: - **Name stores.** A locator is an address. A manifest can point [status](../reference/backends.VerifyReport.md#tether.backends.VerifyReport.status), [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit) and [verify](../reference/ObjectStatus.md#tether.ObjectStatus.verify) at any store your ambient credentials reach, and they sign the requests (a signature does not reveal the secret). [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit) records what it finds, a directory's file names and hashes included, and pushing publishes that. - **Release pins in your stores, with your help.** `gc` releases only pins in its dataset's namespace that no manifest names and that this clone created. `--release-foreign` drops the last condition. In a clone that reuses another dataset's id and names its stores, that releases the other dataset's pins. - **Make commands fail.** Every refusal above is an error: a hostile clone can stop a command, not redirect it. So read a clone's manifests before its first [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit), and a `gc` plan before applying it. The dataset's own repository runs your git hooks on dataset commits; those live in your `.git/`, which no clone brings. # Workspace state `.tether/workspace.toml` is per checkout (per jj workspace or git worktree): ``` toml workspace_id = "7c1e0a4d9b2e4f6a8c1d3e5f7a9b0c2d" bookmark = "relabel" # the dataset bookmark this checkout works on (absent: read-only) last_snapshot_at = "2026-09-04T22:10:03.118Z" [working_refs] # branches that exist (on the trunk: the upstream branches) "zarr/imaging" = "tether.ws.0a1b2c3d.relabel" "db/metrics" = "tether.ws.0a1b2c3d.relabel" [pending_forks] # decided by `new`, created on first writable open "features" = "tether.ws.0a1b2c3d.relabel" [pending_resets."features"] # the head `new` reviewed and agreed to reset on that open version = 9 # (a branch that moved since is refused instead) [base_states."zarr/imaging"] # committed state this working ref corresponds to snapshot_id = "K3F7Q2ZJ1XN4Y8W0" # (differs from the manifest => stale) [fork_points."zarr/imaging"] # base state when the branch was forked (for promote) snapshot_id = "K3F7Q2ZJ1XN4Y8W0" [last_snapshot."zarr/imaging"] # what a plain `status` shows (errors are kept too) snapshot_id = "K3F7Q2ZJ1XN4Y8W0" ``` A writable [open](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.open) moves an entry from `pending_forks` to `working_refs`. `pending_resets` keeps it from resetting a branch that moved since [new](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.new), and `base_states` drive the stale check. Every [open](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.open) re-reads the file, so a long-lived [Repo](../reference/Repo.md#tether.Repo) follows what another process's [new](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.new) wrote. # Operation log `.tether/ops.jsonl` is this checkout's operation log: one JSON line per store-writing command, appended only, like jj's `op log`. The VCS records the manifests, but it cannot see which refs [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit) created, which branches [new](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.new) forked or reset, what `gc` deleted, or where [promote](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.promote) moved a base branch from. Each entry holds the plan it applied, what it did, and what it replaced. `tether ops` lists the entries, and `tether undo` reverses from them. The log is a journal. A store-writing command appends its entry and syncs it *before* the first side effect, appends a progress record as each effect lands, and marks completion with the result. A process killed in between leaves an `INCOMPLETE` entry. `undo` refuses it, `repair --dry-run` says how far it got, and running the command again finishes what is left. The [new](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.new) and `gc` a `drop` runs are journaled as its steps (`parent`). An interruption loses at most the line being written, and the next entry starts on a fresh line. Other commands read the log too, so deleting it loses more than `undo`: - `gc` keeps every pin an unfinished operation in any live checkout set out to create; - [status](../reference/backends.VerifyReport.md#tether.backends.VerifyReport.status) and `ops` flag dataset commits that vanished from the VCS; - the first `gc` seeds the [pin index](reclaiming-storage.md#pins-this-clone-did-not-create) from the logs, so pins whose entries are gone by then are kept as another clone's. # Environment variables Every variable tether reads (`TETHER_REV`, `TETHER_JJ` / `TETHER_GIT`, `TETHER_NATIVE`, `TETHER_PUBLISH_DSN`, `NEON_API_KEY`, the Dolt `*_env` names, the cloud credential chains) is in one table in the [CLI Guide](cli.md#versions-and-environment), with the `GIT_*` variables tether ignores.