# Troubleshooting

Every refusal is a [TetherError](../reference/TetherError.md#tether.TetherError), printed as `error: ...` with exit status 1. Exit 3 means part of the work was done, and 2 is a usage error ([exit codes](cli.md#exit-codes-and-errors)). This page goes by error class, then by the messages the newer refusals print. From Python, catch the same classes: `from tether import StalePlanError`, and [RefMovedError](../reference/errors.RefMovedError.md#tether.errors.RefMovedError) from `tether.errors`. Every [BackendError](../reference/BackendError.md#tether.BackendError) message starts with the object's `[key]` or the backend's `[kind]`.


# By error class

| Error | It means | The way out |
|----|----|----|
| [StaleWorkingCopyError](../reference/StaleWorkingCopyError.md#tether.StaleWorkingCopyError) | An object's committed state changed since this checkout forked its branches (a teammate committed on a `--shared` bookmark, a bookmark moved by hand, a `jj undo`), or the checkout is on no bookmark. Writable [open](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.open) and [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit) refuse; reads work. | `tether new` forks again from the current pins ([Stale working copies](branching-and-writing.md#stale-working-copies)); on no bookmark, `tether new NAME` or `new -b NAME`. |
| [StalePlanError](../reference/StalePlanError.md#tether.StalePlanError) | What the plan saw no longer holds (the manifests, a branch head, history, the bookmark, the checkout), or the plan lacks a required precondition, was edited, or was made by another tether version. Nothing was written. | Plan again and read what moved ([What a plan checks](reclaiming-storage.md#what-a-plan-checks-before-it-applies)). |
| [RefMovedError](../reference/errors.RefMovedError.md#tether.errors.RefMovedError) | A store ref moved between the plan's check and the move. Nothing was written. | Someone else writes to that branch: look at it with the store's tools, then plan again. On a shared bookmark, `tether new --shared` works on the branch as it is. |
| [MergeConflict](../reference/MergeConflict.md#tether.MergeConflict) | [promote](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.promote)'s three-way merge (git, Dolt) stopped on conflicts, named in [conflicts](../reference/PromoteReport.md#tether.PromoteReport.conflicts). The base is untouched and the fast-forwards were held. | Resolve with the system's tools, `tether commit`, then [promote](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.promote) again; or `--strategy ff`, or `tether promote KEY...`. |
| [PinDriftError](../reference/PinDriftError.md#tether.PinDriftError) | A pin's ref points at another state than the manifest recorded: someone moved the tag. | [verify](../reference/ObjectStatus.md#tether.ObjectStatus.verify) reports it `drifted`, and `repair` never overwrites it. Move the tag back, or delete it and let `repair` recreate it. |
| [ImmutableObjectModified](../reference/ImmutableObjectModified.md#tether.ImmutableObjectModified) | A `--file immutable` object's *bytes* changed; `status --snapshot` labels it [error](../reference/ObjectStatus.md#tether.ObjectStatus.error) and exits 1, and `snapshot`, [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit) and `pull` stop. | If the change is legitimate: `tether remove KEY`, `tether add KEY ...`, [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit) ([Drift](pinning.md#drift)). |
| [UnpinnedStateError](../reference/UnpinnedStateError.md#tether.UnpinnedStateError) | `commit --strict` met an Observed object. | Drop `--strict`, or use a store that can hold a pin. |
| [CapabilityError](../reference/CapabilityError.md#tether.CapabilityError) | The backend cannot do what was asked: a writable [open](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.open) of a non-Forkable object, `open --rev` of an Observed one, `--at` on `neon` or `file`. | Check the object's tier ([Backends](backends.md)); open read-only. |
| [BackendError](../reference/BackendError.md#tether.BackendError) | The store said no: credentials, network, a missing ref (`pin missing` on a lazy fork), a busy Neon compute, active writers at [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit), a relative or in-checkout git [path](../reference/handles.GitHandle.md#tether.handles.GitHandle.path). | The `[key]` names the object. Credentials go in [`.tether/secrets.toml`](configuration.md#trust-boundary-tethersecretstoml). A missing ref: [verify](../reference/ObjectStatus.md#tether.ObjectStatus.verify), then `repair`. Active writers: wait, or `commit --force`. Transient: run the command again ([below](#an-interrupted-command)). |
| [MultiObjectError](../reference/MultiObjectError.md#tether.MultiObjectError) | Several objects failed in one fan-out (`snapshot`, [verify](../reference/ObjectStatus.md#tether.ObjectStatus.verify), [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit), `pull`, [new](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.new), [promote](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.promote), `gc`); `gc` and [promote](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.promote) also print what they applied. | Fix each object as its line says. `status --snapshot` labels failures [error](../reference/ObjectStatus.md#tether.ObjectStatus.error) instead, and exits 1. |
| [ConfigError](../reference/ConfigError.md#tether.ConfigError) | An invalid `tether.toml` or manifest, a key the trust boundary forbids, a dataset version that is not this tether's, a tracked per-checkout file, `no such object`. | The message says where a forbidden key belongs. An older dataset: `tether upgrade`; a newer one: upgrade `tether-vcs`. More below. |
| [VcsError](../reference/VcsError.md#tether.VcsError) | jj or git failed or is too old (jj 0.43, git 2.38); a commit hook refused; jj reports a conflicted bookmark or an unresolved `.tether/` conflict (`gc`, `drop`); a new commit lacks a manifest (an ignored directory). | Fix the VCS side (`jj bookmark list`, `jj resolve`, the ignore rule, the hook) and run the command again. |
| [TetherError](../reference/TetherError.md#tether.TetherError) (the lock) | Another command holds the checkout lock (30 s) or the repository lock (60 s); or, on Windows, a writing command. | Wait for, or stop, the other command. Write from a POSIX system or WSL ([Windows](caveats-and-performance.md#windows)). |


# By message

| The message says | What happened | The way out |
|----|----|----|
| `jj tracks .tether/workspace.toml` (or `secrets.toml`, `ops.jsonl`; or `git tracks`) | A clone brought a tracked copy of a per-checkout file, or one was committed by hand. `find` refuses it, and [open](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.open) does when a changed `workspace.toml` became tracked. | Review it (delete a cloned `secrets.toml`), untrack it with the command the message gives (`jj file untrack` / `git rm --cached`), and commit. The file stays on disk. |
| `cannot tell whether jj tracks this checkout's ...` | The VCS found through `PATH`, `TETHER_JJ` or `TETHER_GIT` failed; `secrets.toml`'s `jj_path` is read only after this check. | Put jj or git on `PATH`, or set `TETHER_JJ` / `TETHER_GIT`. |
| `skipping .../ops.jsonl of checkout ...` (a warning) | Another checkout's per-checkout file is tracked there, so `gc` does not count it as that checkout's record. | Untrack it in that checkout. |
| `this plan was made by tether X; this is tether Y: re-run the plan` (or `made by an older tether`) | A saved plan is valid only for the tether version that made it, and this one was saved by another (an upgrade of `tether-vcs` between `--plan` and `--from-plan`), or before plans recorded their version. | Plan again with the tether you will apply it with. |
| `was edited after it was saved` / `made in another checkout` | A saved plan fails its digest, or belongs to another checkout. | Plan again where you will apply it. |
| `has writes since the last commit ... tether commit them first` (promote) | A working branch holds writes the bookmark's commit does not record: uncommitted, undone, or committed with `--no-vcs`. | `tether commit`, then [promote](../reference/testing.ObjectBackend.md#tether.testing.ObjectBackend.promote). |
| `main (...) has commits this bookmark ... does not` / `has conflicting targets` (promote) | Landing would move the trunk bookmark backwards or sideways, or pick one side of a divergent trunk. | Merge or rebase the manifests onto `main` (or `jj bookmark set main -r REV`), or name keys to land a subset. |
| `base branch moved since the plan was made` (promote) | Another writer moved an upstream branch after the plan. | The message says whether planning again helps: it merges where the backend can, and otherwise refuses again unless the base only moved along the fork. |
| `refusing to reset working branches with unpinned writes` (new) | A branch holds writes no commit pins, and the message says whose. | Yours: commit, or `--discard`. A peer's: `tether new --shared` adopts the branch as it is. Writes that do not build on the bookmark's pin can only be discarded. A branch left by a dataset whose repository was lost: `tether new --adopt` ([Recovering a lost dataset repo](#recovering-a-lost-dataset-repo)). |
| `... is gone since the plan was made` / `is superseded by ...` / `no longer builds on ...` (new `--adopt`, `--shared`) | The branch a saved plan adopts was deleted, a store made a newer generation of it (`.2`) because it could not reset it, or it was reset onto something that does not build on the bookmark's pin. Writes on the branch alone never stale the plan. | Plan again: `--adopt` takes the newest generation. |
| `cannot undo ... (new): ... still works on NAME` | The checkout cannot go back from the bookmark that [new](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.new) put it on, because its working copy moved on. | `tether new PREVIOUS` leaves it; the branches stay until `gc --prune-bookmarks`. |
| `skipped workspace.toml: left ... (changed again by a later operation)` (undo ID) | `tether undo ID` restores only the fields nothing has changed since. | Nothing to do; later operations own those fields. |
| `... (drop NAME), which was interrupted ...` (undo, repair) | A `drop` died part-way; a drop is finished, not undone. | Run what the message names: `tether drop NAME` again, or `gc --prune-bookmarks` once the bookmark is gone. |
| `... is checked out in <worktree>` (git) | A fork, reset or promote would move a branch another worktree has checked out. | Check out another branch in that worktree, then run the command again. |
| `diff compares the working copy with its parent, but ...` | The jj working copy is a merge, with two parents. | `tether diff REV` compares one of them with the working tree. |
| `file storage_options: unknown client option ...` | A client option in `secrets.toml` is not one obstore takes, or its value cannot mean what the option needs. | Fix the key or value; any spelling of a known option works. |
| `a newer DuckDB has written this DuckDB-file catalog` (ducklake) | A DuckDB newer than the one tether runs with has written the catalog's metadata file, in a storage format older ones cannot read. DuckDB 2.0 does this to a 1.x catalog at its first write. | Upgrade `duckdb` where tether runs ([DuckLake and DuckDB versions](caveats-and-performance.md#ducklake-and-duckdb-versions)). |
| `tether.toml is version 4; this tether expects 5` | A 0.1.0b1-b3 dataset. | `tether upgrade`, then re-sync the other clones; 0.1.0b3 refuses the upgraded dataset. |
| `'lakefs': N manifest(s) skipped` (a note or warning from `gc`, [verify](../reference/ObjectStatus.md#tether.ObjectStatus.verify)) | History names a backend tether no longer ships. Its pins still count as references. | Nothing; remove the objects in a new commit when convenient. |
| `...takes no expected, so its ref moves are not conditional` (a `RuntimeWarning`) | A third-party backend predates conditional moves. | Update the backend ([Writing a Backend](extending.md#the-protocol)). |


# An interrupted command

A store-writing command journals its plan before the first side effect and marks completion afterwards. A process killed in between leaves an entry marked `INCOMPLETE` in `tether ops`:

- `tether repair --dry-run` lists what the operation got done, and what was planned.
- Running the same command again finishes what is left. [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit) first releases the pins its failed attempt created, unless the VCS commit landed; then the commit stands.
- `tether gc` keeps every pin an unfinished operation names, from any checkout of the repository.
- `tether undo` refuses it, since what it did is not fully known.

The journal: [Operation log](configuration.md#operation-log).


# Warnings [status](../reference/backends.VerifyReport.md#tether.backends.VerifyReport.status) prints

These are not errors, and the line says what to do:

- `(STALE: key, ...)` in the header: see [StaleWorkingCopyError](../reference/StaleWorkingCopyError.md#tether.StaleWorkingCopyError) above.
- `warning: bookmark ... is gone / renamed / moved / left`: the VCS moved the bookmark behind tether's back ([Workspaces and bookmarks](branching-and-writing.md#workspaces-and-bookmarks)).
- `warning: dataset commit ... is no longer in VCS history`: a `jj undo`, `jj abandon` or `git reset` removed a commit tether made. `gc` releases its pins, and `repair` recreates them if the commit comes back ([Undo](branching-and-writing.md#undo)).
- [error](../reference/ObjectStatus.md#tether.ObjectStatus.error) as an object's state: its fingerprint failed, now or at the last snapshot. The reason follows on stderr and [status](../reference/backends.VerifyReport.md#tether.backends.VerifyReport.status) exits 1.


# Recovering a lost dataset repo

The stores keep tether's refs when the repository with the manifests is gone, such as a clone that was never pushed, or a lost disk. Every pin is a ref `tether.<id>.<hash>` (a tag, in most stores), and the hash covers the object, its state and the dataset id. Every bookmark has a working branch `tether.ws.<id>.<bookmark>`; Lance and Neon add `.2`, `.3` when they cannot reset a branch in place. A new dataset under the old id takes them back:

1.  **A fresh repository, the old id, the objects.** In a new jj or git repository, run `tether init --dataset-id ID` with the old id, if you know it (if not, any id: step 2 finds it). Then `tether add` each object with its old locator. Kinds whose locator is a path need no key: `tether add s3://lab/imaging.icechunk --kind icechunk` registers `imaging`.

    ``` bash
    jj git init lab-data && cd lab-data
    tether init --dataset-id 0a1b2c3d
    tether add s3://lab/imaging.icechunk --kind icechunk
    ```

2.  **`tether recover`** lists tether's refs in each object's store by the dataset id they carry: the pin count, and each bookmark with a working branch (and its generation, when there are several). It marks the id `tether.toml` names, prints what to run next, and writes nothing. Each step is a command, its values quoted for the shell, followed by a `#` note; a step with nothing to run is a note alone. If a store can't be listed, it names that object, suggests nothing, and exits 1: fix access, or leave the object out with `tether recover KEY...`. A run scoped to some keys suggests no dataset id and no [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit): a commit pins every object under the current id, which then can no longer change, so an id only a store left out holds would be lost. Its last step is `tether recover` without keys.

        $ tether recover
        dataset id 0a1b2c3d (tether.toml)
          imaging  [icechunk]
            0a1b2c3d (this dataset): 5 pin(s); bookmarks: relabel, sweep
        next steps:
          1. tether commit -m 'Recover main'
             # pins each object's upstream branch on the trunk (main); a state pinned before reuses its tag
          2. tether new -b relabel --adopt -- main && tether commit -m 'Recover relabel'
             # bookmark relabel, made afresh off the trunk: takes its branches as they are, uncommitted writes included, and pins them
          3. tether new -b sweep --adopt -- main && tether commit -m 'Recover sweep'
             # bookmark sweep, made afresh off the trunk: takes its branches as they are, uncommitted writes included, and pins them

    If a manifest already names a pin its store lacks (a tag deleted by hand, say), the first step is `tether repair -- KEY...` for those objects: a [commit](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.commit) does not recreate a pin whose state is unchanged. A run scoped to some keys repairs only the selected ones.

3.  **Set the id, if it is not the old one.** When the refs carry another id and nothing is pinned under this dataset's yet, `recover`'s first step is a note with what to put in `tether.toml`:

          1. # set this dataset's id in tether.toml (nothing is pinned under 12345678 yet):
             # [dataset]
             # id = "0a1b2c3d"

    The id can change only while nothing is pinned under it ([Dataset id](configuration.md#dataset-id)). Once something is, `recover` says so, and the way back is a fresh repository with `tether init --dataset-id`.

4.  **Commit on the trunk.** `tether commit` pins each object's upstream branch. A state the old dataset pinned gets the same pin id, so its tag is reused and no new tag is made.

5.  **Adopt each bookmark, and commit.** `tether new -b NAME --adopt -- main` takes the bookmark's branch as its working ref as it is: no reset, no copy, and writes nobody committed stay. [status](../reference/backends.VerifyReport.md#tether.backends.VerifyReport.status) shows them as [modified](../reference/testing.ObjectDiff.md#tether.testing.ObjectDiff.modified), and `tether commit` pins them. An object whose store has no branch for the bookmark forks from its pin, as [new](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.new) always does. A bookmark whose name had `/`, `.` or a space is listed as its branch spells it (`feat-x-1a2b3c` for `feat/x`), and either name finds the branch. Where the VCS still has the bookmark, `recover` suggests `tether new --adopt -- feat/x`. Where it does not, the original name cannot be read back from the branch: `recover` suggests `tether new -b feat-x-1a2b3c --adopt -- main`, which takes the branch under the escaped name. Every suggested command ends its options with `--`, since a key or a git branch name may start with `-`.

6.  **Copy legacy branches.** Datasets from before bookmarks named working branches per workspace, `tether.ws.<id>.<workspace>.<key>`. `recover` lists them under their dataset id as legacy branches of workspace `<workspace>`, since one may hold the only copy of writes nobody committed. `new --adopt` does not take them, and `restore` refuses the trunk, so each step first moves to a bookmark `recover-<workspace>`, made off the trunk by the workspace's first step and joined by the rest. An existing bookmark of that name is joined only if it holds nothing but an earlier recovery's commits and registers the objects; otherwise the steps make `recover-<workspace>-2` and leave it alone. Then `tether restore --at REF -- KEY` copies the branch onto the bookmark's branch, and `tether commit` pins it before the next step's [new](../reference/vcs.VcsAdapter.md#tether.vcs.VcsAdapter.new), which would reset it otherwise. Objects that share a branch space, such as two Neon databases on one branch, are restored together in one `tether restore --at REF -- KEY1 KEY2`:

          2. tether new -b recover-ab12cd34 -- main && tether restore --at tether.ws.0a1b2c3d.ab12cd34.imaging-3f2a1b -- imaging && tether commit -m 'Recover tether.ws.0a1b2c3d.ab12cd34.imaging-3f2a1b'
             # tether.ws.0a1b2c3d.ab12cd34.imaging-3f2a1b is a legacy branch of workspace ab12cd34 (named before bookmarks) and may hold writes no commit pins: this copies it onto bookmark recover-ab12cd34 and pins it

The VCS history is lost for good: the commit messages, and which states were committed together. The pins survive, but nothing says which of them made up one version of the dataset. An old pin that no new commit reuses, such as a bookmark's earlier commits, is named by no manifest and was not created by this clone. `gc` therefore plans it as `keep-pin` until `tether gc --release-foreign`. To keep one as a starting point, `tether restore KEY --at tether.<id>.<hash>` on a bookmark copies it onto the bookmark's branch.


# Removing tether from a repository

There is no `tether uninstall` yet (it is on the roadmap), so the recipe is manual. The point is the refs tether left in the stores: every pin is `tether.<id>.<hash>` (Icechunk repins add `.2`, `.3`), and every working branch `tether.ws.<id>.<bookmark>`, where `<id>` is the `[dataset] id` in `tether.toml`. Nothing else in a store is tether's.

1.  **Let tether release what it can.** `tether drop NAME --no-dry-run` for every bookmark but the trunk releases their pins and branches; add `--delete-stores` to also delete the stores `add --create` made on them. Then `tether gc --prune-bookmarks --no-dry-run`, with `--release-foreign` if this is the last clone. A created store that a manifest in history still names (one the trunk committed) is kept, never deleted: remove it by hand in step 3.

2.  **Find the stores.** Every commit's manifests name them: `tether export refs.sqlite` puts them in one table (`SELECT DISTINCT kind, uri FROM objects`), or read `.tether/objects/*.toml` at each commit.

3.  **Delete the remaining refs, per store.** History still names the trunk's pins, so they go by hand:

    | Kind | List | Delete |
    |----|----|----|
    | `icechunk` | `repo.list_tags()`, `repo.list_branches()` | `repo.delete_tag(name)`, `repo.delete_branch(name)` |
    | `lance` | `ds.tags.list()`, `ds.branches.list()` | `ds.tags.delete(name)` first, then `ds.branches.delete(name)` (a branch a tag references cannot be deleted) |
    | `git` | `git tag -l 'tether.*'`, `git branch --list 'tether.ws.*'` | `git tag -d NAME`; `git push REMOTE --delete NAME` where pins were pushed; `git branch -D NAME` |
    | `neon` | the project's branch list (console or API), names `tether.*` | the pin branches first (children of the working branches; unprotect the protected ones), then the `tether.ws.*` branches |
    | `iceberg` | `table.refs()` | `table.manage_snapshots().remove_tag(name)` / `.remove_branch(name)` |
    | `dolt` | `SELECT tag_name FROM dolt_tags`, `SELECT name FROM dolt_branches` | `CALL DOLT_TAG('-d', name)`, `CALL DOLT_BRANCH('-D', name)` |
    | `delta`, `ducklake`, `file` | - | nothing: these hold no refs. A `file` directory `add --create` made has a `.tether-owner` marker to remove |

    Delete only names carrying *this* dataset's id. Releasing a ref makes the data eligible for the system's own expiry (`expire_snapshots`, `cleanup_old_versions`); run that afterwards to reclaim the bytes.

4.  **Remove the files.** Delete `tether.toml` and `.tether/` from the working tree and commit. Delete `tether.lock`, `tether-pinned.jsonl`, `tether-created.jsonl` and `tether-touched.jsonl` from the VCS's shared directory (`.jj/repo/` or `.git/`). Every other checkout deletes its own `.tether/`. Old commits keep their manifests as text; rewriting history to remove them is optional, and the VCS's job.