Beads Worker Recovery: Leases, Stale Owners, and Concurrency

A crashed AI worker leaves two separate problems: its task needs a new owner, and its old attempt must lose the ability to finish someone else’s work. Beads provides leases and reclamation for the first problem. For the second, your adapter must use a conditional mutation and keep external actions behind an application-controlled boundary. A recovered task alone does not establish safe recovery.

There is a third problem when work branches and converges: both prerequisites can finish while their dependent stays absent from the ready queue. That is a readiness failure even when the workers completed useful work.

The earlier Beads integration guide explains the planner, skill adapter, task graph, and basic claim loop. This follow-up examines the failure boundary: stop a real synthetic worker, reclaim its lease, give the task to a replacement, replay the stale attempt, and close two prerequisites concurrently. The downloadable lab uses the real CLI against an isolated local database; it does not call a model or operate on a live project.

What changed in Beads, and which version this covers

Beads 1.3.0, released September 15, 2026, introduced the recovery primitives used here: claims with a default five-minute lease, bd heartbeat, bd reclaim, bd unclaim, and atomic updates guarded by the current assignee or status. Beads 1.3.0 release

Beads 1.3.1, released September 30, added a post-commit repair for fan-in: concurrent removal of blockers could otherwise leave a dependent’s stored is_blocked flag set and hide it from bd ready. Its release notes describe coverage across direct Dolt and proxied write routes. Those notes explain the intended fix; a local passing run cannot establish every interleaving or topology. Beads 1.3.1 release

The lab pins the Windows AMD64 Beads 1.3.1 binary at revision c1c4b642a, with a separate Dolt 2.2.0 SQL server on loopback, the external version specified in the pinned Beads documentation. This is a direct server-mode CLI test, not a test of embedded single-writer mode, the Beads HTTP proxy, or federated replicas. Pinned external-server requirement

Sources were checked on October 14, 2026. A relevant newer prerelease, 1.3.2-rc.2, already carries another repair: publishing the blocked-state recheck after its SQL transaction commits. We did not execute that release candidate. Keep that ordering defect in view when assessing 1.3.1; passing ready assertions below do not prove it absent. Release candidate and backport, upstream repair #6876

Expiry is eligibility for recovery, not instant revocation

Consider a worker attempt named worker-a-run1 holding an in_progress task. Its heartbeat refreshes the lease. If the process dies, the clock keeps advancing, but the task does not automatically become open at the expiry timestamp. A supervisor has to run reclamation.

bd reclaim --older-than is a grace period after lease expiry, not an age since the last heartbeat. With the default five-minute TTL and ten-minute grace, the earliest eligibility is about fifteen minutes after the last renewal. The supervisor’s polling cadence adds more delay. --older-than 0s removes the extra grace; it still waits for the lease to expire. Pinned reclaim command

The pinned heartbeat implementation can renew an expired lease if reclamation has not yet removed it and the holder still matches. Expiry and revocation therefore have different meanings. After a replacement with a distinct owner claims the task, the old owner’s heartbeat fails. Pinned lease operations

Event	What the supervisor can conclude
Heartbeat stops	Worker may be dead, paused, partitioned, or overloaded
Lease expires	Task is eligible for reclamation after the configured grace
Reclaim commits	Old ownership has been cleared in this store
Replacement claims	New attempt owns the task; old work must be checked before acceptance

A timeout is a recovery policy, not proof that the previous process has stopped. Choose heartbeat cadence and grace for your actual pauses and network delays. The lab deliberately uses zero grace to keep the experiment near five minutes; that is an experimental setting, not a production recommendation.

Run the isolated recovery lab

Download recovery-lab.mjs. It contains the worker subprocess and assertions in one file. Obtain the Windows AMD64 archives and checksums from the official Beads 1.3.1 release and Dolt 2.2.0 release, verify the downloads, and extract each binary. Node 24.12.0 and an installed Git executable were used for this run. No npm install is required.

Set the executable paths in PowerShell, using the folders where you extracted them:

$env:BD_BIN = (Resolve-Path ./beads/bd.exe).Path
$env:DOLT_BIN = (Resolve-Path ./dolt/bin/dolt.exe).Path
node ./recovery-lab.mjs

The lab creates its own temporary project and Dolt data directory, starts its own loopback server, and initializes Beads without installing hooks or agent instructions. All records are synthetic. Its worker reports that the claim completed before the parent kills the worker process, so the test interrupts execution after a successful claim rather than injecting a database crash mid-write.

It then waits for natural lease expiry. There is no direct SQL timestamp edit, shortened fake TTL, or pretend Beads implementation. The recovered stale attempt is a fresh process using the killed attempt’s saved identity and task ID; Windows does not resurrect the killed process. This models a delayed report or restarted worker retaining obsolete run state.

The ownership transition to inspect

Read the task after every successful transition. The essential sequence is:

1. Claim as attempt A and confirm in_progress with A as assignee.
2. Stop A; a scoped reclaim before expiry must recover nothing.
3. Wait for expiry, then reclaim only the selected task.
4. Confirm open and unassigned; claim as a distinctly named attempt B.
5. Replay A’s heartbeat and conditional completion; both must fail.
6. Confirm B still owns the in_progress task; allow B’s guarded completion.

The isolated harness assigns fixed synthetic IDs, including lab-recovery; in an ordinary worker loop, use the ID returned by creation or claim. Its reclaim invocation supplies both --id and --older-than 0s; it does not sweep unrelated work. ready is a query over eligible tasks, not a status to write into an issue.

Observed results

The exact PowerShell entry point above completed successfully on Windows 11 with the pinned binaries. The final run reclaimed the task 301,524 milliseconds after the worker’s last reported heartbeat. That is one recovery trial; it is not a recovery-latency benchmark.

Check	Final observed result
Scoped reclaim before expiry	Count 0; A remained the owner
Scoped reclaim after expiry	Count 1; task became open, unassigned, and ready
Replayed A heartbeat	Exit 1: task already claimed by worker-B/run1
Replayed A conditional completion	Exit 13 with guard_mismatch: true; B retained in_progress
B conditional completion	Exit 0; task closed
Two prerequisite closes	10/10 rounds: both prerequisites closed and dependent ready
Two claims for one task	10/10 rounds: one success, one ownership-conflict refusal
Ordinary close by former owner	Exit 1; replacement ownership and status preserved
Supervised A → B → A reuse	Old identity’s guard matched; completion exited 0

The stale worker’s output was:

{"heartbeatExit":1,"completionExit":13}

Download the observed results and CLI transcript to inspect the evidence. Local filesystem paths are replaced with placeholders; synthetic IDs, arguments, timings, errors, and output are retained. The lab produces fresh raw evidence in its temporary folder when you run it. Research, writing, artwork, and lab development were AI-assisted; executed outputs and independent source/code review supplied the validation.

Test stale completion on the real command path

The control is a separate synthetic task: assign it to B, then let former owner A issue ordinary bd close. The tested CLI refuses this request with exit 1, identifying the different assignee and actor. That is a useful protection on the actual command path. It would be incorrect to infer ordinary CLI behavior from the storage close function alone: the CLI adds its own preflight policy. Pinned CLI close path

That ownership preflight runs before the close batch, whose items do not carry a saved owner or lifecycle revision. Source inspection therefore distinguishes it from an atomic ownership comparison across a concurrent handover. This lab observes an already changed owner; it does not force or demonstrate a handover between the preflight and write. Preflight and batch request, batch precondition contract

The separate control uses an explicit supervisor --force reassignment to create the handover. Workers should not receive that administrative capability. An adapter that gives them arbitrary CLI flags or lets them impersonate the replacement has failed its own authorization boundary.

For its main recovery completion path, the lab passes the saved attempt identity into both the audit actor and --if-assignee, with --if-status in_progress. This makes the intended condition explicit inside the mutation transaction; an ownership mismatch leaves the row unchanged. A run consisting only of update guard mismatches exits 13. That is a different error contract from ordinary close refusal; inspect the command’s actual exit and JSON rather than treating every failure as a retry request. Pinned guarded update handling

For the ordinary task issues used here, this conditional status transition still enforces the default policy for open children and live direct blockers. It is an application completion pattern after acceptance, not a replacement for every bd close workflow: CLI-only template/pin checks, machine gates, close reasons, and molecule continuation need separate consideration. Your adapter must retain the relevant acceptance and permissions policy. Pinned update request contract, CLI-specific close policy

Give each attempt a distinct identity

If an orchestrator reuses worker-a for attempt 1 and attempt 2, an old --if-assignee worker-a guard can match the new assignment. This is the familiar A → B → A problem: equality of the final name hides intervening ownership changes. The lab includes a separate supervised A → B → A reassignment and checks a completion carrying the reused name. Use a new run value, such as worker-a-run1 and worker-a-run2.

Changing punctuation alone is insufficient: the pinned owner comparison treats separator variants as equivalent. Nor is a unique name a security credential. A worker with arbitrary write access can supply another actor or omit the guard. The adapter must derive the permitted identity from trusted run state and expose only the authorized mutations. Owner comparison contract

The native HTTP close operation offers a different concurrency condition: expected_version, an opaque string revision captured from the claim or read response. Keep the revision associated with that attempt; do not fetch a replacement’s newer revision and attach it to a stale result. A stale revision produces a typed precondition conflict. This is verified source behavior, not an HTTP integration executed by this lab. The revision tracks lifecycle and ownership changes, not every graph or label edit. Pinned OpenAPI contract, revision scope

When two prerequisites finish together

The graph is small: “Release synthetic feature” depends on both “Finish schema” and “Finish tests.” Blocking edges point from the dependent to each prerequisite. Initially the dependent must be missing from bd ready; after just one closes it must still be missing.

The concurrency case uses a fresh graph and two separate bd close processes launched together against the same SQL server. Starting both processes together allows overlap, but does not force an exact SQL interleaving. After both commands return successfully, the harness checks both prerequisite statuses and the dependent’s membership in bd ready.

Beads materializes blocked state to make queue queries practical. Before the 1.3.1 repair, each transaction could see the other blocker as open and preserve the dependent’s blocked flag; after both committed, that stored flag could remain stale. The repair revisits affected dependents after the write commits. Pinned affected-dependent repair

The test’s assertion is stronger than “both close commands exited zero”: it examines the queue the next worker actually reads. It also races two claims for one task and checks that only one wins. Claim exclusivity and dependency freshness are separate invariants; success at one does not imply the other.

For production queue selection, use bd ready --claim with your trusted attempt actor and queue scope. A named bd update <id> --claim checks ownership and status, but does not establish dependency readiness; a separate ready read followed by that claim leaves a graph-change race. The lab uses named claims on fixed synthetic tasks to isolate ownership behavior. It creates durable issues: ephemeral wisps included through --include-ephemeral do not receive the lease used by this recovery protocol. Named-claim contract, ready-claim contract

A readiness check belongs in recovery

When a close reports success but the next stage never starts, inspect the task graph and committed statuses before assuming the scheduler is idle. A scoped graph investigation should answer whether a real live blocker remains or stored readiness is stale. The documented repair command, bd recompute-blocked, is a diagnostic/recovery tool; routinely running it after every operation can hide a regression.

Monitor errors from post-commit repair as well as primary mutation failures. In the pinned store, a repair error can be logged after the original mutation committed. A successful write is not a universal promise that every later queue read is fresh. Blocker additions, reopens, sync merges, and other topologies need their own tests; our concurrent close example does not cover them. Pinned Dolt recheck and error handling

What recovery still cannot promise

Task ownership does not fence an external effect. An old attempt can send an email, publish an artifact, or write another database after its Beads task is reclaimed. If the target accepts that request independently, a Beads guard cannot undo it. Put an attempt generation or revision check at the effect-accepting boundary and use an idempotency key for retries. A fresh ownership read followed by an unconditional external call still has a race between those operations.

For a concrete treatment of expired locks, owner tokens, and version checks, see the n8n external-memory guide. For retries across separately committed systems, the state-ownership comparison explains why the source of truth and outbound delivery need explicit contracts.

Replica scope changes the meaning of liveness. Beads leases are local ephemeral state; heartbeats are not a durable federated event stream. The granting-replica guard is opt-in through BEADS_NODE_ID or local configuration. Clients of one shared store should represent that same replica. --any-replica deliberately overrides the guard; reserve a narrowly scoped override for a replica known to be gone. A stale remote view cannot establish whether its worker is dead. Our single-store test makes no federation claim. Pinned replica-aware reclamation

A stopped worker is not a stopped database. The experiment checks successful writes surviving worker termination and being read by later CLI processes. It does not kill Dolt during commit, remove power, restore backups, simulate partitions, or demonstrate exactly-once execution. Those are different fault models. The agent-testing guide provides the broader validation structure.

The practical completion rule is therefore: accept a result only for the currently authorized attempt, apply its task transition conditionally, and verify the dependent queue after concurrent graph changes. Beads supplies useful coordination primitives. The orchestrator has to make them the only path by which stale work can be accepted.