Do the work
An agent investigates a task, reads its requirements and encounters changes or interruptions. Each method gets the same starting evidence.
Benchmark methods
How we tested continuity.
Configurations, task sets, costs and run records behind the results.
Fresh sessions · Switching tools · Native resume · Compaction · Funded retention · Records
104 trials · eight task families · two coding tools
Fresh sessions: outcomes and API costs.
Crosshead completed 16/16 workflows for $1.62 in model API costs. The tasks tested exact requirements, corrections, conflicting evidence and interrupted work.
| Method | Workflows completed | Total API cost |
|---|---|---|
| Crosshead | 16/16 | $1.62 |
| Agent-written handoff · updated when needed | 16/16 | $1.88 |
| Structured handoff + saved sources | 16/16 | $2.22 |
| Cairntir | 16/16 | $4.92 |
| claude-mem | 16/16 | $4.29 |
| Supermemory Local · configured | 11/16 | $2.89 |
Separate Claude-only auto-memory comparison
In the separate Claude-only comparison, both methods completed 8/8 fresh-session workflows. Crosshead cost $0.91; auto memory cost $1.21. Auto memory carries saved notes into a fresh session; resuming an existing conversation is a different workflow.
Eight synthetic task families created by Crosshead, using GPT-6.1 Sol and Claude Opus 5.5. One run per task and tool pairing. Costs include coding and background model calls; subscriptions, service fees, local compute and operator time are excluded.
Crosshead and both handoff controls reached the 16/16 ceiling. Compared with update-only handoffs, the difference was $0.26 across 16 workflows (13.7%). Against claude-mem it was $2.67 (62.3%). This does not establish a capability advantage over a handoff or savings on a subscription plan.
Collected October 21, 2026. This is the complete 104-trial same-tool lane; the parent run remains at 116/128. No completed cross-tool competitor ranking is inferred.
Tested configuration: free hot memory in an isolated local-only benchmark profile. Ordinary installations sync to the hosted service. Paid context packs, reflection and long-term retrieval were disabled and have no comparative outcome evidence in this panel.
This comparison starts a fresh session within the same tool. See the separate switching-tools results below.
Download the fresh-session results and evidence · How the comparison works
Test design, configurations and supporting evidence
The eight task families are changed-question, decimal-display, effect-recovery, interleaved-routing, negative-exceptions, revision-gate, scoped-correction, source-ancestry. Each task ran once per model and method; Claude Code auto memory was tested only on Claude.
Results by coding tool
| Method | Fresh Codex | Fresh Claude Code |
|---|---|---|
| Crosshead | 8/8 $0.71 | 8/8 $0.91 |
| Agent-written handoff · updated when needed | 8/8 $0.80 | 8/8 $1.08 |
| Structured handoff + saved sources | 8/8 $0.98 | 8/8 $1.24 |
| Cairntir | 8/8 $1.69 | 8/8 $3.24 |
| claude-mem | 8/8 $1.64 | 8/8 $2.66 |
| Supermemory Local · configured | 5/8 $1.09 | 6/8 $1.80 |
| Claude Code auto memory | — | 8/8 $1.21 |
Models, integrations and equal access to evidence
Both coding models use medium reasoning effort through Codex codex-cli 0.159.3 and Claude Code 2.1.302 (Claude Code). Every method receives the same instructions and original source access in isolated workspaces. Source files disappear before continuation; each method can preserve them through its supported memory or handoff mechanism. Fresh sessions cannot recover the old transcript.
Crosshead uses real automatic hooks and free hot memory with hosted synchronization disabled for the benchmark. The agent-written control maintains up to 500 words, updated only when relevant facts change. The structured control can retain 6,000 characters of task state and six source excerpts totaling 12,000 characters through ordinary file writes and retrieval. Crosshead has the same retained-source budget and a 4,200-character automatic delivery budget.
Cairntir 1.12.3 and claude-mem 14.0.1 use their official integrations and local default embeddings. Claude Code auto memory runs with its native instructions enabled. Each coding session has one CPU, 2 GiB of RAM, no swap and 180 seconds. Background capture can take up to 140 seconds for claude-mem and 600 seconds for Supermemory Local. Capture and fresh recall are verified before scored trials.
Supermemory Local configuration
Supermemory Local 0.0.9 uses official Codex 1.0.21 and Claude Code 0.1.13 plugins, GPT-5.1 extraction, v5 REST, instant processing and synchronous capture. These settings support one-shot CLI sessions. Hosted Supermemory is a separate service and was not measured here. Every background extraction call counts toward cost, including OpenAI calls during Claude trials. All five unsuccessful workflows remain in the results.
Cost differences and uncertainty
| Method | Lower API cost | 95% family interval |
|---|---|---|
| Structured handoff + saved sources | 27.0% | 23.0% to 31.3% |
| Supermemory Local · configured | 44.0% | 40.5% to 47.2% |
| Agent-written handoff · updated when needed | 13.7% | 9.6% to 17.6% |
| claude-mem | 62.3% | 61.3% to 63.4% |
| Cairntir | 67.1% | 63.7% to 70.2% |
| Claude Code auto memory | 24.9% | 17.2% to 32.2% |
Native memory is paired only with the same eight Claude workflows. These are vendor-authored synthetic tasks, not an independent holdout or a production success-rate estimate. One repeat and perfect observed scores do not prove equal reliability across future work.
Capture, delivery and task success
Automatic capture preserves the request, observed source excerpts and assistant findings separately. Same-task updates retain prior decisions; conflicting observations remain visible. Capture and delivery are bounded, and incomplete excerpts are marked.
At 20 continuation boundaries, 0 immediately preceding excerpts and 4 excerpts from earlier sessions were not delivered in full. The downloadable audit retains these observations separately from task scores. A passing task does not certify complete retention, and an omitted earlier excerpt is not automatically a missing requirement.
Collection scope and reproducibility
All 104 planned same-tool observations are included, regardless of outcome. The 128-trial execution also contains cross-tool comparisons; 12 of those remain unrun. This publication covers the completed same-tool lane and makes no cross-tool ranking from it. Candidate, model settings and scoring stayed fixed; the product was not tuned on these results.
The scoped-correction prompt was clarified to distinguish loss of source-file access from expiry of the requirements. Every observation for that family was replaced across methods and hosts, including passes. Original outcomes, transport interruptions, the zero-cost integration setup correction and all retained charges remain in the download. No valid task failure was dropped.
Shared test procedure
Start a fresh session
The next agent gets the working files and the method’s saved context. Some workflows also switch between Claude Code and Codex.
Check the outcome
Hidden tests check the finished work. We count the API cost of every session and any background capture, then inspect what reached the next agent.
Switch tools · six tasks · two repeats · both directions
Switching tools
Crosshead completed 12/12 workflows in each direction, matching written handoffs, at 21–22% lower API cost. Cold restarts completed 8/12 in each direction.
This six-task regression comparison uses a handoff written at every session boundary. Its savings apply to that control; the fresh-session comparison above uses handoffs updated only when needed.
Switching tools: results and method
| Model / direction | Method | Passed | Input + output tokens | API cost (USD) |
|---|---|---|---|---|
| GPT → Claude | Cold restart | 8/12 | 1,041,337 | $1.07 |
| GPT → Claude | Written handoff | 12/12 | 1,250,118 | $1.34 |
| GPT → Claude | Crosshead · fresh session | 12/12 | 1,034,465 | $1.05 |
| Claude → GPT | Cold restart | 8/12 | 1,155,509 | $1.12 |
| Claude → GPT | Written handoff | 12/12 | 1,127,693 | $1.45 |
| Claude → GPT | Crosshead · fresh session | 12/12 | 971,141 | $1.13 |
Crosshead used 17.3% fewer tokens and cost 21.5% less for GPT → Claude; 13.9% fewer tokens and 22.0% less for Claude → GPT. The task-bootstrap cost-savings intervals were 16.9–26.5% and 17.3–26.8%, respectively. Native resume cannot continue the other tool’s conversation. These development tasks were available during product fixes.
Keep context through compaction · two-task pilot
Native compaction pilot
Crosshead completed 4/4 workflows through two native compactions and 4/4 with fresh sessions. Update-only handoffs also completed all four fresh-session cases.
In the fresh sessions, Crosshead cost 5.9% less on GPT and 10.3% less on Claude than those update-only handoffs.
A controlled two-task pilot with manual compaction and outside-workspace transcript recovery disabled. It tests retained context under those conditions; default automatic compaction and dynamic timing remain unmeasured.
Compaction: results and method
| Model / direction | Method | Passed | Input + output tokens | API cost (USD) |
|---|---|---|---|---|
| GPT-6.1 Sol | Native compaction | 2/2 | 1,532,821 | $0.93 |
| GPT-6.1 Sol | Crosshead + native compaction | 2/2 | 1,386,720 | $0.88 |
| GPT-6.1 Sol | Written handoff | 2/2 | 1,681,919 | $1.04 |
| GPT-6.1 Sol | Crosshead · fresh session | 2/2 | 1,462,747 | $0.98 |
| Claude Opus 5.5 | Native compaction | 1/2 | 931,194 | $2.94 |
| Claude Opus 5.5 | Crosshead + native compaction | 2/2 | 1,138,668 | $3.27 |
| Claude Opus 5.5 | Written handoff | 2/2 | 1,127,826 | $1.84 |
| Claude Opus 5.5 | Crosshead · fresh session | 2/2 | 658,702 | $1.65 |
We tested a disappearing behavior contract and a corrected contract, with two large irrelevant logs between investigation and implementation. Resumed conditions used real native manual compaction twice. Fresh-session conditions kept Crosshead memory or a handoff note updated only when relevant facts changed. Context pressure stayed below the default full-window threshold.
Native compaction passed 3/4 under the transcript restriction; the failed Claude case lost exact contract rules. With same-session transcript recovery allowed in a targeted follow-up, Claude retrieved the contract and passed ($1.57). That diagnostic was selected after the failure and is excluded from comparison rates.
The records retain calibration attempts and two Codex runs affected by skipped compaction hooks. Both Codex cases were rechecked with verified hooks; all attempts count toward the campaign budget. Two regression tasks do not establish a general success rate or an optimal compaction point.
Continue the conversation, or start fresh · 24 tasks · two repeats
Native-resume regression
Crosshead completed 48/48 workflows on each model, matching native resume and written handoffs. Cold restarts completed 38/48 on each.
Cold restarts failed 10 runs across five distinct tasks per model. Each of the 24 tasks ran twice for each model and method; repeats are not additional tasks.
Resume is a good choice when you want to continue the existing conversation in the same tool. It cost less in this comparison: Crosshead used 17.4% more API spend on Claude and 58.1% more on GPT. Crosshead gives you continuity when you want a fresh thread or a different coding tool.
Native resume: results and method
| Model / direction | Method | Passed | Input + output tokens | API cost (USD) |
|---|---|---|---|---|
| Claude Opus 5.5 | Cold restart | 38/48 | 2,749,605 | $3.81 |
| Claude Opus 5.5 | Written handoff | 48/48 | 3,406,591 | $5.49 |
| Claude Opus 5.5 | Native resume | 48/48 | 2,494,588 | $3.31 |
| Claude Opus 5.5 | Crosshead · fresh session | 48/48 | 2,675,191 | $3.89 |
| GPT-6.1 Sol | Cold restart | 38/48 | 5,577,704 | $4.23 |
| GPT-6.1 Sol | Written handoff | 48/48 | 6,011,964 | $4.81 |
| GPT-6.1 Sol | Native resume | 48/48 | 4,962,301 | $2.64 |
| GPT-6.1 Sol | Crosshead · fresh session | 48/48 | 5,108,872 | $4.17 |
These synthetic development/regression tasks were used during memory fixes. The written control saves up to 500 words at every boundary, including unrelated interludes. Native resume retains the original conversation; a cold restart receives only workspace state.
The campaign retains 480 comparison records: 384 main trials, 72 switching trials and 24 quota rechecks. Eight main trials were interrupted by an Anthropic quota outage. All methods and repeats for the three affected families were rechecked, including passes; the displayed totals use those complete replacement blocks.
Separate controlled replay · October 26, 2026
What happens beyond free working history?
Using normal host-adapter delivery, funded assist completed 0/12 continuations; free memory completed 0/12 after the same recorded histories deliberately exceeded the 12-checkpoint free limit.
Both arms received explicit approval to save the contract durably. Funding allowed those writes, but current Claude Code and Codex turn hooks request hot memory only. Warm and cold archives were outside the receiver’s retrieval scope. Funding alone did not restore the older contract in this configuration.
This is a retention-capacity diagnostic, not a general paid-tier advantage. Three vendor-authored development tasks, two models, two repeats. Captured histories were replayed through the runtime and service; receivers used the shared host adapter, including its free fallback, normal scopes and budgets. This is not a native CLI capture trial. The backend used local Postgres and object storage with real OpenAI embeddings.
Correction: the first attempt’s 12/12 funded versus 0/12 free comparison was invalid and is withdrawn. Its harness omitted the free adapter fallback and requested broader paid retrieval. The corrected attempt preserves every new outcome; the original artifacts remain available for audit.
Corrected-attempt provider usage, priced at frozen rates, cost $0.0775. Simulated service consumption was 2,736 credits ($2.736), reported separately; no production account was funded. Counting both attempts, provider and nominal service accounting totals $5.646, within the approved $25 limit.
Corrected outcomes, delivered context, model responses and frozen plan · Invalid first attempt and reason for withdrawal.
Scope and earlier comparisons
Earlier runtime losses remain published. In the completed 124-trial competitor baseline, Crosshead passed 25/28 versus 28/28 for update-only handoffs and cost $3.70 versus $2.97. One cross-tool contract-loss failure has not been remeasured with the improved runtime. The later panel uses a different frozen runtime and tasks; it does not erase those outcomes.
The native CLI comparisons above test free memory; the separate controlled replay tests a specific paid retention path. The local-only benchmark configuration is distinct from the hosted sync used by ordinary installations. Model API cost is not a measurement of subscription bills or usage-limit relief.
Each comparison answers a different workflow question. Results include model API usage, with cached input counted once and reasoning included in output. Service fees, subscriptions, local compute and operator time are excluded. Task success and context delivery are checked separately.
These are vendor-authored synthetic tasks. Sample sizes, controls and runtime versions differ by comparison; the records below identify each frozen configuration. They help you assess the tested workflows, rather than predict a production success rate.
Downloadable results and versioned run records
Runs per task: the fresh-session and earlier 60-trial decision-transfer panels ran once per task, model and condition. The native-resume regression ran twice per task, model and condition; its switching-tools comparison ran twice per task, direction and condition. The compaction pilot ran once per task, model and condition, with two compaction events in each resumed workflow. The October 26 retention pilot ran twice per task, model and condition.
- 104-trial fresh-session comparison: all planned same-tool outcomes, matched costs, capture and delivery audits, source hashes and recovery records. The parent 128-trial competitor matrix is unfinished.
- 60-trial decision-transfer comparison: automatic capture and two written-handoff methods, including four cross-tool workflows per method.
- Switching tools, compaction and native resume: the protocols, source reports and hashes behind the three workflow comparisons above. These use their own frozen runtime snapshots, with Codex 0.159.3 and Claude Code 2.1.301.
- 124-trial competitor run record: the pre-fix runtime, including all failed workflows and their recorded causes. Its six task families are cancellation-window, event-watermark, idempotency-expiry, rollout-precedence, unicode-slugs and reservation-expiry.
- Competitor integration pilot and Supermemory Local pilot: frozen integration and calibration records.
Deterministic public and hidden tests score behavior, with integrity checks protecting the tests. Outgoing requests verify Crosshead delivery. Runtime, fixture and scorer hashes are frozen for each campaign; results from different campaigns are not pooled.
Model API prices used in the records
| Model | Input | Cache read | Cache write | Output |
|---|---|---|---|---|
| GPT-6.1 Sol | $2.00 | $0.10 | $2.50 | $10.00 |
| Claude Opus 5.5 | $4.00 | $0.20 | $5.00 / 5 min $8.00 / 1 hr | $20.00 |
These frozen rates match the official GPT-6.1 Sol and Claude Opus 5.5 price pages checked October 26, 2026. Published costs use recorded usage and each campaign’s frozen rates; later price changes do not rewrite historical results. Integration-specific background models and their charges remain in the campaign records.