TreasureBench Agents v1

Public conformance

The implementation provides deterministic generation across 138 canonical cells and 552 prompt variants, closed information boundaries, five registered architecture orchestrators, strict structured actions, separate exact-rational metrics, hashed and redacted traces, AES-256-GCM custody, 12 contamination probe classes, authorization guards, and independent verification.

The original deterministic offline rehearsal covers 10 public tasks across all five architectures: 50 cases pass, Method A and Method B agree, and all 24 original instrument corruptions reject. The repaired rehearsal preserves those 50 cases, adds independent Method C and action-budget range checks, and rejects all 28 instrument corruptions. The live public calibration exercised both required direct-provider adapters over the same 10 public tasks and five architectures: 100 route-cases passed protocol and Method A/B checks. These remain adapter and instrument-conformance records, not scientific model results.

Task families

• Common Source Acquisition
• One Reader Versus Broadcast Attention
• Point Versus Shortlist Sharing
• Consensus Collapse Versus Portfolio Recovery
• Threshold Team Formation

Sealed-pilot closeout

The fixed encrypted pilot covered 50 tasks, five architectures, two exact direct model snapshots, and 500 private runs. Its redacted ledger contains 3,037 attempts, 3,035 successes, and two preserved Anthropic errors. Usage was 2,004,503 input tokens and 392,889 output tokens; total cost was USD 11.5702435.

The provider phase closed before unsealing. The output lock covers 3,545 encrypted objects. Post-lock Method A and independent Method B have zero disagreements, contamination findings are zero, all 76 registered corruptions reject, and redaction, private-leak scanning, and cost reconciliation pass. Two reconstructed protocol errors and the provider failures require sealed-pilot-quarantined-provider-failure.

Redacted repair adjudication

The exact owner-authorized read-only diagnostic verified the original output lock, custody commitments, append-only logs, final encrypted audit package, and retained-state immutability. It inspected exactly 500 locked traces. The aggregate reconstruction found 137 runs with invalid final cardinality, 266 invalid final agent outputs including 265 over-budget outputs, 138 metric records affected by credited extra actions, and 57 legacy coverage range violations.

The transient provider-service event recovered and created no terminal run. The other provider event is terminal but remains unknown-terminal because the minimum retained evidence cannot safely distinguish its exact cause. It accounts for one protocol-invalid run; the other reconstructed protocol error is a separate downstream failure. The prospective instrument now requires exactly one final action per agent, independent Method C conformance, range checks, and zero terminal or protocol-failure tolerance.

Fresh repair-confirmation closeout

The separately registered campaign treasurebench-agents-v1-repair-confirmation-v1, batch tb-agents-v1-repair-confirmation-v1-b01, stopped at its first public canary. One direct OpenAI request returned terminal HTTP 400 class schema-or-parameter; Anthropic received zero calls, reported input and output tokens are zero, cost is USD 0.00, and no private run occurred.

The provider phase is closed and six retained objects are bound by output lock sha256:8102a6c1b6bda003336d5503136dfe29301b04cb8f35e7740edd8d56f0eb3c1d. No task seed, private task, answer, task or answer key, task or answer ciphertext, or custody manifest was created. The campaign and batch cannot be retried, repaired, reopened, reused, rescored, executed, or reauthorized.

Its then-next public-only task was TreasureBench exact provider-schema conformance repair and public-canary gate; AO-0004 completed that separate engineering gate without reopening either campaign.

Provider-schema conformance

AO-0004 R4 completed as conformance-pass-both-complete-schemas. OpenAI and Anthropic each passed their minimal and complete provider-specific TreasureBench action schemas, and independent post-parse validation passed the canonical exactly-one-final-action semantic contract.

The four ordered calls used 1,603 input tokens and 275 output tokens and cost USD 0.0086685: OpenAI USD 0.0031875 and Anthropic USD 0.005481. No bisection ran. R3's bounded failure at the exact 128-token ceiling remains preserved and unresolved beyond an output-truncation hypothesis; the prospectively frozen R4 256-token execution passed. R1 and R2 remain superseded and unused; R3 and R4 are consumed and immutable.

This is provider-schema and semantic-contract engineering conformance, not scientific evidence, model-performance evidence, peer review, external validation, or a performance comparison. Private and scientific state did not change.

Fresh repair-confirmation v2 closeout

AO-0006 campaign treasurebench-agents-v1-repair-confirmation-v2, batch tb-agents-v1-repair-confirmation-v2-b01, passed both direct-provider public canaries and then stopped at custody creation with decision fresh-pilot-v2-quarantined-engineering-only and public-safe class custody-creation-failure.

The two calls used 1,349 input tokens and 253 output tokens and cost USD 0.0076095: OpenAI USD 0.0027975 and Anthropic USD 0.004812. No private architecture/model run occurred. The provider phase is closed, and eight safely lockable objects are preserved under output lock sha256:127a9c796459c7627f6fd90b92ef1587ad0f6b1910b4ff255c2ceb976f3ab25f. No call followed the lock and no material was unsealed.

A new seed and task and answer keys exist only in retained private state; no task ciphertext, answer ciphertext, or custody manifest was created. The campaign and batch cannot be retried, repaired, reopened, reused, rescored, spliced, executed, or reauthorized.

AO-0007 custody diagnosis and prospective repair

The one-use, owner-authorized AO-0007 diagnostic verified the AO-0006 retained boundary without mutation, identified v2-campaign-absent-from-private-generation-permit-allowlist as the exact custody failure, and permanently closed private-read authority. It disclosed no private value or host path and made zero credential reads, provider calls, or spend.

The repair is prospective only: AO-0006 remains quarantined and unchanged. The production custody path now passes all 20 required synthetic live-mode classes, independent AES-256-GCM verification, and six negative classes. No new private pilot, scientific run, claim, performance evidence, ranking, or base-campaign authority was created.

Fresh repair-confirmation v3 closeout

AO-0008 campaign treasurebench-agents-v1-repair-confirmation-v3, batch tb-agents-v1-repair-confirmation-v3-b01, passed both direct-provider public canaries, wholly fresh custody, and the private prefix. The fixed full batch then stopped with decision fresh-pilot-v3-quarantined-engineering-only and public-safe class fixed-full-batch-failure.

The provider phase made 3,067 calls, used 2,304,303 input tokens and 444,085 output tokens, and cost USD 13.1861145: OpenAI USD 4.5952575 and Anthropic USD 8.590857. Every hard cap held. Provider access is closed, and 3,576 retained objects are preserved under output lock sha256:e52055b08ca3a8acb1cfb6ac608c6e601f3c618352900f92bf91c5ffc4718dbb. No call followed the lock and no material was unsealed.

The partial private batch is not evaluated or published. The campaign and batch cannot be retried, repaired, reopened, reused, rescored, spliced, executed, or reauthorized.

AO-0009 aggregate fixed-batch diagnosis

The exact R2 owner-gated diagnostic ran once, authenticated all 450 fixed-full-batch traces, verified the output lock, immutable inventory, append-only ledgers, complete response identity correspondence, unique 2/50/450 trace partition, all 500 pairing records, and retained-state immutability, then permanently closed private-read authority.

The public aggregate reports 32 protocol-nonconforming traces and 32 parse/schema-repair exhaustion traces. Terminal-provider attempts, contamination traces, invalid final cardinalities, cap triggers, and integrity failures are zero. Frozen causal precedence selects agent-protocol-policy-decision-required.

AO-0009 made no protocol acceptance, normalization, retry, invalid-output credit, or scoring change. Any prospective agent-protocol change required a separate explicit owner decision and task. AO-0008 remains permanently quarantined, private-read authority is closed, and at AO-0009 closeout v4 was not registered.

AO-0010 policy v2 and fresh v4 closeout

AO-0010 registers forward-only policy treasurebench-agents-v1-protocol-validity-policy-v2 and wholly fresh campaign treasurebench-agents-v1-repair-confirmation-v4, batch tb-agents-v1-repair-confirmation-v4-b01. Historical policies, schemas, identities, locks, and decisions remain unchanged.

The policy separates batch-integrity validity, missing provider terminals, protocol-valid output, and protocol-invalid output. A completed response still nonconforming after one schema-only repair remains in every intended-pairing denominator without replacement, semantic retry, invented action, or invalid-output credit. Protocol-invalid traces alone do not quarantine. Registered metric-specific feasible bounds and architecture-contrast bounds cover all intended pairings; complete-case effects cannot be presented as unconditional.

The authorization-free 50-task/500-pairing rehearsal classifies all 500 pairings, constructs 6,400 pairing metric intervals and 120 architecture-contrast bounds, and passes independent reconstruction. Zero, mixed, and all-invalid synthetic batches pass without protocol-invalid-only quarantine; provider-terminal and integrity failures still quarantine. All 104 relevant corruptions reject.

The exact authorized v4 attempt passed both public canaries and wholly fresh custody, then stopped during the private prefix under class provider-terminal-missing. Provider access closed after 116 calls, 58,636 input tokens, 13,275 output tokens, and USD 0.3470610. The verified lock preserves 150 retained objects under sha256:88c442bf6cb5e1b99a893808d423b488d2b1062543a49fcfb32b77f276abaa1c. No call followed the lock and no material was unsealed.

The incomplete real batch has no protocol classification, Methods A/B/C, metric bounds, or performance interpretation. V4 is permanently quarantined and cannot be retried, replaced, repaired, reopened, reused, rescored, spliced, executed, unsealed, or reauthorized. No performance result, scientific run, claim, ranking, release, submission, or base campaign is created.

AO-0011 policy v3 and fresh v5 closeout

AO-0011 freezes forward-only policy treasurebench-agents-v1-provider-outcome-policy-v3 under unchanged protocol-validity policy v2 and completes campaign treasurebench-agents-v1-repair-confirmation-v5, batch tb-agents-v1-repair-confirmation-v5-b01.

Both public canaries, wholly fresh 50-task custody, the private prefix, and all 500 intended pairings completed. Terminal classification contains 496 protocol-valid and 4 protocol-invalid pairings, zero provider-operational missingness, zero contract/safety failure, and no circuit-breaker trigger.

One transient-provider first attempt selected and completed the frozen 5-second fallback before the sole identical retry recovered. No third, changed, duplicate, semantic, outcome-dependent, or replacement retry occurred. Provider access closed after 3,058 calls, 2,290,993 input tokens, 437,027 output tokens, and USD 13.0413660.

The verified output lock covers 4,067 objects under sha256:e18e7f8173f9ac0026f74cd6ff9b577010abdf65620ad269606f6862ff16e47b. No call followed the lock. Post-lock replay/unseal, Methods A/B/C, independent provider classification and all-pairing bounds, exact identity and cost correspondence, contamination, corruptions, redaction, and retained-state checks pass.

This is aggregate engineering status only. No task, answer, prompt, provider output, raw trace, task-level metric, model/provider/architecture/family performance, comparison, ranking, composite, scientific run, claim, paper result, release, submission, DD-023, or base campaign is published or created.

Evidence boundary

No private task, answer, seed, key, prompt, output, raw trace, task-level metric, dimension breakdown, sensitivity calculation, provider comparison, ranking, leaderboard, or composite is public. Encrypted custody and the final encrypted audit package are retained for 365 days. No hidden reasoning was requested or stored.

This is non-inferential DD-010 instrument engineering. DD-023 remains unallocated; no study, claim, immutable scientific run, paper result, or base-campaign authority exists.

Versions and next gate

Owner

DD-010 instrument

Content

Explicit selection from preserved v1, v2, or v3

Agent protocol

agents-v1

Protocol-validity policy

treasurebench-agents-v1-protocol-validity-policy-v2

Provider-outcome policy

treasurebench-agents-v1-provider-outcome-policy-v3

Generator

agents-task-generator-v1

Original pilot lock

sha256:1d487723f7587e8e2fa865682e6f6cc473cf2da4967b837dedf3952cddfcbfab

Repair-confirmation v1 lock

sha256:8102a6c1b6bda003336d5503136dfe29301b04cb8f35e7740edd8d56f0eb3c1d

Repair-confirmation v2 lock

sha256:127a9c796459c7627f6fd90b92ef1587ad0f6b1910b4ff255c2ceb976f3ab25f

Repair-confirmation v3 lock

sha256:e52055b08ca3a8acb1cfb6ac608c6e601f3c618352900f92bf91c5ffc4718dbb

Repair-confirmation v4 lock

sha256:88c442bf6cb5e1b99a893808d423b488d2b1062543a49fcfb32b77f276abaa1c

Repair-confirmation v5 lock

sha256:e18e7f8173f9ac0026f74cd6ff9b577010abdf65620ad269606f6862ff16e47b

The custody path is repaired, policies v2 and v3 are frozen, the original pilot and v1 through v4 remain permanently quarantined, and v5 is engineering complete. Any separate future task requires a new explicit owner decision and wholly new identities. The 200-task base campaign remains blocked.

Download the provider-schema conformance summary · Download the redacted operational summary · Download the implementation summary · Download the preserved instrument registration · DD-010 study record · TreasureBench overview