DD-010
TreasureBench (DD-010; historical identifier DiscoveryBench)
TreasureBench is the formal public suite for the versioned exact golden benchmark with enforced protocol information boundaries; frozen DiscoveryBench identifiers and all five quarantined Agents v1 pilots remain unchanged, while AO-0011 completes a wholly fresh 500-pairing v5 engineering pilot under forward-only provider-outcome policy v3 and unchanged protocol-validity policy v2 without publishing task-level performance or creating scientific evidence.
Completed finite studyChecked independently
The question
Can a versioned benchmark exactly reproduce the project's validated bounded fixtures while enforcing protocol information capabilities and reporting task-level metric vectors without collapsing them into an unexplained score?
What we found
- DD-C-0055: DiscoveryBench v1 contains 15 exact golden tasks, 13 capability-declared built-in protocols, and 19 versioned metrics; all 16 declared compatible task/protocol pairs exactly reproduce their registered project values, while 179 incompatible pairs remain explicit exclusions and adversarial hidden-information access is rejected.
- DD-C-0069: DiscoveryBench v2 preserves all v1 defaults and exact vectors while extending the registry to 20 tasks, 21 capability-declared protocols, and 27 metrics; all 28 declared compatible pairs exactly reproduce their registered values and all 392 incompatible pairs remain explicit exclusions.
- DD-C-0087: DiscoveryBench v3 preserves every v1 and v2 exact metric vector while extending the opt-in registry to 24 tasks, 29 capability-declared protocols, and 39 metrics; all 36 compatible pairs exactly reproduce their registered values and all 660 incompatible pairs remain explicit exclusions among 696 candidate pairs.
What this result covers
Checked independently. The formal evidence wording and registry state remain available below.
Technical details
- Study phase
complete-bounded-study- Registry status
complete-v1-v3-with-five-agents-v1-pilots-quarantined-and-ao0011-fresh-v5-engineering-complete-under-policies-v2-v3- Evidence status
v1-v2-preserved-v3-independently-reproduced-exact-matrix; original-agents-v1-pilot-permanently-quarantined; repair-confirmation-v1-quarantined-at-terminal-public-canary; AO-0004-R4-provider-schema-and-semantic-contract-engineering-conformance-passed; repair-confirmation-v2-quarantined-at-custody-after-two-passing-public-canaries-with-zero-private-runs; AO-0007-one-use-custody-diagnostic-and-prospective-live-mode-conformance-passed; repair-confirmation-v3-quarantined-during-fixed-full-batch-with-provider-phase-closed-3576-object-lock-and-no-unseal; AO-0009-one-use-aggregate-r2-diagnostic-authenticated-450-fixed-batch-traces; AO-0010-forward-only-protocol-validity-policy-v2-frozen-and-repair-confirmation-v4-quarantined-at-private-prefix-provider-terminal-missing-with-provider-phase-closed-150-object-lock-and-no-unseal; AO-0011-forward-only-provider-outcome-policy-v3-and-fresh-v5-engineering-complete-with-500-terminal-pairings-496-protocol-valid-4-protocol-invalid-zero-provider-missing-zero-contract-safety-no-circuit-breaker-provider-phase-closed-4067-object-lock-and-post-lock-verification; AO-0012-R4-owner-authorization-consumed-at-pre-ingress-hard-stop-and-never-reusable; AO-0012-R5-owner-authorization-consumed-after-credential-ingress-at-control-plane-secret-deletion-ambiguity-hard-stop-before-pod-model-endpoint-inference-or-billing; AO-0012-R6b-owner-authorization-consumed-at-one-query-inventory-ambiguity-with-no-deletion-inference-or-spend-and-current-empty-secrets-tab-qualified-owner-observation-only; AO-0012-R7-owner-authorization-consumed-at-fresh-account-inventory-boundary-with-zero-resource-mutation-model-inference-billing-or-spend; no-closeout-is-performance-or-scientific-evidence
Reproducible evidence
20260721T183014Z_DD-010_ce930050_8ec718c24220260721T230249Z_DD-010_add85590_56c61a219520260722T054447Z_DD-010_d265e480_6930915b02
Files and data
- TreasureBench Agents v1 instrument contract with frozen DiscoveryBench compatibility identifiers
- Forward-only protocol-validity, batch-acceptance, and all-intended-pairing metric-bound policy for prospective v4 use
- Public-safe wholly fresh v4 engineering registration and authority boundary
- Public-safe complete v4 rehearsal, protocol-validity scenarios, independent bounds, corruption, and redaction outcomes
- Redacted fresh v4 private-prefix provider-terminal failure and permanent quarantine after passing public canaries and wholly fresh custody
- Redacted fresh v5 engineering-complete aggregate with all 500 terminal pairings, deterministic retry-delay audit, provider closure, output lock, and post-lock verification; not scientific or task-level performance evidence
- Offline implementation readiness decision and evaluation authorization boundary
- Agents v1 ownership, evidence boundary, and exact next gate
- Redacted aggregate pilot protocol and provider-error adjudication
- Redacted fresh repair-confirmation public-canary failure and permanent quarantine
- Redacted fresh v2 custody-stage failure and permanent quarantine after two successful public canaries
- Redacted fresh v3 fixed-full-batch failure and permanent quarantine after passing public canaries, custody, and private prefix
- Redacted one-use custody diagnosis, prospective repair, independent verification, and complete synthetic live-mode conformance; not scientific or performance evidence
- Redacted one-use aggregate fixed-batch diagnosis and agent-protocol policy gate; not scientific or performance evidence
- Public provider-schema and semantic-contract engineering conformance outcome; not scientific or model-performance evidence
- Benchmark evidence report
- Versioned task schema
- Versioned selective-attention task schema
- Versioned Program V4 threshold, dynamic, and mechanism task schema
- Frozen DiscoveryBench v3 public-safe registry underlying the TreasureBench display
- Complete v3 exact task registry
- Complete v3 capability-declared protocol registry
- Complete v3 task-level metric registry
- All 36 v3 compatible exact vectors
- All 696 capability-isolated v3 candidate pairs
- Independent value, schema, provenance, count, and capability verification
- Value, compatibility, and leakage corruption rejections
What comes next
Preserve the R7 inventory-boundary hard stop and consumed authorization; require a fresh explicit owner decision before any local repair gate, provider query, or live retry; keep PR