Representative panel rubric calibration — 2026-10-01¶
This is a reproducible, source-curated calibration of all 24 complete task rubrics. Correct cases use eighteen text exemplars with source grounding and six actual protected-test-qualified patches. Incomplete cases omit a required distinction; wrong cases assert a plausible contrary fact. Every rubric criterion is reviewed against the exact artifact and frozen rubric digest. Real campaign answers still require blinded semantic adjudication; this replay is not an automatic semantic classifier.
All 72 answer probes bind exact response and rubric digests to source-curated adjudications. Correct exemplars support every required criterion; incomplete and wrong responses fail the substantive and requested-coverage criteria. The table highlights each task's decisive distinction. The full exemplars are in the versioned calibration JSON. Applied patch behavior is also checked independently in the exact admitted image.
| Task | Correct claim | Incomplete claim | Subtly wrong claim | Decisive distinction |
|---|---|---|---|---|
| panel-n1 | UNSET aliases Sentinel.UNSET; the enum member is an object-valued singleton. | UNSET is a sentinel. | UNSET and Sentinel.UNSET are separate enum members. | alias identity |
| panel-n2 | Lower.render lowercases; Upper.render uppercases; Adapter.render delegates to the plugin interface. | Lower.render lowercases text. | Upper.render lowercases text, while Lower.render uppercases it. | method owner |
| panel-n3 | index.ts exports wrapper.convert as format; wrapper calls impl.transform; consumer.display calls format; unrelated.format is a different declaration. | The public format is re-exported from wrapper.convert. | The public format re-export points to unrelated.format. | re-export target |
| panel-d1 | Default enum pickling reconstructs by value; a fresh object cannot equal the original singleton value; copying already preserves identity. | Pickling fails during enum reconstruction. | Copying and deep copying fail for the same reason as pickling. | copy versus pickle behavior |
| panel-d2 | compileRe uses opts.flags || (opts.nocase ? i : empty); empty flags fall back to nocase, nonempty flags override that fallback. | nocase selects the i flag when flags are absent. | An explicitly empty flags string suppresses the nocase fallback. | empty flags fallback |
| panel-d3 | convert uses options.enabled || true, so false is overwritten; transform itself correctly preserves text when enabled is false. | The false option is overwritten. | impl.transform overwrites enabled=false; wrapper.convert preserves false. | faulty boundary |
| panel-t1 | onResult precedes no-match and ignore decisions; ignored matches call onIgnore, never onMatch; accepted matches call onMatch. | Ignored matches call onIgnore. | An ignored matching path fires onMatch before onIgnore. | callback ordering |
| panel-t2 | The parser passes search_profile through ContextRequest; active_similarity_search_profile selects the configured profile and rejects unknown values; CLI overrides task selection, profile configuration uses effective precedence. | The CLI forwards search_profile to ContextRequest. | Unknown search profiles silently select the default profile. | invalid profile handling |
| panel-t3 | main builds environment and argument maps; config.Resolve applies defaults then environment then arguments; store.Format formats MiXeD. | config.Resolve chooses a mode. | SERVICE_MODE overrides an explicit mode argument. | configuration precedence |
| panel-i1 | Direct callers include symbol, symbols, references, documentation_findings, context_for_task, impact_analysis, repository_map, arch, emb, docs, _call_edges; callers/callees dispatch through _call_edges; capabilities, index_status and symbol_evidence bypass _query; warm _ConnectionExecutor.execute forwards a supplied connection. | MCPAdapter._query has direct callers. | capabilities and index_status dispatch through MCPAdapter._query. | bypass paths |
| panel-i2 | Structural query connections and warm adapters must preserve execute semantics; backend implementations and query contract tests must be included, with unresolved dynamic use explicitly scoped. | BackendQueryConnection.execute affects query execution. | Changing BackendQueryConnection.execute cannot affect warm query adapters. | warm adapter impact |
| panel-i3 | consumer.display consumes runtime format; type-consumer.defaults consumes Options only; wrapper.convert propagates runtime options; unrelated.format is a negative control. | consumer.display calls runtime format. | type-consumer.defaults invokes runtime format, while consumer.display only imports Options. | runtime versus type consumers |
| panel-p1 | The correct patch uses name-based enum reduction; every member and full Option retain identity, name and default. | Make Sentinel pickle by name. | Patch only Sentinel.UNSET so pickle works; leave the other enum members and Option defaults untested. | full member and Option behavior |
| panel-p2 | The seed disables the isIgnored branch. Restore ignore handling and independently check matching, nonmatching and ignored paths. | Restore the isIgnored check. | After invoking onIgnore, return true and also invoke onMatch for an ignored path. | ignored match result and callback |
| panel-p3 | Format must reject only negative limits; zero bypasses truncation; positive truncation and case conversion remain unchanged. | Make zero a valid limit. | Treat every nonpositive limit as unlimited, including negative values. | negative limit remains invalid |
| panel-f1 | CLI validates a bounded result count and passes it to the existing core result limit; default behavior remains ten complete items. | Add --max-results to context. | Set --max-results default to 100 and truncate the rendered answer at that many characters. | item limit and default |
| panel-f2 | Adapter delegates optional capability discovery through the public plugin interface with a false fallback for legacy plugins. | Add Adapter.capabilities(). | Require every legacy render-only plugin to implement capabilities(), otherwise raise AttributeError. | legacy plugin fallback |
| panel-f3 | The suffix travels across re-export, wrapper and implementation; old two-argument and one-argument calls preserve outputs. | Add a suffix to transform. | Append the suffix before conversion and require it in every existing call. | suffix ordering and compatibility |
| panel-u1 | The public picomatch API returns a matcher; guides must include positive and negative matches and distinguish empty/nonempty flags. | picomatch returns a matcher for a pattern. | Explicit empty flags disable nocase and make the example case-sensitive. | flags precedence |
| panel-u2 | service.render(text, mode=upper) supports upper/lower and raises ValueError for unknown modes; Adapter accepts a Plugin. | service.render supports upper and lower modes. | An unsupported mode returns the original text without an error. | unsupported mode result |
| panel-u3 | Defaults are upper/10; environment overrides defaults; command arguments override environment; current Format rejects zero. | SERVICE_MODE sets the mode. | Environment settings override explicit command arguments, and zero is already accepted. | precedence and known zero defect |
| panel-e1 | No such API is declared in the frozen source; absence is scoped to the inspected repository and usable index, not every external plugin. | teleport_repository was not found. | The index proves teleport_repository cannot exist in any external Codira plugin. | scope of negative evidence |
| panel-e2 | Disabling the text analyzer omits the file from the index; the file still exists and describes environment/argument precedence. | operator-notes.txt is missing from the index. | The index omission proves operator-notes.txt is absent from the repository. | index coverage versus source existence |
| panel-e3 | Cursors bind query/profile/page size/generation; old continuation is invalid after reindexing, and source expansion rejects modified files before indexing. | Reindex after changing transform. | A cursor from before reindex remains valid for the updated generation. | generation-bound cursor |
Bank SHA-256: e78dacbba51aee517e40200307e27ffc526d2c5f8f84eae55aadefe548eada67. Calibration SHA-256: 53d4918cebac27bb473e84135a80fee67d12c671aca5af95641cf8d36ffc91da.
Patch and feature tasks additionally require their frozen protected probes to pass on the actual applied patch. The preparation receipt records correct, incomplete and subtly wrong applied patch cases; their protected traces remain in ignored durable evidence.