{ "schema": "asn.public-local-code-reading-replication.v1", "created_at": "2026-10-24T00:04:38.861Z", "prepared_suite": { "schema": "asn.local-model-holdout-suite.v1", "suite_id": "custom-local-code-reading-replication-20261024-v1", "source_kind": "operator_authored_json", "oracle_method": "Twelve new synthetic JavaScript traces across four condition families. Each expected JSON result is independently recomputed by an explicit deterministic oracle and by executing only the supplied snippet in a Node VM sandbox with a 100 ms limit. No model assistance is used to create or verify expected values.", "oracle_review_state": "mechanically_checked_operator_authored", "source_evidence_sha256": "7d6fe090da9da1c31974f4a9be44ca95cbbcc7cd76a140e1a226ca7b03ff1519", "suite_sha256": "3994ad0306058b5e658d8fec36c782c5e7786248f35c99d3936aab37bc4e5c27", "tasks": [ { "id": "replication-state-partition-01.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "ready_count": { "type": "integer" }, "blocked_count": { "type": "integer" }, "other_count": { "type": "integer" } }, "required": [ "ready_count", "blocked_count", "other_count" ], "additionalProperties": false }, "expected": { "ready_count": 2, "blocked_count": 2, "other_count": 3 }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nfunction countStates(rows) {\n return rows.reduce((counts, row) => {\n if (row.state === 'ready') counts.ready++;\n else if (row.state === 'blocked') counts.blocked++;\n else counts.other++;\n return counts;\n }, { ready: 0, blocked: 0, other: 0 });\n}\nconst result = countStates([\n {state:'ready'}, {state:'failed'}, {state:'blocked'},\n {state:'ready'}, {state:'cancelled'}, {state:'blocked'}, {state:'pending'}\n]);\n\nTrace each row through the accumulator and return the final three counts using the exact JSON field names.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nfunction countStates(rows) {\n return rows.reduce((counts, row) => {\n if (row.state === 'ready') counts.ready++;\n else if (row.state === 'blocked') counts.blocked++;\n else counts.other++;\n return counts;\n }, { ready: 0, blocked: 0, other: 0 });\n}\nconst result = countStates([\n {state:'ready'}, {state:'failed'}, {state:'blocked'},\n {state:'ready'}, {state:'cancelled'}, {state:'blocked'}, {state:'pending'}\n]);\n\nFor every state, apply the ordered branches in `countStates`. Report how many are ready, blocked, and in all remaining states combined. Return only the required JSON object.", "critical_fields": [ "ready_count", "blocked_count", "other_count" ], "condition_family": "branch_and_accumulator", "definition_sha256": "73b26664d791c48e6764c85adea43be6bcca76ce1d5e3c59ec46626ca2e0aece" }, { "id": "replication-state-priority-02.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "p1_count": { "type": "integer" }, "p2_count": { "type": "integer" }, "p3_count": { "type": "integer" }, "closed_count": { "type": "integer" } }, "required": [ "p1_count", "p2_count", "p3_count", "closed_count" ], "additionalProperties": false }, "expected": { "p1_count": 2, "p2_count": 2, "p3_count": 1, "closed_count": 1 }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nfunction priority(rows) {\n return rows.reduce((out, row) => {\n if (row.urgent && row.open) out.p1++;\n else if (row.open && row.users >= 3) out.p2++;\n else if (row.open) out.p3++;\n else out.closed++;\n return out;\n }, {p1:0,p2:0,p3:0,closed:0});\n}\nconst result = priority([\n {urgent:true,open:true,users:1}, {urgent:false,open:true,users:4},\n {urgent:true,open:true,users:5}, {urgent:false,open:true,users:1},\n {urgent:false,open:false,users:8}, {urgent:false,open:true,users:3}\n]);\n\nApply the branches in order to all six rows. Return the four final bucket counts with the required keys.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nfunction priority(rows) {\n return rows.reduce((out, row) => {\n if (row.urgent && row.open) out.p1++;\n else if (row.open && row.users >= 3) out.p2++;\n else if (row.open) out.p3++;\n else out.closed++;\n return out;\n }, {p1:0,p2:0,p3:0,closed:0});\n}\nconst result = priority([\n {urgent:true,open:true,users:1}, {urgent:false,open:true,users:4},\n {urgent:true,open:true,users:5}, {urgent:false,open:true,users:1},\n {urgent:false,open:false,users:8}, {urgent:false,open:true,users:3}\n]);\n\nWalk the six records through `priority` exactly as written; a record is assigned to only its first matching branch. Give counts for P1, P2, P3, and closed as JSON.", "critical_fields": [ "p1_count", "p2_count", "p3_count", "closed_count" ], "condition_family": "branch_and_accumulator", "definition_sha256": "ce2233c3b8c6a16f8cb5d21a3602385d6f15dd4b497056b2b362ec58678bf6d1" }, { "id": "replication-state-threshold-03.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "accepted_count": { "type": "integer" }, "review_count": { "type": "integer" }, "rejected_count": { "type": "integer" } }, "required": [ "accepted_count", "review_count", "rejected_count" ], "additionalProperties": false }, "expected": { "accepted_count": 2, "review_count": 3, "rejected_count": 2 }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nfunction summarize(rows) {\n const out = {accepted:0,review:0,rejected:0};\n for (const row of rows) {\n if (row.score >= 80 && row.verified) out.accepted++;\n else if (row.score >= 50) out.review++;\n else out.rejected++;\n }\n return out;\n}\nconst result = summarize([\n {score:80,verified:true}, {score:99,verified:false},\n {score:49,verified:true}, {score:50,verified:false},\n {score:79,verified:true}, {score:100,verified:true}, {score:10,verified:false}\n]);\n\nTrace every record through the threshold and verification checks. Return exact counts for accepted, review, and rejected.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nfunction summarize(rows) {\n const out = {accepted:0,review:0,rejected:0};\n for (const row of rows) {\n if (row.score >= 80 && row.verified) out.accepted++;\n else if (row.score >= 50) out.review++;\n else out.rejected++;\n }\n return out;\n}\nconst result = summarize([\n {score:80,verified:true}, {score:99,verified:false},\n {score:49,verified:true}, {score:50,verified:false},\n {score:79,verified:true}, {score:100,verified:true}, {score:10,verified:false}\n]);\n\nHow many rows increment each field of `out`? Remember that the review branch runs when the first condition fails but score is still at least 50. Output only JSON.", "critical_fields": [ "accepted_count", "review_count", "rejected_count" ], "condition_family": "branch_and_accumulator", "definition_sha256": "c874bc4e3902d6d562b0e2f21b0bffc8904b7ce365908567080f3eeb2a7aa98e" }, { "id": "replication-array-transform-04.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "selected_count": { "type": "integer" }, "tripled_sorted": { "type": "string" }, "sum": { "type": "integer" } }, "required": [ "selected_count", "tripled_sorted", "sum" ], "additionalProperties": false }, "expected": { "selected_count": 2, "tripled_sorted": "12,18", "sum": 30 }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nconst values = [4, 9, 2, 9, 7, 1, 6];\nconst selected = values.filter(n => n >= 4 && n % 2 === 0);\nconst result = selected.map(n => n * 3).sort((a,b) => a-b);\n\nEvaluate the filter, map and numeric sort in order. Return the selected count, final array and its sum.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nconst values = [4, 9, 2, 9, 7, 1, 6];\nconst selected = values.filter(n => n >= 4 && n % 2 === 0);\nconst result = selected.map(n => n * 3).sort((a,b) => a-b);\n\nKeep even values of at least four, triple each retained value, sort numerically ascending, then report the count, array and total as JSON.", "critical_fields": [ "selected_count", "sum" ], "condition_family": "collection_transformations", "definition_sha256": "2390600effcd8ab566538d57f74a66f49cedaf6d8eb59ca8c03d3fd166570de8" }, { "id": "replication-array-transform-05.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "valid_count": { "type": "integer" }, "total_bytes": { "type": "integer" }, "largest_bytes": { "type": "integer" }, "ignored_count": { "type": "integer" } }, "required": [ "valid_count", "total_bytes", "largest_bytes", "ignored_count" ], "additionalProperties": false }, "expected": { "valid_count": 3, "total_bytes": 38, "largest_bytes": 19, "ignored_count": 2 }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nfunction summarize(rows) {\n const valid = rows.filter(row => row.status === 'ok');\n const total = valid.reduce((sum,row) => sum + row.bytes, 0);\n return {count:valid.length,total,largest:Math.max(...valid.map(row => row.bytes))};\n}\nconst result = summarize([\n {status:'ok',bytes:12}, {status:'error',bytes:900},\n {status:'ok',bytes:7}, {status:'ok',bytes:19}, {status:'timeout',bytes:50}\n]);\n\nEvaluate `summarize` and return valid count, byte total, largest valid row, and ignored count using the required keys.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nfunction summarize(rows) {\n const valid = rows.filter(row => row.status === 'ok');\n const total = valid.reduce((sum,row) => sum + row.bytes, 0);\n return {count:valid.length,total,largest:Math.max(...valid.map(row => row.bytes))};\n}\nconst result = summarize([\n {status:'ok',bytes:12}, {status:'error',bytes:900},\n {status:'ok',bytes:7}, {status:'ok',bytes:19}, {status:'timeout',bytes:50}\n]);\n\nOnly rows with status exactly `ok` contribute to the sum and maximum. Calculate the four requested values from the five rows and output JSON only.", "critical_fields": [ "valid_count", "total_bytes", "largest_bytes" ], "condition_family": "collection_transformations", "definition_sha256": "f7e169ade63db7e7aefaabe2ebf2995809d36cb717de2c87c15e8ea6c8cac90c" }, { "id": "replication-array-transform-06.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "regions": { "type": "string" }, "region_count": { "type": "integer" }, "first_region": { "type": "string" } }, "required": [ "regions", "region_count", "first_region" ], "additionalProperties": false }, "expected": { "regions": "east|north|west", "region_count": 3, "first_region": "east" }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nfunction uniqueSorted(rows) {\n return [...new Set(rows.filter(row => row.enabled).map(row => row.region))]\n .sort((a,b) => a.localeCompare(b));\n}\nconst result = uniqueSorted([\n {region:'west',enabled:true}, {region:'east',enabled:false},\n {region:'north',enabled:true}, {region:'west',enabled:true},\n {region:'east',enabled:true}, {region:'north',enabled:false}\n]);\n\nApply filter, map, Set deduplication and locale-aware sort. Return the resulting list, count and first item.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nfunction uniqueSorted(rows) {\n return [...new Set(rows.filter(row => row.enabled).map(row => row.region))]\n .sort((a,b) => a.localeCompare(b));\n}\nconst result = uniqueSorted([\n {region:'west',enabled:true}, {region:'east',enabled:false},\n {region:'north',enabled:true}, {region:'west',enabled:true},\n {region:'east',enabled:true}, {region:'north',enabled:false}\n]);\n\nKeep enabled records only; collect each region once and sort the names using the shown comparator. Give the sorted regions, their number, and the first region as JSON.", "critical_fields": [ "region_count", "first_region" ], "condition_family": "collection_transformations", "definition_sha256": "88a1e6412cf787ffb60e3f5f1d941fce08cc87bb3b10c615c23032284181e337" }, { "id": "replication-summary-percentile-07.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "valid_count": { "type": "integer" }, "median_ms": { "type": "integer" }, "p75_ms": { "type": "integer" }, "ignored_count": { "type": "integer" } }, "required": [ "valid_count", "median_ms", "p75_ms", "ignored_count" ], "additionalProperties": false }, "expected": { "valid_count": 5, "median_ms": 30, "p75_ms": 40, "ignored_count": 1 }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nfunction summarize(rows) {\n const values = rows.filter(row => row.ok).map(row => row.ms).sort((a,b) => a-b);\n const middle = Math.floor(values.length / 2);\n const median = values.length % 2 ? values[middle] : (values[middle-1] + values[middle]) / 2;\n const p75 = values[Math.ceil(0.75 * values.length) - 1];\n return {count:values.length,median,p75};\n}\nconst result = summarize([\n {ok:true,ms:90}, {ok:false,ms:5}, {ok:true,ms:10},\n {ok:true,ms:40}, {ok:true,ms:30}, {ok:true,ms:20}\n]);\n\nTrace filtering and numeric sorting, then use the exact median and p75 index formulas in the code. Return all four summary fields.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nfunction summarize(rows) {\n const values = rows.filter(row => row.ok).map(row => row.ms).sort((a,b) => a-b);\n const middle = Math.floor(values.length / 2);\n const median = values.length % 2 ? values[middle] : (values[middle-1] + values[middle]) / 2;\n const p75 = values[Math.ceil(0.75 * values.length) - 1];\n return {count:values.length,median,p75};\n}\nconst result = summarize([\n {ok:true,ms:90}, {ok:false,ms:5}, {ok:true,ms:10},\n {ok:true,ms:40}, {ok:true,ms:30}, {ok:true,ms:20}\n]);\n\nExclude the failed sample, sort the remaining latencies numerically, calculate the code-defined median and nearest-rank p75, and report counts. Output only the exact JSON object.", "critical_fields": [ "valid_count", "median_ms", "p75_ms" ], "condition_family": "numeric_summary_semantics", "definition_sha256": "9bd1a99a20e401d66c54886494238a6bf935599c9219e07e2095f8c75855d46f" }, { "id": "replication-summary-percentile-08.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "included_count": { "type": "integer" }, "median_lower_ms": { "type": "integer" }, "p90_ms": { "type": "integer" }, "excluded_cancelled": { "type": "integer" } }, "required": [ "included_count", "median_lower_ms", "p90_ms", "excluded_cancelled" ], "additionalProperties": false }, "expected": { "included_count": 6, "median_lower_ms": 10, "p90_ms": 20, "excluded_cancelled": 1 }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nfunction summarize(rows) {\n const values = rows.filter(row => row.state !== 'cancelled').map(row => row.ms).sort((a,b) => a-b);\n const median = values[Math.floor((values.length - 1) / 2)];\n const p90 = values[Math.ceil(0.9 * values.length) - 1];\n return {count:values.length,median,p90};\n}\nconst result = summarize([\n {state:'ok',ms:8}, {state:'cancelled',ms:1}, {state:'timeout',ms:20},\n {state:'ok',ms:4}, {state:'ok',ms:12}, {state:'error',ms:16}, {state:'ok',ms:10}\n]);\n\nOnly cancelled rows are excluded. Follow the lower-middle median and p90 formulas literally, then return the four values.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nfunction summarize(rows) {\n const values = rows.filter(row => row.state !== 'cancelled').map(row => row.ms).sort((a,b) => a-b);\n const median = values[Math.floor((values.length - 1) / 2)];\n const p90 = values[Math.ceil(0.9 * values.length) - 1];\n return {count:values.length,median,p90};\n}\nconst result = summarize([\n {state:'ok',ms:8}, {state:'cancelled',ms:1}, {state:'timeout',ms:20},\n {state:'ok',ms:4}, {state:'ok',ms:12}, {state:'error',ms:16}, {state:'ok',ms:10}\n]);\n\nSort every non-cancelled latency, including timeouts and errors. Use index floor((n-1)/2) for the median and ceil(0.9n)-1 for p90; report the sample and exclusion counts.", "critical_fields": [ "included_count", "median_lower_ms", "p90_ms" ], "condition_family": "numeric_summary_semantics", "definition_sha256": "da0e184f1218944d25fc20b68e1af595e888e3a0e70d52e31166a76fbbd69bb8" }, { "id": "replication-summary-percentile-09.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "groups": { "type": "string" }, "group_count": { "type": "integer" }, "total_ms": { "type": "integer" } }, "required": [ "groups", "group_count", "total_ms" ], "additionalProperties": false }, "expected": { "groups": "boreal:12|coast:10", "group_count": 2, "total_ms": 22 }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nfunction summarize(rows) {\n const groups = new Map();\n for (const row of rows) {\n if (!row.ok) continue;\n groups.set(row.region, (groups.get(row.region) || 0) + row.ms);\n }\n return [...groups.entries()].sort((a,b) => a[0].localeCompare(b[0]));\n}\nconst result = summarize([\n {region:'boreal',ok:true,ms:7}, {region:'coast',ok:true,ms:4},\n {region:'boreal',ok:false,ms:100}, {region:'coast',ok:true,ms:6},\n {region:'boreal',ok:true,ms:5}, {region:'delta',ok:false,ms:3}\n]);\n\nFollow the success filter and Map accumulation. Return alphabetically sorted region totals, group count and total latency.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nfunction summarize(rows) {\n const groups = new Map();\n for (const row of rows) {\n if (!row.ok) continue;\n groups.set(row.region, (groups.get(row.region) || 0) + row.ms);\n }\n return [...groups.entries()].sort((a,b) => a[0].localeCompare(b[0]));\n}\nconst result = summarize([\n {region:'boreal',ok:true,ms:7}, {region:'coast',ok:true,ms:4},\n {region:'boreal',ok:false,ms:100}, {region:'coast',ok:true,ms:6},\n {region:'boreal',ok:true,ms:5}, {region:'delta',ok:false,ms:3}\n]);\n\nAdd only successful row durations by region, then sort the `[region,total]` pairs by region name. Report the pairs, number of groups and grand total as JSON.", "critical_fields": [ "group_count", "total_ms" ], "condition_family": "numeric_summary_semantics", "definition_sha256": "e8e59f69a737202da4bad2fe35eace360849fc2219110b4f693385e338a49258" }, { "id": "replication-retry-backoff-10.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "retry_0_ms": { "type": "integer" }, "retry_1_ms": { "type": "integer" }, "retry_2_ms": { "type": "integer" }, "retry_4_ms": { "type": "integer" }, "retry_6_ms": { "type": "integer" } }, "required": [ "retry_0_ms", "retry_1_ms", "retry_2_ms", "retry_4_ms", "retry_6_ms" ], "additionalProperties": false }, "expected": { "retry_0_ms": 0, "retry_1_ms": 300, "retry_2_ms": 600, "retry_4_ms": 2400, "retry_6_ms": 2400 }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nfunction delay(retry, base, cap) {\n if (!Number.isInteger(retry) || retry < 1) return 0;\n return Math.min(cap, base * 2 ** (retry - 1));\n}\nconst values = [0, 1, 2, 4, 6].map(n => delay(n, 300, 2400));\n\nCalculate each call to `delay` and return the five values in `values` with the required field names.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nfunction delay(retry, base, cap) {\n if (!Number.isInteger(retry) || retry < 1) return 0;\n return Math.min(cap, base * 2 ** (retry - 1));\n}\nconst values = [0, 1, 2, 4, 6].map(n => delay(n, 300, 2400));\n\nTrace the invalid-retry guard, exponential multiplier and 2,400 ms cap. Give delays for retries 0, 1, 2, 4 and 6 as JSON.", "critical_fields": [ "retry_1_ms", "retry_4_ms", "retry_6_ms" ], "condition_family": "retry_and_delay_state", "definition_sha256": "f947078b4301cde6b591ef4e5c77fc6be3d57f772f46afaba626727f9beae000" }, { "id": "replication-retry-classification-11.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "success_count": { "type": "integer" }, "retry_count": { "type": "integer" }, "final_count": { "type": "integer" } }, "required": [ "success_count", "retry_count", "final_count" ], "additionalProperties": false }, "expected": { "success_count": 1, "retry_count": 3, "final_count": 3 }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nfunction summarize(results) {\n return results.reduce((out, row) => {\n if (row.status === 200) out.success++;\n else if (row.status === 429 || row.status >= 500) out.retry++;\n else out.final++;\n return out;\n }, {success:0,retry:0,final:0});\n}\nconst result = summarize([\n {status:200}, {status:429}, {status:503},\n {status:404}, {status:500}, {status:204}, {status:408}\n]);\n\nApply the ordered status-code branches and return the success, retry and final counts.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nfunction summarize(results) {\n return results.reduce((out, row) => {\n if (row.status === 200) out.success++;\n else if (row.status === 429 || row.status >= 500) out.retry++;\n else out.final++;\n return out;\n }, {success:0,retry:0,final:0});\n}\nconst result = summarize([\n {status:200}, {status:429}, {status:503},\n {status:404}, {status:500}, {status:204}, {status:408}\n]);\n\nBucket each of the seven HTTP statuses: 200 succeeds; 429 or any status at least 500 is retryable; all others are final. Return the three totals as JSON.", "critical_fields": [ "success_count", "retry_count", "final_count" ], "condition_family": "retry_and_delay_state", "definition_sha256": "b0560ff352c44f950d9090771ece8d9c411cf41a3d7cbb29dc349bc70f990c19" }, { "id": "replication-retry-window-12.h1", "task_class": "code_reading", "schema": { "type": "object", "properties": { "checks": { "type": "string" }, "accepted_count": { "type": "integer" }, "rejected_count": { "type": "integer" } }, "required": [ "checks", "accepted_count", "rejected_count" ], "additionalProperties": false }, "expected": { "checks": "true,false,true,false", "accepted_count": 2, "rejected_count": 2 }, "prompt": "Read the JavaScript code below statically; do not run external tools.\n\nfunction withinWindow(start, duration, deadline) {\n return start + duration <= deadline;\n}\nconst checks = [\n withinWindow(0, 30, 30),\n withinWindow(10, 20, 29),\n withinWindow(8, 12, 21),\n withinWindow(4, 0, 3)\n];\nconst result = {checks, accepted:checks.filter(Boolean).length};\n\nEvaluate the inclusive deadline comparison for all four calls. Return the Boolean array and accepted/rejected counts.", "paraphrase_prompt": "Trace the supplied JavaScript without changing it or using external tools.\n\nfunction withinWindow(start, duration, deadline) {\n return start + duration <= deadline;\n}\nconst checks = [\n withinWindow(0, 30, 30),\n withinWindow(10, 20, 29),\n withinWindow(8, 12, 21),\n withinWindow(4, 0, 3)\n];\nconst result = {checks, accepted:checks.filter(Boolean).length};\n\nFor each interval, test whether start plus duration is less than or equal to the deadline. Include boundary equality as written, then report the checks and totals.", "critical_fields": [ "accepted_count", "rejected_count" ], "condition_family": "retry_and_delay_state", "definition_sha256": "b207fd9ef889777c73769acbec978055b6199203032a4963661edfdd840681a2" } ], "protocol": { "task_count": 12, "condition_families": [ "branch_and_accumulator", "collection_transformations", "numeric_summary_semantics", "retry_and_delay_state" ], "prompt_variants": [ "canonical", "paraphrase" ], "repetitions_per_task_per_variant": 3, "models": [ "llama3.2:3b" ], "expected_model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "maximum_calls": 72, "provider_calls": 0, "provider_spend_microusd": 0, "request_timeout_ms": 60000, "maximum_runtime_ms": 1200000, "success_threshold": { "whole_case_passes_per_variant_at_least": 11, "critical_field_errors": 0, "paired_verdict_flips": 0 }, "failure_rule": "Any critical-field error on any repeated run, fewer than 11 of 12 distinct cases passing in either prompt variant, or any paired verdict flip rejects the generalization hypothesis.", "operator_approval_required": true, "production_ranking_eligible": false } }, "checkpoint": { "header": { "kind": "header", "schema": "asn.local-model-holdout-checkpoint.v1", "suite_id": "custom-local-code-reading-replication-20261024-v1", "suite_sha256": "3994ad0306058b5e658d8fec36c782c5e7786248f35c99d3936aab37bc4e5c27", "models": [ { "name": "llama3.2:3b", "digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72" } ], "repetitions_per_task_per_variant": 3, "prompt_variants": [ "canonical", "paraphrase" ], "request_timeout_ms": 60000, "max_runtime_ms": 900000, "entry_sha256": "40a8f457d9877192bf2aa7030803cc7126fc956cf87e232458e1101bacd5edaf" }, "rows": [ { "model": "llama3.2:3b", "model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "task_id": "replication-state-partition-01.h1", "task_class": "code_reading", "variant": "canonical", "replicate": 1, "run_sequence": 0, "model_order_position": 0, "state": "verified_fail", "verification_method": "independent_exact_holdout_oracle.v1", "verification_reason": "value_mismatch", "output": { "ready_count": 2, "blocked_count": 2, "other_count": 2 }, "output_sha256": "f8f8b22500b861a8097b5e010c0777b81d6b5a8110783e1931be747f2bdaf5f4", "inference_latency_ms": 9285.226, "reported_timings_ms": { "prompt_eval_duration": 5048.839, "eval_duration": 4236.387, "total_duration": 16036.883 }, "elapsed_ms": 16042 }, { "model": "llama3.2:3b", "model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "task_id": "replication-state-priority-02.h1", "task_class": "code_reading", "variant": "canonical", "replicate": 1, "run_sequence": 1, "model_order_position": 0, "state": "verified_fail", "verification_method": "independent_exact_holdout_oracle.v1", "verification_reason": "value_mismatch", "output": { "p1_count": 2, "p2_count": 2, "p3_count": 2, "closed_count": 0 }, "output_sha256": "534314e186a243a53654d5ef857aff5ad8023efb6adc45ef66bf5665f045f3ac", "inference_latency_ms": 12001.29, "reported_timings_ms": { "prompt_eval_duration": 5786.839, "eval_duration": 6214.451, "total_duration": 12021.519 }, "elapsed_ms": 12026 }, { "model": "llama3.2:3b", "model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "task_id": "replication-state-threshold-03.h1", "task_class": "code_reading", "variant": "canonical", "replicate": 1, "run_sequence": 2, "model_order_position": 0, "state": "verified_fail", "verification_method": "independent_exact_holdout_oracle.v1", "verification_reason": "value_mismatch", "output": { "accepted_count": 3, "review_count": 2, "rejected_count": 2 }, "output_sha256": "3244b8aa76eda1901101a2c8d6199919e7c32c2299aa76ad0eaaa64d6292ada0", "inference_latency_ms": 10543.199, "reported_timings_ms": { "prompt_eval_duration": 5525.484, "eval_duration": 5017.715, "total_duration": 10569.772 }, "elapsed_ms": 10574 }, { "model": "llama3.2:3b", "model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "task_id": "replication-array-transform-04.h1", "task_class": "code_reading", "variant": "canonical", "replicate": 1, "run_sequence": 3, "model_order_position": 0, "state": "execution_error", "error_code": "The_operation_was_aborted_due_to_timeout", "elapsed_ms": 60021 }, { "model": "llama3.2:3b", "model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "task_id": "replication-array-transform-05.h1", "task_class": "code_reading", "variant": "canonical", "replicate": 1, "run_sequence": 4, "model_order_position": 0, "state": "verified_fail", "verification_method": "independent_exact_holdout_oracle.v1", "verification_reason": "value_mismatch", "output": { "valid_count": 3, "total_bytes": 138, "largest_bytes": 900, "ignored_count": 2 }, "output_sha256": "dcff304d53c562f5c53d285335f526822132de13c0cef495395eb0ffeda18c64", "inference_latency_ms": 13265.813, "reported_timings_ms": { "prompt_eval_duration": 6653.375, "eval_duration": 6612.438, "total_duration": 13725.322 }, "elapsed_ms": 13743 }, { "model": "llama3.2:3b", "model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "task_id": "replication-array-transform-06.h1", "task_class": "code_reading", "variant": "canonical", "replicate": 1, "run_sequence": 5, "model_order_position": 0, "state": "verified_fail", "verification_method": "independent_exact_holdout_oracle.v1", "verification_reason": "value_mismatch", "output": { "regions": "['east', 'north', 'west']", "region_count": 3, "first_region": "east" }, "output_sha256": "ed5007cd48c05aebf24c988a918b5b1f3a998e96bd507026a5ac9555da31f92c", "inference_latency_ms": 10656.296, "reported_timings_ms": { "prompt_eval_duration": 4992.154, "eval_duration": 5664.142, "total_duration": 10679.539 }, "elapsed_ms": 10683 }, { "model": "llama3.2:3b", "model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "task_id": "replication-summary-percentile-07.h1", "task_class": "code_reading", "variant": "canonical", "replicate": 1, "run_sequence": 6, "model_order_position": 0, "state": "verified_fail", "verification_method": "independent_exact_holdout_oracle.v1", "verification_reason": "value_mismatch", "output": { "valid_count": 6, "median_ms": 25, "p75_ms": 35, "ignored_count": 2 }, "output_sha256": "8f607b7e502c47721bc45fafacfd8d82bee79d5bb6d7a02710a07e41c0904e7d", "inference_latency_ms": 15282.143, "reported_timings_ms": { "prompt_eval_duration": 8499.738, "eval_duration": 6782.405, "total_duration": 15311.793 }, "elapsed_ms": 15316 }, { "model": "llama3.2:3b", "model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "task_id": "replication-summary-percentile-08.h1", "task_class": "code_reading", "variant": "canonical", "replicate": 1, "run_sequence": 7, "model_order_position": 0, "state": "verified_fail", "verification_method": "independent_exact_holdout_oracle.v1", "verification_reason": "value_mismatch", "output": { "included_count": 6, "median_lower_ms": 8, "p90_ms": 12, "excluded_cancelled": 1 }, "output_sha256": "2601a4d9a959ba8df8c834c9878c0e2218168df857549a18bf8200772b0659f2", "inference_latency_ms": 15282.54, "reported_timings_ms": { "prompt_eval_duration": 7454.127, "eval_duration": 7828.413, "total_duration": 15315.244 }, "elapsed_ms": 15319 }, { "model": "llama3.2:3b", "model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "task_id": "replication-summary-percentile-09.h1", "task_class": "code_reading", "variant": "canonical", "replicate": 1, "run_sequence": 8, "model_order_position": 0, "state": "verified_fail", "verification_method": "independent_exact_holdout_oracle.v1", "verification_reason": "value_mismatch", "output": { "groups": "{'boreal': 112, 'delta': 3, 'coast': 10}", "group_count": 3, "total_ms": 120 }, "output_sha256": "b1da66bb5c116c9c24ecf5de291de235179796e76c4010219e67ad4b837779d4", "inference_latency_ms": 16163.737, "reported_timings_ms": { "prompt_eval_duration": 7600.735, "eval_duration": 8563.002, "total_duration": 16193.156 }, "elapsed_ms": 16196 }, { "model": "llama3.2:3b", "model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "task_id": "replication-retry-backoff-10.h1", "task_class": "code_reading", "variant": "canonical", "replicate": 1, "run_sequence": 9, "model_order_position": 0, "state": "verified_fail", "verification_method": "independent_exact_holdout_oracle.v1", "verification_reason": "value_mismatch", "output": { "retry_0_ms": 0, "retry_1_ms": 600, "retry_2_ms": 1200, "retry_4_ms": 2400, "retry_6_ms": 4800 }, "output_sha256": "774c3d50949a0f6cc6a5b1fe7c9ca8dc6ffb3ef7d107e573ad8eb074019ae482", "inference_latency_ms": 16189.466, "reported_timings_ms": { "prompt_eval_duration": 4669.997, "eval_duration": 11519.469, "total_duration": 16256.578 }, "elapsed_ms": 16302 }, { "model": "llama3.2:3b", "model_digest": "a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72", "task_id": "replication-retry-classification-11.h1", "task_class": "code_reading", "variant": "canonical", "replicate": 1, "run_sequence": 10, "model_order_position": 0, "state": "verified_fail", "verification_method": "independent_exact_holdout_oracle.v1", "verification_reason": "value_mismatch", "output": { "success_count": 1, "retry_count": 2, "final_count": 4 }, "output_sha256": "4bbdf9dc7fa7bb2ceb969e4a96fb84cdae9b2318b033a26c002204c04e174d02", "inference_latency_ms": 13557.467, "reported_timings_ms": { "prompt_eval_duration": 7062.298, "eval_duration": 6495.169, "total_duration": 13597.411 }, "elapsed_ms": 13601 } ], "sha256": "68a7a8e8475d3cabd535034cdb580b317050df359fc0b39732ea301036e87fc0" }, "audit": { "schema": "asn.local-model-holdout-checkpoint-audit.v1", "created_at": "2026-10-24T00:04:38.861Z", "suite_id": "custom-local-code-reading-replication-20261024-v1", "suite_sha256": "3994ad0306058b5e658d8fec36c782c5e7786248f35c99d3936aab37bc4e5c27", "suite_source_sha256": "85497d37cefe8ede1563eb1eb1e5d9636299ab2047b4dcf40a95cbc8f52c37bd", "checkpoint_path": "brain/evidence/local-code-reading-replication-20261024.checkpoint.jsonl", "checkpoint_sha256": "68a7a8e8475d3cabd535034cdb580b317050df359fc0b39732ea301036e87fc0", "checkpoint_complete_rows": 11, "expected_runs": 72, "coverage": 0.1527777777777778, "states": { "verified_pass": 0, "verified_fail": 10, "execution_error": 1 }, "critical_failures": [ { "run_sequence": 0, "task_id": "replication-state-partition-01.h1", "task_class": "code_reading", "condition_family": "branch_and_accumulator", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "other_count", "expected": 3, "observed": 2 }, { "run_sequence": 1, "task_id": "replication-state-priority-02.h1", "task_class": "code_reading", "condition_family": "branch_and_accumulator", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "p3_count", "expected": 1, "observed": 2 }, { "run_sequence": 1, "task_id": "replication-state-priority-02.h1", "task_class": "code_reading", "condition_family": "branch_and_accumulator", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "closed_count", "expected": 1, "observed": 0 }, { "run_sequence": 2, "task_id": "replication-state-threshold-03.h1", "task_class": "code_reading", "condition_family": "branch_and_accumulator", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "accepted_count", "expected": 2, "observed": 3 }, { "run_sequence": 2, "task_id": "replication-state-threshold-03.h1", "task_class": "code_reading", "condition_family": "branch_and_accumulator", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "review_count", "expected": 3, "observed": 2 }, { "run_sequence": 4, "task_id": "replication-array-transform-05.h1", "task_class": "code_reading", "condition_family": "collection_transformations", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "total_bytes", "expected": 38, "observed": 138 }, { "run_sequence": 4, "task_id": "replication-array-transform-05.h1", "task_class": "code_reading", "condition_family": "collection_transformations", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "largest_bytes", "expected": 19, "observed": 900 }, { "run_sequence": 6, "task_id": "replication-summary-percentile-07.h1", "task_class": "code_reading", "condition_family": "numeric_summary_semantics", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "valid_count", "expected": 5, "observed": 6 }, { "run_sequence": 6, "task_id": "replication-summary-percentile-07.h1", "task_class": "code_reading", "condition_family": "numeric_summary_semantics", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "median_ms", "expected": 30, "observed": 25 }, { "run_sequence": 6, "task_id": "replication-summary-percentile-07.h1", "task_class": "code_reading", "condition_family": "numeric_summary_semantics", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "p75_ms", "expected": 40, "observed": 35 }, { "run_sequence": 7, "task_id": "replication-summary-percentile-08.h1", "task_class": "code_reading", "condition_family": "numeric_summary_semantics", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "median_lower_ms", "expected": 10, "observed": 8 }, { "run_sequence": 7, "task_id": "replication-summary-percentile-08.h1", "task_class": "code_reading", "condition_family": "numeric_summary_semantics", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "p90_ms", "expected": 20, "observed": 12 }, { "run_sequence": 8, "task_id": "replication-summary-percentile-09.h1", "task_class": "code_reading", "condition_family": "numeric_summary_semantics", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "group_count", "expected": 2, "observed": 3 }, { "run_sequence": 8, "task_id": "replication-summary-percentile-09.h1", "task_class": "code_reading", "condition_family": "numeric_summary_semantics", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "total_ms", "expected": 22, "observed": 120 }, { "run_sequence": 9, "task_id": "replication-retry-backoff-10.h1", "task_class": "code_reading", "condition_family": "retry_and_delay_state", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "retry_1_ms", "expected": 300, "observed": 600 }, { "run_sequence": 9, "task_id": "replication-retry-backoff-10.h1", "task_class": "code_reading", "condition_family": "retry_and_delay_state", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "retry_6_ms", "expected": 2400, "observed": 4800 }, { "run_sequence": 10, "task_id": "replication-retry-classification-11.h1", "task_class": "code_reading", "condition_family": "retry_and_delay_state", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "retry_count", "expected": 3, "observed": 2 }, { "run_sequence": 10, "task_id": "replication-retry-classification-11.h1", "task_class": "code_reading", "condition_family": "retry_and_delay_state", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "final_count", "expected": 3, "observed": 4 } ], "decision": "failure_condition_met", "operator_stop_reason": null, "first_critical_failure": { "run_sequence": 0, "task_id": "replication-state-partition-01.h1", "task_class": "code_reading", "condition_family": "branch_and_accumulator", "model": "llama3.2:3b", "variant": "canonical", "replicate": 1, "field": "other_count", "expected": 3, "observed": 2 }, "authority": { "local_only": true, "model_calls_after_stopping": false, "external_calls": false, "provider_spend_microusd": 0, "production_routing_changed": false, "money_movement": false }, "semantics": { "partial_evidence_only": true, "checkpoint_rows_hash_checked": true, "oracle_recomputed_independently": true, "stopped_run_is_not_a_complete_holdout_artifact": true }, "audit_sha256": "ffe1d862716a23d02ee64586c6e7109accd66365bb5bde6240573c9ae7d40864" }, "semantics": { "local_only": true, "external_provider_calls": false, "provider_spend_microusd": 0, "production_measurement_written": false, "production_routing_changed": false, "authority_expansion": false, "complete_holdout": false, "partial_evidence_only": true, "raw_prompts_and_outputs_included": true }, "artifact_sha256": "d4e44fc34e2a02fbaeedefc9e9f3402e9f2dc6bf4d0f42e3fe6aa829e9c7c0f2" }