LOCAL EXPLORATORY STUDY · 30 INFERENCES · ZERO PROVIDER SPEND
Small models struggled to trace short code snippets exactly.
On 24 October 2026, three already-installed Ollama models answered five operator-authored JavaScript tracing tasks. Each task used a canonical prompt and a meaning-preserving paraphrase. A deterministic oracle independently computed each expected JSON result.
Exact whole-output matches: Llama 3.2 3B passed 3 of 10 prompts; DeepSeek Coder and Qwen 2.5 1.5B passed 0 of 10 each. No model passed both prompt variants on a task. These results describe this small authored suite only; they do not rank models generally.
5 TASKS · 2 PROMPTS EACH · 3 MODELS
Exact output matches
| Local model build | Exact prompts | Prompt pairs fully correct | Mean inference time |
|---|---|---|---|
DeepSeek Coder · installed digest 3ddd2d3fc8d2…a915d3 | 0 / 10 | 0 / 5 | 8.20 s |
| Qwen 2.5 1.5B · Q4_K_M | 0 / 10 | 0 / 5 | 7.44 s |
| Llama 3.2 3B · Q4_K_M | 3 / 10 | 1 / 5 | 13.39 s |
The only fully correct pair was Llama 3.2 3B on the latency-summary snippet. It returned the exact expected object under both prompt variants once each. That single pair is a candidate signal for replication, not evidence of reliable code-reading capability. The study has no independent reviewer, blinded tasks, repeated trials, hardware comparison or generalization set.
NEW TASKS · PARTIAL REPLICATION · LOCAL ONLY
The candidate did not generalize to new code traces.
A follow-up suite added 12 new synthetic traces across four condition families. It planned three repeats of two prompt variants for Llama 3.2 3B. After 11 of 72 planned attempts, the checkpoint audit recorded zero exact passes, ten mismatches and one execution error. The first verified attempt already had a critical-field mismatch, so the replication hypothesis was rejected and remaining calls were stopped.
This is a partial, single-model replication on operator-authored tasks. Only the first canonical attempts were reached; no paraphrase results were collected. It supports no general estimate of code-reading capability and no model ranking.
- Model identity
- Llama 3.2 3B · installed digest
a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72. - Execution and spend
- 11 saved local attempts through Ollama loopback; 60-second request cap and 15-minute total cap. Zero external provider calls and zero provider spend. Local electricity and hardware opportunity cost are unmeasured.
- Evidence receipt
- Open the complete prepared suite, saved checkpoint rows and independently recomputed audit ↗. Public artifact SHA-256:
d4e44fc34e2a02fbaeedefc9e9f3402e9f2dc6bf4d0f42e3fe6aa829e9c7c0f2. Checkpoint SHA-256:68a7a8e8475d3cabd535034cdb580b317050df359fc0b39732ea301036e87fc0. Suite SHA-256:3994ad0306058b5e658d8fec36c782c5e7786248f35c99d3936aab37bc4e5c27. - Production status
- This failed, partial local result is excluded from production measurements, rankings and routing. It made no production writes and did not change system authority.
What went wrong?
All three models missed the retry-backoff values and URL-normalization result on both prompts. They also failed the feature-flag and latency-summary tasks, with the exception noted above. On the status-counter task, Llama passed the paraphrased prompt but failed the canonical prompt, demonstrating a prompt-sensitive verdict on this fixture.
All 30 runs produced parseable, schema-constrained JSON and completed without execution errors. This separates reasoning mistakes from formatting or runtime failures.
Protocol and evidence
- Suite
- Five synthetic JavaScript snippets covering retry backoff, status counters, URL normalization, feature flags and latency summaries. Expected values were recomputed by explicit deterministic functions and checked by executing the supplied snippets in a sandbox with a 100 ms timeout.
- Execution
- Ollama 0.33.3 via
127.0.0.1, temperature 0, seed 42, one paired prompt run per task and model, maximum 60 seconds per request and 15 minutes total. 30 local inferences; zero external calls and zero provider spend. - Model identities
- DeepSeek Coder digest
3ddd2d3fc8d2b5fe039d18f859271132fd9c7960ef0be1864984442dc2a915d3; Qwen 2.5 1.5B digest65ec06548149b04c096a120e4a6da9d4017ea809c91734ea5631e89f96ddc57b; Llama 3.2 3B digesta80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72. - Evidence receipt
- Full machine-readable runs, prompts, outputs, timings and model digests: open the JSON artifact ↗. Artifact SHA-256:
7d6fe090da9da1c31974f4a9be44ca95cbbcc7cd76a140e1a226ca7b03ff1519. Suite SHA-256:9220fa2ef8740ed16221e7803f5b1bebb51defdefabc21389f75bbb35fe7ae05. - Production status
- This evidence remains a local exploratory study. It does not update the production intelligence index, model routing, public rankings or autonomous authority. Local electricity and hardware opportunity cost are unmeasured.
Five operator-authored synthetic snippets cannot establish general code comprehension, coding skill or task utility. Each prompt pair shares the same underlying snippet and is correlated; no confidence interval is meaningful for population performance here. Exact match measures only the declared output schema and oracle. Preserve the failures as well as the three passes.