LOCAL FIXTURE STUDY · LOOPBACK ONLY · 4 OPERATOR-AUTHORED TASKS

What do small local models get right?

Two fixed local model digests ran on the same four mechanically checked tasks: arithmetic, structured extraction, two prompt phrasings per task and one run per phrasing.

Across eight outputs each: Qwen 2.5 1.5B exactly matched 4/8; Llama 3.2 3B matched 3/8. There were no execution errors. The small difference is descriptive and does not support a model ranking.

16 VERIFIED OUTPUTS · 4 FIXTURES · 2 PROMPT VARIANTS

Exact complete-object matches

FIXTURE STUDY · NOT A RANKING

Each fixture required an exact match of the complete JSON object. A correct field inside an otherwise incorrect object does not count as a pass.

Operator-authored fixture	Qwen 2.5 1.5B	Llama 3.2 3B
Pack total · arithmetic	2 / 2 exact	1 / 2 exact
Discounted invoice · arithmetic	0 / 2 exact	0 / 2 exact
Waybill fields · extraction	0 / 2 exact	0 / 2 exact
Contract terms · extraction	2 / 2 exact	2 / 2 exact
All fixtures · 8 outputs each	4 / 8 exact · 0 errors	3 / 8 exact · 0 errors

There are only four operator-authored fixtures. Each task was run once with each prompt phrasing, so the paired attempts are not independent examples. The fixtures are not blinded third-party benchmarks and do not establish performance on real customer work.

Both phrasings passed on 2/4 fixtures for Qwen and 1/4 for Llama. Neither model passed all fixtures; this report does not identify a general winner.

Protocol and provenance

Source: Operator-authored JSON; oracle mechanically checked against literal source fields and arithmetic.
Task coverage: Four fixtures across verified arithmetic and structured extraction; two prompt phrasings and one repetition per phrasing.
Runtime: Ollama 0.33.3 on loopback; 60-second request timeout and 15-minute total cap. No external calls, spend or production writes.
Qwen identity: qwen2.5:1.5b · Q4_K_M · digest 65ec06548149…ddc57b
Llama identity: llama3.2:3b · Q4_K_M · digest a80c4f17acd5…b5b8b72
Suite hash: 1032d3c6f775…3509326

Inspect the hash-checked machine-readable receipt ↗

Source artifact SHA-256: 47bb7f3bec88094e1d09b35954b35e71d9ed2d85f1584bc960f715a9b90cb3ad. Inspect the complete local evidence, including prompts, expected values and model outputs, in the reproducibility artifact ↗.

These local fixtures remain separate from the live production index and provider rankings.