TASK-CONDITIONED EVIDENCE · THREE VERIFIED LOCAL STUDIES
The better model changes with the task.
We ran the same pinned Qwen 2.5 1.5B and Llama 3.2 3B builds on contract extraction, instruction following and tool calls. Their results differ by task and by what counts as success.
This is a reading guide to three separate studies, not a pooled score or model ranking. The samples, metrics and denominators are different. All runs were local; they do not establish hosted-service behavior or customer value.
Browse every study and receipt ↗ · Return to the live barometer ↗
25 CONTRACTS · 50 ATTEMPTS PER MODEL
Contract-clause extraction · CUAD
Qwen matched 3 of 49 valid outputs; Llama matched 5 of 45. There were 1 and 5 execution errors respectively. Their 95% Wilson intervals overlap widely: 2.1–16.5% and 4.8–23.5%.
Strict reproduction of an annotated clause is not legal correctness. The result is descriptive for these selected public contracts.
Read the method and task results ↗ · Verify the aggregate receipt ↗
17 COMPLETE PAIRS · 30 ATTEMPTS PER MODEL
Instruction following · IFEval rules
Among the 17 prompts both models completed, Qwen passed 9 and Llama passed 16. Across all 30 attempts, Qwen returned 30 outputs (15 passes); Llama returned 17 (16 passes, 1 failure) and had 13 execution errors.
Counting only returned outputs hides the completion gap. These five pinned checks are a narrow subset of IFEval, not its official benchmark score.
Read the held-out report and timeout details ↗ · Verify the audited summary ↗
12 TASKS · ONE ATTEMPT PER MODEL
Tool calling · BFCL V4
Qwen selected the expected function in 11/12 cases and passed exact function-plus-argument checks in 8/12. Llama selected the function in 12/12 but passed exact argument checks in 2/12.
No tool was executed. Function choice and typed-argument accuracy are different measurements; this small public slice does not establish production capability.
Read the protocol and per-task outcomes ↗ · Verify the 12-task receipt ↗
What these studies say together
They do not identify one overall winner. In these samples, Llama returned more exact CUAD spans and passed more completed IFEval prompts, but it also had more CUAD execution errors and 13 IFEval errors. On BFCL, it selected tools more often while Qwen produced more exact argument sets. Which result matters depends on the task and whether incomplete execution is acceptable.
Do not average these rows: CUAD spans, IFEval format rules and BFCL calls are not a shared unit. The models used pinned local digests `65ec06548149…ddc57b` and `a80c4f17acd5…b5b8b72`. Small, selected public test slices may have training exposure. All three studies remain outside the production measurement index.
Inspect a strictly paired five-contract CUAD comparison ↗ · Open the complete methods catalog ↗