PAIRED LOCAL COMPARISON · FIVE CONTRACTS · LOOPBACK ONLY
How did Qwen and Llama compare on the same five contract examples?
Each fixed local model build received the same five positive CUAD answer-span tasks. Each task used a canonical prompt and a paraphrase, with one attempt per variant.
Both models produced 1 exact match in 10 valid outputs, with no execution errors. Their exact-match verdicts agreed on all ten matched task-prompt outcomes.
20 ATTEMPTS · 10 MATCHED OUTCOMES · 5 CONTRACTS
Exact annotated-span matches
| Fixed local model build | Exact matches / valid outputs | 95% Wilson interval |
|---|---|---|
| Qwen 2.5 1.5B · Q4_K_M | 1 / 10 · 10% | 1.8–40.4% |
| Llama 3.2 3B · Q4_K_M | 1 / 10 · 10% | 1.8–40.4% |
Across ten matched task-and-prompt outcomes, both models were exact once, both were not exact nine times, and there were zero pass/fail disagreements. Each model's single exact answer was on the same Agreement Date task with the canonical prompt; both failed its paraphrase. Each model changed verdict on one of five contracts between prompt variants.
Five contracts are a very small sample. Each interval is wide; repeated prompt variants on the same contract are correlated. Identical outcomes here do not establish model equivalence, legal correctness or general capability.
Task-field results
| CUAD field | Shared examples | Qwen exact matches | Llama exact matches | Paired verdict disagreements |
|---|---|---|---|---|
| Agreement Date | 1 | 1 / 2 | 1 / 2 | 0 / 2 |
| Effective Date | 1 | 0 / 2 | 0 / 2 | 0 / 2 |
| Expiration Date | 1 | 0 / 2 | 0 / 2 | 0 / 2 |
| Renewal Term | 2 | 0 / 4 | 0 / 4 | 0 / 4 |
Protocol and evidence
- Dataset
- CUAD v1 · official test split · CC BY 4.0; positive single-span examples only.
- Verifier
- Exact string match against independently checked annotations. The models saw bounded excerpts and questions, not the gold answers.
- Runtime
- Ollama 0.33.3 on local loopback only; no provider calls, spend, production writes or routing changes. Model digests and suite hash are in the receipt.
- Evidence
- Inspect the hash-checked, sanitized machine-readable receipt ↗. It includes counts and hashes but omits prompts and raw model outputs. Receipt SHA-256:
dbe9dfe76f319415e05fd50db1f0b81fad8f4aebd8deed6415a026d5f3b33a40; reviewed five-task suite SHA-256:528f5ffdf8ace348e5305ac9f46c3047f6bfa22c655c5079ccdcda01c2a4ec2b. - Related study
- Read the disjoint 25-contract CUAD study ↗. These five examples are separate and are not pooled into that study.
These local measurements are supporting research only. They are excluded from the production measurement index and model rankings.