EXPLORATORY LOCAL COMPARISON · 27 OCT 2026 · CUAD
Same six contracts. Different exact-match results.
Qwen 2.5 1.5B matched more official answer spans than Llama 3.2 3B in this small local run. This is evidence about these six examples and model builds, not a general model ranking.
24 NEW LOCAL INFERENCES · $0 PROVIDER SPEND · NOT A PRODUCTION SCORE
RESPONSE-LEVEL RESULTS · 12 ATTEMPTS PER PROMPT
Exact annotated-span reproduction
| Local model build | Canonical prompt | Paraphrased prompt | Exact responses | Execution errors |
|---|---|---|---|---|
| Qwen 2.5 1.5B · Q4_K_M | 6 / 12 · 3 / 6 contracts passed both repeats | 4 / 12 · 2 / 6 contracts passed both repeats | 10 / 24 | 0 |
| Llama 3.2 3B · Q4_K_M | 0 / 12 · 0 / 6 contracts passed both repeats | 2 / 12 · 1 / 6 contracts passed both repeats | 2 / 24 | 0 |
Wilson 95% intervals for the contract-level “both repeats exact” outcome are wide: Qwen canonical 18.8–81.2%, paraphrase 9.7–70.0%; Llama canonical 0–39.0%, paraphrase 3.0–56.4%. Six tasks are too few to support a broad winner claim.
| CUAD example | Qwen · canonical | Qwen · paraphrase | Llama · canonical | Llama · paraphrase |
|---|---|---|---|---|
| Agreement date · A | 2 / 2 | 2 / 2 | 0 / 2 | 0 / 2 |
| Agreement date · B | 2 / 2 | 2 / 2 | 0 / 2 | 2 / 2 |
| Effective date | 2 / 2 | 0 / 2 | 0 / 2 | 0 / 2 |
| Expiration date | 0 / 2 | 0 / 2 | 0 / 2 | 0 / 2 |
| Renewal term | 0 / 2 | 0 / 2 | 0 / 2 | 0 / 2 |
| Notice period to terminate renewal | 0 / 2 | 0 / 2 | 0 / 2 | 0 / 2 |
Each cell counts exact outputs out of two repeated answers for that prompt. This table reports only the six pinned public examples; no contract text, prompts or raw model outputs are exposed.
SEPARATE FOLLOW-UP · LLAMA ONLY · LOCAL OLLAMA
A later Llama-only rerun returned 19 of 24 answers.
This is a separate rerun of the same six contracts, not part of the matched Qwen–Llama comparison above. Of 19 returned outputs, 2 exactly matched the CUAD annotations and 17 did not; five requests timed out at 60 seconds and remain unknown task outcomes.
| Prompt variant | Attempts | Returned | Exact matches | Verified mismatches | Timeouts |
|---|---|---|---|---|---|
| Canonical | 12 | 11 | 0 | 11 | 1 |
| Paraphrased | 12 | 8 | 2 | 6 | 4 |
| Total | 24 | 19 | 2 | 17 | 5 |
The exact-match rate is 2/19 returned outputs (10.5%). Timeouts are not counted as answer failures. Unlike the original complete Llama run (2/24 with no timeouts), this rerun has incomplete outputs; the runs are not pooled. It adds no new contract coverage, cannot rank models and does not populate the production barometer.
Inspect the sanitized run receipt and task-level counts ↗. Prompts, contract excerpts and raw outputs are not published.
Protocol, provenance and limits
- Source
- CUAD v1 official test split, CC BY 4.0; six positive, single-span examples. Dataset and archive hashes are included in the receipt.
- Protocol
- Same six pinned tasks, canonical and paraphrased prompts, two repeats per prompt, temperature 0, seed 42, serial Ollama loopback. Runs used Ollama 0.33.3 on separate days; provider spend was $0.
- Model builds
- Qwen 2.5 1.5B · Q4_K_M ·
65ec06548149b04c096a120e4a6da9d4017ea809c91734ea5631e89f96ddc57b
Llama 3.2 3B · Q4_K_M ·a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72. - Independent verification
- Each of 48 outputs was checked against the pinned CUAD span oracle. The receipt includes suite and run-artifact SHA-256 values and the verifier identity.
- Latency
- Local Ollama prompt and generation time only; excludes hosted latency, queueing, electricity and hardware opportunity cost.
Exact annotation reproduction does not establish legal correctness, full-document search, review time saved, customer utility or hosted-model behavior. The models ran on separate days; this is a matched protocol, not a simultaneous randomized trial. Public test-set training exposure is unknown.
NEW VERIFIED RUN · 28 OCT 2026 · SAME SIX TASKS
The approved Llama run matched 2 of 24 annotations.
Both builds returned all 24 outputs on the same six pinned contract tasks. Exact annotation matches by prompt variant:
| Prompt variant | Qwen 2.5 1.5B | Llama 3.2 3B |
|---|---|---|
| Canonical | 6 / 12 | 0 / 12 |
| Paraphrase | 4 / 12 | 2 / 12 |
| All attempts | 10 / 24 | 2 / 24 |
The models ran on separate dates; this was not a simultaneous randomized trial. Two repetitions reuse each of only six contracts, so the 24 attempts per model are not 24 independent tasks. Results do not establish legal correctness, hosted behavior or a general model winner.
Inspect the sanitized Llama receipt ↗ · Inspect the Qwen baseline receipt ↗
The 28 Oct run is exploratory local evidence, not a production measurement, rank or routing input.
Three separate local Llama executions reproduced the same output hashes and exact-verification verdicts for all 24 attempts; each had 2 exact matches and no execution errors. This supports repeatability for this one local setup, adds no task coverage and does not establish broad capability. Inspect the sanitized three-run repeatability receipt ↗.
The matched comparison and both local reruns stay outside production measurements, resource rankings and automated routing.
Inspect the sanitized task-level receipt ↗ · ← Browse all studies