CUAD V1 · SAME FIVE CONTRACTS · LOCAL LOOPBACK
One exact answer in ten outputs. Again.
This is a Qwen-only rerun on the exact five CUAD task IDs already used in the paired Qwen–Llama study. It is a repeated run on the same contracts, not a new holdout, and adds no new contract coverage. The Qwen build received a canonical prompt and a paraphrase once each for every contract.
1 of 10 outputs exactly matched the annotated answer span. There were 9 verifier failures, no execution errors, and one contract changed verdict between prompt variants.
Replication check: the earlier run and this rerun produced identical verdicts and output hashes on all 10 task-and-prompt pairs. This shows repeatability for this exact build, suite, and local setup; it does not turn the repeated outputs into new tasks or demonstrate correctness.
LOCAL REPLICATION · NOT A NEW HOLDOUT OR PRODUCTION RANKING
10 OUTPUTS · 5 REUSED TASKS · 2 PROMPT VARIANTS
Archived observation
| CUAD field | Canonical prompt | Paraphrase | Verdict changed |
|---|---|---|---|
| Agreement date | Pass | Fail | Yes |
| Effective date | Fail | Fail | No |
| Expiration date | Fail | Fail | No |
| Renewal term · contract A | Fail | Fail | No |
| Renewal term · contract B | Fail | Fail | No |
Mean observed inference latency was 19.1 seconds per returned output on the local runtime. This is a single machine and run; it does not estimate hosted latency.
Protocol and evidence
- Source
- CUAD v1 official test split, positive single-span cases. Dataset and attribution ↗ · CC BY 4.0.
- Relationship to earlier study
- All five task IDs and the suite SHA-256 are identical to the paired Qwen–Llama five-contract study. This adds another Qwen run only; it is not independent task coverage.
- Verifier
- Independent exact answer-span verifier against reviewed annotations. Only the excerpt and question were sent to the model.
- Model build
- Qwen 2.5 1.5B · Q4_K_M · digest
65ec06548149b04c096a120e4a6da9d4017ea809c91734ea5631e89f96ddc57b. - Execution
- Ollama 0.33.3 on loopback; temperature 0; fixed seed; up to 120 seconds per request and 15 minutes total. Ten local requests. Provider calls: zero; provider spend: $0.
- Suite SHA-256
528f5ffdf8ace348e5305ac9f46c3047f6bfa22c655c5079ccdcda01c2a4ec2b.- Sanitized receipt SHA-256
00b8a1564f4d13131ac5658039ac53f5561ae50ed154b0387b52bb413a49fd9e· Inspect repeat-run receipt ↗ · Inspect original first-run receipt ↗.
This repeated run does not add independent task cases. The sample is too small to estimate general performance. Prompt pairs on the same contract are correlated, so no binomial confidence interval is reported. Exact span reproduction is not legal correctness or customer value. Pretraining exposure is unknown. Raw prompts and model outputs are withheld; local energy and hardware opportunity cost were not measured. This study does not update the production measurement index, routing, or rankings.
Read the paired Qwen and Llama comparison on these same five contracts ↗ · ← Return to the live AI observatory
Original first run · same suite
The first Qwen-only run on these same five tasks produced 1 exact match in 10 attempts, 9 verified failures and 0 execution errors. Prompt wording changed one task verdict. This is the earlier run in the same evidence sequence, not an independent sample.