TWO-RUN LOCAL REPLICATION · 6 CUAD CONTRACTS · 26–27 OCT 2026
One prompt rewrite changed an exact-match result — twice.
In the first run, Qwen 2.5 1.5B produced 6 exact matches in 12 canonical-prompt attempts and 4 in 12 paraphrased-prompt attempts. A separate 24-call repeat reproduced every output and verdict exactly. The same six contracts passed consistently on three canonical and two paraphrased cases; one passed only canonically, none only under paraphrase.
48 LOCAL INFERENCES · TWO RUNS · NOT A MODEL RANKING
FIRST-RUN TASK-LEVEL RESULTS · TWO REPEATS PER PROMPT
Exact annotated-span reproduction
| Prompt condition | Exact attempts | Contracts passing both repeats | 95% Wilson interval across contracts | Errors |
|---|---|---|---|---|
| Canonical | 6 / 12 | 3 / 6 | 18.7–81.3% | 0 |
| Paraphrased | 4 / 12 | 2 / 6 | 9.7–70.0% | 0 |
Intervals use six contract-level outcomes, not twelve correlated attempts per wording. They are wide and overlap. Two contracts passed with both prompts, one passed only with canonical wording, none passed only with the paraphrase, and three failed with both.
| CUAD category | Canonical | Paraphrase | Task outcome |
|---|---|---|---|
| Agreement date · contract 1 | 2 / 2 pass | 2 / 2 pass | Both |
| Agreement date · contract 2 | 2 / 2 pass | 2 / 2 pass | Both |
| Effective date | 2 / 2 pass | 0 / 2 pass | Canonical only |
| Expiration date | 0 / 2 pass | 0 / 2 pass | Neither |
| Renewal term | 0 / 2 pass | 0 / 2 pass | Neither |
| Notice period | 0 / 2 pass | 0 / 2 pass | Neither |
In the first run, all 24 responses were returned and independently checked against the official annotated answer spans: 10 exact matches, 14 verified failures and 0 execution errors. Mean local inference latency was 11.04 seconds, median 10.91 seconds and p95 23.32 seconds.
Repeatability check · second run on 27 Oct 2026
The approved repeat used the same pinned model digest, six-contract suite, prompt variants and deterministic settings. All 24 output hashes and independently recomputed verdicts matched the first run; the repeat again produced 10 exact spans, 14 mismatches and zero execution errors.
This supports repeatability for one model build and this fixed six-contract prompt suite. It adds no independent contract coverage and does not establish general capability. The repeat used local Ollama only, with $0 provider spend.
Protocol and evidence
- Source
- CUAD v1 official test split · CC BY 4.0; positive annotated spans only. Model training exposure is unknown.
- Model build
- Qwen 2.5 1.5B · Q4_K_M · digest
65ec06548149b04c096a120e4a6da9d4017ea809c91734ea5631e89f96ddc57b. - Protocol
- Six distinct contracts; canonical and paraphrased prompts; two repetitions per prompt. Ollama 0.33.3 on local loopback; temperature 0; seed 42; 60-second request timeout; 15-minute run cap.
- Independent evidence
- Suite SHA-256
611a4e3fbfacc9dc67f16d462c5d8fcb1953e30d140926f0226d0357f614150c; first-run artifact SHA-256ed5a091b14c279a12eb4f441b75c9aef3154cc35d885dea5a91dbac30b9ca296; repeat artifact SHA-2561afd0c0d209af87dab29fafe9000339036863097c57702e33fd128d859e9c5a3; sanitized two-run receipt with task-level verdicts and verification hashes ↗.
This six-contract public sample is exploratory. Exact annotation reproduction is not legal correctness, full-document review, customer utility or hosted-model behavior. Raw excerpts and model outputs are not published. The result does not enter production measurements, routing or rankings.