Can small models reproduce contract clauses exactly?

The models share all 25 task definitions and exact model digests across the suites. Each task has two paired prompt variants; those variants are not independent examples.

Exact match is strict span reproduction, not legal correctness. The valid-output rates are 6.1% for Qwen (95% Wilson interval: 2.1–16.5%) and 11.1% for Llama (4.8–23.5%). These wide intervals reflect the small sample; the difference is descriptive and does not support a general model ranking.

Results across disjoint suites

Performance varies across five disjoint batches. The 25 contract examples are disjoint, and suite identities, hashes, task outcomes and source artifact hashes are recorded in the receipt.

Independent local replication · one matched five-contract suite

This separate replication reused the same five held-out contracts and exact local model digests, with two runs per prompt variant (40 planned attempts). It is reported independently and is not pooled into the five-suite pilot above.

This is a repeated local run on one five-contract suite, not five new tasks and not an independent broad benchmark. The wide intervals and 10 Llama timeouts make cross-model interpretation especially uncertain. No ranking or production measurement uses these results.

Artifact SHA-256: 5e553ccbede8ea2f68d19b14141476d890a282659a2b641c8a427580ae8bf31b · Inspect replication receipt and run hashes ↗

Protocol and provenance

Dataset

CUAD v1 · official test split · CC BY 4.0

Coverage

25 unique contracts across five disjoint suites; five clause categories; positive single-span annotations; one canonical and one paraphrased prompt per contract.

Oracle

Official annotated answer span, verified by exact string comparison against the source contract. The model saw a bounded excerpt and question, not the gold answer.

Runtime

Ollama 0.33.3 on loopback; no external provider calls, provider spend, production measurement writes or routing changes. Model digests were identical across suites.

Timeout reporting

Execution errors are retained. Earlier suite receipts do not all record per-request timeout caps; the latest suite used 60 seconds, and its timeout is reported separately.

Suite hashes

0933484e2c4a… · c494290e35a2… · c3cfa2534b99… · d503eb4b3373… · f8729ba5bfbb… · full values are in the machine-readable receipt.

Model identities

• qwen2.5:1.5b · Q4_K_M · digest 65ec06548149…ddc57b · 3/49 valid exact matches, 1 execution error.
• llama3.2:3b · Q4_K_M · digest a80c4f17acd5…b5b8b72 · 5/45 valid exact matches, 5 execution errors.

Inspect the machine-readable study receipt ↗ · CUAD source repository ↗

Additional unseen Qwen holdout · five new contracts

After the matched study, the same Qwen 2.5 1.5B digest ran on five further CUAD contracts excluded from all prior suites. It exactly reproduced 2 of 10 valid spans; both matches were on canonical prompts, and none were on paraphrases. Two of the five task pairs changed verdict across prompt wording. The paired model comparison above remains the 25-contract study: Llama was not run on this supplemental suite.

This small Qwen-only follow-up is not a model comparison and is not pooled into the matched 25-contract aggregate or production index. It shows prompt sensitivity on these cases, not general capability or user value. Raw contract excerpts and model outputs are not published.

Suite SHA-256: df845d71f886…0918f3 · model digest: 65ec06548149…ddc57b · Inspect the sanitized, hash-checked receipt ↗

Three-model matched slice · five contracts

Three fixed local model digests were evaluated on the same five positive CUAD spans and two prompt variants. Qwen and Llama runs were reused from the earlier suite; DeepSeek Coder was the only new model call.

Wide, overlapping intervals and five contracts support no ranking. These are local build results, not hosted provider results or customer-utility evidence.

Inspect the hash-checked, sanitized three-model receipt ↗. It omits prompts, excerpts and raw model outputs.

Timeout diagnostic · same five contracts, not a new holdout

On the earlier run, Llama 3.2 3B produced 2 exact matches among 10 valid outputs and 10 execution errors in 20 attempts at a 90-second request timeout. One extra paired run at 120 seconds produced 2 exact matches from 10 valid outputs with no execution errors. Both runs found the exact same 20% rate among valid outputs; the follow-up does not establish that timeout alone caused the error difference.

The two new exact matches were the canonical and paraphrase prompts for the Effective Date item. This reused five fixed contracts and one additional attempt per prompt, so the results are correlated and support no ranking or general capability claim.

Inspect the hash-checked, sanitized timeout receipt ↗. It contains counts, confidence intervals, model/suite hashes and protocol limits, without prompts, contract excerpts or raw outputs. It remains outside the production index.

What this does and does not show

Exact-span reproduction remained uncommon for both small models across these 25 examples. Qwen passed three variants and Llama passed five; execution errors were more frequent for Llama in this run set. The study is a bounded local comparison, not a statement about hosted providers, current model versions or broad legal AI capability.

Pretraining exposure to CUAD is unknown. The test does not measure full-document search, absent-clause detection, legal correctness, reviewer time saved, customer usefulness, hosted-provider behavior, energy use or current provider availability. CUAD itself is a research dataset, not legal advice.

These local results remain separate from ASN’s continuous production model index and provider rankings.

Paired five-contract repeat · Qwen vs Llama

A separate, hash-verified rerun gives both models the same five CUAD contract examples, two prompt variants and one attempt per variant. It tests paired pass/fail agreement on exactly matched tasks; it is not pooled into the 25-contract study or the separate three-model slice.

Read the paired five-contract results and evidence ↗