AI SOURCE NETWORKINDEPENDENT AI BAROMETERPUBLIC STUDY CATALOG · METHODS AND RECEIPTS
Browse task-level comparisons, protocols and machine-readable evidence. Studies below use fixed public examples or bounded local fixtures; each states what was measured and what remains unknown.
These studies are not the live production ranking. Local and third-party results stay separate from the production index. Small samples, timeouts and failures remain visible.
Open the live evidence index ↗ · Read the method and limits ↗
LATEST VERIFIED LOCAL RESULT · 28 OCT 2026
The approved run returned all 24 outputs across six CUAD contracts: 0/12 exact matches on canonical prompts and 2/12 on paraphrases. The matched Qwen baseline scored 10/24 on the same suite. A separate three-run repeatability study from 27 Oct matched all 24 output hashes pairwise; it does not include this 28 Oct run.
Only six distinct contracts were tested, with repeated prompts. Exact annotation matching is not legal correctness, and neither study supports a general model ranking. A separate Llama rerun with five timeouts remains unpooled.
Read the comparison and limits ↗ · Inspect the 28 Oct run receipt ↗ · Inspect the matched comparison receipt ↗ · Inspect the separate 27 Oct repeatability receipt ↗ · Inspect the separate timeout rerun ↗
SOURCE-BACKED TASKS
TOOL CALLING · BFCL V4
Five-case pilot plus a disjoint 12-task extension. In the extension, Qwen selected the expected function in 11/12 cases and passed exact typed-argument checks in 8/12; Llama scored 12/12 and 2/12. No downstream tool was executed.
Public benchmark slice; one attempt per model and task. Wide uncertainty; no general ranking or customer-utility claim.
Read protocol and task failures ↗DOCUMENT EXTRACTION · CUAD
Twenty-five unique contracts across five disjoint suites, with canonical and paraphrased prompts. Qwen matched 3/49 valid outputs; Llama matched 5/45. Execution errors are reported separately.
Exact span match is not legal correctness. Public test-set exposure is unknown; the small sample does not support a general model ranking.
Read the 25-contract study ↗Paired Qwen–Llama comparison ↗ · Qwen repeatability on the same five tasks ↗
INSTRUCTION FOLLOWING · IFEVAL
Across 30 attempts, Qwen passed 15; Llama passed 16 of 17 returned outputs and timed out on 13. On the 17 prompts both returned, passes were Qwen 9 and Llama 16. The report applies five fixed formatting rules to a selected public prompt slice.
Different completion counts make the overall rates incomparable; this is a small local slice, not the official benchmark score. Timeouts are unresolved, not verifier failures.
Read the held-out checks ↗BOUNDED LOCAL FIXTURES
BFCL V4 · FIVE-CASE PILOT
On five public examples, both local builds selected the expected function; 5/10 calls also had exact typed arguments. This dated pilot overlaps the tool-selection extension and remains here as the baseline.
Inspect the hash-checked five-case receipt ↗DETERMINISTIC ARITHMETIC · SYNTHETIC
On three synthetic unit-price examples, the checker accepted all three valid cases and rejected nine injected inconsistencies. An archived model-output replay contains four outputs across two cases.
This tests arithmetic consistency, not document extraction, customer usefulness or general model capability. No model calls or production ranking.
Inspect the hash-checked summary ↗ · Inspect verifier evidence ↗CODE READING
Five operator-authored traces, two prompt phrasings each: Llama passed 3/10 exact checks; Qwen and DeepSeek Coder passed 0/10 each. The follow-up stopped at its predeclared critical-failure rule; partial results are retained.
Read the comparison and stopped replication ↗TASK FIXTURES
Bounded deterministic checks for a small set of structured and policy tasks. Useful for protocol development; not a representative user workload.
Read the four-task comparison ↗These studies reuse six public contracts to isolate wording or repeatability; they do not add broad task coverage.
Qwen canonical vs paraphrased prompts ↗ · Inspect the sanitized receipt ↗
Arena’s human-preference and agent-evaluation tracks use different methods and scales. ASN reports them as third-party data, does not independently verify them, and does not combine them with local studies or production scores.