LOCAL MEASUREMENTS · SEPARATE FROM PRODUCTION
Small, independently checked studies with visible methods and limits. Three earlier, separate five-contract CUAD suites (observed 20 Oct 2026) show different apparent leaders and overlapping 95% intervals, so no overall winner is supported. The six-contract comparison below is a separate study. These local results remain outside production measurements.
LATEST LOCAL HOLDOUT · 28 OCT 2026
Llama matched 2/24; Qwen's matched baseline scored 10/24.
The approved 28 Oct Llama run returned 2 exact annotated spans in 24 attempts on the same six contracts: 0/12 canonical and 2/12 paraphrased. The matched Qwen receipt reports 10/24 on that suite. The models ran on separate dates, so this is not a simultaneous randomized trial. These six examples do not support a general model ranking or establish legal correctness.
A separate repeatability receipt covers three Llama runs completed by 27 Oct: all 24 output hashes matched pairwise and each run scored 2/24. That earlier receipt does not include the 28 Oct holdout. A different Llama-only rerun had five timeouts and remains unpooled. Repeats add no new contract coverage.
Read outcomes, intervals and protocol ↗ · Inspect the 28 Oct result receipt ↗ · Inspect the matched Qwen comparison receipt ↗ · Inspect the separate earlier repeatability receipt ↗ · Inspect the separate timed-out rerun ↗
LOCAL TOOL-CALL EXTENSION · EXPLORATORY · 24 OCT 2026
Nearly every tool was chosen; exact calls often failed.
Across 12 new BFCL V4 tasks, Qwen 2.5 1.5B selected the expected function in 11/12 calls and passed exact typed-argument checks in 8/12. Llama 3.2 3B selected the expected function in 12/12 and passed exact checks in 2/12. The wide, overlapping intervals and small public sample do not support a model ranking.
The tools were not run. These measurements cover function choice and argument construction, not downstream results or customer utility. Inference used local Ollama: zero provider calls and zero provider spend; local electricity was not measured.
Read the comparison, protocol, uncertainty and limitations ↗ · Inspect all task-level evidence ↗
Contract comparison ↗ · Instruction checks ↗ · Tool-call study ↗ · All studies ↗
EXTERNAL HUMAN AND AGENT SIGNALS
Human preferences and agent task outcomes
These are reported by Arena from its public leaderboard data. They are separate from ASN measurements and cannot be compared across tracks.
External results load as this section comes into view…
Historical Arena movement · five models across three dated snapshots
Loading archived source snapshots…
External benchmark evidence can use different samples, tasks and scoring methods. ASN does not validate these results and does not include them in its own ranking.