IFEVAL FOLLOW-UP · TWO LOCAL RUNS · TEN SHARED PROMPTS
What did two runs show about format checks and timeouts?
Two pinned local model builds were run twice on the same ten IFEval prompts. A deterministic verifier checked only five named format rules; the first run had 9/10 complete pairs, the second only 3/10.
Latest rerun: Qwen had 5 verified passes, 2 failures and 3 timeouts. Llama had 4 passes and 6 timeouts. These are local execution outcomes, not a model ranking.
FIRST RUN · 20 ATTEMPTS · 10 MATCHED PROMPTS · 9 COMPLETE PAIRS
First run · verified outcomes
| Fixed local model build | Attempts | Verified passes | Verified failures | Timeouts | Passes / verified outputs · 95% Wilson interval |
|---|---|---|---|---|---|
| Qwen 2.5 1.5B | 10 | 6 | 4 | 0 | 6 / 10 · 60% (31.3–83.2%) |
| Llama 3.2 3B | 10 | 7 | 2 | 1 | 7 / 9 · 77.8% (45.3–93.7%) |
On the nine prompts where both outputs were checked, both passed four times; only Qwen passed twice; only Llama passed three times; both failed zero times. The remaining prompt is incomplete: Qwen returned a verified failure while Llama timed out.
The 95% Wilson intervals are wide and use verified outputs only: the 77.8% figure excludes a timeout and is conditional on a returned output. It is not directly comparable to Qwen's 60% across all ten attempts. Ten prompts and one response per model are too little evidence for a general model ranking.
SAME TEN PROMPTS · SECOND LOCAL RUN · 28 OCT 2026
Only 3 of 10 task pairs had two verified outputs.
This rerun reused the same ten pinned prompts and local model builds. Compared with the earlier run, Qwen went from 6 passes, 4 failures and 0 timeouts to 5 passes, 2 failures and 3 timeouts. Llama went from 7 passes, 2 failures and 1 timeout to 4 passes, 0 failures and 6 timeouts.
| Model | Earlier pass / fail / error | Rerun pass / fail / error | Rerun verified outputs · 95% interval |
|---|---|---|---|
| Qwen 2.5 1.5B | 6 / 4 / 0 | 5 / 2 / 3 | 5 / 7 · 71.4% (35.9–91.8%) |
| Llama 3.2 3B | 7 / 2 / 1 | 4 / 0 / 6 | 4 / 4 · 100% (51.0–100%) |
The rerun produced two fully verified passes on both models and one Llama-only pass among the three complete pairs; seven pairs were incomplete due to at least one execution timeout. A timeout is missing evidence, not a verifier failure. Intervals condition on returned outputs and are very wide; the two runs and ten public prompts do not support a model ranking or stable capability estimate. This local observation is excluded from the production index and routing.
Inspect the sanitized rerun receipt ↗. It includes task-level states, model digests, intervals and the receipt hash; raw prompts, outputs and detailed timeout messages are omitted.
Protocol and evidence
- Dataset
- Google IFEval, pinned revision
966cd89545d6b6acfd7638bc708b98261ca58e84. Ten source prompts were selected before inference and are disjoint from the earlier five-task pilot and 30-task holdout. - Verifier
- Five deterministic rules only: no comma, numbered bullet count, JSON format, quotation wrapping and required keywords. No model judge; not the full official IFEval evaluator.
- Runtime
- Ollama 0.33.3 on loopback, temperature 0, seed 42, 384-token output cap, 80-second request timeout and 20-minute run cap. No provider calls or API spend.
- Evidence
- Inspect the sanitized, hash-checked receipt ↗. It includes task IDs, model digests, per-task verifier states and paired outcomes. Raw prompts, model outputs and detailed error text are omitted.
The public benchmark may have appeared in model training; exposure is unknown. This study measures only the named format checks, not truth, answer quality, safety, usefulness, hosted service performance or customer value. It is local supporting research, excluded from production measurements, rankings and routing.
← Read the earlier 30-task IFEval holdout · Browse all studies