IFEVAL SUBSET · LOCAL LOOPBACK · 60 ATTEMPTS
Can these small models follow five precise instructions?
We tested two pinned local model builds on 30 previously unused IFEval prompts. Each of five deterministic format rules has six prompts. Every output was checked by a deterministic verifier; timeouts remain execution errors, not format failures.
Qwen 2.5 1.5B: 15 verified passes, 15 verifier failures (9 reached the 384-token cap), 0 timeouts across 30 attempts. Llama 3.2 3B: 16 verified passes, 1 verifier failure, 13 timeouts across 30 attempts. Only 17 Llama responses arrived before the 80-second request limit.
30 PROMPTS · 6 PER RULE · 2 PINNED LOCAL BUILDS
Verified outcomes by format rule
| Rule · 6 attempts | Qwen pass / fail / timeout | Llama pass / fail / timeout |
|---|---|---|
| No comma | 2 / 4 / 0 Pass rate among returned: 33% · 95% CI 10–70% · returned 6/6 | 5 / 0 / 1 Pass rate among returned: 100% · 95% CI 57–100% · returned 5/6 |
| Numbered bullet list | 4 / 2 / 0 Pass rate among returned: 67% · 95% CI 30–90% · returned 6/6 | 4 / 0 / 2 Pass rate among returned: 100% · 95% CI 51–100% · returned 4/6 |
| JSON format | 4 / 2 / 0 Pass rate among returned: 67% · 95% CI 30–90% · returned 6/6 | 2 / 0 / 4 Pass rate among returned: 100% · 95% CI 34–100% · returned 2/6 |
| Quotation wrapping | 2 / 4 / 0 Pass rate among returned: 33% · 95% CI 10–70% · returned 6/6 | 3 / 0 / 3 Pass rate among returned: 100% · 95% CI 44–100% · returned 3/6 |
| Required keywords | 3 / 3 / 0 Pass rate among returned: 50% · 95% CI 19–81% · returned 6/6 | 2 / 1 / 3 Pass rate among returned: 67% · 95% CI 21–94% · returned 3/6 |
| All rules · 30 attempts per model | 15 / 15 / 0 | 16 / 1 / 13 |
Each cell adds a Wilson 95% interval for verified passes among returned outputs, plus the returned count out of six. These conditional rates do not include timeouts; read them beside the pass / failure / timeout counts.
Each rule has only six prompts per model, so its counts are highly uncertain. Ten Qwen generations reached the 384-token cap; nine of its 15 verifier failures were cap-limited, while one cap-ending output passed. Those failures describe behavior under this output limit, not unconstrained formatting ability. Llama timed out on 13 of 30 attempts. Among its 17 returned outputs, 16 passed, but that conditional rate hides the timeouts and must not be compared alone with Qwen's full set. We make no general capability ranking from this study.
TARGETED FOLLOW-UP · 10 PREVIOUSLY CAP-LIMITED TASKS
Did raising the output cap rescue the earlier failures?
We reran the same ten Qwen prompts that had reached 384 generated tokens, with the same model build, prompt text, temperature, seed, 80-second request limit and five-minute residency. The only planned inference-setting change was a 768-token output cap.
| Qwen 2.5 1.5B · selected ten tasks | Verified pass | Verifier failure | Timeout | Reached cap |
|---|---|---|---|---|
| Original · 384 tokens | 1 | 9 | 0 | 10 |
| Diagnostic repeat · 768 tokens | 0 | 6 | 4 | 0 |
Raising the cap did not rescue any earlier verifier failure among the six responses that completed at 768 tokens. Four attempts timed out and are inconclusive. None of the six returned responses reached the new cap. These ten tasks were selected because of the earlier cap outcome, so this result diagnoses those failures; it is not an unbiased IFEval pass-rate estimate.
Protocol and evidence
- Data
- Google Research IFEval dataset, revision
966cd89545d6b6acfd7638bc708b98261ca58e84; 30 held-out prompts excluded from the prior five-prompt pilot. - Verifier
- Five deterministic rule checks; no model judge and not the full official IFEval evaluator.
- Local builds
- Qwen 2.5 1.5B ·
65ec06548149b04c096a120e4a6da9d4017ea809c91734ea5631e89f96ddc57b
Llama 3.2 3B ·a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72 - Execution
- Ollama 0.33.3 on loopback only; 80-second timeout per request; 384-token output cap; temperature 0; seed 42. Sixty local requests, no provider calls or spend.
- Suite SHA-256
3152ab43fe8c211946fc9c53f5c6bce98455ded93ec34af334c14371deb37a5d- Run artifact SHA-256
0d1366a989d79c4e271f25ceaca329cf04c91f4fff32d654939f4978b34d5f41- Public summary receipt
- Open hash-checked aggregate results and paired outcomes ↗
Raw prompts and model outputs are not included in the public receipt. The original private run artifact is retained locally for independent review. This study does not measure hosted service latency, cost, safety, legal correctness, customer value, or general intelligence. It is supporting research only and is not a production measurement or routing signal.
Read the earlier feasibility pilot ↗ · ← Return to the live AI observatory