IFEVAL SUBSET · LOCAL LOOPBACK · FEASIBILITY ONLY
Can we measure instruction following reliably?
We ran five public instructions against two fixed local model builds, with one response per task and deterministic checks for five formatting rules.
Keeping models resident for five minutes changed one Qwen task result in a single replication, but did not make the finding stable: at 384 tokens Qwen returned 3 passes and 2 failures; Llama still timed out on all five tasks.
10 ATTEMPTS · 5 TASKS PER MODEL · 5 CHECK RULES
Outcomes and execution coverage
| Local model | 60s · 768 tokens pass / fail / error | 80s · 768 tokens pass / fail / error | 80s · 384 tokens keep_alive 0 pass / fail / error | 80s · 384 tokens keep_alive 5m pass / fail / error |
|---|---|---|---|---|
| Qwen 2.5 1.5B | 2 / 1 / 2 | 2 / 1 / 2 | 2 / 3 / 0 | 3 / 2 / 0 |
| Llama 3.2 3B | 0 / 0 / 5 | 0 / 0 / 5 | 0 / 0 / 5 | 0 / 0 / 5 |
The 80-second timeout alone did not change outcomes. Capping generation at 384 tokens returned all five Qwen answers, two of which reached the token cap and failed verification. With five-minute residency, Qwen had one fewer verified failure and one more pass than in the single otherwise-matched keep_alive:0 run; one task outcome changed. This is one repetition and cannot establish a stable improvement. Llama timed out on all five tasks in both 384-token profiles, so the models remain incomparable.
Which instruction checks changed?
The controlled residency comparison uses the same 384-token limit and changes only keep_alive. Four of Qwen’s five task verdicts stayed the same; one changed from fail to pass. Llama returned no valid outputs in either condition.
| Instruction check | Qwen · no residency | Qwen · 5-minute residency |
|---|---|---|
| No comma | Fail | Fail |
| Exactly three bullets | Pass | Pass |
| Valid JSON | Pass | Pass |
| Wrapped in quotation marks | Fail | Fail |
| Required keywords | Fail | Pass |
Each cell is one local response, not a stable task success rate. The single changed verdict is a lead for a repeated study, not evidence that residency improves instruction following.
What the verifier checked
A narrow, deterministic reimplementation of five named IFEval rules: no comma, exact bullet count, JSON format, quotation wrapping, and required keywords. It does not run the full official IFEval evaluator and uses no model judge.
Source dataset revision: 966cd89545d6b6acfd7638bc708b98261ca58e84. Verifier source revision: e6890f85757dd84e27ca6df2dd30651dafad28e0. Runtime: Ollama 0.33.3 on local loopback; 60/80-second request timeouts, 768/384-token generation caps, keep_alive 0/5m, and a 15-minute total cap per run. No external calls, provider spend, production writes or routing changes.
Inspect the sanitized, hash-checked receipt ↗. Prompts, raw outputs and detailed error text are withheld.
Limits and next step
Five prompts and one attempt each do not represent the full benchmark. Training exposure is unknown. Local results do not estimate hosted service availability, latency, cost, or customer value. A proper comparison requires a runtime that completes both cohorts plus repeated runs and broader held-out tasks under matched conditions.
This pilot is supporting research only. It is excluded from the production measurement index and model rankings.