Both models chose the right tool.
Half the calls had exact arguments.

Choosing the right tool is only one part of a usable tool call. The argument types and values must also be correct. The first column below measures the function choice; the second requires a full oracle match.

The 95% Wilson intervals for exact calls are wide and overlap (Qwen 23–88%; Llama 12–77%). The one-example difference does not support a model ranking. Function-choice accuracy is 5/5 for both models here, also too small to generalize.

Where arguments failed

Wall-clock request medians were 17.0 s for Qwen and 18.5 s for Llama on this host. They include local model/runtime conditions and are not general speed estimates. No selected tool was executed, so this study measures call selection and arguments—not the freshness, correctness or usefulness of any downstream tool result.

A deterministic, disjoint 12-task extension of the five-case pilot above tested the same installed model builds on new public BFCL V4 examples. Qwen selected the expected function in 11/12 responses and passed the exact function-plus-arguments check in 8/12. Llama selected the expected function in 12/12 and passed the exact check in 2/12.

The exact-call 95% Wilson intervals are wide and overlap (Qwen 39–86%; Llama 5–45%). The task sample is small, public and potentially present in training data. These results describe only these model builds and prompts; they do not establish a general winner, production ranking or customer utility.

All 10 Llama failures were invalid argument schemas. Qwen's three argument mismatches followed the correct function selection; its fourth failure was a response without exactly one tool call. This small sample shows why tool selection alone does not establish a usable tool call.

No tool was executed. We tested function selection and typed arguments, not whether a downstream service exists, returns fresh data or solves the user's task.

Source and selection

BFCL V4 multiple-function category, pinned to the source commit above. Twelve tasks were deterministically selected by a published hash rule from 67 eligible cases, excluding the five pilot tasks.

Protocol

One response per model-task pair; Ollama loopback; temperature 0; seed 42; 60-second request cap; 20-minute total cap. 24 local calls, zero provider calls and $0 provider spend. Electricity and hardware opportunity cost were not measured.

Model builds

Qwen 2.5 1.5B 65ec06548149b04c096a120e4a6da9d4017ea809c91734ea5631e89f96ddc57b; Llama 3.2 3B a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72.

Independent evidence

Plan SHA-256 0df8705c39854f68123162d101f2692290afeddb15f9598195cfd0ed6510468e; local run SHA-256 057f7d2b5c0d92a2dae26f497f91f80c4f57835fd171868fd08e5ff6cfce514c; public receipt SHA-256 1009ad7b6206b7fd80a5e7b8d59a4781ac12796d1797c45dc85f2c64668042a3. Inspect all 12 tasks, expected arguments, model calls and verification outcomes ↗.

Production status

Exploratory evidence only. It is excluded from the production measurement index, routing and rankings. It did not execute tools, write production data or change system authority.

Protocol, provenance and limits

Source

Berkeley Function Calling Leaderboard (BFCL), V4 multiple-function selection category, pinned to commit 58f57e9124ea981403792dd51e00a6577e621fae (Apache-2.0). Model exposure to these public examples during training is unknown.

Grading

Independent exact check of one expected function, schema-valid argument types and source-oracle values. The correct function and full-call result are shown separately.

Execution

Ollama loopback only; one response per model and example; temperature 0; fixed seed; 60-second request timeout; 15-minute total cap. Provider calls and spend: zero. Local electricity cost was not measured.

Source hashes

Case file aef168155ebd74b7ac2401198b201343bc7d16d7a3d7e0d4e6d8ee82c6969b2a; answer file 244e00ce9395df948bcafc7bee64e8f9c87ef70887587d83cae45b13699f3047. The original run receipt preserved two mistyped hashes; the pinned files and all five task mappings have now been reverified. Inspect the source hash correction and task map ↗.

Model builds

Qwen 2.5 1.5B 65ec06548149b04c096a120e4a6da9d4017ea809c91734ea5631e89f96ddc57b; Llama 3.2 3B a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72.

Evidence receipt

Inspect task-level calls, typed arguments, scores and integrity hash ↗. Receipt SHA-256: 1aeac6517005551fba535520a9cc129a4ba1c71486e73c28f5618b4488dd903b.

Production status

This exploratory local pilot is excluded from production measurements, rankings and routing. It made no production writes and changed no system authority.

← Return to the live AI observatory