The question
We wanted a small model to drive a Mac: read the mail, check the calendar, find a file, change a setting. Nobody had published the numbers we needed to pick one. When an agent has thirty two tools and one sentence of instruction, which models choose correctly, and does paying more help?
How it works
Every model gets the same 55 tasks, the same 32 tools and the same prompt, and picks one tool at a time against a simulated Mac. A task passes when that machine ends in the right state, so a model cannot score by describing what it would have done. Eight of the tasks cannot be done with any tool, and those are there to see who admits it.
All fifteen are small models, the dearest being claude-haiku-4.5 at $0.0055 a question. No flagship tier was tested: no GPT-5, no Opus, no Gemini Pro. Four run on the machine itself through llama.cpp, three from Liquid AI’s LFM2 family and one Qwen3-4B, all quantised to Q4.
Results
A fraction of a cent per question is hard to hold in your head, so set it to however many questions you would actually ask.
| Model | Score | Refused | Speed | A day | A month | |
|---|---|---|---|---|---|---|
| gpt-oss-20bopenai | 52/55 | 8/8 | 7.81s | <1c | 3c | |
| glm-5.3-flashz-ai | 52/55 | 8/8 | 6.07s | <1c | 11c | |
| gemini-3.1-flash-litegoogle | 51/55 | 6/8 | 3.16s | 1c | 40c | |
| kimi-k2.5moonshotai | 51/55 | 8/8 | 12.45s | 3c | 82c | |
| qwen3.7-flashqwen | 50/55 | 7/8 | 4.68s | <1c | 4c | |
| deepseek-v4-flashdeepseek | 50/55 | 5/8 | 7.58s | <1c | 5c | |
| ministral-8b-2512mistralai | 50/55 | 5/8 | 2.46s | <1c | 7c | |
| claude-haiku-4.5anthropic | 49/55 | 5/8 | 3.06s | 11c | $3.28 | |
| glm-4.7-flashz-ai | 45/55 | 5/8 | 7.93s | 1c | 19c | |
| claude-3-haikuanthropic | 43/55 | 2/8 | 3.3s | 3c | 79c | |
| gemini-2.5-flash-litegoogle | 42/55 | 4/8 | 2.22s | <1c | 8c | |
| LFM2.5-2.6Bruns on your Mac | 39/55 | 0/8 | 4.02s | free | free | |
| Qwen3-4Bruns on your Mac | 38/55 | 3/8 | 6.54s | free | free | |
| LFM2.5-1.2B-Instructruns on your Mac | 27/55 | 0/8 | 1.01s | free | free | |
| LFM2-1.2B-Toolruns on your Mac | 25/55 | 5/8 | 0.35s | free | free |
Refused counts the eight tasks that cannot be done with any tool, where declining is the right answer. Costs are what the gateway billed during the run, divided by 55, so they carry that run’s prompt sizes. A longer question costs more than a short one.
What we found
Price told us nothing. gpt-oss-20b scored 52/55 at $0.000058 a question. claude-haiku-4.5 scored 49 at 95 times the price. The top seven sit within two tasks of each other.
Refusing separates them. The leaders decline all eight impossible tasks. LFM2.5-2.6B and LFM2.5-1.2B-Instruct decline none and invent an action instead, while scoring respectably everywhere else, which is how one overall number hides it.
The tool-tuned model came last. LFM2-1.2B-Tool is fine tuned for tool calling and scored 25/55, the lowest here, and it is the fastest at 0.35s. It emits well formed tool calls quickly and picks the wrong tool.
So Celeritas ships LFM2.5-2.6B. 39/55, free on the machine, 13 behind the leader. The app prints both numbers in its settings, and anyone who wants the top of the table can paste their own gateway key.
Limitations
- We built the suite around our own launcher and we ship one of the models in it. Read the order with that in mind.
- No flagship model was tested, so this says what it costs to get close to the ceiling, never where the ceiling is.
- Development split, one trial per model, no error bars. The top seven are within two tasks of each other, and one task is 1.8 points.
- The simulated Mac is our own, and a real one will disagree somewhere we have not found.
- Latency was measured on one machine. Compare the scores across models, not the seconds.
Reproducing it
Each row came from a run record holding the prompt, the tool chosen, the arguments and the outcome for all 55 tasks. The harness points at any OpenAI shaped gateway:
CELERITY_BASE=<gateway>/v1 CELERITY_KEY=<key> \ python3 bench/run.py --model <id> --split dev
A run that hits a transport failure is abandoned rather than scored. An early parallel run produced a complete and entirely false leaderboard because rate limits were counted as wrong answers.