How fast can a personal AI setup actually be?
TL;DR
- Small coding models answered in roughly 10–14 seconds per prompt in the first Pavilion test. Eight Qwen3.5 variants averaged 22–32 seconds with no request errors across their 40 coding prompts.
- The setup matters as much as the model name. Qwen2.5 Coder 3B averaged 70.5 seconds in the first run and 21.04 seconds in the next day's run, after the test order changed.
- We cannot assign a fixed “router tax.” The 0.5B model was about 30% slower than an earlier direct test in run one, but about 7% slower in run two. These were different runs on different dates, with loading and runtime differences.
- A reply is not a correct solution. The scores below check requested words and formatting. They do not prove that the generated code works.
An older gaming laptop can be useful for local AI. The useful question is how long your actual prompt takes to finish, including the software between you and the model. These tests measured that complete request path on Pavilion, an HP laptop with an Intel i7-9750H, a GTX 1050 with 4 GB of VRAM and 16 GB of RAM.
The results below cover Pavilion. They do not establish performance for the Mac, Predator, a CPU VPS or a combined four-machine cluster. They are historical measurements, not a statement about those machines' current configuration.
What we measured
On 28 April 2026, the test sent one greeting and five coding questions to each of sixteen model configurations through Weeyuga's OpenAI-compatible request endpoint. The router selected an inference engine on Pavilion. Qwen3.5 0.8B used llama.cpp; most other entries used Ollama. This therefore compares deployed configurations, including their engines and settings.
The questions asked for a disk-space guard script, an IPv4 validator with tests, a safe nginx reload, a YAML validation plan and SSH lockout diagnosis. Each question ran once per model. The first run recorded 96 calls in total: 81 without a recorded request error and 15 with one. Those totals include the greetings; the table reports only the five coding questions.
The first run: seconds per coding question
| Model group | Mean time per question | Reported output tok/s | Request errors | Format passes | Keyword coverage |
|---|---|---|---|---|---|
| Qwen3.5, eight variants from 0.8B to 35B-A3B | 22–32 s | 30–41 | 0 / 40 | 3–4 / 5 per variant | 72–85% |
| Qwen2.5 Coder 0.5B / 1.5B | 10–14 s | 18–38 | 0 / 10 | 3 / 5 each | 82–90% |
| Qwen2.5 Coder 3B | 70.5 s | 3.2 | 0 / 5 | 3 / 5 | 90% |
| Qwen2.5 Coder 14B | All five timed out at the six-minute limit | — | 5 / 5 | 0 / 5 | 0% |
| Qwen3 4B, thinking configuration | 263 s | 1.65 | 0 / 5 | 0 / 5 | 0% |
| Qwen3 8B / 14B, thinking configurations | 157–353 s, completed calls only | — | 3 / 5 and 4 / 5 | 2 / 5 and 1 / 5 | 40% and 16% |
| Qwen2.5 3B | 28.0 s | 10.5 | 0 / 5 | 3 / 5 | 90% |
“22 seconds” means the average time for one question, not the entire five-question packet. The 35B-A3B entries are mixture-of-experts models; their names do not mean all weights fit into 4 GB of VRAM. Memory placement was not established by these latency figures.
The Qwen3.5 configurations were the most consistently responsive group in this particular first run. That observation is useful for shortlisting a configuration to try. Five questions and one run per question are too little evidence to name a universal best model.
The next run changed the picture
The first run's 0.5B model averaged 10.4022 seconds through the router, compared with 7.99 seconds in a separate 11 April direct-to-Ollama GPU test. That is an observed difference of about 30%. The original article treated it as the router's fixed overhead. The evidence does not support that conclusion.
On 29 April, the test ran a complete greeting-plus-five-question block for each model before moving to the next. Its raw ledger gives these means:
| Model | Earlier direct test, 11 April | Router run, 28 April | Router run, 29 April |
|---|---|---|---|
| Qwen2.5 Coder 0.5B | 7.99 s | 10.4022 s | 8.5332 s |
| Qwen2.5 Coder 3B | 13.00 s | 70.5146 s | 21.0400 s |
The second run's 0.5B result is about 7% above the older direct test, while the 3B result is about 62% above it. Both are much closer than in the first run. Run order, model residency, request settings and device state can affect the difference. We did not vary only the router while keeping everything else fixed.
Model eviction and reloading were a plausible explanation for some of the slowdown. The planned comparison of Ollama and llama.cpp with one resident model stalled on a service-wrapper problem and remained incomplete. Neither run establishes how much latency each router component adds.
Explore the second run or download its complete call ledger. The published first-run viewer uses a partial original ledger; the fuller historical report is linked in the evidence notes below.
What the scores tell us
Keyword coverage is the fraction of expected text markers found in the response. Format passes check a requested output shape. In the older harness, a usable answer simply meant that the response was nonempty. These are inexpensive automated checks; they are not execution tests, expert review or an accuracy score.
For example, the first 0.5B router run had about 82% keyword coverage and three format passes out of five; the earlier direct run had 80% and three out of five. Similar scores on five prompts do not prove that routing preserves correctness. In the second router run, keyword coverage was about 77%, again with three format passes.
Qwen3 4B illustrates another limitation. It completed HTTP requests but spent the output budget on thinking text, yielding no markable final answer on the five questions. The 2,048-token output limit bounded the responses; it did not make them useful. Request success, response usefulness and solution correctness are separate checks.
What this means for a similar laptop
The measured small coding models are reasonable starting points for short, single-user tasks on this hardware. Try the actual request path you intend to use and include the first request as well as later ones. A fast warm response can hide a slow load, and an error-free response can still be wrong.
These runs do not measure long-context reliability, continuous background operation, concurrent users, voice latency or the effect of buying a newer GPU. Adding another computer also does not automatically combine the two GPUs' memory into one model's memory pool.
Evidence and limits
- Detailed first-run historical report: the 28 April results, per-model tables and audit notes. Driver revision
9934892. - First-run catalog entry and original partial ledger: this original file is incomplete and must not be mistaken for all 96 calls.
- Second-run catalog entry and complete 29 April ledger: the source of the recalculated 8.5332-second and 21.04-second means above.
- Earlier 0.5B direct GPU report and how we measure.
Each configuration had one greeting and five different coding prompts, with one observation per prompt. The configured context was 4,096 tokens, temperature 0.1 and output budget 2,048 tokens; the harness imposed a six-minute request timeout. These runs do not provide repeat-run percentiles.
The first run's primary JSONL stopped after 18 records; the source analysis reconstructed the 96 calls from its stdout log. Some original per-call fields could not be recovered, including prompt-token counts, response lengths and marker lists. The second run captured all 96 call records. Later comparisons should state which ledger they use and preserve errors alongside completed calls.