Gemma on older NVIDIA GPUs: Vulkan helped; our drafting experiments did not
TL;DR
- Vulkan improved measured generation speed from 25.34 to 32.15 tokens/second for Gemma 4 E4B on Predator, and from 23.44 to 34.46 for E2B on Pavilion: gains of about 27% and 47%.
- There was a tradeoff: the source notes report prompt processing about 65% slower with Vulkan. Faster output does not guarantee a faster response to a long prompt.
- Our speculative-decoding attempts did not produce a deployable speedup. They encountered rejected drafts, memory pressure, slow CPU drafting and unsupported software paths.
- This is a dated result for the tested builds. It does not prove that speculative decoding can never work on Pascal or that the measured Vulkan gain is the hardware's ceiling.
We wanted an older laptop to answer faster without buying new hardware. One candidate was speculative decoding: a smaller “drafter” suggests several next tokens, and the main model checks them together. This can save work when the guesses are accepted quickly enough. We also investigated Google's purpose-built multi-token prediction (MTP) assistant. These are related approaches, but they have different model and runtime requirements.
The 6 May experiments used Predator with a GTX 1060 6 GB and Pavilion with a GTX 1050 4 GB, as recorded in the source report. Hardware labels here describe that historical test configuration. The experiments were not a complete survey of every speculative-decoding implementation.
The simpler change that helped
Changing llama.cpp's backend from CUDA to Vulkan improved output-token generation in the recorded Gemma 4 configurations:
| Machine in the test | Model | Backend | Generation tok/s | Comparison |
|---|---|---|---|---|
| Predator, GTX 1060 | Gemma 4 E4B Q4_K_M | CUDA | 25.34 | E4B baseline |
| Predator, GTX 1060 | Gemma 4 E4B Q4_K_M | Vulkan | 32.15 | About 27% above the same-model CUDA result |
| Predator, GTX 1060 | Gemma 4 E2B Q4_K_M | Vulkan | 56.90 | Smaller model; not a same-model backend comparison |
| Pavilion, GTX 1050 | Gemma 4 E2B Q4_K_M | CUDA | 23.44 | Partial baseline in source notes |
| Pavilion, GTX 1050 | Gemma 4 E2B Q4_K_M | Vulkan | 34.46 | About 47% above that baseline |
The original notes associate the result with CUDA Flash Attention limitations on compute capability 6.1 and link llama.cpp issue 7055. The timing table shows a backend difference; it does not isolate one kernel as the cause.
The omitted tradeoff matters: the same source reports roughly 65% slower prompt evaluation with Vulkan. Prompt evaluation reads your input; generation writes the answer. Short prompts with longer answers may benefit from this balance, while long documents can make the slower input stage more important. The source does not give a prompt-length crossover or a complete end-to-end comparison for every row.
What happened in the drafting experiments
The earlier version of this article described several attempts differently from its linked research notes. The account below follows those notes, including their unfinished paths.
1. A small Gemma 3 drafter had no accepted drafts
Using Gemma 3 270M to draft for Gemma 4 in llama.cpp produced 0% draft acceptance. The recorded result was 22.6 tokens/second versus a 26.0-token/second target-only baseline, about 13% slower. The notes identified differing tokenizer metadata and special-token classifications despite equal vocabulary sizes. Matching vocabulary size alone was not enough for this pairing.
2. A same-family GPU drafter used too much memory
Gemma 4 E2B avoided the cross-family pairing but needed substantial memory alongside the E4B target. The source reports an out-of-memory attempt and another configuration at 17 tokens/second, about 33% below the 26-token/second baseline. The saved notes attribute the slowdown to memory pressure; this was not a controlled measurement of paging.
3. Moving the drafter to the CPU was slower still
The CPU-resident E2B drafter reached 7.0 tokens/second, roughly 73% below the target-only baseline. In that setup, waiting for the CPU to propose tokens cost more time than verification saved. This result concerns that model, CPU and configuration.
4. The official Gemma 4 assistant was unsupported in the tested llama.cpp path
The source identifies the official Gemma4AssistantForCausalLM architecture, not a Gemma 3 MTP model. Its converter/runtime path was unavailable in the tested llama.cpp setup, so the small official assistant could not be evaluated there. A conversion failure is a software compatibility result, not a measured inference speed.
5. PyTorch support was more specific than we first thought
The investigation corrected an initial claim that PyTorch had dropped Pascal entirely. Pavilion's tested CUDA 12.6 wheel reported sm_61 in its architecture list. Wheel channel, toolkit and version matter. The research notes still list a Transformers assistant-model experiment as in progress; they do not establish a completed, timed failure for that path. Predictions about its speed must remain predictions.
6. The lookup-based alternative was absent from the installed binaries
The production build, llama.cpp b8951, did not provide the lookup tools described in the notes. Testing that approach would have required different binaries or a custom build. It was therefore unavailable in this setup, rather than benchmarked and found slow.
How to use this result
For the recorded short-prompt Gemma workloads, Vulkan with Q4_K_M weights was the configuration with a measured generation improvement. The smaller E2B model also reduced memory pressure. These observations give a useful starting point for a similar machine, but the final choice should account for input length, model quality and total response time.
The evidence supports leaving the unsuccessful drafting configurations out of this deployment. It does not support saying that all future Pascal speculative decoding is impossible, that tensor cores are a universal requirement, or that a 27–47% gain is the maximum this hardware can ever achieve.
Sources and limits
The measurements and attempt descriptions come from the 6 May source research notes. Those notes record llama.cpp b8951/b9041, Transformers 5.8.0 and the tested PyTorch variants. They combine measured timings, compatibility checks and hypotheses; they do not include a full repeated-trial dataset for every claim.
Related material is available in the public archive. Treat the date and exact build as part of the result. Software support may have changed since these experiments.
All findings · Gemma 4 E2B benchmarks · Gemma 4 E4B benchmarks