# Gemma 4 Multi-Token-Prediction (MTP) on Pascal — Negative Findings

**Author:** mac/claude-main
**Date:** 2026-05-06
**Hardware tested:** Predator (GTX 1060 6 GB), Pavilion (GTX 1050 4 GB)
**Tools tested:** llama.cpp build 8951 + b9041 (CUDA + Vulkan), HuggingFace Transformers 5.8.0, PyTorch 2.0.1+cu118 / 2.4.0+cu118 / 2.11.0+cu126, bitsandbytes 0.49.2

> **One-line takeaway:** Don't burn time chasing a 3x MTP speedup for Gemma 4 on
> Pascal-class GPUs. The official Google drafter doesn't load in llama.cpp;
> generic llama.cpp `--model-draft` yields 0–13% slowdown not speedup; CPU-side
> drafting tanks throughput; Pascal lacks tensor cores so the speedup math
> doesn't pencil even when it works elsewhere. **Switch to Vulkan backend
> instead — that's where the real Gemma 4 win lives on Pascal.**

---

## What we wanted

A 3× tokens/sec speedup for Gemma 4 E4B serving on Predator/Pavilion via
Multi-Token Prediction (speculative decoding with a drafter model verifying
multiple tokens per target forward pass).

## What actually worked (positive finding for context)

**Switching llama.cpp from CUDA backend to Vulkan backend** on Pascal nets
+27% to +47% generation throughput because Flash Attention is broken on
compute capability 6.1 in llama.cpp's CUDA backend
([issue #7055](https://github.com/ggml-org/llama.cpp/issues/7055)) and Vulkan
goes around it. The actual measured wins:

| Machine | Model | Backend | tg t/s | Note |
|---|---|---|---|---|
| Predator GTX 1060 | Gemma 4 E4B Q4_K_M | CUDA | 25.34 | original baseline |
| Predator GTX 1060 | Gemma 4 E4B Q4_K_M | **Vulkan** | **32.15** | +27% |
| Predator GTX 1060 | Gemma 4 E2B Q4_K_M | **Vulkan** | **56.90** | 2.2× baseline |
| Pavilion GTX 1050 | Gemma 4 E2B Q4_K_M | CUDA | 23.44 | partial baseline |
| Pavilion GTX 1050 | Gemma 4 E2B Q4_K_M | **Vulkan** | **34.46** | +47% |

**This is the real Gemma 4 deployment on Pascal — Vulkan + Q4_K_M, no MTP.**

## What we tried for MTP and why it didn't work

### Attempt 1 — llama.cpp `-md` with Gemma 3 270M as drafter

**Hypothesis:** llama.cpp's generic `--model-draft` flag accepts any drafter
that shares vocab with the target. Smallest available Gemma small-model is
Gemma 3 270M.

**Result:** Drafter loaded fine. **0% draft acceptance** in measurements.
End-to-end generation 13% slower than baseline (22.6 t/s vs 26.0 t/s on
Predator with cleared VRAM).

**Root cause:** Vocabulary mismatch. Gemma 4 GGUF has
`tokenizer.ggml.model = "gemma4"` while Gemma 3 270M GGUF has
`tokenizer.ggml.model = "llama"`. Vocab sizes match (262,144) but special-token
classifications diverge (`token_type[0..5]` differ between the two).
Result: same token ID encodes a different actual token in each model. Every
draft is rejected, all overhead, no payoff.

**Don't repeat.** Verify `tokenizer.ggml.model` field matches between target
and drafter before bothering to benchmark. The vocab-size match is necessary
but **not** sufficient.

### Attempt 2 — llama.cpp `-md` with Gemma 4 E2B as drafter, GPU-resident

**Hypothesis:** Use a Gemma-4-family drafter so vocab matches. E2B is the
smallest official Gemma 4 (5B params, ~3 GB at Q4).

**Result:** Drafter is too big to fit alongside the target in 4–6 GB VRAM.
On GTX 1060 6 GB: target Q4 (~4.2 GB) + drafter Q4 (~3 GB) = OOM. On the
attempt that did squeeze in (different quants), VRAM saturation forced the
GPU to spill to shared system memory — generation collapsed to 17 t/s
(33% slower than baseline).

**Don't repeat.** A drafter that needs ~50% of target's VRAM defeats the
"small drafter, big target" assumption MTP relies on. There is no smaller
official Gemma 4 model.

### Attempt 3 — llama.cpp `-md` with Gemma 4 E2B as drafter, CPU-resident (`-ngld 0`)

**Hypothesis:** Move drafter to CPU to avoid VRAM contention. CPU runs the
small drafter, GPU runs the big target. Mainstream MTP pattern on
asymmetric drafter/target hardware.

**Result:** **7.0 t/s** — 73% slower than the GPU-only baseline of 26 t/s.

**Root cause:** GTX 1060 / 1050 era CPUs (Pavilion has Haswell-era cores) are
slow at running a 5B-param model. Each target verify cycle waits for CPU
drafter to produce 16 candidate tokens. Even at high acceptance, drafter
latency dominates total time.

**Don't repeat on this hardware tier.** CPU-drafter MTP only works when the
CPU is fast enough to generate drafts faster than the GPU can verify them.
On a 4–6-year-old laptop CPU paired with a Pascal GPU, the math doesn't pencil.

### Attempt 4 — Google's official Gemma 4 MTP drafter (`Gemma4AssistantForCausalLM`)

**Hypothesis:** Google released purpose-built drafters for each Gemma 4 size
(`google/gemma-4-E4B-it-assistant`, `gemma-4-E2B-it-assistant`, etc.) — these
use a centroid-prediction architecture fused with the target backbone, much
better acceptance than a separate small model.

**Result:** Cannot load in llama.cpp. The architecture
`Gemma4AssistantForCausalLM` is **not implemented** in llama.cpp's GGUF
converter or runtime. The 159 MB safetensors won't convert via
`convert_hf_to_gguf.py`.

**Workaround tried:** Switch frameworks. HuggingFace Transformers 5.8.0
natively implements `gemma4_assistant` model class. But transformers 5.8.0
requires PyTorch ≥ 2.4, and we initially believed PyTorch 2.x had dropped
SM_61 (Pascal compute capability 6.1). See attempt 5.

### Attempt 5 — HuggingFace Transformers + assistant_model API (PyTorch path)

**Hypothesis:** PyTorch on Pascal is dead anyway (claim from session
context).

**Result of investigation (corrected on the same day by Mila's research):**
**The claim was wrong.** PyTorch wheels on the **cu126** channel still
include SM_61 SASS through PyTorch 2.11.x. Verified by running
`torch.cuda.get_arch_list()` on Pavilion: returns
`['sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']`.
Only **cu128/cu129/cu130** wheels (CUDA toolkit 12.8+, current default for
PyTorch 2.9+) dropped Pascal — and that's a CUDA-toolkit-side decision,
not a PyTorch project decision.

**Right install for Pascal:**

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
```

See [`docs/RESEARCH/pytorch_pascal_sm61_2026-05-06.md`](pytorch_pascal_sm61_2026-05-06.md)
for the full architecture matrix.

**However** — even with PyTorch happy, the *library ecosystem above
PyTorch* is gated above Pascal:
- **flash-attention v2/v3** require SM_80 (Ampere) — no fallback
- **vLLM / SGLang** depend on flash-attention in critical paths — useless on Pascal
- **bitsandbytes 4-bit** has slow fallback paths only on SM_61
- **PyTorch SDPA** works but is slow vs Ampere

So even with the assistant_model API loading the official drafter via
transformers, expected speedup on Pascal is 1.5×–2× at best (not the 3×
seen on Ampere+ hardware), because the underlying matmul kernels lack
tensor-core acceleration.

**Don't repeat the "PyTorch dropped SM_61" assumption.** Check
`torch.cuda.get_arch_list()` empirically before designing around the
constraint.

### Attempt 6 — n-gram self-speculative decoding (no separate drafter)

**Hypothesis:** llama.cpp's `--lookup-cache-dynamic` mode uses the target
model's own n-gram history as an in-band "drafter", no second model needed,
no vocab issue, no VRAM cost.

**Result:** **Feature not present in llama.cpp build 8951** (the production
build on Predator/Pavilion). The `llama-lookup` and `llama-lookup-stats`
tools that previously implemented this were dropped from official prebuild
distributions. Would need a custom rebuild to test.

**Don't repeat.** If you're on official llama.cpp prebuilt Windows
binaries, n-gram self-spec is unavailable.

---

## Independent confirmations from sister research

[`docs/RESEARCH/trinity_30b_a3b_speedup_2026-05-06.md`](trinity_30b_a3b_speedup_2026-05-06.md)
(Mila's 30B-A3B speedup research) reaches a parallel negative finding for
MoE models: **spec decoding on Qwen3-30B-A3B is net-negative on RTX 3090**
across 19 tested configs (per
[thc1006 benchmark](https://github.com/thc1006/qwen3.6-speculative-decoding-rtx3090)),
even at 100% draft acceptance. MoE expert-loading overhead per drafted
token defeats the amortization that makes spec decoding profitable on
dense models. Different model, different reason — but same direction:
spec decoding is a research-grade finding, not a deploy-and-forget win on
consumer GPUs.

---

## What to do instead (deploy these, not MTP)

For Gemma 4 on Pascal-class hardware:

1. **Switch llama.cpp backend to Vulkan** — `+27%` to `+47%` generation
   throughput, free.
   - Download: [`llama-b9041-bin-win-vulkan-x64.zip`](https://github.com/ggml-org/llama.cpp/releases/tag/b9041)
   - Trade-off: Vulkan prompt-eval is 65% slower than CUDA prompt-eval, so
     prompt-heavy workloads (long context, RAG) keep using CUDA.
2. **Pick the model size that fits in VRAM** — Gemma 4 E2B (5B params)
   fits in both 4 GB and 6 GB, generates at 34–57 t/s. Forcing E4B (7.5B)
   onto a 4 GB GPU spills to CPU and halves throughput.
3. **Skip MTP**. The 3× speedup that Google advertises requires sm_80+
   tensor cores. Don't waste cycles on it for Pascal deployments.

## Reproducibility — exact commands that demonstrated the negative findings

Detailed per-attempt commands and outputs are preserved in
[`coordination/messages/2026-05-06T121601Z-mac-claude-main-broadcast-pavilion-vulkan-backend-47pct-gemma4-win.md`](../../coordination/messages/2026-05-06T121601Z-mac-claude-main-broadcast-pavilion-vulkan-backend-47pct-gemma4-win.md)
(positive finding) and the same-day session transcript thread.

## Open follow-on questions (parked, non-blocking)

1. **Pavilion bnb-4bit + transformers + assistant_model E2B test** — in
   flight at time of writing. If it lands a real speedup, append to this
   doc. If it doesn't beat the Vulkan 34.46 t/s baseline, that's another
   negative for the file.
2. **Vulkan dual-device offload via `--tensor-split`** — Pavilion's Vulkan
   sees both the GTX 1050 and the Intel UHD Graphics 630. Could split
   layers across both for another 20–30%. Not tested. Trinity P2.
3. **Gemma 4 + llama.cpp custom build with `Gemma4AssistantForCausalLM`
   support** — would unlock the 159 MB official drafter for llama.cpp
   users. Multi-day C++ work. Not warranted for our hardware tier.

---

## Public-safe attestation

This doc contains no internal IPs, credentials, or secrets. All hostnames
and paths are generic (`Pavilion`, `Predator`, `D:\WeeyugaModels\…`).
External links are to public repos and HF model cards. Safe to mirror to
`benchmarks.weeyuga.com/research/failed-experiments/` after Tomas's
re-edit pass.