On June 5, 2026, Google DeepMind published a new set of checkpoints on Hugging Face for the entire Gemma 4 model family, all trained with Quantization-Aware Training (QAT). According to Google DeepMind, this cuts VRAM requirements by roughly 72% compared to the original BF16 models, while keeping output quality very close to the uncompressed originals.
For teams running local AI on existing hardware, this changes the calculus significantly.
What Is QAT — and Why Does It Beat Standard Quantization?
Standard post-training quantization (PTQ) compresses a finished model to 4 bits after training. That process inevitably loses information — the model wasn't trained to account for those rounding errors.
QAT works differently. During training itself, the effects of quantization are simulated, so the model learns to compensate for them before weights are ever frozen. The result is a 4-bit model that behaves much more like its full-precision counterpart.
The Unsloth team, which contributed detailed benchmark comparisons for Gemma 4 QAT, reports accuracy improvements of over 15% versus standard PTQ conversion at the same compression level. In practical terms: fewer hallucinations, better instruction-following, more stable multi-turn behavior.
Google DeepMind's stated goal was to maximize on-device performance — letting developers get BF16-level results from hardware that can only fit 4-bit weights.
Model Sizes and Memory Requirements
The Gemma 4 QAT family covers five sizes. Using the recommended UD-Q4\K\XL GGUF format (per Unsloth's documentation), approximate memory requirements are:
| Model | Notes | ~Memory |
|---|---|---|
| Gemma 4 E2B | Ultra-lightweight | ~3 GB |
| Gemma 4 E4B | Best quality for 8 GB GPUs | ~5 GB |
| Gemma 4 12B | Balanced | ~7 GB |
| Gemma 4 26B-A4B | MoE, 3.8B active/token | ~15 GB |
| Gemma 4 31B | Highest quality | ~18 GB |
The 26B-A4B variant is the standout. It uses a Mixture-of-Experts (MoE) architecture, activating only 3.8 billion parameters per token. This means inference speed and memory behavior are close to a 4B model, while output quality reflects the full 26B parameter count. According to reported community measurements, the 26B-A4B QAT model loads in approximately 15 GB at Q4\K\XL — fitting comfortably on a 16 GB Apple Silicon Mac or a 16 GB GPU such as an RTX 4060 Ti.
Getting Started: Three Setup Paths
With Ollama (recommended for quick deployment)
Ollama 0.24.0 supports the Gemma 4 QAT models. The exact model tags are listed on the Ollama Model Hub. A typical pull looks like:
# For 16 GB systems (26B-A4B QAT):
ollama pull gemma4:27b-a4b-q4_k_m
# For 8 GB systems (E4B QAT):
ollama pull gemma4:e4b-q4_k_mOnce pulled, the model runs fully offline. No internet connection needed, no API keys, no data leaving your machine.
With llama.cpp
llama.cpp has full support for the QAT GGUFs. The UD-Q4\K\XL files are available from the UnslothAI repository on Hugging Face:
./llama-cli -m gemma4-26b-a4b-UD-Q4_K_XL.gguf \
--ctx-size 8192 \
--n-gpu-layers 999The --n-gpu-layers 999 flag offloads all layers to Metal GPU on Apple Silicon, giving a notable speed boost versus CPU-only inference.
With vLLM (for shared team servers)
For organizations where multiple users need simultaneous access, vLLM ≥ 0.22.0 supports the QAT checkpoints in HuggingFace format:
vllm serve google/gemma-4-27b-a4b-qat \
--quantization bitsandbytes \
--max-model-len 8192A single server with 16 GB VRAM is sufficient for teams of 5–20 concurrent users on typical office workloads.
Why This Matters for On-Premise AI Deployments
Until now, the local AI landscape forced a difficult tradeoff. Small models (3B–8B) ran fast on consumer hardware but fell short on complex tasks — reasoning, multilingual output, nuanced instruction-following. Large models (70B+) delivered quality but needed 80–128 GB of RAM, pushing organizations toward dedicated server hardware.
Gemma 4 QAT 26B-A4B lands in the gap between those two extremes.
For data-sensitive use cases, the implications are direct:
- GDPR compliance: No user data is sent to a third-party API. Queries stay within your network, eliminating the need for data processing agreements with AI vendors.
- Confidentiality: Internal documents, client correspondence, financial records — none of it touches an external server.
- Cost predictability: No per-token billing. Hardware costs are fixed; usage costs are electricity.
- Resilience: The model runs without an internet connection, suitable for air-gapped environments or locations with unreliable connectivity.
Real-World Scenarios
Legal firm (20 employees): Contract review and clause extraction using Gemma 4 QAT 26B-A4B on an existing Mac Studio M2 Ultra (64 GB). Client data never leaves the building.
Development agency (8 developers): Code review, inline documentation generation, and ticket triage. On a 16 GB MacBook Pro M4, as reported by practitioners running similar MoE setups, the model generates responses at near 4B-equivalent speeds — practical for real-time pair programming assistance.
Manufacturing SMB (40 employees): Automated summarization of supplier correspondence in German, English, and Spanish. Runs on existing workstations, no cloud integration required.
What Changes From the Original Gemma 4?
The original Gemma 4 checkpoints released earlier in 2026 were already capable models. The QAT versions are not new architectures — they are the same Gemma 4 family, but the quantization process was embedded into training rather than applied afterward.
The practical difference: standard 4-bit GGUF conversions of the original Gemma 4 showed the typical quality degradation you'd expect from PTQ. The QAT versions don't. That's the upgrade that matters.
Getting Started at Your Organization
If your team has been waiting for local AI quality to reach a level suitable for production use — contract analysis, customer communication drafting, internal knowledge search — Gemma 4 QAT 26B-A4B is a credible option on hardware that may already be sitting on desks.
A proper pilot takes an afternoon to set up and a week to evaluate meaningfully. We help organizations design that evaluation, identify the right use case, and assess fit with existing workflows before committing to infrastructure changes.
Related reading: Local AI overview · Data sovereignty · kAIra Toolkit · Training and workshops