Run GPT-OSS 120B Locally: Fable-5 Distilled GGUF Guide

On 22 June 2026, AutoTrust AI posted on X: "Our first community model is live." Behind that announcement: a community GGUF of OpenAI's GPT-OSS 120B, LoRA-tuned with knowledge distilled from Claude Fable-5 and converted to the GGUF format that runs on llama.cpp, LM Studio, and Ollama. The release logged 243 downloads in its first 24 hours, a clear signal that the local LLM community had been waiting for exactly this.

For businesses running local AI, the implications are concrete: a 120-billion-parameter model is now available in GGUF form, deployable entirely on-premise on hardware that already exists or is procurable in a European SME context.

What Is GPT-OSS 120B, and What Does "Fable-5 Distilled" Mean?

GPT-OSS 120B is OpenAI's openly licensed language model with a Mixture-of-Experts (MoE) architecture: 36 layers, 128 experts in total, 4 active per token generation step. This design means only a small fraction of parameters is activated at each inference step, substantially reducing memory and compute demands compared to a dense 120B model.

"Fable-5 Distilled" refers to the LoRA fine-tuning process applied by AutoTrust AI: supervised training on code session data using knowledge distilled from Anthropic's Claude Fable-5. The training focused on coding scenarios; RLHF was not applied, which, as documented on Hugging Face, means a somewhat lighter safety alignment than commercial instruction-tuned models typically carry.

The GGUF format enables execution via llama.cpp, Ollama, LM Studio, GPT4All, and MLX-LM, cross-platform on macOS, Windows, and Linux, with backends for CPU, CUDA, Metal, Vulkan, and ROCm.

Hardware Requirements: What Do You Actually Need?

The model in MXFP4 quantisation sits at approximately 61 GB of model data. Add compute buffers (~2.7 GB) and KV cache (from ~0.3 GB at 8,192 tokens). According to the llama.cpp community guide, total memory at an 8,192-token context is roughly 64 GB, rising to ~68.5 GB at 131,072 tokens.

Practical hardware options for European SMEs:

• Mac Studio M3 Ultra (128-192 GB unified memory): Recommended. Community reports on Hacker News put a maxed-out Mac Studio at 60-100 tokens/sec for short contexts. Speed drops as context grows due to memory pressure.
• Mac Studio M3 Max (96 GB): Works with constraints. Context windows above 32k tokens may create memory pressure.
• NVIDIA DGX Spark GB10 (128 GB): Well suited. GPU-optimised inference typically outperforms Apple Silicon at equivalent memory capacity.
• Desktop with 16-24 GB GPU + large system RAM: Possible via CPU offloading with --n-cpu-moe N in llama.cpp. Generation slows to roughly 5-15 tokens/sec, usable for asynchronous tasks like document analysis.

Benchmark reference points from community measurements in the llama.cpp discussion: NVIDIA RTX Pro 6000 (96 GB VRAM) reaches approximately 196 tokens/sec generation; M2 Ultra (192 GB, 76 GPU cores) reaches approximately 80 tokens/sec.

Setup: llama.cpp and Ollama

For systems with 64 GB+ VRAM or unified memory, run the model without CPU offloading:

llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048

For systems with 16 GB VRAM and sufficient system RAM:

llama-server -hf ggml-org/gpt-oss-120b-GGUF \
  --ctx-size 32768 --jinja -ub 4096 -b 4096 --n-cpu-moe 32

The --n-cpu-moe flag tells llama.cpp to offload MoE layers to CPU, the key optimisation for memory-constrained setups. Adding -fa (Flash Attention) improves throughput further.

For Ollama and LM Studio, the community GGUF is available directly from Hugging Face.

Important, prompt format: GPT-OSS 120B expects the Harmony format using the <|channel|> protocol. Standard ChatML or Alpaca-style prompts produce substantially degraded outputs, as reported by practitioners in the llama.cpp community thread. Plan for prompt adaptation before integrating the model into an existing application.

Use Cases and What to Be Aware Of

AutoTrust AI's Fable-5 Distilled variant was trained primarily on code session data and performs well at:

• Automated code review and technical documentation: At 120B scale, the model demonstrates noticeably more nuanced understanding of architectural decisions than smaller models.
• Internal development assistants with full data sovereignty: No request logs at external providers, no API dependency, no token limits.
• Analysis and summarisation of technical or legal documents: The 128k-token context window allows processing entire codebases or large contract bundles in a single call.

For general reasoning and text generation, the model also performs well, though the coding focus of the training dataset is worth keeping in mind when evaluating it against non-technical workloads.

Data Sovereignty: The Structural Argument for On-Premise

The fundamental argument for local deployment holds here as it does for smaller models: all inference runs on your own hardware. No request leaves your network; no company data reaches an external service. Based on our reading of the GDPR, this represents a meaningful risk reduction under Article 32, particularly relevant for law firms, healthcare providers, accountancies, and industrial companies processing confidential documents.

Cloud APIs require that every request pass through external servers. With a locally hosted model, no prompt leaves your network. That is a structural difference that shows up directly in GDPR risk documentation.

More on how local AI infrastructure aligns privacy goals with operational efficiency on our data sovereignty page.

Who Should Consider the Jump to GPT-OSS 120B?

GPT-OSS 120B in the Fable-5 Distilled variant is not an entry-level model. If you are already running Llama 4, Qwen3, or Gemma 4 on a Mac Studio M3 with Ollama, this model offers a quality step up, with proportionally higher hardware requirements.

The move makes practical sense for organisations that:

• Already operate Mac Studio M3 Ultra or equivalent hardware
• Need coding assistance or document analysis at maximum model quality without cloud dependency
• Are prepared to integrate the Harmony prompt format into their application layer

For SMEs at the start of their local AI journey, smaller models, Llama 3.3 70B or Qwen3 32B, remain a more pragmatic entry point that delivers strong results on more modest hardware.

If you are planning the step to 120B-class local models or want an independent review of your current local AI stack, get in touch.