The local LLM conversation has quietly shifted. Teams that set up Ollama on a Mac Studio last year now ask a different question: not which model to run, but which runtime to use. The model is the same. The hardware is the same. The software layer in between is where the performance gap opens up.
Practitioners on X have been comparing inference backends on identical Apple Silicon hardware over the past weeks. A post by Alex Jones highlighted "two open-source MLX inference servers worth knowing about if you run LLMs on Mac": mlx-serve and MTPLX. Both replace Ollama's llama.cpp path with direct MLX acceleration — and the difference in measured throughput is significant.
What mlx-serve Is
mlx-serve is an inference server written in Zig, a systems language comparable to C. There is no Python runtime, no interpreter overhead. The self-contained binary is approximately 7 MB. It ships with a CLI server and MLX Core, a native macOS menu-bar application with chat, agent mode, and tool calling built in.
The architectural distinction from Ollama matters here. Ollama processes models through llama.cpp, which uses a generalised compute path that runs on macOS but is not specifically tuned for Apple's Unified Memory Architecture. mlx-serve talks directly to Apple's MLX framework and the Metal GPU kernels underneath it. That is where the performance advantage comes from.
GGUF vs. Native MLX Weights
GGUF is a quantisation format developed for llama.cpp. Its strength is portability — it works on Linux, Windows, and macOS alike. On Apple Silicon, however, the compute path goes through llama.cpp's generalised routines rather than MLX's chip-specific optimisations.
MLX-native weights are adapted for Apple's framework directly. Arithmetic runs on the same compute units, addresses the same memory subsystem, and benefits from optimisations Apple has built specifically for the M-chip generation. The practical result: more tokens per second from the same hardware, at no additional cost.
Performance: What the Benchmarks Show
According to the project's benchmarks on an Apple M4 Max, using identical 4-bit weights, mlx-serve achieves:
- Qwen 3.6 27B: approximately 29 tok/s baseline, up to 58.4 tok/s with Multi-Token Prediction (MTP) enabled
- Gemma 4 E4B: approximately 118 tok/s
- +48% geomean across 18 workload combinations compared to LM Studio 0.4.15
- Prefill acceleration: 1.8–2.5× faster than LM Studio on smaller Gemma models
The companion tool MTPLX focuses specifically on MTP-based speculative decoding. Community-reported measurements describe approximately 63 tok/s on Qwen 3.6 27B on an M5 Max, and a roughly 2.6× speedup on MacBook Pro hardware compared to the standard MLX baseline.
Multi-Token Prediction Explained
Standard speculative decoding runs a small, fast draft model ahead of the main model, generating several candidate tokens. The large model then verifies all candidates in a single batched pass — when guesses are correct, you receive several tokens for the compute cost of one.
mlx-serve and MTPLX apply a specific optimisation: they use the model's own built-in MTP heads as the drafter, rather than loading a separate draft model. Modern Qwen 3.5/3.6 and Llama 3.x checkpoints already include these heads. The result is zero additional RAM consumption and no second model to manage. According to project benchmarks, this delivers +15–26% on code tasks and +44–61% on agentic loops via Prompt Lookup Decoding.
API Compatibility: A True Drop-In for Ollama
Teams already using Ollama do not need to change their tooling. mlx-serve implements the Ollama wire protocol in full:
/api/chatand/api/generate— identical to Ollama/v1/chat/completions— OpenAI-compatible/v1/messages— Anthropic Messages API
Applications pointing at http://localhost:11434 (Ollama's default) work immediately after switching to http://localhost:11234. Open WebUI, Raycast, Obsidian, and similar tools require no further configuration changes.
Installation
brew install mlx-serve
mlx-serve run qwen3.6:27bModel download and server startup happen automatically. For teams running a Mac Mini or Mac Studio as a shared inference server, mlx-serve can be registered as a launchd service so it starts on boot without manual intervention.
What Else mlx-serve Handles
mlx-serve is not limited to text generation. According to the project's documentation, it also supports:
- Image generation and editing: FLUX.2, Krea-2
- Video generation: LTX-Video 2.3
- Speech synthesis with voice cloning: Qwen3-TTS
- KV-cache quantisation (4-bit/8-bit): reduces memory footprint by a factor of 2–4
- Prefix caching: system prompts and multi-turn conversations are cached, making repeat requests significantly faster
- Prompt Lookup Decoding: reported +44–61% on agentic loops
The last two points are directly relevant for SMBs running multi-agent workflows or repetitive document-processing pipelines on local infrastructure.
Why This Matters for SMBs
The typical Freshlab entry-level setup is a Mac Mini M4 or Mac Studio M3 Ultra. With Ollama and llama.cpp, Qwen 3.6 27B on an M4 Max reportedly runs at approximately 20–25 tok/s based on community data. With mlx-serve and MTP enabled, the same hardware reaches up to 58 tok/s — more than double, at zero additional hardware cost.
For a five-person team that daily summarises internal documents, drafts proposals, or handles customer queries with a local LLM, response time drops from 8–10 seconds to 3–4 seconds per request. Across ten daily requests per person, that recovers more than an hour of waiting time per week — on infrastructure that keeps every token on-premise.
As we outline on our local AI page and in our writing on data sovereignty, the primary advantage of running LLMs locally is not just token cost savings. It is the complete exclusion of external data transmission. mlx-serve changes nothing about that guarantee — it simply makes the same protection faster.
Does mlx-serve Fully Replace Ollama?
Not universally, yet. Ollama has a broader model ecosystem, more mature model management tooling, and a larger community. Teams primarily using GGUF models or niche checkpoints will remain well-served by Ollama. mlx-serve includes an embedded llama.cpp fallback for GGUF — but the performance advantage is clearest with native MLX weights.
For SMBs running Qwen 3.x, Gemma 4, or Llama 3.x on Apple Silicon, mlx-serve is today a measurably better-performing choice.
If you want to find out whether your current setup would benefit from a runtime switch — or which stack fits your team's specific workload best — we are happy to help: start a pilot project.