You invested in a Mac Studio M3 Ultra for local AI, and the token throughput feels lower than benchmark tables promised. The gap is real and it is a software problem, not a hardware one. An AI researcher put it plainly on X: "Performance isn't about the hardware, it's about three software layers stacked on top of it." (source) Testing MLX, Ollama, llama.cpp, and the new vllm-mlx on identical hardware, the performance differences were significant, and entirely down to software choices.
For European businesses evaluating on-premise AI infrastructure, this matters directly: the inference stack you deploy can double or halve the effective throughput of the same machine.
The Three Software Layers
To understand why two identical Mac Studios perform very differently, track three layers:
1. Quantization layer, how much are the model weights compressed?<br> Q4KM is fast and memory-efficient. Q8_0 preserves more output quality at nearly double the memory cost. F16 uses full precision but demands the most unified memory. This choice alone can shift throughput by a factor of 1.5 to 2 before the inference engine even enters the picture.
2. Compute layer, what framework drives the actual inference?<br> MLX uses Apple's native Metal framework directly and is tuned for Apple Silicon's unified memory architecture. llama.cpp uses Metal offloading but is a generic cross-platform tool, not Apple-specific. Whether Metal shaders are fully or only partially utilised determines whether inference runs GPU-accelerated or falls back to the CPU, a difference of several times in speed.
3. Serving layer, how is the model exposed to client applications?<br> Ollama wraps llama.cpp with a REST server: convenient, but with measurable overhead. Calling mlx-lm or llama.cpp directly removes that abstraction. Under high request volume, the difference compounds quickly.
The interplay of these three layers, not the chip specification alone, determines the token-per-second rate that users actually experience.
MLX, Ollama, llama.cpp, and vllm-mlx Compared
MLX-LM: Apple Silicon Native
MLX is Apple's own machine learning framework, built specifically for the unified memory architecture of Apple Silicon. mlx-lm provides the language model interface on top of it.
Community benchmarks report that a Mac Studio M3 Ultra 192 GB running Llama 3.3 70B at Q4KM quantization reaches between 45 and 65 tokens per second with MLX-LM, the highest figures reported for this hardware class. MLX calls Metal shaders directly, eliminating the middleware overhead present in other stacks.
The trade-off: mlx-lm is a command-line library and Python package, not a turnkey server. It suits technical teams building Python-based backends or developers embedding models directly into pipelines.
Ollama: Convenience at a Cost
Ollama is the most popular local LLM runner, and for good reason: a single command downloads and serves any supported model, and its REST API is directly compatible with Open WebUI, LangChain, LlamaIndex, and most integration frameworks.
Under the hood, Ollama uses llama.cpp as the inference backend. That convenience wrapper costs roughly 20-30 % of throughput compared to calling llama.cpp or mlx-lm directly, according to community reports. For a single developer working alone, this is usually acceptable. For a team with concurrent requests, the overhead compounds.
For most European SMBs deploying an internal chatbot or a document Q&A system, Ollama remains the practical default, far easier to operate than a raw mlx-lm deployment. See the kAIra Tools for how we integrate it into a full local AI stack.
llama.cpp Direct: Control Without a Wrapper
llama.cpp is the C++ reference implementation for GGUF-format models. Calling it directly, without the Ollama serving layer, typically recovers 10-20 % throughput. On Apple Silicon, llama.cpp uses its ggml-metal backend; when Metal offloading is correctly configured, performance approaches MLX-LM levels on many models.
The portability advantage is meaningful for organisations with mixed environments: GGUF models run unchanged on Mac, Linux, and Windows. If the Mac Studio is the primary inference server and Linux machines serve as fallback, the same weights work across the entire fleet.
vllm-mlx: Production Throughput for Concurrent Teams
vllm-mlx is the newest entrant in this comparison, and the most relevant for teams with multiple simultaneous users. vLLM is a production-grade inference server known for its PagedAttention mechanism, which efficiently shares GPU memory across parallel requests. The MLX port brings that batching advantage to Apple Silicon.
For a single user in a chat session, vllm-mlx shows similar per-request latency to mlx-lm. With five to ten team members querying the same Mac Studio simultaneously, early community reports suggest vllm-mlx can increase total system throughput by 30-50 % over Ollama, because requests are batched intelligently rather than queued sequentially.
Quantization: The Overlooked Lever
Alongside inference engine choice, the quantization format is the most underappreciated performance lever:
- Q4KM: Best efficiency for most use cases. Llama 3.3 70B at Q4KM occupies approximately 40 GB of unified memory, leaving substantial headroom on a 192 GB M3 Ultra. For document analysis, summarisation, and chat, quality differences versus Q8_0 are minimal in practice.
- Q8_0: Better quality, nearly double the memory footprint. Recommended for code generation and reasoning tasks where numerical precision matters.
- F16: Full precision; rarely necessary for SMB deployments.
An M3 Ultra 192 GB can run models up to approximately 405 B parameters at Q4KM, making it the most capable desktop machine available for local AI without dedicated server hardware.
Which Stack for Which Scenario?
| Scenario | Recommended stack |
|---|---|
| Developer, Python integration | mlx-lm (direct) |
| Team chatbot with a UI, 5-20 users | Ollama + Open WebUI |
| Concurrent requests, production grade | vllm-mlx |
| GGUF portability across Mac + Linux | llama.cpp direct |
This table is a starting point. The optimal stack for your specific environment depends on network topology, model size, user headcount, and which applications the model needs to connect to.
GDPR and Data Sovereignty
Regardless of which stack you choose, every inference call runs entirely on local hardware. No token leaves the corporate network; no third-country data transfer under GDPR Art. 44 takes place. That fundamental advantage of data sovereignty holds whether the serving layer is MLX-LM, Ollama, or vllm-mlx.
Based on our reading of the EU AI Act, operators running open-weight models locally are in many deployment scenarios exempt from the most burdensome GPAI provider obligations. The exact classification depends on your specific use case, this is informational commentary, not legal advice.
Want to know which local AI stack fits your team's workflow and headcount? As part of a pilot project, we assess your requirements and recommend the optimal inference stack, concrete, priced, and GDPR-compliant.