Local LLM Setup: Let Your Hardware Choose the Runtime and Model

Most people ask the wrong first question when setting up a local LLM. As developer Michael Guo writes on X, the common instinct is "Which model format should I use?" when the correct starting point is "What hardware do I have?" Hardware determines your viable runtime options. Runtime options determine the model formats that work. Model formats determine which specific models can fit in your available memory.

Getting this chain backwards — picking a popular model, then discovering your runtime doesn't support it, then finding it won't fit in your VRAM — is the most common friction point in local LLM setup. This guide maps the chain clearly so you can make one informed decision pass instead of cycling through trial-and-error.

The Decision Chain: Hardware → Runtime → Format → Model

Memory is the hard constraint in local LLM inference. The model must be fully loaded — or substantially offloaded — into the memory pool that your inference runtime can access. Mismatches either prevent loading entirely or force a fallback to slow CPU paths.

Answer these four questions in sequence:

1. What hardware do you have? (Apple Silicon, NVIDIA GPU, CPU-only)
2. Which runtimes perform best on that hardware? (Ollama with MLX, vLLM, llama.cpp)
3. What model format does that runtime use? (MLX weights, GGUF, safetensors)
4. Which models fit that format within your memory budget?

Working through these in order rather than starting with a model popularity list prevents almost all first-time setup failures.

Tier 1: Apple Silicon Macs

Apple Silicon uses a Unified Memory Architecture (UMA) where RAM and GPU memory share the same physical pool. A Mac Studio M3 Ultra with 192 GB of memory can address that entire pool for model inference — no system-GPU split, no PCIe bandwidth overhead between CPU and graphics memory.

Recommended runtime: Ollama (which gained native MLX backend support in early 2026) or MLX-LM directly for lower-level control.

Model format: MLX quantised weights for native MLX pipelines, or GGUF via Ollama's standard interface.

Recommended models by available RAM:

Practitioners report inference rates of 20–40 tokens/second for quantised 70B models on Mac Studio M3 Ultra — sufficient for team-facing production workloads. Smaller models on a Mac Mini M4 (16–32 GB) reportedly reach 60–100+ tokens/second in published community benchmarks, making them fast enough for interactive use across multiple concurrent users.

For most SMB use cases — document analysis, customer-email drafting, internal knowledge search, code review assistance — a Mac Mini M4 (32 GB) or Mac Studio M3 Max (64 GB) is the practical sweet spot. Larger memory configurations make sense when running multiple simultaneous users, very long context windows, or models in the 70B+ class at higher quality levels.

Single-user vs. team setups on Apple Silicon

For a single user, Ollama alone is sufficient. For teams, Open WebUI adds a browser-based frontend with session management, per-user model switching, and a familiar chat interface — all without any cloud dependency. The Ollama + Open WebUI combination is the most commonly reported Apple Silicon team stack in practitioner discussions from mid-2026. More about setting up team access is covered in our training resources.

Tier 2: NVIDIA GPUs

NVIDIA cards expose dedicated VRAM that is separate from system RAM. If a model doesn't fit entirely in VRAM, the remainder spills over PCIe to CPU RAM — functional, but significantly slower.

Recommended runtimes:

• Ollama — simple setup, handles one request at a time cleanly, best for individual users and small teams, uses GGUF format
• vLLM — production-grade concurrent request handling via continuous batching, best for teams with consistent API demand, uses safetensors format
• llama.cpp — maximum configuration control, GGUF format, supports partial layer offloading between VRAM and CPU RAM

Recommended models by available VRAM:

A used RTX 3090 (24 GB VRAM) was available for approximately €400–600 in mid-2026, as reported by practitioners in local LLM communities — making it the most cost-efficient path to 32B-class models on NVIDIA hardware. For teams running a shared internal LLM API, vLLM is worth the additional setup: its continuous batching engine handles 10 simultaneous requests significantly faster than serialising them, with published benchmarks showing 2–4× throughput gains over single-request-at-a-time setups.

Tier 3: CPU-Only and Low-Power Hardware

Pure-CPU inference is slower but still functional for exploration, low-frequency batch tasks, or hardware without a dedicated GPU.

Recommended runtime: llama.cpp or Ollama's CPU backend.

Model format: GGUF, heavily quantised (Q2–Q4).

Practical model choices:

• Phi-4 Mini (3B): Microsoft's compact reasoning model, runs on any modern laptop
• Gemma3 1B: Google's smallest release, very fast on CPU for simple tasks
• Qwen2.5 3B: Strong multilingual capability including German and Spanish at minimal memory requirements

Reported inference rates on CPU-only hardware range from 2–8 tokens/second depending on the machine and quantisation level. Adequate for asynchronous batch processing; challenging for interactive multi-user chat.

Data Sovereignty as a Business Factor

For EU-based businesses, the hardware decision is also a compliance decision. Running all inference locally means no inputs, no outputs, and no processing metadata leave your own infrastructure. There are no API calls to US-based cloud providers, no per-token exposure of business data, and no dependency on the pricing or usage policy changes of external vendors.

This is the core value of local AI deployment for European SMBs operating under GDPR. Our overview of what data sovereignty means in practice is at /data-sovereignty.html. For a broader introduction to local AI for business, see /local-ai.html.

Hardware Funding in Europe

European businesses investing in AI infrastructure may be able to access regional support programs. In Spain, Kit Digital covers AI tooling and implementation services for qualifying SMEs. In Germany, BAFA digital consulting grants and KfW digitalisation loans apply to AI infrastructure investments. Based on our reading of current program guidelines, hardware procurement and implementation consulting are potentially eligible items; exact eligibility depends on company type and current program availability.

Start with the Hardware You Have

The right local LLM decision does not begin with a ranked list of popular models. It begins with an honest inventory of available memory and hardware generation. Apple Silicon, NVIDIA GPU, or CPU-only — every path leads to a productive outcome when runtime and model format are matched to the hardware.

To map your specific setup — existing hardware, team size, use cases — to a working local LLM architecture, the Freshlab pilot project process covers exactly this: from current infrastructure to a production-ready deployment.