Running a serious local LLM does not require enterprise hardware. A used NVIDIA RTX 3090 with 24 GB of VRAM has become the community's go-to recommendation for SMBs entering local AI in 2026, capable enough for 32B-class models like Qwen3:32B and Gemma4:26B, affordable enough to stay within a realistic IT budget.
A widely-shared local LLM hardware guide published on X this week listed the RTX 3090 as the first hardware recommendation for teams building their first local AI server, with a complete build estimated at $1,000-1,500, or roughly €1,300-1,750 in European markets. That estimate has been corroborated by multiple community sources and aligns with what practitioners are currently seeing in the used GPU market.
This article explains why 24 GB VRAM is the meaningful threshold, which models fit, what performance to expect, and how to think about the total cost of ownership compared to cloud APIs.
Why 24 GB VRAM Is the Turning Point
Language model inference is fastest when the entire model fits in GPU VRAM. When VRAM runs out, the inference engine spills model weights into system RAM, and throughput collapses, typically by a factor of 5-15.
With consumer GPUs, the VRAM tiers roughly correspond to these model classes (Q4 quantization):
- 8 GB (RTX 3060 12 GB, RTX 4060): 7B, 12B models
- 16 GB (RTX 3080 Ti, RTX 4070): 13B, 20B models
- 24 GB (RTX 3090, RTX 4090): 27B, 34B models
- 48 GB (RTX 6000 Ada, dual 3090): 70B models
The jump from 16 GB to 24 GB is the most impactful: it opens the 27B, 32B class, which in practice delivers measurably better reasoning, instruction-following, and multilingual output than 13B models, without requiring a professional-grade GPU costing €3,000+.
Which Models Run on a Single RTX 3090
Based on community measurements published in 2026:
| Model | VRAM at Q4 | Notes |
|---|---|---|
| Qwen3:32B | ~20 GB | Strong reasoning, multilingual DE/EN/ES |
| Qwen3.6-27B | ~17 GB | Vision-language: understands images too |
| Gemma4:26B | ~18 GB | Apache 2.0 licence, commercially free |
| Gemma4:12B | ~8 GB | Multimodal (text + images), 256K context |
| Llama3.3:70B | ~43 GB | Does not fit on a single 24 GB card |
Qwen3.6-27B stands out as the current community favourite for business use: it is the first dense open-weight vision-language model that runs comfortably on consumer hardware, meaning it can process both written text and document screenshots in the same inference call.
Performance: What to Expect
Community measurements for Qwen3.6-27B Q4\K\M on a single RTX 3090 (sources: runaihome.com, explore.n1n.ai):
- 35-40 tok/s with Ollama (standard setup, no tuning)
- 78-85 tok/s with vLLM and speculative decoding using Qwen3-0.6B as a draft model
For most SMB workflows, the Ollama baseline is sufficient: drafting a 300-word email or summarising a two-page document takes 8-12 seconds at 35 tok/s, fast enough for interactive use. The vLLM configuration becomes worthwhile when three or more team members access the model concurrently.
Complete Build Specification for 2026
Based on community hardware guides, a practical local AI server built around the RTX 3090 looks like this:
GPU: NVIDIA RTX 3090 (used), approx. €700-900 24 GB GDDR6X memory, 936 GB/s bandwidth. Used units with clean histories are available on eBay and specialist refurb platforms. Check cooling performance and verify GPU load history before purchase.
CPU + Motherboard, approx. €350-550 Any modern desktop CPU with PCIe 4.0 × 16 works: AMD Ryzen 7 7700X or Intel Core i7-13xxx. LLM inference is GPU-bound; the CPU is not a bottleneck.
RAM: 64 GB DDR5, approx. €150-200 Model loading temporarily touches system memory. 32 GB is technically sufficient for single Q4 27B models; 64 GB adds headroom for multiple models and parallel workloads.
Storage: 2 TB NVMe SSD, approx. €100 A single 32B GGUF file is 20-22 GB. 2 TB lets you keep 5-10 models ready without managing storage.
OS: Ubuntu 24.04 LTS, free NVIDIA CUDA, Ollama, and vLLM are best supported and documented on Linux.
Total: approx. €1,300-1,750, one-time. No subscriptions, no API keys, no data leaving your premises.
Software Stack: Ollama for the First Hour, vLLM for Production
Starting with Ollama (under one minute to a running model):
curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen2.5:32bOllama auto-detects the NVIDIA GPU and serves an OpenAI-compatible REST API at localhost:11434, compatible with Open WebUI, VS Code extensions, and most internal tooling. For single-user development and initial testing, Ollama is the right choice.
Scaling with vLLM (for concurrent team access):
When more than two or three team members access the model simultaneously, vLLM's continuous batching and PagedAttention keep throughput consistent under load. For a step-by-step guide to running a shared local AI server, see our local AI deployment overview.
The GDPR Case for Local Inference
Every request sent to a cloud API leaves your organisation. Even with contractual data processing agreements in place, routing employee queries, client documents, or strategic text through a third-party API is not an acceptable arrangement in many regulated contexts.
A locally hosted LLM eliminates the issue at the architectural level:
- No third-country transfer (GDPR Art. 44 ff.): data stays within your EU legal perimeter.
- No data processing agreement with a cloud provider required.
- Full control over model version, prompt logs, response data, and update timing.
This matters for law firms, healthcare providers, accountancies, HR departments, and any organisation with obligations under GDPR Art. 32 or sector-specific data handling rules.
For a deeper look at the regulatory framing, see data sovereignty with local AI.
Cloud API vs. Own Hardware: The Economics
A team of five making regular use of a GPT-4-class cloud API, say, 50-100 requests per day at 2,000 tokens each, can reasonably expect to spend €200-400 per month at current list prices. Over three years, that is up to €14,400.
A local RTX 3090 workstation costs approximately €1,500 upfront, with electricity adding roughly €3-5 per month during active use. Based on our reading of typical SMB usage patterns, break-even typically falls between six and fourteen months. Actual results depend heavily on usage intensity and workload type, these figures are illustrative, not guaranteed.
For SMBs eligible for EU digitisation support schemes, hardware investments in operational AI infrastructure may qualify under applicable national programmes. The specifics vary by country and scheme; we recommend verifying eligibility with an accredited consultant.
When the RTX 3090 Is Not the Right Choice
- 70B-class models without quality compromise: Dual RTX 3090 in NVLink or a 48 GB professional card (NVIDIA RTX 6000 Ada, ~€3,500-4,000) are the next step.
- High concurrency (20+ simultaneous users): Consider a dedicated GPU server or a distributed inference setup.
- No internal Linux expertise: Without someone who can manage CUDA configuration and ongoing maintenance, the real cost of a local server includes IT time, plan for that or bring in outside support.
If you want to evaluate which local AI hardware configuration makes sense for your team, including a data sovereignty audit and a concrete use case analysis, get in touch.