A Full Production AI System — No Licensing Bill
Developers and practitioners across the AI community are increasingly reporting the same discovery: a complete, production-ready AI stack is deployable in 2026 at zero licensing cost. The building blocks are mature, well-documented, and run entirely on your own hardware — no data upload to the cloud, no monthly API invoices, no lock-in to a US provider.
What sounded like a niche experiment two years ago has become a credible alternative for small and medium-sized businesses. According to reported figures from community trackers, Ollama crossed 52 million monthly downloads in Q1 2026. Developer surveys suggest that roughly 42 percent of developers now run at least part of their LLM workloads entirely on local machines.
The question is no longer whether this approach works — it is how quickly your organisation can adopt it.
The Stack
The most widely discussed configuration combines:
- LLM server: Ollama (open source, MIT license) running on
localhost:11434 - Models: Google Gemma 4, Meta Llama 3.3, or Mistral Small 4 — all under permissive licenses
- Orchestration: LangGraph or CrewAI for multi-step agent workflows
- RAG layer: LlamaIndex as the framework, ChromaDB or Qdrant as the local vector store
- Embeddings: nomic-embed-text (274 MB, 8 192-token chunks) — also local
Every component is open source. Ollama handles model management with a single command (ollama pull gemma4); LangGraph and CrewAI orchestrate tasks, tool calls, and decision loops; ChromaDB or Qdrant persist embedding vectors on your own storage without any external connection.
The result: a complete AI system that incurs no ongoing licensing or usage costs after the initial hardware investment.
Choosing a Model: Gemma 4, Llama 3.3, or Mistral Small 4?
Google Gemma 4 (Apache 2.0 license) is, according to Google DeepMind's official technical blog, the first open-weight model family where agentic capability — tool calling, multi-step planning, structured outputs — is a first-class design goal rather than an afterthought. The 12B and 27B variants run on a standard workstation or a Mac Studio and deliver strong results for document analysis, structured data extraction, and FAQ handling.
Meta Llama 3.3 offers strong all-around performance and is a natural fit for teams with powerful hardware — a Mac Studio M4 Ultra (128 GB unified memory) or a Linux workstation with an NVIDIA RTX 4090 (24 GB VRAM) — who want the 70B variant. Smaller variants (8B, 32B) run on more modest machines.
Mistral Small 4 stands out for its compact footprint and fast inference on consumer-grade hardware. For an SMB entry point, the community consistently recommends Gemma 4 (27B) via Ollama: capable enough for real tasks, GDPR-safe, and free.
Inference Speed
According to community measurements, Gemma 4 27B on a Mac Studio M3 Ultra typically delivers 20–40 tokens per second, depending on quantisation level (Q4 to Q8). On an NVIDIA RTX 4090 under Linux, reported figures for comparable models range from 30 to 60 tokens per second. These speeds make local inference practical for most interactive use cases.
Orchestration with LangGraph and CrewAI
Once the model is running via Ollama, LangGraph enables complex, stateful workflows: the agent reads documents, fills in tables, makes decisions, and calls tools — all offline. LangGraph describes agent workflows as directed graphs, which simplifies debugging and extension.
CrewAI is particularly suited to multi-agent systems where distinct roles (research, analysis, summarisation, quality review) collaborate toward a shared goal. Both frameworks offer official Python integrations and connect to Ollama via the OpenAI-compatible API — a single base_url parameter is all that changes.
Typical SMB use cases:
- Answering internal queries (HR FAQ, IT helpdesk, company policies)
- Processing documents (invoices, contracts, supplier correspondence)
- Extracting data from unstructured reports or forms
- Summarising meeting notes or customer requests
According to practitioner reports, setup time with an in-house Python developer is two to four hours for a working prototype.
Local RAG with LlamaIndex and ChromaDB
Retrieval-Augmented Generation (RAG) lets the model query company-owned documents rather than relying solely on training knowledge. LlamaIndex indexes PDF, Word, and HTML files; ChromaDB or Qdrant store the resulting vectors locally on your own server.
The result: an AI assistant that knows your internal manuals, product catalogues, technical documentation, or customer correspondence — without a single character of that data leaving your network. For a software company this could mean the assistant knows all internal coding guidelines. For a law firm: all non-personal template texts and precedents.
GDPR Compliance as a Competitive Edge
For European SMBs, this is the decisive point. Because all processing happens on-premise, there is no need for a Data Processing Agreement with a US cloud provider, and no transfer of personal data to a third country under GDPR.
Based on our reading of the EU AI Act and current GDPR guidance, a fully local stack significantly reduces documentation obligations and simplifies your compliance posture. Customers and business partners can easily verify — on request — where their data is processed. That is a concrete argument in B2B sales conversations and public tenders.
The EU AI Act imposes substantial transparency and documentation requirements for certain high-risk applications. A local stack that runs exclusively on your own servers and transfers no personal data to third parties materially reduces your regulatory risk profile.
Hardware and Cost
Entry-level options:
| Hardware | Memory | Best for |
|---|---|---|
| Mac Mini M4 Pro | 48 GB unified | Gemma 4 27B (Q4/Q8) |
| Mac Studio M3 Ultra | 96–192 GB | Llama 3.3 70B |
| Linux workstation + RTX 4090 | 24 GB VRAM | Gemma 4 27B, Mistral Small 4 |
Prices from approximately €1,600 (Mac Mini M4 Pro) to around €6,000 (Mac Studio M3 Ultra) — a one-time investment with no ongoing licensing costs.
Running costs: electricity (typically 30–150 W under load) plus occasional maintenance. No API subscription, no per-token charges.
Break-even: Compared to a typical SMB cloud API subscription — reported at €50–€500 per month in community cost analyses — break-even falls at six to twelve months. After that, the stack runs for free.
Open-Weight Ecosystem Maturity
A telling indicator of how far the ecosystem has come: Gemma 4 launched with day-one support from Ollama, vLLM, llama.cpp, MLX, LM Studio, Hugging Face Transformers, SGLang, and several other inference runtimes. There is no longer any meaningful delay between a model release and its availability for local deployment.
This maturity matters for SMBs. Choosing a local-first architecture today does not mean falling behind — the open-weight community is shipping at the same pace as proprietary cloud providers, and often faster on hardware optimisation for Apple Silicon and NVIDIA consumer GPUs.
Where to Go Next
Freshlab has deployed comparable local AI stacks in pilot projects with European SMBs — covering hardware selection, model setup, RAG configuration, and staff onboarding as a complete package. Our Kaira Toolkit provides a production-ready foundation for exactly this stack.
For background on the legal and technical case for on-premise AI, visit our data sovereignty page or explore our local AI overview.
If you want to know whether a local AI stack fits your business, get in touch — we will walk you through what is realistic in a free initial conversation.