LocalAI — the open-source project maintained by Enrico Massone (mudler) — has shipped a significant release this month. Per the project's GitHub page, the latest version brings a suite of features that cross the threshold from developer tool to enterprise-grade platform: distributed inference with prefix-cache routing, a real-time WebRTC voice assistant, enterprise security with NATS JWT and TLS/mTLS, and 60 text-to-speech voices across 42 languages.
For SMBs considering local AI infrastructure without cloud dependencies, this is a development worth paying close attention to.
What Is LocalAI?
LocalAI is an open-source, OpenAI-compatible API that runs entirely on your own hardware — CPU, GPU, or Apple Silicon — without any data leaving your environment. It supports text models (Llama 3.3, Qwen 2.5, Gemma 4, DeepSeek-V3), image generation, speech processing, and now real-time voice, all through a single unified API.
The practical advantage: organisations using cloud APIs today can swap the endpoint to a local LocalAI instance with a single configuration change, without modifying application code. This makes migration from cloud to on-premise significantly more tractable.
Key New Features
Distributed Inference and Enterprise Security
The most significant update for production deployments. Per the project documentation:
- Prefix-cache-aware routing: Requests sharing common prefixes benefit from KV-cache reuse across calls — particularly relevant for document Q&A workloads where the system prompt repeats across many queries
- Production-ready request router with auto-sized batches for embedding and reranking workloads
- DS4 layer-split distributed inference: Large models (70B+) can be split across multiple GPUs or machines, so no single node needs to hold the entire model in memory
- NATS JWT auth + TLS/mTLS: Proper authentication and encrypted inter-node communication for multi-machine deployments
- Resumable file uploads for robust model distribution even on unstable network connections
The security layer matters for real enterprise deployments. With NATS JWT, individual services and user groups can be authorised at fine granularity — not a single shared API key for everyone on the network.
Real-Time Voice Assistant with WebRTC
LocalAI now ships a complete real-time voice assistant — fully local, no cloud services required. Per the project page:
- A Go client with a full bidirectional talk-back voice loop, including tool calling
- Streaming of the complete LLM → TTS → transcription pipeline in real time
- Configurable WebRTC ICE candidates for flexible network topologies
In practice: meetings, customer calls, or dictation can be transcribed and responded to on a local server, with no audio leaving the organisation's infrastructure. For GDPR-sensitive sectors — healthcare, legal, HR — this is a meaningful advantage over cloud-based transcription and voice services.
Speech Processing: 60 Voices, 42 Languages
The new CrispASR backend makes LocalAI a comprehensive local voice platform. Per the project documentation:
- 60 Piper TTS voices across 42 languages — German, Spanish, English, French, and many more
- parakeet.cpp ASR with NeMo-compatible segment-level timestamps for precise, timestamped transcripts
- Multilingual streaming via the Nemotron-3.5 model for real-time multi-language transcription
- Dynamic batching for concurrent transcription requests under load
This enables transcription and speech synthesis in the target language without routing any audio through an external service.
What This Means for SMBs
No cloud dependency — for text, voice, image, or object detection. This is the same principle underlying Freshlab's local AI approach: one infrastructure, full control, GDPR compliance without compromise.
OpenAI API compatibility is an underrated advantage. Organisations that already have applications built on the OpenAI API can migrate to local inference by changing a single endpoint URL. This substantially reduces vendor lock-in and eliminates exposure to future pricing changes from US providers.
Team deployment without per-seat licensing: LocalAI operates as an internal service for the entire organisation. With the new request router and NATS auth, different teams and services can access the same local LLM stack in isolation — no per-user fee, no usage-based API invoice.
Cost structure: Local operation incurs no per-token costs. According to community reports, total cost of ownership — hardware, electricity, maintenance — typically becomes competitive with cloud APIs after 12–18 months for five or more users. Actual numbers vary substantially by hardware configuration and usage pattern.
LocalAI vs. Ollama vs. Microsoft Foundry: Which Fits When?
LocalAI is not the only approach to local AI, and each tool has its place:
- Ollama is simpler to set up, ideal for single users and rapid prototyping. Fewer production features.
- Microsoft Foundry Local (available since June 2026) integrates deeply with Windows and Visual Studio Code — well suited to Windows-centric development environments.
- LocalAI is the broadest platform: text, voice, image, video, agents — all through one API, with distributed mode and enterprise auth.
For SMBs covering more than one use case — from customer support chatbot to meeting transcription to document search — LocalAI is the most complete local AI platform currently available.
Use Cases for SMBs
Law firms and consulting: Transcribe client calls, summarise contracts, search internal knowledge bases — all on a local server, no data shared externally, no per-query billing.
Manufacturing and trades: Voice-driven work order documentation, automated post-visit reports, quality inspection logging — with 60 voices in 42 languages built in and ready to use.
Finance and accounting: RAG-powered search over client documents, tax filings, or export data. With LocalAI's real-time voice input, dictation becomes a first-class interface rather than an afterthought.
Getting Started
A structured pilot project is the most effective entry point. LocalAI runs on a Mac Studio M3 Ultra, an existing Linux server with GPU, or even CPU-only. The first step is an audit: which use cases matter most to the organisation, which models fit those cases, and what hardware is available or needs to be procured.
For teams without prior AI experience, our training programmes build the foundation needed to actually deploy — not just install — a local AI stack. Data sovereignty is structurally guaranteed by LocalAI, but it needs to be designed in from the start, not bolted on after the first privacy incident.
To explore what a local LocalAI stack would look like for your organisation, get in touch.