The documents a law firm or medical practice most wants to search with AI are exactly the ones that may never leave the premises. Client files, contracts, expert opinions, patient records. This is where the usual cloud assistants fail, and where local AI gets interesting.
RAG stands for Retrieval Augmented Generation. In plain terms, the language model answers questions not from its training knowledge but from your own documents, with a citation. You ask "What is the notice period in the framework agreement with supplier X?", and the system returns the answer plus the passage it came from.
The pivotal question is where that search happens. With the well-known cloud services, your documents travel to third-party servers for processing. With a local RAG installation, every file stays on your own hardware.
What a local RAG stack actually is
Three components are in play, and in a local install all of them run on-premise:
- Embedding model: turns your documents into searchable vectors. Compact open-weight models such as nomic-embed-text or mxbai-embed run comfortably on a Mac Studio or a mid-range workstation.
- Vector database: stores those vectors and retrieves the relevant passages for a given question. ChromaDB and Qdrant are common self-hosted choices.
- Language model: composes the answer from the retrieved passages. Through Ollama you run local models here, so not a single token leaves your network.
None of these components needs an internet connection at runtime. That is the difference that matters in regulated professions. We made the broader argument that the software layer, not the raw hardware, decides day-to-day usefulness when we looked at structured data, see querying your ERP with a local LLM. With unstructured documents the same holds, and the privacy lever is even larger.
Why cloud RAG is fraught for regulated firms
Lawyers, doctors, tax advisers and other professionals bound by confidentiality operate under strict secrecy duties. In Germany, breaching protected secrets is a criminal matter under section 203 of the Criminal Code. Based on our reading, that is precisely why external IT providers were long a legal grey zone for these professions.
The 2017 Professional Secrets Protection Act eased this. Section 43e of the Federal Lawyers' Act now expressly permits engaging external service providers, but under conditions: the provider must be carefully selected, bound to confidentiality in text form, and the need-to-know principle applies. For a cloud vendor that means a web of contracts, data-processing agreements and ongoing oversight.
For health data, GDPR adds another layer. Health data falls under the special categories of personal data in Article 9 and carries heightened protection. Outside counsel agreements and patient-data clauses frequently prohibit sending the protected material to a third-party processor at all.
Here is the crux. A local RAG install largely sidesteps the service-provider question, because there is no external processor. Nobody to vet, no cloud contract to enforce, no third-party transfer to justify. The file stays in the building. That does not dissolve your general obligations, but it removes an entire category of risk.
The local stack in daily use
A realistic setup for a mid-sized firm or clinic looks like this:
- A Mac Studio or a GPU workstation in the server room, reachable only on the internal network.
- Ollama as the model server, plus a model in the 8B to 32B parameter range depending on hardware.
- A local embedding model and a self-hosted vector database over your document corpus.
- A simple intranet web front end where staff type their questions.
Quality depends less on the model than on preparation. How you split documents into meaningful chunks, how you re-rank hits, how you check answers against the source, that determines whether the system holds up in practice. A carefully built small stack beats a large model paired with poor retrieval.
The other common use case in these firms is transcription: automatically minuting client meetings and case reviews, again without any cloud. We covered that in running Whisper locally for meeting notes.
Limits, stated honestly
Local RAG AI is not a set-and-forget tool. Setup takes time, someone has to maintain the models and the database, and a locally hosted 14B model does not answer at the level of the very largest cloud models. For finding and summarising facts from your own corpus, practitioners report the quality is usually more than sufficient; for complex legal reasoning, less so.
And local AI can be wrong too. Answers with a citation help, because you can verify on the spot, but professional responsibility stays with the human. Treat the system as a fast research assistant, not a decision maker.
If you are weighing how much data control your firm genuinely needs, our page on data sovereignty sets out the wider frame.
Wondering whether a local RAG assistant pays off for your confidential documents? Talk to us about a pilot project we run together on your own file base.