For regulated RAG, the model choice comes down to a trade between data sensitivity, answer quality and cost — and the right answer is usually different per project, which is why the most important decision is to stay model-agnostic. RAGSuite lets you route across OpenAI, Anthropic, Mistral, Gemini or a local model via Ollama, and use Compare Models to choose with evidence rather than by reputation.
The decision has three axes
Data sensitivity sets the hard constraints: if data must never leave your network, a local model via Ollama is the answer. Answer quality differs by task — long-context reasoning, multilingual (which matters for German), concise extraction — and is best measured on your content. Cost and operations trade pay-per-use hosted APIs against hardware you provision and run.
The options, at a glance
| Option | Where inference runs | Best when |
|---|---|---|
| OpenAI / Anthropic / Gemini | Hosted API | Quality matters and the project’s data can use a hosted model |
| Mistral | Hosted or self-hosted | You want strong European options or to self-host weights |
| Ollama (local) | Inside your network | Data must never leave — full isolation, air-gapped |
Why one model rarely fits everything
A simple way to decide
For each project, in order
- Can the data leave the network? If no, use a local model via Ollama and stop here.
- If yes, which model answers best on this corpus? Use Compare Models to find out.
- Does the cost and operational shape fit? Confirm the trade before rollout.
Do this per project, revisit as models improve, and keep the option to switch. See the quality loop for how to keep measuring once you have chosen, and the quality-loop page for Compare Models in the product.
Frequently asked questions
Should we just pick one model for everything?
Rarely the best move. A public-documentation assistant and an HR-records assistant have different sensitivity, quality needs and cost tolerance. Forcing both onto one model compromises one of them; routing per project lets each use what fits.
How does Compare Models help?
It runs the same queries across candidates so you can see, on your own corpus, which gives better-grounded answers before you commit — turning 'which model is best?' into 'which is best for this project?', which you can actually measure.
Sources & further reading
- Ollama — run open models locally — the local-model path
- RAGSuite — Compare Models & the quality loop — decide with evidence