RAG Over Internal Documents: What Decides Answer Quality, From Three of Our Projects
Short answer (November 2026). In a RAG system over company documents, answer quality is decided less by the language model than by five engineering choices: what goes into the knowledge base at all, how documents are split and searched, what happens when nothing relevant is found, who is allowed to see which document, and whether every search is logged. Even commercial legal research tools built on RAG still produced wrong or unsupported answers in 17% to 33% of test queries in a Stanford study, so "the model reads our documents" is not a quality guarantee. Combining vector search with keyword search, as we do, is one of the cheapest improvements: Anthropic measured 49% fewer failed retrievals with contextual embeddings plus BM25, and 67% with reranking added. The rule that matters most is simple: no source, no answer.
We have built retrieval into three products: a customer-facing AI assistant that answers from a client's own documents, an assistant for accountants in Kazakhstan that works with the Tax Code and accounting legislation, and CostQuest, an AI estimation tool for founders. This article collects what we learned, without our own metrics: the decisions, the mistakes we caught, and the tests we now run before a knowledge base goes live.
Three projects, three kinds of knowledge
| Who asks | What the knowledge is | How we retrieve it | Main safeguard | |
|---|---|---|---|---|
| AI Intake Agent | customers of a business, in chat and messengers | the business's own documents: PDF, DOCX, scans | vector search as a tool the agent calls, filtered by client in code | the assistant says a colleague will confirm, hands the question to a person and continues the conversation |
| Accounting assistant, Kazakhstan | accountants | the Tax Code and accounting legislation | hybrid search: vectors plus full text | a fixed "not found in the documents" answer, no guessing |
| CostQuest | founders estimating a product | a catalogue of reusable features with historical effort baselines | a database query by product category, no vectors | only catalogue rows reach the prompt, nothing is matched by similarity |
The first is a hosted service: a business connects its Instagram, Messenger, Telegram, e-mail or website chat, uploads its documents, and the assistant collects enquiries and answers questions. One of its users is RadioLance, a remote radiology service we also built the platform for; there the assistant works as a coordinator on the public site and answers from the service's own materials, while never interpreting a study or giving a diagnosis.
Decision 1: not everything belongs in RAG
The most useful decision we made was to take knowledge out of the vector store.

- Prices are arithmetic, not search. In the radiology assistant, price was the most common question. Left to retrieval, a query that named the price zones scored below our relevance threshold, so the assistant escalated and promised a colleague's answer that the knowledge base already held. We moved the tariff grid into a prompt template that each client can edit in the admin panel. A price change is now an edit, not a reindex.
- Structured data wants a query. CostQuest estimates rest on a catalogue of features grouped by product category, each with effort baselines by developer level. Selecting rows by category is exact and explainable; embedding them would only add a way to pick the wrong rows.
- State is not knowledge. "Which documents are still missing?" is answered from the request's own state, which the assistant already sees. No search needed.
- Live data needs a tool. Delivery status or stock levels change by the minute. They belong behind an API or an MCP server, not in an index that is a copy from last night.
What is left for RAG is long text: laws, policies, manuals, contracts, clinical service descriptions. That is where it is strong.
Decision 2: retrieval is a tool, not a bigger prompt
The tempting shortcut is to paste the documents into the system prompt while the knowledge base is small. We decided against it in the intake assistant for three reasons.
- The search query depends on the customer's question, which does not exist when the system prompt is built. As a tool, the agent reformulates the question into a query and searches when it needs to.
- Long prompts get worse, not just more expensive. In the NoLiMa benchmark, 11 of 13 models that claim at least 128,000 tokens of context fell below half of their short-input score at 32,000 tokens, and models are known to miss facts placed in the middle of long inputs ("lost in the middle").
- A tool returns chunk ids, so every answer can cite its document. Pasted context cannot.
A test pins the size of the rendered system prompt, so nobody can later "just include the top documents" without the change showing up.
Decision 3: hybrid search for documents full of numbers
Legislation is the worst case for pure vector search. Accountants ask about "Article 412", a form number or a rate, and embeddings are good at meaning but weak at exact identifiers. In the Kazakhstan assistant we run two searches for every question, a vector search and a full-text search, and merge the ranked lists with reciprocal rank fusion, a simple method from 2009 that needs no tuning of score scales. The BEIR benchmark found plain keyword ranking (BM25) a robust baseline that dense models often fail to beat outside their training domain, and in Anthropic's measurements adding BM25 to contextual embeddings took the reduction in failed retrievals from 35% to 49%.
Two smaller things made a visible difference in the same project:
- Follow-up questions. "And for individual entrepreneurs?" means nothing on its own. The search query includes the user's last few questions, while the answer is still generated only from what that search returns.
- Language. Questions and sources are not always in the same language, and the answer must come in the user's language whatever the source language is. That is one instruction in the prompt, but it has to be tested in every combination.
Decision 4: chunks that remember where they came from
Every chunk carries its document, page and position, so an answer can say "Tax Code, page 87" instead of "according to the documents". Our chunking is deliberately boring: around a thousand characters, split on paragraph boundaries, never mid-word, with some overlap.
Boring has a purpose. Chunking is deterministic, so chunk ids are stable: reindexing a document replaces its vectors instead of duplicating them, and when a new version is shorter, the leftover chunks are deleted. Deleting a document deletes its vectors, which matters for the GDPR as much as for quality: an outdated policy that still answers questions is a silent bug.
For legislation, versions are the real problem. Kazakhstan adopted a new Tax Code on 18 July 2025, in force from 1 January 2026, replacing the code of 2017, and it has already been amended since. A knowledge base over law has to know which version of a document is in force on which date, or it will answer correctly about the wrong year.
Scanned PDFs are read with a multimodal model, page by page, with a hard page limit per document, so a 200-page scan cannot quietly run up a bill.
Decision 5: the refusal path is a feature

Both document projects share one rule: no source, no answer.
- A relevance floor. Results below a similarity threshold are dropped before the model sees them. If nothing is left, the model is not called at all.
- Mandatory citations. The model must put a source number after each statement. Sources it did not cite are removed from what the user sees, so the list of references is the list of evidence.
- A fixed "not found" answer. If the sources do not contain the answer, the model must reply with one exact phrase, which the code recognises. A confidence label comes from the best retrieval score, not from the model's opinion of itself.
- A handover that does not stop the conversation. In the customer-facing assistant, a miss raises a flag for a person and the assistant says so plainly, then keeps collecting the enquiry in the same message. An assistant that freezes on an unknown question is worse than one that says "I'll check with a colleague".
The threshold itself is the one number that cannot be decided from first principles. Ours started as a guess and was tuned on real documents. One trap we found: the same embedding model gives systematically lower similarity scores for Russian text than for English, so a threshold copied from an English example silently turns a Russian knowledge base into an escalation machine.
We are honest about what code enforces and what it does not. Code enforces the threshold, the tenant filter, the audit and the handover. That the model does not embellish the retrieved text is a prompt rule, not a guarantee. The mitigation is observability.
Decision 6: access control lives in code, not in the prompt
The worst failure of a multi-client knowledge base is one client's documents answering another client's customer. In the intake assistant all clients share one vector collection, and every search is filtered by the client's id. The id comes from the conversation, never from a tool argument, so the model cannot be talked into reading another client's documents: the client is simply not something it can say. The search function has no default for the client id; forgetting it is an error, not a leak. A test with identical documents in two organisations proves it. OWASP lists this risk as "Vector and Embedding Weaknesses" and recommends permission-aware vector storage with strict logical partitioning.
The same applies to roles inside one company: if a junior employee may not open a contract, the assistant must not quote it to them. Filter by permissions at search time; do not ask the model to keep a secret it has already read.
Decision 7: retrieved text is data, not instructions
A document can contain "ignore previous instructions". In a customer-facing system, documents come from the client and, through e-mail and uploads, sometimes from the client's customers. Retrieved text is passed to the model clearly marked as data, below the system rules, and it can never change them. OWASP ranks prompt injection, including this indirect kind through files, as the top risk for LLM applications and notes that RAG does not fully prevent it. This reduces the risk; it does not remove it, which is one more reason the assistant has no tools that write anything without a person approving.
Decision 8: log every search, misses included
Every search writes an audit record: the query, the best score and the documents returned, including searches that found nothing. This is what turns "the assistant said something odd" from folklore into a bug with an address: the record shows whether the right document was found, found with a weak score, or never found. The distribution of best scores is also what we used to tune the threshold. Without that log, a knowledge base cannot be improved, only argued about.
Every embedding and OCR call is logged with its token count too, so the cost of "reindex everything" is a number before anyone presses the button. Embeddings themselves are cheap (OpenAI lists text-embedding-3-small at $0.02 per million tokens); OCR of scans and the answering model are the larger costs.
The test set we run before go-live
We write a test document whose numbers are invented, so a correct answer can only come from the knowledge base, never from the model's general knowledge. Then we ask:
| Check | Example | Expected |
|---|---|---|
| Direct hit | "How much is an export declaration?" | the invented price, with a citation |
| Fact in another chunk | "How long is the Hamburg to Istanbul run?" | the right chunk, not the first one |
| False premise | "Is it true the first five days of storage are free?" | a correction, three days not five, with a citation |
| Plausible but absent | "What is your rate for sea freight to Alexandria?" | no invented rate; handover to a person |
| Outside the knowledge base | "Where is my cargo right now?" | handover without searching |
| Conversation continues | the two answers above | the assistant also asks the next question it needs |
| No duplicate handovers | the same question again | no second flag |
| Client boundary | the first question from another client's chat | not answered from the first client's document |
The false premise is the most valuable check. Confidently confirming something wrong is the most expensive mistake a RAG system makes, and when it fails the cause is usually the prompt, not the search. For larger sets, the RAGAS metrics are a useful vocabulary: faithfulness (is every statement supported by the retrieved text), answer relevance and context relevance, as defined in the original paper; the current ragas library adds context precision and recall.
Cloud API or local models
For the accountants' assistant, confidentiality decided the architecture. The same code runs against a cloud API or entirely on the client's server with open models: BGE-M3 for embeddings (MIT licence, more than 100 languages) and a Qwen model for answers, served by Ollama, with Qdrant as the vector store. Local models answer more slowly and less fluently, but no document leaves the building. For most EU companies a cloud model under a data processing agreement and an EU region is enough; for tax, legal and medical archives the local option is worth pricing in from the start. Either way, the Hamburg data protection authority notes that access, erasure and rectification requests can relate to the input and output of an AI system, including its database queries, so deleting a document has to delete its vectors.
Checklist
- List the questions users actually ask and sort them by where the answer lives: state, template, database, documents or a live system.
- Put only long text into RAG.
- Combine vector and keyword search when documents contain article numbers, codes or product names.
- Keep document, page and version on every chunk; make reindexing idempotent and deletion real.
- Set a relevance floor and tune it on your own documents and languages.
- Require a citation for every statement and a fixed answer for "not found".
- Hand unanswered questions to a person without stopping the conversation.
- Filter by client and permissions in code, at search time.
- Treat retrieved text as data; give the assistant no unapproved write actions.
- Log every search, including misses, and every embedding call with its cost.
- Test with invented facts, a false premise, an absent fact and a second client before go-live.
- If you serve people in the EU, tell them they are talking to an AI. Article 50 of the AI Act requires this from 2 August 2026; the Digital Omnibus did not postpone it.
How we can help
We design and build custom RAG applications over company documents, from a prototype on your own files to a production system with access control, audit and on-premise models. If you want an assistant for your customers rather than a project, the AI Intake Agent runs on your channels with your documents.
For background on the architecture itself, see our earlier guide to RAG for business leaders and the checklist for deciding whether you need custom RAG.
FAQ
What is RAG over internal documents?
Retrieval-augmented generation: before answering, the system searches the company's own documents for relevant passages and gives only those to the language model, which answers from them and cites them. The model does not need to be retrained when documents change; the index is updated instead.
Why does a RAG system still give wrong answers?
Usually because the right passage was not found, a weak match was used anyway, or the model added to what the passage says. A relevance threshold, mandatory citations, a fixed "not found" answer and an audit of every search address all three.
Do we need a vector database for everything?
No. Prices, schedules and other small stable data work better in a prompt template, structured records behind a database query, and live data behind an API or MCP tool. A vector database is for long text.
Is vector search enough, or do we need keyword search too?
For documents full of article numbers, codes and names, combine both. Vector search finds meaning; keyword search finds exact identifiers. Reciprocal rank fusion merges the two lists without tuning.
Can a RAG system run without sending documents to the cloud?
Yes. Open embedding and language models can run on your own server. Answers are slower and less polished than from the largest cloud models, but no document leaves your infrastructure.
How do we test a RAG system before launch?
Use a test document with invented facts so answers cannot come from the model's general knowledge, and include a false premise, a plausible question the documents do not answer, a question outside their scope and a query from a second client or role.
Sources
- Magesh, Surani, Dahl, Suzgun, Manning, Ho, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Journal of Empirical Legal Studies, 2025
- Anthropic, Introducing Contextual Retrieval, September 2024
- Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, NeurIPS 2021
- Cormack, Clarke, Büttcher, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods, SIGIR 2009
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 2024
- Modarressi et al., NoLiMa: Long-Context Evaluation Beyond Literal Matching, ICML 2025
- Baker McKenzie, Kazakhstan: New Tax Code, September 2025
- OWASP, LLM01:2025 Prompt Injection and LLM08:2025 Vector and Embedding Weaknesses
- Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation, EACL 2024; ragas metrics
- BAAI, bge-m3 model card
- OpenAI, API pricing, checked October 2026
- Hamburg Commissioner for Data Protection, Discussion paper: Large Language Models and personal data, July 2024
- EU AI Act, Article 50; Lewis Silkin, The Digital Omnibus on AI enters into force today, July 2026
Categories
About the author
Share
Need a project estimate?
Drop us a line, and we provide you with a qualified consultation.