When staff ask a general AI chatbot about leave policy, the returns procedure, or the terms of a supplier contract, the answer often sounds convincing but does not match how your company actually works. The reason is simple: the model has never read your internal documents. It answers from general knowledge, not from the SOPs and contracts that apply in your office.
This is where RAG comes in. With RAG, AI answers from company documents and cites its sources, so every answer can be checked. This article explains how RAG works, what you need to prepare, and the limits worth understanding before you start.
What RAG is and how AI answers from company documents
RAG stands for Retrieval-Augmented Generation. Before the AI answers, the system first searches for the document passages most relevant to the question and hands them to the model as source material. The model is instructed to answer only from that material and to say which document each part of the answer came from.
The flow looks roughly like this:
- Company documents are split into small passages and stored in a search index, usually with semantic search that understands meaning rather than matching exact words.
- An employee asks a question, for example "how many days of maternity leave do contract staff get?"
- The system retrieves the most relevant passages, taking the asker's access rights into account.
- The model writes an answer from those passages and includes citations, such as the document name and section.
- If nothing relevant is found, the AI should say it does not know rather than make something up.
The real value is in the citations. Staff do not have to take the answer on trust; they can open the source and check for themselves.
Document preparation: the foundation people underestimate
RAG answer quality depends heavily on the quality of the documents you feed it. If your folders hold three conflicting versions of the same SOP, the AI will be confused in exactly the way a new hire would be.
A few things to sort out first:
- Decide the official source. Choose the documents that are actually in force and archive old versions.
- Check formats. Scanned PDFs need OCR. Tables and appendices need checking to make sure they are read correctly.
- Add metadata. Effective date, document owner, and department help the system pick the right source.
- Set an update process. When a policy changes, the index must be updated too.
Chunking, access control, and personal data
Chunking
Chunking is how documents are split into passages. Passages that are too small lose context; passages that are too large make retrieval less precise and raise processing costs. A good approach usually follows the document's own structure, such as chapters, clauses, or subheadings, so each passage carries one complete idea.
Access control
This is the part you cannot skip. If payroll documents are only meant for HR, RAG must not retrieve passages from them to answer a question from another department. Permissions have to be enforced at the retrieval step, before any passage is sent to the model, not merely hidden in the interface.
Personal data
Company documents often contain personal data such as NIK, NPWP, or home addresses. This kind of data should be masked before it is sent to a model, in line with your privacy policy and the principles of Indonesia's Personal Data Protection Law (UU PDP).
Evaluating answer quality
A RAG system needs measurable testing. Build a list of real employee questions with the correct answers and their source documents. Then check two things separately:
- Retrieval: did the system find the right passages?
- Answer: did the AI answer faithfully from those passages, with correct citations and nothing added?
Separating the two tells you where the problem lies. A wrong answer is often not the model's fault; the right document was simply never retrieved. Rerun these tests every time you change the chunking approach, the model, or the prompt.
Once the system is in use, collect feedback too. Unanswered questions point to documents that do not exist yet, while answers flagged as wrong point to passages that need fixing. This record makes the system more useful month by month and shows document owners which sections need to be clearer.
Limits and pitfalls to watch for
RAG reduces made-up answers but does not eliminate them. Common pitfalls include:
- Aggregate questions. RAG suits "what is the rule" but not "what were total sales last month". Questions like that are better answered from a database or operational system.
- Conflicting documents. When two sources disagree, the AI may pick the wrong one. Curation still matters.
- Unchecked citations. Citations only help if people get into the habit of opening them, especially for important decisions.
- Inflated expectations. RAG is a tool for finding and summarising, not a substitute for legal, financial, or HR expertise.
Building AI that understands your documents
RAG works best when it starts with one clearly defined document set, such as operational SOPs or HR policies, with a limited group of users and regular testing. Once quality is proven, coverage can expand to other departments and documents.
Averoa helps companies build search with citations, per-team access control, and personal data masking through our Company AI service. If you also want to ask questions directly of operational data such as HR and finance, take a look at Sage in Trellis, which works over your own company data and is isolated per company.
