AI Assistant for Company Documents: How RAG Works, Simply

Almost every company has documents nobody reads end to end: operating SOPs, product manuals, leave policies, claims guides. When staff need an answer, they ask the most senior colleague nearby, and the answer can vary depending on who they ask. AI assistants that answer from company documents are built for exactly this problem. The most common approach is called RAG, short for retrieval-augmented generation. The name sounds technical, but the idea is simple.

RAG in one paragraph

Picture a new hire who writes well but hasn't memorised company rules yet. Whenever someone asks a question, they first find the relevant page in the filing cabinet, read it, and answer based on that page. RAG works the same way. The system searches for the document passages most relevant to the question, then hands those passages to an AI model as material for the answer. The model isn't asked to remember your documents, only to read the parts that were found.

Step 1: splitting documents into chunks

You can't feed a long document in full for every question; it's slow and expensive. So documents are split into small pieces, usually a few paragraphs each. This is called chunking. Chunk size matters: too small and the context disappears, too large and the important part gets buried among other text.

The quality of the source documents matters just as much. Blurry scanned PDFs, messy tables, or two versions of an SOP that contradict each other will produce answers that are just as messy. Cleaning up documents before they go in is often the most valuable work in the whole project.

Step 2: finding the relevant chunks

Each chunk is turned into a list of numbers that captures its meaning, called an embedding, and stored in a database. When someone asks a question, the question is converted the same way, and the system looks for chunks whose meaning is closest. Because it matches meaning rather than exact words, "how many days off do I get?" can still find a section titled "annual leave entitlement".

There are several search techniques. One of them, MMR (maximal marginal relevance), picks chunks that are relevant but don't repeat each other, so the model gets more varied material to work from.

Step 3: answers grounded in the source

The retrieved chunks go to the AI model with a firm instruction: answer only from this material, and say plainly when the answer isn't there. An answer built this way is called grounded.

• Show the source. Include the document or section used, so staff can check it themselves.
• Allow "I don't know". It's better for the assistant to say "not found in the documents" than to invent something.
• Limit the scope. An assistant for warehouse SOPs shouldn't also be answering payroll questions.
• Respect access rights. Documents only managers may read shouldn't be searchable by everyone.

Evaluation: don't just try it a few times

The most common way to judge an AI assistant is to try it a few times and feel satisfied. That's misleading, because an answer that sounds convincing isn't necessarily correct. A more honest approach is a test set: a list of real questions along with the correct answers according to the documents. Every time the documents or settings change, you run the test set again.

Some tools help measure the important things automatically, such as RAGAS. Tools like this check whether answers stay faithful to the retrieved chunks and whether those chunks were actually relevant to the question. The scores are useful for comparing versions, but important questions still deserve a human check.

A real example: our PDF question-answering prototype

We built a PDF question-answering prototype that follows this exact flow. A user uploads a PDF through a web interface, the document is chunked and stored in PostgreSQL with pgvector, retrieval uses MMR, and the Llama 3.3 model via Groq writes the answer from the retrieved chunks. Quality is measured with RAGAS against a set of test questions. The backend is FastAPI and the interface is Streamlit.

To be clear, it is still a prototype and hasn't been deployed for business use. It shows the flow works end to end. A company deployment would still need per-user access rights, a tidy process for updating documents, logging, and testing against that company's own documents. The same principle of grounding answers in official material is used by Numekara's AI tutor, which answers from published lessons and official solutions.

Which documents to start with

• SOPs and work guides that new staff ask about all the time.
• Product manuals and technical FAQs for sales or support teams.
• Finalised internal policies, such as leave, business travel, or purchasing.

Avoid starting with documents that contain personal data about employees or customers. If you need them later, look into your obligations under UU No. 27 Tahun 2022 tentang Pelindungan Data Pribadi (Indonesia's Personal Data Protection Law) and consult a professional.

Next steps

1. Pick one limited set of documents that people ask about most.
2. Clean them up: remove old versions and make sure the text is machine-readable.
3. Build a test set of real questions with correct answers.
4. Pilot it with a small group of staff, collect the wrong answers, and fix the sources or settings.