A quick refresher on RAG
Retrieval-augmented generation (RAG) combines search with a large language model. When someone asks a question, the system first retrieves the most relevant passages from your documents, then asks the model to write an answer using only those passages, ideally with links back to the source.
It’s a powerful way to put company knowledge at people’s fingertips without retraining a model. But the quality of the answer depends almost entirely on the quality of what gets retrieved.
Problem 1: messy documents
Real knowledge bases contain scanned PDFs, multi-column layouts, spreadsheets, slide decks and old Word files. If text is extracted badly, tables lose their structure and headings get separated from the content they describe. The model then answers from fragments.
Good document processing, including OCR for scans and careful handling of tables, is usually the single biggest improvement you can make.
Problem 2: finding the right passage
Pure vector (semantic) search is good at meaning but can miss exact part numbers, policy codes or product names. Keyword search is the opposite. Most reliable systems combine both and then re-rank the results before passing them to the model.
Problem 3: permissions
Not everyone should see every document. An enterprise RAG system has to respect the same access rules as your file shares and intranet, so a question from one team never surfaces content meant for another. This needs to be designed in from the start.
Problem 4: knowing if it works
“It looks right” is not a test. Build a set of real questions from your team with known correct answers, measure how often the system gets them right, and re-run the test whenever documents or settings change. Showing sources with every answer also lets users check the result for themselves.
Problem 5: keeping it current
Documents change every week. Plan how new, updated and deleted files will flow into the index automatically, otherwise the assistant will confidently quote last year’s policy.
None of these problems are unsolvable. They just need the same engineering care as any other production system.
Related: Enterprise RAG, AI copilots · More insights