The common assumption is that private AI means a rack, a cooling problem, and a capital request with several zeros. For document question-answering in a normal office, that assumption is wrong — and the reason is that most of the work is not the model.
Where the effort actually goes
Answering "what is our notice period for senior staff" is mostly a retrieval problem. The system has to find the right two or three passages out of thousands. That is search, and search is cheap. The language model only has to read those passages and phrase an answer, which is a small job compared with generating text from scratch.
- Documents are split into passages and indexed when they are uploaded — this is the heavy step, and it happens once per document.
- A hybrid of keyword and meaning-based search narrows thousands of passages down to a handful.
- A re-ranking step reorders those candidates so the best passage is used first.
- A compact, quantised model turns the selected passages into an answer with citations.
Where the limits are — honestly
- Long creative writing and complex multi-step reasoning are better on large cloud models.
- Indexing time and answer latency depend on the model, hardware, file size, OCR workload, and concurrent use; benchmark your own representative documents.
- Scans with poor image quality produce poor OCR, and therefore poor answers.
The practical test
Run your own worst document through it before you commit. A blurry scanned contract and a 300-page policy manual will tell you more than any specification sheet.