Building an Internal AI Knowledge Assistant: Sources, Permissions and Trust

An internal AI knowledge assistant helps staff ask questions about approved company information and find the relevant sources. It should respect the same access boundaries as the underlying documents and make it easy to check an answer. Uploading a shared drive and adding a chat box is not enough.

This approach is often called retrieval-augmented generation, or RAG: the application retrieves relevant material and supplies it as context when generating a response. The practical question for a business owner is whether that process produces useful, verifiable answers for the people who need them.

Start with a bounded knowledge collection

Choose a coherent set of information, such as current service procedures or approved product guidance. Assign an owner to each source. Remove obsolete versions and resolve contradictory instructions before building an answer interface.

Decide which questions are outside the assistant's remit. If a procedure does not cover an unusual case, the correct response may be to point the employee to a person rather than assemble a confident guess.

Make permissions part of retrieval

A user should only retrieve material they are authorised to access. This check belongs in the application and data-access layer; it should not depend on telling the model to keep a secret. Consider departmental boundaries and access changes when staff move roles or leave.

Test with at least two accounts that have different permissions. Ask each account questions that would require restricted material. Also check whether titles, snippets, citations or cached answers could reveal information the user should not see.

Show sources people can actually inspect

A useful answer includes links to the source passages and indicates when the information was reviewed or updated. A citation is only valuable if it supports the statement beside it. Reviewers need to check both the generated answer and whether the linked passage justifies it.

Give staff a straightforward way to report an incorrect answer or outdated source. Route those reports to an owner. Otherwise the same problem can be rediscovered repeatedly without anyone fixing the source of it.

Keep document content separate from authority

Documents can contain misleading instructions, including text that asks a tool to ignore its rules or reveal other information. Treat retrieved content as information to assess, not permission to perform an action. Start with read-only answers and avoid connecting the first release to email sending, payments or record changes.

The OWASP prompt-injection guidance describes why instructions embedded in external content require explicit controls. Application-level permissions remain necessary even when the model behaves well in a demonstration.

Evaluate the questions staff really ask

Create a test set containing common questions, ambiguous questions, outdated terminology and questions with no answer in the approved sources. Assess factual correctness, source support, appropriate refusal and time taken to reach a usable answer.

Repeat those checks when the document collection, retrieval settings or model changes. Include the effort of maintaining the knowledge base when estimating benefit. For some teams, improving document organisation and conventional search may be sufficient.

Choose a manageable first release

A sensible pilot covers one team and one approved collection, with named owners and feedback from actual users. Compare existing products and custom applications before commissioning a build.

I can help scope the knowledge, permissions and evaluation plan through AI consultancy and integration. Tell me what your team struggles to find.