A vector index forgets who was allowed to read each document

The idea

A RAG pipeline chunks documents from a wiki, a drive or a ticket system, embeds them and stores the vectors. The content comes along. The access rules usually do not. Retrieval then returns the nearest chunks by similarity, regardless of who is asking.

Permission-aware retrieval fixes this. Each chunk carries the access control data of its source document, and every query is filtered by the caller's identity before results are returned. The point is the order of operations. Once a chunk is in the context window, it has already been disclosed. Asking the model to withhold it is a request, not a control.

An example

Picture an internal assistant indexed over a company wiki and an HR drive. An engineer asks for the salary bands for senior roles.

Without filtering, the compensation spreadsheet from the HR drive is the closest match. Its chunks land in the prompt and the model answers accurately.

With permission-aware retrieval, the flow changes:

• At ingestion, each chunk stores allowed_principals: the user and group IDs that can read the source document.
• At query time, the service resolves the engineer's groups from the identity provider, using the authenticated session.
• The vector query includes a metadata filter that matches only chunks whose allowed_principals intersect that set.
• The HR chunks never come back. The model answers from the public career ladder page, or says it found nothing relevant.

Where it breaks

Staleness is the main failure. Permissions change in the source long before the index notices. A revoked user can keep reading through the assistant until the next sync.

Filtering after retrieval is the second. If you take the top results and then drop the forbidden ones, a restricted user may get almost nothing back. The pattern also invites someone to "temporarily" skip the filter. Filtering inside approximate nearest neighbour search has its own cost: very selective filters can hurt recall, depending on the index.

Source ACL models also flatten badly. Deny rules, inherited folder permissions and anyone-with-the-link sharing do not map cleanly onto a list of principal IDs.

Derived artifacts leak around the filter. Cached answers, conversation summaries, long-term memory and semantic caches can all carry restricted content across users. Fine-tuning on the corpus is worse, because you cannot filter weights.

If the corpus is genuinely public, all of this is overhead you do not need.

In practice

I would treat retrieval as an authorization boundary and build it that way:

• Copy ACLs from the source system at ingest. If a chunk has no ACL, fail closed and do not index it.
• Apply the filter inside the vector query. Never rely on the prompt, and never filter after generation.
• Take identity from the authenticated session, never from model output or tool arguments.
• Sync permission changes on their own fast path, separate from re-embedding content, and process revocations first.
• For highly sensitive sources, re-check access against the source API at read time before a chunk enters the prompt.
• Key every cache and memory store by the principal set that produced it.
• Write tests with fixture users who hold different access, and assert on the retrieved chunk IDs, not the generated text.
• Log the chunk IDs served per request, so an audit can answer who saw what.

In a RAG system, authorization happens at retrieval, because anything that reaches the context window has already been disclosed.