Your AI RAG might be a drag on your security policy

An AI knowledge search tool such as a RAG system that provides answers based on sensitive documents must know about the user, not just their query. Checking only the clearance level per document is not enough when documents also carry compartments (extra restrictions a user must hold separately).

We built a retrieval engine that checks clearance and compartment for both users and chunks (paragraphs), and tested it on 520 declassified US intelligence releases. Across 58 questions, a clearance level-only check gave access to a Top Secret user with no compartments to 524 of 1,160 top paragraphs which were beyond that user's access permissions; our engine denied all of them, and a leak test found no leaks in 2,320 requests.

The problem

Basic retrieval-augmented generation finds information by how closely it matches a query. In most permission-aware systems only one clearance level is used, usually a ranked system such as Official Use Only < Confidential < Secret < Top Secret. However, this is not sufficient if you must fulfill compartmentized requirements and clearance-level permissions.

We measured a baseline against a filter that checks a single level on each document. The documents carry two kinds of restriction at once. One is a level, such as Secret. The other is a set of compartments, such as NOFORN or SI, which a user must also hold. In this system, a user with a higher level than a paragraph might still not be able to access it.

What we built

We built a retrieval engine that reads each paragraph's level and compartments from the document's own markings, labels each chunk at ingest, filters by permission before ranking, and labels each answer with the combined label of the chunks it came from.

We ran it on declassified documents that the Office of the Director of National Intelligence publishes in its IC on the Record database. Of 520 releases, 336 yielded labelled text, for a total of 17,346 paragraphs, each labelled from the document's own paragraph markings. The engine indexes a sample of up to 1,000 paragraphs per label, 5,457 in all.

Performance

A leak test put 58 questions to each of the 40 clearances that the corpus's labels allow, a total of 2,320 queries. All of those queries yielded only paragraphs within the user's access level.

Next we changed the engine's permission checking to ignore compartments. This resulted in 4,542 leaked paragraphs, showing that enforcing compartments enforces the correct access controls.

An answer to a query can become more sensitive than any of the individual paragraphs in that answer. In 5 of the 2,320 requests, the answer's label ranked above every paragraph in it. In one example, paragraphs marked (S//NF/ORCON) and (TS//NF/SI) combined into an answer labelled (TS//NF/ORCON/SI), which is more restrictive.

Out of scope

Fifty of the 520 files had no text layer; we skipped them rather than run OCR. Another 134 had no readable banner or paragraph mark, and we excluded them as well. In the documents that remained, 3,118 paragraphs had a missing or unreadable mark and inherited a document-level label.

We did not measure the quality of answers at each clearance, nor tried labelling schemes other than this one.

See it run

In a 15-minute walkthrough we run the engine live, answer one question for readers at different clearances, and break the permission check so you can watch the audit catch it. We reply within 24 hours.

Source documents: public releases from ODNI's IC on the Record database (link: https://www.intelligence.gov/ic-on-the-record-database). They are US government works in the public domain. ODNI has no involvement in this work.