Five Targeted Documents Cut Classical RAG Retrieval Scores by Up to 61.5%

A controlled benchmark of retrieval-augmented generation (RAG) systems found that five documents tailored to each query reduced classical retrievers’ top-10 ranking scores by 40.2% to 61.5%.

The evaluation used 623 queries from SciFact and NFCorpus, two research benchmarks. In isolated-target tests, each query was paired with its own five targeted documents. The result does not describe five documents added once to a shared collection for all 623 queries.

The study measured nDCG@10, a score for how well a system ranks relevant documents among its first 10 results. The reported decline applied to classical retrievers under the isolated-target setup. These controlled tests do not establish how often such attacks occur in deployed search systems.

For BM25, a keyword-based ranking method, repeating the full query in a document explained most of the retrieval damage. The test also included the dense retriever BAAI/bge-small-en-v1.5, referred to as BGE-small. Under query-only poisoning, it had 100% target exposure. That means at least one targeted document appeared in the top 10 for every evaluated query; it does not mean all top-10 results were poisoned.

Adding malicious payloads barely changed BGE-small’s ranking.

Overlap-based filters could block full-query poisons. The tested filters also reduced clean retrieval quality. MMR offered little to no protection in the tested setups.

Posted Nov. 3, the preprint measures retrieval failures only. It did not test whether poisoned results changed AI-generated answers or whether models followed instructions hidden in those documents. A retrieval score decline therefore does not establish a corresponding failure rate in AI answers.