TRIS: A Tri-Layer Retrieval Integrity Sieve Against Knowledge Poisoning

RAG (Lewis et al., 2020) augments LLMs with non-parametric memory: a query $q$ is embedded by $f_{Q}$ and the top-$k$ documents in a corpus $\mathcal{D}$ are retrieved by $s(q,d)=\mathrm{sim}(f_{Q}(q),f_{D}(d))$, after which an LLM conditions on that evidence. Dense retrievers such as DPR (Karpukhin et al., 2020) and Contriever (Izacard et al., 2022) are now standard. RAG offers stronger factuality than parametric-only LLMs (Gao et al., 2023), but trusts retrieved evidence by default, opening a new attack surface.

Classical data poisoning targets training pipelines (Geiping et al., 2021; Steinhardt et al., 2017; Gu et al., 2019). Retrieval-stage poisoning is a newer threat that requires only corpus access (Zou et al., 2025; Shafran et al., 2025; Edemacu et al., 2025).

Output-side methods (Song et al., 2024) detect post-generation mismatches but cannot prevent retrieval dominance. TrustRAG (Zhou et al., 2025) re-ranks via a single learned trust score (with an optional LLM-consistency check); effective on simple attacks, weaker on MS-MARCO (Section 6). TRIS differs architecturally, not just numerically: three independent off-the-shelf checks rather than one learned scorer, each targeting a different constraint, evaluated against an explicit adaptivity taxonomy (Section 3.2). We scope that difference precisely: Layer 2 targets verbatim and near-duplicate injection — the common case — where TrustRAG has no dedicated structural filter (ROUGE-L overlap is its closest analogue). Under paraphrased triggers Layer 2 fires on roughly zero documents per query and TrustRAG’s more lenient check catches more, so we claim Layer 2 for the common attack, not as a paraphrase defense (Section 6.5).

RobustRAG (Xiang et al., 2024) gives isolate-and-aggregate decoding certified against $k$-corruption, but strict isolation harms clean accuracy on multi-hop queries. Query paraphrasing helps only marginally, since dense retrievers map paraphrases into similar embedding regions (Zou et al., 2025). Perplexity detection fails because LLM-generated payloads are often more fluent than genuine web text (Shafran et al., 2025). Prompt injection (Greshake et al., 2023) is related but distinct: our payloads encode a false fact, not an instruction.

A RAG pipeline consists of (i) a retriever $\mathcal{M}_{\mathrm{ret}}$ that embeds $q$ and retrieves the top-$k$ from corpus $\mathcal{D}$, and (ii) a generator $\mathcal{M}_{\mathrm{gen}}$ that conditions its output on $\mathcal{D}_{k}$. The generator trusts retrieved evidence by default, matching production deployments over mutable corpora.

The adversary may (a) inject documents into $\mathcal{D}$, (b) craft adversarial triggers that maximize similarity with $q^{*}$ under $\mathcal{M}_{\mathrm{ret}}$ (verbatim query repetition in the black-box setting; HotFlip (Ebrahimi et al., 2018) gradient optimization in the white-box setting), and (c) embed malicious payloads that express $y_{\mathrm{adv}}$ as an authoritative encyclopedic claim. These capabilities instantiate the Split-and-Merge construction of Zou et al. (2025): a poisoned document concatenates a retrieval-optimized Trigger with a generation-optimized Payload, independently optimizing retrieval dominance and output steering.

The adversary’s goals are to dominate retrieval for $q^{*}$, override the LLM’s internal knowledge, and induce a targeted output $y_{\mathrm{adv}}\neq y_{\mathrm{true}}$ (Zou et al., 2025; Shafran et al., 2025). The defender (i) cannot retrain $\mathcal{M}_{\mathrm{ret}}$ or $\mathcal{M}_{\mathrm{gen}}$, (ii) cannot modify $\mathcal{D}$ beyond lightweight preprocessing, and (iii) must operate as a middleware layer — constraints reflecting deployed enterprise settings.

A reviewer might reasonably ask whether TRIS (Section 4) withstands attackers who know about it. We distinguish four levels of adaptivity, of which the evaluated PoisonedRAG attacks are Level 0:

L0 (static): the attacker is unaware of the defense — PoisonedRAG black-box and white-box, our headline setting. L1 (architecture-aware): the attacker knows TRIS is deployed but not its parameters, and may paraphrase rather than repeat $q$ (defeating Layer 2) and write encyclopedic-style payloads (degrading Layer 3); Layer 1’s geometric check still applies. L2 (judge-aware): the attacker also knows $\mathcal{J}$ and jointly optimizes for rank under $\mathcal{R}$ and majority-cluster membership under $\mathcal{J}$ — a conjunction constrained by the geometric independence of the two spaces, and compounded by ensembling or rotating $\mathcal{J}$. L3 (full white-box): the attacker also knows Layer 2’s thresholds and $y_{\mathrm{int}}$, and must craft a payload contradicting $y_{\mathrm{int}}$ that the LLM finds equally plausible.

We now evaluate Level 1 empirically (Section 6.5); Levels 2 and 3 remain conceptual only, and closing that gap is the most important remaining limitation of our evaluation (Section Limitations). A core claim of this work — now supported empirically for Level 1 — is that defeating an $n$-layer orthogonal defense requires the attacker to satisfy the conjunction of $n$ constraints in non-aligned objective landscapes, which substantially enlarges the attack search space even without certified bounds.

We exclude (i) training-set poisoning, (ii) prompt-injection attacks where the user is malicious, and (iii) attacks that require control of LLM weights.

Poisoned documents are optimized against a specific retriever encoder, and adversarial trigger optimization (Ebrahimi et al., 2018; Wallace et al., 2019) exploits the geometry of that encoder rather than true semantic content. Layer 1 is therefore designed as majority-keep in an architecturally independent geometry: rather than assuming poisons fail to cluster near the query under a second encoder, it assumes nothing about where they land and simply retains the majority cluster. We frame this as a design property — it drops the requirement that poisons form a tight, separable cluster in the retriever’s own space — not as a source of empirical gain; Appendix B.3 tests the geometry directly and finds it is not the differentiating factor.

Layer 1 re-embeds the top-$k$ retrieved documents with a judge model $\mathcal{J}$ that is architecturally distinct from the retriever $\mathcal{R}$ — we use Sentence-BERT all-MiniLM-L6-v2 (Reimers and Gurevych, 2019). After $L_{2}$-normalization we run K-Means with a small cluster count $C$ and identify the majority cluster; we sweep $C\in\{2,3,5\}$ in Section 6 and find $C=3$ the default sweet spot. Poisons typically appear as isolated micro-clusters, high-density off-distribution clusters, or low-similarity outliers. Documents outside the majority cluster are dropped, with a fallback retaining all documents when no clear majority exists.

Layer 2 targets the structural footprint of Split-and-Merge: Trigger and Payload need not share semantic content, but must coexist in one document, producing lexical irregularities at the prefix. For each surviving document we compute token-level Jaccard similarity between $q$ and the first $T=20$ tokens, bigram/trigram overlap between $q$ and the prefix, and length/repetition patterns; if Jaccard or $n$-gram overlap exceeds a configurable threshold (we use $0.8$), the document is discarded. Unlike perplexity filtering — which fails because LLM payloads are highly fluent (Shafran et al., 2025) — this detects the poison’s functional decomposition. Evasion is cheap at the retrieval stage, however: Section 6.5 shows a paraphrased trigger keeps near-identical rank while falling below the threshold, shifting the burden onto Layers 1 and 3. Layer 2 costs $\sim\!8$ ms per query.

Layer 3 verifies semantically by prompting an LLM (the generator $\mathcal{M}_{\mathrm{gen}}$ in self-knowledge mode, or a separate verifier) to judge each document in isolation, which defeats the dominance effect when many retrieved passages repeat the same poisoned claim; LLMs retain strong parametric priors even when retrieval corrupts the final answer (Longpre et al., 2021). The verifier (i) answers $q$ without context to capture its parametric belief $y_{\mathrm{int}}$, (ii) summarizes each document’s claim relevant to $q$, and (iii) judges it compatible with $y_{\mathrm{int}}$, contradictory, or uncertain; only strong contradictions without evidence of a legitimate update are dropped. Crucially, if the verifier is not confident — or the call fails or returns a malformed response — the document fails open and is retained, so Layer 3 can never remove a document it cannot judge, and verifier unavailability degrades it to a no-op rather than discarding evidence under uncertainty (this underlies the valid-response-only rows in Section 5 and Table 4). Two invocation modes exist: always-on (used for L3 ablation rows) and adaptive, invoked only when Layers 1 and 2 disagree.

The three layers are complementary in coverage rather than strictly additive: triggers that evade Layer 1 are typically caught by Layer 2’s prefix overlap check, and payloads that survive Layer 2 trigger contradictions in Layer 3. Composing all three does not strictly dominate every pairwise combination, however, since Layer 3 occasionally discards clean passages whose phrasing differs from the LLM’s prior.

We therefore recommend a single default: L1+L2 with Layer 3 in adaptive mode, invoked only where Layers 1 and 2 disagree. It attains the lowest black-box ASR ($3.0\%$ on NQ) at $\sim\!12$ ms/query, and adaptive invocation recovers most of Layer 3’s white-box and architecture-aware robustness without paying always-on latency on every query. Deployments with a known threat profile should override this: white-box-dominant settings use L1+L3 always-on ($27.8\%$ white-box ASR),and settings expecting trigger-paraphrasing adversaries should enable Layer 3 always-on instead (Section 6.5 gives the resulting ASR, accuracy, and latency). Abstract numbers correspond to the L1+L2 default unless stated. TRIS is a drop-in Python module wrapping the retrieval call: model-agnostic, batched-cache friendly, falling back to unfiltered $\mathcal{D}_{k}$ with a logged warning when all documents are filtered.

Unless a table caption states otherwise, each cell aggregates $10$ iterations of $M=10$ queries ($100$ trials). The reduced-scale tables — Tables 2, 3, 7, and 9 — print their own $n$ in the caption and are single runs at that scale, not $10$-iteration aggregates; differences of a few points in those tables should not be read as resolved. Within each iteration we re-sample queries, re-generate poisons, and re-run the pipeline, capturing variance from poison generation, retrieval order, and LLM decoding (despite temperature $0$, GPT-3.5-turbo shows residual output variation). Pairwise differences $>\!3$ points are stable across re-runs; smaller ones are within noise. Clean Recall@5 of $0.180$ reflects our $100$-query Wikipedia subset.

Table 1 reports clean accuracy and ASR across all three datasets under the black-box LM-targeted attack. Vanilla RAG is catastrophically vulnerable: ASR ranges from $64.0\%$ (MS-MARCO) to $87.0\%$ (HotpotQA), and clean accuracy collapses from $74$–$82\%$ to $13$–$33\%$ under attack. TRIS restores robustness across all three datasets: ASR drops to $3.0\%$ on NQ, $14.0\%$ on HotpotQA, and $4.0\%$ on MS-MARCO, while CleanAcc recovers by $41$, $45$, and $44$ points respectively over the attacked system and matches or exceeds every baseline on all three datasets. The residual gap to the no-attack ceiling is $2$ points on NQ and $6$ on MS-MARCO, but $16$ on HotpotQA, where Layer 3 over-filters multi-hop evidence.

Table 1 fixes each baseline at one configuration, but both expose a tuning knob that moves them along an ASR–utility frontier, so a single-point comparison can flatter either system. Table 2 sweeps RobustRAG’s threshold $\alpha$, and Table 3 re-evaluates TrustRAG with its LLM-consistency check enabled. Both are reduced-scale ($n=30$): enough to locate a frontier, not to resolve a few points.

Tuned fairly, RobustRAG is far stronger than Table 1 suggests. At $\alpha=0.2$ it reaches $6.7\%$/$36.7\%$ (ASR/CleanAcc) on NQ and $3.3\%$/$66.7\%$ on HotpotQA — competitive with TRIS on NQ and better on both axes on HotpotQA. We therefore do not claim ASR dominance over a fairly-tuned RobustRAG. Two caveats cut in opposite directions: these are $n=30$ against TRIS’s $n=100$, and our RobustRAG uses heuristic rather than LLM-based keyword extraction (Section 5).

What does separate the systems at comparable ASR is verification cost: TRIS’s L1+L2 default needs one MiniLM pass and one generation call ($\sim\!0.35$ s/query), against RobustRAG’s per-document isolation ($31$–$36$ calls/query) — a $\sim\!36\times$ reduction. This is regime-specific, not a uniform win: under paraphrased triggers TRIS needs Layer 3, whose cost returns to that range (Section 6.5).

TrustRAG’s optional LLM-consistency check — its closest analogue to Layer 3 — is likewise disabled in Table 1. Enabled, under the adaptive attacks of Section 6.5, the two systems are indistinguishable at $n=30$: every margin is one query wide, with TrustRAG marginally ahead on trigger-paraphrased poisons (Table 3).

We therefore withdraw any claim of an adaptive-ASR advantage over a fairly-configured TrustRAG, and rest the comparison on architecture instead: a dedicated structural layer, an explicit fail-open verdict in Layer 3, and a modular composition whose layers toggle independently at known cost.

A further ablation isolates how much Layer 1’s independent embedding geometry contributes. Swapping only that space — MiniLM versus the retriever’s own Contriever, $n=100$, both datasets — leaves ASR statistically unchanged (NQ $32.0\%$ vs. $33.0\%$, $p=0.88$; HotpotQA $44.0\%$ vs. $53.0\%$, $p=0.20$), whereas Layer 2 accounts for the large effect ($p<10^{-7}$; Appendix B.3). We report this as a negative result about our own design rationale: the independent geometry is a robustness property — it drops the assumption that poisons form a separable cluster in the retriever’s space — not the source of the measured gain.

To understand how the Sieve works, Table 6 (Appendix B.1) reports retriever-level metrics on NQ. Vanilla poisoning eliminates Recall@5 and drives PoisonedMRR to its maximum of $1.000$ — a poison at rank 1 for every query. The Sieve fully inverts this: Recall@5 and Clean MRR return to clean-corpus baselines and PoisonedMRR falls to $0.000$. Recall@50 is unchanged throughout, confirming that poisons remain in the candidate pool but no longer reach the visible context: the Sieve reorders rather than discards. TrustRAG achieves the same retriever-side outcome on NQ, consistent with its strong NQ end-to-end numbers; the Sieve’s advantage emerges downstream on harder datasets.

Table 4 isolates the contribution of each layer on NQ under both black-box and white-box attacks. Three findings stand out.

Section 3.2 defines four levels of attacker adaptivity. This section empirically evaluates Level 1 (architecture-aware); Levels 2 (judge-aware) and 3 (full white-box) remain conceptual only, and implementing them is planned future work.

Varying poisons per query from $1$ to $5$ on NQ (Table 8, Appendix B.4), baseline ASR climbs from $0\%$ to $66\%$ while the Sieve holds below $10\%$; extending to $6$–$20$ poisons ($n=25$; Table 9) leaves TRIS ASR flat at $4.0\%$. Because Layers 1 and 2 filter independently rather than by majority vote, denser poisoning cannot tip the defense the way it could tip a voting aggregator. Full-scale replication remains future work (Section Limitations).

Sweeping retrieval depth $k\in\{5,10,20,50\}$ against cluster count $C\in\{2,3,5\}$ (Table 10, Appendix B.5), the defense is unstable at $k=5$ (ASR $85$–$93\%$, clean accuracy $9$–$17\%$) because K-Means partitions on too few candidates are unreliable. From $k=10$ onward ASR stabilizes at $3$–$4\%$ and clean accuracy rises monotonically to $78\%$ at $k=50$; $C=3$ is our default, and $C=5$ performs comparably on this sweep — we fix $C=3$ as the more conservative setting.

Layer 2 adds $\sim\!8$ ms/query and Layer 1 $\sim\!13$ ms; Layer 3 dominates at $\sim\!2.9$ s/query, making always-on ($\sim\!15$ s) acceptable for high-risk queries but not high-throughput search — under live retrieval its added cost is similar ($\sim\!16$–$19$ s/query; Section 6.5). Appendix C traces one successful defense and one failure, showing where the residual HotpotQA ASR originates.