Module: SDD-B09 — Prompt Injection Detection Models Diagram count: 5 Tool: Mermaid (primary). Each diagram validated in Mermaid Live Editor.
Type: Comparison / architecture Purpose: The foundational visual. Prompt-injection detection models fall into two families with distinct tradeoffs. Family 1 (dedicated classifier) is a small fine-tuned model — fast, cheap, fixed decision boundary, ages with its training corpus. Family 2 (secondary-LLM-as-detector) is a second LLM prompted to judge — flexible, expensive, and itself injectable. The choice is the first design decision a deployer makes, and it determines the latency, cost, false-positive, and evasion profile of the defense. Reading the diagram: Two parallel lanes. The untrusted content enters both. The dedicated classifier returns a fast binary verdict; the secondary LLM returns a reasoned verdict at higher cost. The teal callout marks the load-bearing difference: the dedicated classifier's boundary is fixed but probeable; the secondary LLM is flexible but carries the full injection surface of any LLM.
flowchart LR
UC["UNTRUSTED CONTENT<br/>retrieved doc · email · web page · tool result"]:::warn
subgraph F1["FAMILY 1 — DEDICATED CLASSIFIER"]
direction TB
C1["DeBERTa / RoBERTa fine-tune<br/>Llama Prompt Guard (86M / 22M)<br/>ProtectAI · Lakera Guard"]
C2["Binary verdict: injection? Y/N<br/>fast (~19-600ms) · cheap · tunable threshold"]
C3["FIXED DECISION BOUNDARY<br/>ages with training corpus<br/>probeable by adaptive attacker"]:::danger
C1 --> C2 --> C3
end
subgraph F2["FAMILY 2 — SECONDARY-LLM-AS-DETECTOR"]
direction TB
L1["A second LLM (cheaper than primary)<br/>prompted: 'does this contain<br/>overriding instructions?'"]
L2["Reasoned verdict + explanation<br/>flexible · re-promptable · feeds human review"]
L3["ITSELF AN LLM — full injection surface<br/>the content it checks can inject IT<br/>constrain output · no tool access"]:::danger
L1 --> L2 --> L3
end
UC --> F1
UC --> F2
F1 --> VERDICT["VERDICT: quarantine / sanitize / allow"]
F2 --> VERDICT
classDef warn fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
classDef danger fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
style UC fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
style F1 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style F2 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style C1 fill:#101018,stroke:#5eead4,color:#e4e4e8
style C2 fill:#101018,stroke:#5eead4,color:#e4e4e8
style L1 fill:#101018,stroke:#5eead4,color:#e4e4e8
style L2 fill:#101018,stroke:#5eead4,color:#e4e4e8
style VERDICT fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
Note: The dedicated classifier is the default choice for high-volume, low-latency gates; the secondary LLM is the choice for ambiguous, low-volume review where reasoning adds value. Most production deployments use the dedicated classifier as the primary gate and the secondary LLM as a second-pass reviewer for flagged-ambiguous content. Neither is a boundary; both are layers with residuals.
Type: Comparison / overlay Purpose: The most misunderstood property of detection models. Vendors report 95–99%+ benchmark accuracy; real-world, false-positive-constrained, adversarially-adapted accuracy is 84–90%. The gap is not a measurement artifact — it is an adversarial property with three causes: distribution shift (benchmarks are in-distribution; attackers are out-of-distribution), the false-positive tax (real traffic is 99% benign; a 1% FPR blocks one in a hundred legitimate inputs), and adversarial adaptation (the detector's surface is probed and learned). The number you operate on is the last. Reading the diagram: Three stacked bands. The top (benchmark) is the vendor number. The middle (deployment) is the number at a tolerable false-positive ceiling. The bottom (adversarial) is the number against an adaptive, out-of-distribution attacker — the one the red-team must measure and the one no model card reports.
flowchart TB
subgraph BENCH["BENCHMARK ACCURACY — the vendor number"]
B1["In-distribution test set<br/>95% to 99%+ detection"]
B2["Llama Prompt Guard 2: AUC ~0.998<br/>ProtectAI: 95-99%+ on held-out set<br/>Instruction-detection (arXiv:2505.06311): 99.6% in-domain"]
end
subgraph DEPLOY["DEPLOYMENT ACCURACY — at a tolerable false-positive ceiling"]
D1["Real traffic, real FPR<br/>84% to 90% detection"]
D2["ProtectAI ~90% in independent testing<br/>Prompt Guard: 26.7% detection at 50% FPR in one eval<br/>ProtectAI blocks ~1 in 3 legit users at aggressive threshold"]
end
subgraph ADV["ADVERSARIAL ACCURACY — out-of-distribution, adaptive attacker"]
A1["The number that matters<br/>lower still — and decays over time"]
A2["Out-of-domain: 96.9% (curated, not adaptive)<br/>arXiv:2504.11168: high benchmark ≠ robustness<br/>Galileo: 56% of production LLMs successfully injected"]
end
BENCH -->|"distribution shift"| DEPLOY
DEPLOY -->|"false-positive tax + adversarial adaptation"| ADV
OPERATE["THE NUMBER YOU OPERATE ON<br/>out-of-distribution, FP-constrained,<br/>adversarially-adapted detection rate<br/>NO VENDOR PUTS THIS ON THE MODEL CARD"]:::danger
ADV --> OPERATE
classDef danger fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
style BENCH fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#e4e4e8
style DEPLOY fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
style ADV fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#e4e4e8
style B1 fill:#101018,stroke:#82e0aa,color:#e4e4e8
style B2 fill:#101018,stroke:#82e0aa,color:#e4e4e8
style D1 fill:#101018,stroke:#f0a868,color:#e4e4e8
style D2 fill:#101018,stroke:#f0a868,color:#e4e4e8
style A1 fill:#101018,stroke:#f08080,color:#e4e4e8
style A2 fill:#101018,stroke:#f08080,color:#e4e4e8
style OPERATE fill:#14141f,stroke:#f08080,stroke-width:2px,color:#f08080
Note: The 15-point gap between benchmark and deployment is the single most important number for a senior engineer evaluating a detection model. A detector that reports 99% benchmark accuracy and operates at 85% real-world accuracy is a useful layer; a detector whose deployer believes the 99% number is a deployment waiting to fall.
Type: Decision flow / adversarial model Purpose: The evasion dynamic has a nastier variant than simple detector evasion. An attacker crafting an indirect payload against a detector-defended agent faces two targets: the payload must (a) evade the detector (read as benign) AND (b) compromise the primary model (execute as injection). The dual constraint is a targeting signal — the detector's decision boundary is a fixed artifact the attacker can probe, and a payload engineered into the detector's false-negative region while still carrying the injection is the attacker's goal. The correlated case (shared model class) makes this easier. Reading the diagram: The payload enters and must pass two gates. The red path is the attacker's success (evades both). Note the correlation arrow: when detector and primary model share a model class, an evasion against one has elevated probability against the other — the independence assumption fails.
flowchart TB
PAYLOAD["ADVERSARIAL PAYLOAD<br/>crafted against the detector surface"]:::warn
PAYLOAD --> Q1{"Evades DETECTOR?<br/>(classified benign)"}
Q1 -->|"NO — flagged"| CAUGHT["CAUGHT<br/>quarantined / sanitized / refused"]:::good
Q1 -->|"YES — false negative"| Q2{"Compromises PRIMARY MODEL?<br/>(executes as injection)"}
Q2 -->|"NO — too weak"| USELESS["USELESS TO ATTACKER<br/>evaded detector but moved nothing"]:::good
Q2 -->|"YES — executes"| SUCCESS["ATTACKER SUCCESS<br/>dual-injection satisfied"]:::danger
CORR["CORRELATED CASE<br/>detector + primary share model class<br/>evasion against one → elevated prob. against other<br/>independence assumption FAILS"]:::danger
Q2 -.->|"shared Llama-family / tokenizer / corpus"| CORR
CORR -.->|"raises P(SUCCESS)"| SUCCESS
classDef warn fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
classDef danger fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
classDef good fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
style PAYLOAD fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
style Q1 fill:#101018,stroke:#5eead4,color:#e4e4e8
style Q2 fill:#101018,stroke:#5eead4,color:#e4e4e8
style CAUGHT fill:#101018,stroke:#82e0aa,color:#e4e4e8
style USELESS fill:#101018,stroke:#82e0aa,color:#e4e4e8
style SUCCESS fill:#101018,stroke:#f08080,color:#f08080
Note: The dual constraint helps the defender against unsophisticated attackers (a naive jailbreak often trips the detector). Against an adaptive attacker, the constraint is a targeting problem: probe the detector's boundary, find the false-negative region, and craft the payload to sit there while still moving the primary model. The correlated-shared-model-class case is the worst — the same evasion technique often works on both.
Type: Layered architecture Purpose: The detector is the highest-value single layer for indirect-injection defense, but it is a layer, not a boundary. This diagram shows the composition from B2's stack: deterministic layers (input/output rails, IronCurtain, harness scope gate) bound the worst case; model-based layers (detector Layer 4, primary model refusal) bound the volume. The only layers without an evasion surface are the deterministic ones. The end-to-end bypass rate is the product of the residuals, and the harness gate is the floor. Reading the diagram: Untrusted content enters at the top. Each layer either catches or passes the injection. The teal layers are deterministic (no evasion surface); the danger layers are model-based (evasion surface). The detector is Layer 4 — high value, but not the boundary. The harness scope gate at the bottom is the hard stop.
flowchart TB
UC["UNTRUSTED CONTENT"]:::warn
L1["LAYER 1-2 — INPUT RAILS + DETERMINISTIC BOUNDARY<br/>NeMo input rails (SDD-B08) · IronCurtain (SDD-B05)<br/>catches gross violations · bounds the worst case<br/>DETERMINISTIC — no evasion surface"]:::teal
L4["LAYER 4 — THE DETECTION MODEL (this deep-dive)<br/>dedicated classifier OR secondary-LLM-as-detector<br/>catches the BULK of indirect injections<br/>MODEL-BASED — evasion surface, residual exists"]:::danger
L5["LAYER 5 — OUTPUT RAILS + PRIMARY MODEL REFUSAL<br/>backstop if injection evades Layer 4<br/>MODEL-BASED — evasion surface, residual exists"]:::danger
HC["HARNESS CONTROLS — SCOPE GATE<br/>tool-call arg validation · evidence classifier<br/>even a compromised model cannot take the disallowed action<br/>DETERMINISTIC — the hard stop, the floor"]:::teal
UC --> L1
L1 -->|"passes"| L4
L4 -->|"misses (residual)"| L5
L5 -->|"model complies"| HC
HC -->|"disallowed action BLOCKED"| STOP["FLOOR — the action never happens"]:::good
L4 -.->|"quarantine / sanitize / refuse"| CAUGHT["CAUGHT AT LAYER 4"]:::good
classDef warn fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
classDef teal fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
classDef danger fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
classDef good fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
style UC fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
style STOP fill:#101018,stroke:#82e0aa,color:#82e0aa
style CAUGHT fill:#101018,stroke:#82e0aa,color:#82e0aa
Note: The math: if the detector catches 90% at a tolerable FPR, the deterministic boundary catches 50% of the remainder, output rails catch 60% of what reaches the model, and the harness gate is the hard stop — the end-to-end bypass is the product of the residuals, floored by the gate. No single layer is trusted; the composition bounds the residual. This is the only architecture that holds under an adaptive adversary, because every model-based layer has an evasion surface.
Type: Process / measurement flow Purpose: A detector is measured the way every defense layer in this course is measured: bypass rate over N attempts under fixed parameters, with the false-positive rate measured on legitimate traffic. The methodology has five steps, and the deliverable is a defensible number — not a vendor page claim. The out-of-distribution probe (step 1's held-out set) and the correlated-bypass test (step 5) are the two steps teams skip and the two that determine whether the measurement is honest. Reading the diagram: Five sequential steps. The teal steps are the standard measurement; the danger steps are the adversarial reality checks. The output is a measurement statement with in-distribution, out-of-distribution, false-positive, and end-to-end numbers.
flowchart TB
S1["STEP 1 — ASSEMBLE THE CORPUS<br/>in-distribution (techniques detector claims to catch)<br/>+ HELD-OUT out-of-distribution set (novel techniques)<br/>the held-out set is the number that matters"]:::danger
S2["STEP 2 — PER-DETECTOR BYPASS RATE<br/>false-negative rate at deployed threshold<br/>in-distribution rate vs out-of-distribution rate"]:::teal
S3["STEP 3 — FALSE-POSITIVE TAX<br/>run LEGITIMATE untrusted content (emails, code, docs)<br/>record FPR · set threshold where FPR is tolerable<br/>detection rate at THAT threshold = the real number"]:::danger
S4["STEP 4 — END-TO-END BYPASS RATE<br/>full stack: detector + primary + rails + harness<br/>fraction of injections → disallowed action<br/>THIS is whether the deployment is defensible"]:::teal
S5["STEP 5 — CORRELATED-BYPASS TEST<br/>if detector + primary share model class<br/>test whether detector-evasion also evades primary<br/>end-to-end rate > independence product = correlated"]:::danger
S1 --> S2 --> S3 --> S4 --> S5
S5 --> OUT["DEFENSIBLE MEASUREMENT<br/>'Detector L4: 91% in-dist / 74% OOD at 2% FPR<br/>end-to-end bypass 3.2% full stack / 5.8% correlated<br/>harness gate is the floor'"]:::good
classDef teal fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
classDef danger fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
classDef good fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
style S1 fill:#101018,stroke:#f08080,color:#e4e4e8
style S2 fill:#101018,stroke:#5eead4,color:#e4e4e8
style S3 fill:#101018,stroke:#f08080,color:#e4e4e8
style S4 fill:#101018,stroke:#5eead4,color:#e4e4e8
style S5 fill:#101018,stroke:#f08080,color:#e4e4e8
style OUT fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
Note: The two danger steps (out-of-distribution probe, correlated-bypass test) are the ones that distinguish an honest measurement from a vendor reproduction. A team that runs steps 1–4 on in-distribution traffic only will report a detection rate that bears no relationship to the rate an adaptive attacker achieves. The held-out set and the correlation test are where the real number lives.
# Diagrams — SDD-B09: Prompt Injection Detection Models
**Module**: SDD-B09 — Prompt Injection Detection Models
**Diagram count**: 5
**Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live).
---
## Diagram 1 — The Two Detector Families: Dedicated Classifier vs. Secondary-LLM-as-Detector
**Type**: Comparison / architecture
**Purpose**: The foundational visual. Prompt-injection detection models fall into two families with distinct tradeoffs. Family 1 (dedicated classifier) is a small fine-tuned model — fast, cheap, fixed decision boundary, ages with its training corpus. Family 2 (secondary-LLM-as-detector) is a second LLM prompted to judge — flexible, expensive, and itself injectable. The choice is the first design decision a deployer makes, and it determines the latency, cost, false-positive, and evasion profile of the defense.
**Reading the diagram**: Two parallel lanes. The untrusted content enters both. The dedicated classifier returns a fast binary verdict; the secondary LLM returns a reasoned verdict at higher cost. The teal callout marks the load-bearing difference: the dedicated classifier's boundary is fixed but probeable; the secondary LLM is flexible but carries the full injection surface of any LLM.
```mermaid
flowchart LR
UC["UNTRUSTED CONTENT<br/>retrieved doc · email · web page · tool result"]:::warn
subgraph F1["FAMILY 1 — DEDICATED CLASSIFIER"]
direction TB
C1["DeBERTa / RoBERTa fine-tune<br/>Llama Prompt Guard (86M / 22M)<br/>ProtectAI · Lakera Guard"]
C2["Binary verdict: injection? Y/N<br/>fast (~19-600ms) · cheap · tunable threshold"]
C3["FIXED DECISION BOUNDARY<br/>ages with training corpus<br/>probeable by adaptive attacker"]:::danger
C1 --> C2 --> C3
end
subgraph F2["FAMILY 2 — SECONDARY-LLM-AS-DETECTOR"]
direction TB
L1["A second LLM (cheaper than primary)<br/>prompted: 'does this contain<br/>overriding instructions?'"]
L2["Reasoned verdict + explanation<br/>flexible · re-promptable · feeds human review"]
L3["ITSELF AN LLM — full injection surface<br/>the content it checks can inject IT<br/>constrain output · no tool access"]:::danger
L1 --> L2 --> L3
end
UC --> F1
UC --> F2
F1 --> VERDICT["VERDICT: quarantine / sanitize / allow"]
F2 --> VERDICT
classDef warn fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
classDef danger fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
style UC fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
style F1 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style F2 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style C1 fill:#101018,stroke:#5eead4,color:#e4e4e8
style C2 fill:#101018,stroke:#5eead4,color:#e4e4e8
style L1 fill:#101018,stroke:#5eead4,color:#e4e4e8
style L2 fill:#101018,stroke:#5eead4,color:#e4e4e8
style VERDICT fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
```
> **Note**: The dedicated classifier is the default choice for high-volume, low-latency gates; the secondary LLM is the choice for ambiguous, low-volume review where reasoning adds value. Most production deployments use the dedicated classifier as the primary gate and the secondary LLM as a second-pass reviewer for flagged-ambiguous content. Neither is a boundary; both are layers with residuals.
---
## Diagram 2 — The Benchmark-vs-Deployment Gap
**Type**: Comparison / overlay
**Purpose**: The most misunderstood property of detection models. Vendors report 95–99%+ benchmark accuracy; real-world, false-positive-constrained, adversarially-adapted accuracy is 84–90%. The gap is not a measurement artifact — it is an adversarial property with three causes: distribution shift (benchmarks are in-distribution; attackers are out-of-distribution), the false-positive tax (real traffic is 99% benign; a 1% FPR blocks one in a hundred legitimate inputs), and adversarial adaptation (the detector's surface is probed and learned). The number you operate on is the last.
**Reading the diagram**: Three stacked bands. The top (benchmark) is the vendor number. The middle (deployment) is the number at a tolerable false-positive ceiling. The bottom (adversarial) is the number against an adaptive, out-of-distribution attacker — the one the red-team must measure and the one no model card reports.
```mermaid
flowchart TB
subgraph BENCH["BENCHMARK ACCURACY — the vendor number"]
B1["In-distribution test set<br/>95% to 99%+ detection"]
B2["Llama Prompt Guard 2: AUC ~0.998<br/>ProtectAI: 95-99%+ on held-out set<br/>Instruction-detection (arXiv:2505.06311): 99.6% in-domain"]
end
subgraph DEPLOY["DEPLOYMENT ACCURACY — at a tolerable false-positive ceiling"]
D1["Real traffic, real FPR<br/>84% to 90% detection"]
D2["ProtectAI ~90% in independent testing<br/>Prompt Guard: 26.7% detection at 50% FPR in one eval<br/>ProtectAI blocks ~1 in 3 legit users at aggressive threshold"]
end
subgraph ADV["ADVERSARIAL ACCURACY — out-of-distribution, adaptive attacker"]
A1["The number that matters<br/>lower still — and decays over time"]
A2["Out-of-domain: 96.9% (curated, not adaptive)<br/>arXiv:2504.11168: high benchmark ≠ robustness<br/>Galileo: 56% of production LLMs successfully injected"]
end
BENCH -->|"distribution shift"| DEPLOY
DEPLOY -->|"false-positive tax + adversarial adaptation"| ADV
OPERATE["THE NUMBER YOU OPERATE ON<br/>out-of-distribution, FP-constrained,<br/>adversarially-adapted detection rate<br/>NO VENDOR PUTS THIS ON THE MODEL CARD"]:::danger
ADV --> OPERATE
classDef danger fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
style BENCH fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#e4e4e8
style DEPLOY fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
style ADV fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#e4e4e8
style B1 fill:#101018,stroke:#82e0aa,color:#e4e4e8
style B2 fill:#101018,stroke:#82e0aa,color:#e4e4e8
style D1 fill:#101018,stroke:#f0a868,color:#e4e4e8
style D2 fill:#101018,stroke:#f0a868,color:#e4e4e8
style A1 fill:#101018,stroke:#f08080,color:#e4e4e8
style A2 fill:#101018,stroke:#f08080,color:#e4e4e8
style OPERATE fill:#14141f,stroke:#f08080,stroke-width:2px,color:#f08080
```
> **Note**: The 15-point gap between benchmark and deployment is the single most important number for a senior engineer evaluating a detection model. A detector that reports 99% benchmark accuracy and operates at 85% real-world accuracy is a useful layer; a detector whose deployer believes the 99% number is a deployment waiting to fall.
---
## Diagram 3 — The Dual-Injection Problem
**Type**: Decision flow / adversarial model
**Purpose**: The evasion dynamic has a nastier variant than simple detector evasion. An attacker crafting an indirect payload against a detector-defended agent faces two targets: the payload must (a) evade the detector (read as benign) AND (b) compromise the primary model (execute as injection). The dual constraint is a targeting signal — the detector's decision boundary is a fixed artifact the attacker can probe, and a payload engineered into the detector's false-negative region while still carrying the injection is the attacker's goal. The correlated case (shared model class) makes this easier.
**Reading the diagram**: The payload enters and must pass two gates. The red path is the attacker's success (evades both). Note the correlation arrow: when detector and primary model share a model class, an evasion against one has elevated probability against the other — the independence assumption fails.
```mermaid
flowchart TB
PAYLOAD["ADVERSARIAL PAYLOAD<br/>crafted against the detector surface"]:::warn
PAYLOAD --> Q1{"Evades DETECTOR?<br/>(classified benign)"}
Q1 -->|"NO — flagged"| CAUGHT["CAUGHT<br/>quarantined / sanitized / refused"]:::good
Q1 -->|"YES — false negative"| Q2{"Compromises PRIMARY MODEL?<br/>(executes as injection)"}
Q2 -->|"NO — too weak"| USELESS["USELESS TO ATTACKER<br/>evaded detector but moved nothing"]:::good
Q2 -->|"YES — executes"| SUCCESS["ATTACKER SUCCESS<br/>dual-injection satisfied"]:::danger
CORR["CORRELATED CASE<br/>detector + primary share model class<br/>evasion against one → elevated prob. against other<br/>independence assumption FAILS"]:::danger
Q2 -.->|"shared Llama-family / tokenizer / corpus"| CORR
CORR -.->|"raises P(SUCCESS)"| SUCCESS
classDef warn fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
classDef danger fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
classDef good fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
style PAYLOAD fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
style Q1 fill:#101018,stroke:#5eead4,color:#e4e4e8
style Q2 fill:#101018,stroke:#5eead4,color:#e4e4e8
style CAUGHT fill:#101018,stroke:#82e0aa,color:#e4e4e8
style USELESS fill:#101018,stroke:#82e0aa,color:#e4e4e8
style SUCCESS fill:#101018,stroke:#f08080,color:#f08080
```
> **Note**: The dual constraint helps the defender against unsophisticated attackers (a naive jailbreak often trips the detector). Against an adaptive attacker, the constraint is a targeting problem: probe the detector's boundary, find the false-negative region, and craft the payload to sit there while still moving the primary model. The correlated-shared-model-class case is the worst — the same evasion technique often works on both.
---
## Diagram 4 — Defense-in-Depth: The Detector as Layer 4
**Type**: Layered architecture
**Purpose**: The detector is the highest-value single layer for indirect-injection defense, but it is a layer, not a boundary. This diagram shows the composition from B2's stack: deterministic layers (input/output rails, IronCurtain, harness scope gate) bound the worst case; model-based layers (detector Layer 4, primary model refusal) bound the volume. The only layers without an evasion surface are the deterministic ones. The end-to-end bypass rate is the product of the residuals, and the harness gate is the floor.
**Reading the diagram**: Untrusted content enters at the top. Each layer either catches or passes the injection. The teal layers are deterministic (no evasion surface); the danger layers are model-based (evasion surface). The detector is Layer 4 — high value, but not the boundary. The harness scope gate at the bottom is the hard stop.
```mermaid
flowchart TB
UC["UNTRUSTED CONTENT"]:::warn
L1["LAYER 1-2 — INPUT RAILS + DETERMINISTIC BOUNDARY<br/>NeMo input rails (SDD-B08) · IronCurtain (SDD-B05)<br/>catches gross violations · bounds the worst case<br/>DETERMINISTIC — no evasion surface"]:::teal
L4["LAYER 4 — THE DETECTION MODEL (this deep-dive)<br/>dedicated classifier OR secondary-LLM-as-detector<br/>catches the BULK of indirect injections<br/>MODEL-BASED — evasion surface, residual exists"]:::danger
L5["LAYER 5 — OUTPUT RAILS + PRIMARY MODEL REFUSAL<br/>backstop if injection evades Layer 4<br/>MODEL-BASED — evasion surface, residual exists"]:::danger
HC["HARNESS CONTROLS — SCOPE GATE<br/>tool-call arg validation · evidence classifier<br/>even a compromised model cannot take the disallowed action<br/>DETERMINISTIC — the hard stop, the floor"]:::teal
UC --> L1
L1 -->|"passes"| L4
L4 -->|"misses (residual)"| L5
L5 -->|"model complies"| HC
HC -->|"disallowed action BLOCKED"| STOP["FLOOR — the action never happens"]:::good
L4 -.->|"quarantine / sanitize / refuse"| CAUGHT["CAUGHT AT LAYER 4"]:::good
classDef warn fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
classDef teal fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
classDef danger fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
classDef good fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
style UC fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
style STOP fill:#101018,stroke:#82e0aa,color:#82e0aa
style CAUGHT fill:#101018,stroke:#82e0aa,color:#82e0aa
```
> **Note**: The math: if the detector catches 90% at a tolerable FPR, the deterministic boundary catches 50% of the remainder, output rails catch 60% of what reaches the model, and the harness gate is the hard stop — the end-to-end bypass is the product of the residuals, floored by the gate. No single layer is trusted; the composition bounds the residual. This is the only architecture that holds under an adaptive adversary, because every model-based layer has an evasion surface.
---
## Diagram 5 — The Measurement Methodology
**Type**: Process / measurement flow
**Purpose**: A detector is measured the way every defense layer in this course is measured: bypass rate over N attempts under fixed parameters, with the false-positive rate measured on legitimate traffic. The methodology has five steps, and the deliverable is a defensible number — not a vendor page claim. The out-of-distribution probe (step 1's held-out set) and the correlated-bypass test (step 5) are the two steps teams skip and the two that determine whether the measurement is honest.
**Reading the diagram**: Five sequential steps. The teal steps are the standard measurement; the danger steps are the adversarial reality checks. The output is a measurement statement with in-distribution, out-of-distribution, false-positive, and end-to-end numbers.
```mermaid
flowchart TB
S1["STEP 1 — ASSEMBLE THE CORPUS<br/>in-distribution (techniques detector claims to catch)<br/>+ HELD-OUT out-of-distribution set (novel techniques)<br/>the held-out set is the number that matters"]:::danger
S2["STEP 2 — PER-DETECTOR BYPASS RATE<br/>false-negative rate at deployed threshold<br/>in-distribution rate vs out-of-distribution rate"]:::teal
S3["STEP 3 — FALSE-POSITIVE TAX<br/>run LEGITIMATE untrusted content (emails, code, docs)<br/>record FPR · set threshold where FPR is tolerable<br/>detection rate at THAT threshold = the real number"]:::danger
S4["STEP 4 — END-TO-END BYPASS RATE<br/>full stack: detector + primary + rails + harness<br/>fraction of injections → disallowed action<br/>THIS is whether the deployment is defensible"]:::teal
S5["STEP 5 — CORRELATED-BYPASS TEST<br/>if detector + primary share model class<br/>test whether detector-evasion also evades primary<br/>end-to-end rate > independence product = correlated"]:::danger
S1 --> S2 --> S3 --> S4 --> S5
S5 --> OUT["DEFENSIBLE MEASUREMENT<br/>'Detector L4: 91% in-dist / 74% OOD at 2% FPR<br/>end-to-end bypass 3.2% full stack / 5.8% correlated<br/>harness gate is the floor'"]:::good
classDef teal fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
classDef danger fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
classDef good fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
style S1 fill:#101018,stroke:#f08080,color:#e4e4e8
style S2 fill:#101018,stroke:#5eead4,color:#e4e4e8
style S3 fill:#101018,stroke:#f08080,color:#e4e4e8
style S4 fill:#101018,stroke:#5eead4,color:#e4e4e8
style S5 fill:#101018,stroke:#f08080,color:#e4e4e8
style OUT fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
```
> **Note**: The two danger steps (out-of-distribution probe, correlated-bypass test) are the ones that distinguish an honest measurement from a vendor reproduction. A team that runs steps 1–4 on in-distribution traffic only will report a detection rate that bears no relationship to the rate an adaptive attacker achieves. The held-out set and the correlation test are where the real number lives.