{ "module": "SDD-B09 — Prompt Injection Detection Models", "course": "2B — Securing & Attacking Harnesses and LLMs", "version": "1.0.0", "duration_minutes": 35, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What are the two families of prompt-injection detection models?", "options": [ "Regex filters and API rate limiters.", "Dedicated classifier models (small fine-tunes like DeBERTa/RoBERTa, Llama Prompt Guard, ProtectAI) and the secondary-LLM-as-detector approach (a second LLM prompted to judge content).", "Input rails and output rails.", "Deterministic boundaries and harness scope gates." ], "answer_index": 1, "rationale": "The two families are the first design decision a deployer makes. Family 1 (dedicated classifier) is fast, cheap, tunable, with a fixed decision boundary that ages with its corpus. Family 2 (secondary-LLM-as-detector) is flexible, re-promptable, expensive, and itself an LLM carrying the full injection surface. The default deployment: dedicated classifier as the high-volume gate, secondary LLM as a second-pass reviewer for ambiguous content." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "What signal do prompt-injection detection models classify?", "options": [ "Whether the input contains profanity.", "Overriding instructions — text that attempts to change the model's task, role, or constraints (direct, indirect, encoded, multi-step injections).", "Whether the input is in English.", "Whether the input exceeds a token length threshold." ], "answer_index": 1, "rationale": "The detection target is overriding instructions — text that changes the model's task, role, or constraints. This is subtler than 'is this malicious.' A classifier is trained on labeled injection/benign datasets and is only as good as the diversity of injections in its training corpus, which ages — a 2026 technique is not in a 2024-trained model's distribution. This is why out-of-domain accuracy determines real-world performance." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "Which accuracy number should a senior engineer operate on when evaluating a detection model?", "options": [ "The benchmark AUC reported on the model card (95–99%+).", "The in-distribution test-set accuracy.", "The out-of-distribution, false-positive-constrained, adversarially-adapted detection rate — the lowest number, which no vendor reports.", "The vendor's marketing-quoted detection percentage." ], "answer_index": 2, "rationale": "Benchmark accuracy is 95–99%+ (in-distribution). Deployment accuracy at a tolerable false-positive ceiling is 84–90%. Adversarial, out-of-distribution accuracy is lower still and decays as the adversary adapts. The number you operate on is the last — out-of-distribution, false-positive-constrained, adversarially-adapted — and it is the one vendors report least. No vendor puts it on the model card; the red-team must measure it." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "A vendor pitches a dedicated classifier quoting '98%+ detection at sub-50ms latency' from its model card. Your client wants to deploy it as the sole injection defense for their RAG agent. What is the correct assessment?", "options": [ "98% is sufficient — deploy it as the sole boundary; no further layers needed.", "The 98% is a benchmark (in-distribution) number; real-world, FP-constrained, adversarially-adapted accuracy is 84–90% and decays. Deploying it as the SOLE boundary means a single evasion compromises the agent. It should be Layer 4, composed with a deterministic boundary (IronCurtain), output rails, and the harness scope gate.", "Reject the classifier entirely — dedicated classifiers are useless against adaptive attackers.", "Deploy it but disable the threshold so it blocks nothing, avoiding false positives." ], "answer_index": 1, "rationale": "The model-card number is benchmark/in-distribution. The real number is 10–15 points lower and adversarially decays. Deploying a model-based detector as the sole boundary is the core anti-pattern — a single evasion (which an adaptive attacker will find by probing the fixed decision boundary) compromises the agent with no backstop. The detector is Layer 4; it must be composed with deterministic layers (IronCurtain, harness scope gate) that have no evasion surface." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "Your detection model reports 99% recall at 1% false-positive rate. Your RAG pipeline processes 10,000 legitimate documents per day, ~99% benign. How many legitimate documents per day does the detector flag as injection at 1% FPR, and what is the operational consequence?", "options": [ "0 — 1% FPR is negligible at this volume.", "Roughly 99 legitimate documents per day flagged as injection (1% of ~9,900 benign). The operational consequence: the business must either tolerate the false-positive tax (quarantining/sanitizing ~99 legit docs/day) or raise the threshold, which drops recall and the real detection rate.", "10,000 — everything is flagged.", "1 document per day, which is acceptable without measurement." ], "answer_index": 1, "rationale": "At 1% FPR on ~9,900 benign documents, ~99 legitimate documents are falsely flagged per day. This is the false-positive tax: the cost of running the detector on real traffic where the vast majority is benign. A deployment that cannot tolerate this tax raises the threshold, which drops recall, which drops the real detection rate. The operational accuracy is the accuracy at the FPR the business can absorb — invariably lower than the benchmark." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You deploy a Llama Prompt Guard classifier as the detector and a Llama-family model as the primary agent. Per-layer bypass rates multiply to 2%. Is 2% your expected end-to-end bypass rate?", "options": [ "Yes — the product of per-layer rates is the end-to-end rate, always.", "No — likely HIGHER than 2%. When detector and primary model share a model class (both Llama-family, shared tokenizer/corpus), an evasion against the detector has elevated probability against the primary. The independence assumption fails; the correlated-bypass test may show the end-to-end rate exceeds the product.", "No — likely LOWER than 2%, because two Llama models are stronger together.", "It is impossible to determine without retraining both models." ], "answer_index": 1, "rationale": "The independence assumption (overall bypass = product of per-layer rates) fails when layers share a model class. An evasion technique against the Llama-family detector has an elevated probability of also evading the Llama-family primary's refusal — shared architecture, tokenizer quirks, training-corpus overlap. The correlated-bypass test (measurement step 5) directly tests this: if end-to-end exceeds the independence product, the layers are correlated and 2% is optimistic." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You use a raw LLM as your secondary detector, prompted with 'Does this text contain overriding instructions? Explain your reasoning.' An attacker submits a retrieved document containing an injection targeting the detector LLM itself. What is the failure mode and the control?", "options": [ "No failure — the detector LLM is separate from the primary, so it is safe.", "Failure: the untrusted content injects the DETECTOR LLM, causing it to output a false 'no injection' verdict, passing the payload to the primary. Control: constrain the detector's output to a structured schema (no free-text reasoning the content can manipulate), no tool access, no context carryover, fixed detection prompt not addressable by the content, and measure the detector's own evasion rate.", "Failure: the detector becomes faster. Control: add more memory.", "No control exists — secondary-LLM detectors should never be used." ], "answer_index": 1, "rationale": "The secondary-LLM-as-detector is itself an LLM carrying the full injection surface. A payload sophisticated enough to compromise the primary may also compromise the detector — the dual-injection problem where the detector is the first target. The control is to constrain the detector's output (structured verdict, not free text), remove tool access and context carryover, fix the detection prompt outside the evaluated content, and measure the detector's evasion rate as a model with its own surface." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "You measure your detector: 95% detection on in-distribution injections, 74% on a held-out out-of-distribution set, at 2% FPR. End-to-end bypass with the full stack is 3.2%, rising to 5.8% under correlated-bypass testing. Which number do you report to the client as the deployment's residual risk?", "options": [ "95% — the in-distribution detection rate.", "5% (100% - 95%) — the in-distribution miss rate.", "5.8% — the correlated end-to-end bypass rate, because it reflects the adversarial, out-of-distribution, correlation-aware reality an adaptive attacker achieves. The harness scope gate is the floor beneath it.", "2% — the false-positive rate, since that is the operational constraint." ], "answer_index": 2, "rationale": "The defensible residual-risk number is the one that reflects the adversarial reality: out-of-distribution, false-positive-constrained, end-to-end, AND correlated-bypass. 5.8% is the rate an adaptive attacker with knowledge of the shared model class achieves against the full stack. The in-distribution 95% is the ceiling, not the floor. The harness scope gate is the deterministic floor beneath the 5.8% — it bounds the worst case regardless of detector evasion." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "Your team deployed a ProtectAI DeBERTa detector 14 months ago and has not re-evaluated it. New evasion techniques have been published since. What is the failure mode and the control?", "options": [ "No failure — the detector's decision boundary is fixed and does not change.", "Adversarial adaptation: the detector's surface has been probed and learned; evasion techniques published since deployment are out-of-distribution for the 14-month-old checkpoint; effective accuracy has decayed. Control: re-measure the out-of-distribution bypass rate on a cadence (quarterly or after each new public evasion technique); retrain, re-prompt, swap architecture, or add to ensemble when the rate exceeds tolerance. Treat the detector as a decaying asset.", "The detector has improved on its own through use. No action needed.", "Replace the detector with a regex filter, which does not decay." ], "answer_index": 1, "rationale": "A detector is a decaying asset. Its decision boundary is fixed at training time; the adversarial ecosystem learns it over time, and new evasion techniques are out-of-distribution for an old checkpoint. The control is cadenced re-measurement of the out-of-distribution bypass rate and updating the detector (retrain, re-prompt, swap, ensemble) when the rate exceeds tolerance. A static detector's real bypass rate 14 months post-deployment is materially higher than at deployment." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does external placement of the detection model (running it outside the agent's process) NOT protect against evasion, and what DOES it protect against?", "options": [ "It protects against both evasion and disablement equally.", "External placement protects against DISABLE — a compromised agent cannot reach the detector process to turn it off (the Course 1 Module 0.2 principle). It does NOT protect against EVADE — an adversarial input crafted against the detector's decision boundary causes the detector to run but classify the payload as benign. The detector is a model; external enforcement stops disablement, not evasion.", "It protects against evasion but not disablement.", "It protects against neither — placement is irrelevant to detector security." ], "answer_index": 1, "rationale": "This is the load-bearing distinction. External evaluation (the governance-beneath-the-agent principle) ensures a compromised agent cannot disable or modify the detector — the detector runs regardless of agent state. But the detector is still a model with a decision boundary, and an input engineered against that boundary is classified as benign while the detector runs normally. External enforcement stops DISABLE; it does not stop EVADE. This is SDD-B08's insight restated at the detector layer." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "In the dual-injection problem, why is the dual constraint (evade detector AND compromise primary) a targeting signal for adaptive attackers rather than purely a defender advantage?", "options": [ "It is purely a defender advantage — two constraints are always better than one.", "The detector's decision boundary is a FIXED, probeable artifact. An adaptive attacker can query the detector (directly or via side channel), map its false-negative region, and craft a payload that sits in that region while still carrying the injection that compromises the primary. The detector's existence as a model publishes its attack surface (model card, architecture, latency, probing behavior), turning the constraint into a solvable targeting problem.", "The dual constraint makes attacks impossible, so it is not a targeting signal.", "The dual constraint only matters for secondary-LLM detectors, not dedicated classifiers." ], "answer_index": 1, "rationale": "The dual constraint helps against naive attackers (a jailbreak that works often trips the detector). Against adaptive attackers, the detector's fixed decision boundary is a target: probe it, map the false-negative region, and craft the payload to sit there while still moving the primary model. The detector publishes its attack surface through its model card, architecture, latency profile, and probing behavior. The constraint is a targeting problem, not an impenetrable barrier." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "Which layers in the defense-in-depth stack have NO evasion surface, and why is this the architectural property that makes the stack hold under an adaptive adversary?", "options": [ "The detection model and the primary model's refusal training — they are the strongest layers.", "The deterministic layers: IronCurtain's policy boundary (SDD-B05) and the harness scope gate. They enforce rules via code/logic, not model classification, so there is no decision boundary to probe or evade. Every model-based layer (detector, rails, primary refusal) has an evasion surface; the deterministic layers bound the worst case and are the floor. This is why the composition holds — the product of residuals is floored by a layer with no residual.", "The input rails and output rails — they are externally evaluated.", "The evidence classifier and the retrieval store — they are data, not models." ], "answer_index": 1, "rationale": "The only layers without an evasion surface are the deterministic ones: IronCurtain (a policy enforced via code/logic, not model classification) and the harness scope gate (which blocks disallowed actions regardless of model state). Every model-based layer — the detector, the input/output rails, the primary model's refusal — has a decision boundary an adaptive attacker can probe and evade. The deterministic layers bound the worst case and serve as the floor beneath the product of model-layer residuals. This is why defense-in-depth holds under an adaptive adversary." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "Construct the five-step measurement methodology for a detection model. Which two steps do teams most often skip, and why do those two determine whether the measurement is honest?", "options": [ "Steps 1 and 2 (assemble corpus, per-detector rate). They are skipped because they are too easy.", "Steps 1 (the held-out out-of-distribution set) and 5 (the correlated-bypass test). These are skipped because they require extra work and often produce uncomfortable numbers. They determine honesty because: the held-out set is the only probe of out-of-distribution accuracy (the number that matters); the correlation test is the only check of whether the independence assumption holds. Without them, the measurement reproduces the vendor's in-distribution benchmark and bears no relationship to the rate an adaptive attacker achieves.", "Steps 3 and 4 (false-positive tax, end-to-end). They are skipped because they require production traffic.", "No steps are commonly skipped — all five are standard practice." ], "answer_index": 1, "rationale": "The two steps teams skip are the held-out out-of-distribution set (step 1) and the correlated-bypass test (step 5). They are skipped because they require extra effort (assembling novel techniques, testing shared-model-class transfer) and often produce numbers lower than the benchmark. They are the honesty checks: the held-out set is the only probe of out-of-distribution accuracy (the adversarial number that matters), and the correlation test is the only check of whether the independence assumption holds. Without them, the measurement is a vendor reproduction, not a deployment assessment." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why will detection-model out-of-distribution accuracy improve but never reach 100% against an adaptive adversary, and what is the architectural implication?", "options": [ "OOD accuracy will reach 100% once corpora are large enough; no architectural implication.", "OOD accuracy improves via larger/diverse corpora and ensembles, BUT an adaptive adversary's goal is to find inputs the model misclassifies, concentrated where training is sparse — a moving target. The detector's boundary is a fixed artifact probeable per checkpoint; the adversary automates evasion synthesis. No classifier achieves zero false-negatives against an unconstrained adversary. Implication: detection is NECESSARY but NOT SUFFICIENT — the architecture must compose the detector with DETERMINISTIC layers (IronCurtain, harness gate) that have no evasion surface.", "OOD accuracy is irrelevant; only in-distribution matters.", "Detection models will be replaced entirely by deterministic layers within a year." ], "answer_index": 1, "rationale": "OOD accuracy improves but never reaches 100% against an adaptive adversary because the adversary's target (misclassified inputs) is a moving frontier where training is sparse, and the detector's fixed boundary is probeable per checkpoint with automatable evasion synthesis. The architectural implication is decisive: detection is necessary (it catches the bulk of unsophisticated injections cheaply) but not sufficient (it has an irreducible residual). The architecture must compose the detector with deterministic layers that have no evasion surface — the detector bounds volume, deterministic layers bound the worst case." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "Design the full defense-in-depth composition for an agent that retrieves untrusted web content, with a detection model as Layer 4. Specify which layers are model-based (evasion surface) and which are deterministic (no evasion surface), and how the end-to-end bypass rate is bounded.", "options": [ "Detection model only — it is sufficient as the sole layer.", "L1-2: NeMo input rails (model-based) + IronCurtain deterministic boundary (NO evasion surface) — catch gross violations, bound worst case. L4: detection model (model-based, evasion surface) — catch the BULK of indirect injections from retrieved web content, quarantine/sanitize/refuse flagged content. L5: output rails (model-based) + primary model refusal (model-based) — backstop if L4 evaded. Harness: scope gate + tool-call arg validation + evidence classifier (DETERMINISTIC, NO evasion surface) — hard stop, the floor. End-to-end bypass = product of model-layer residuals, FLOORED by the harness gate. The detector bounds volume; deterministic layers bound the worst case.", "IronCurtain only — deterministic layers are sufficient without any model-based detection.", "Primary model refusal only — the model's safety training is the only layer needed." ], "answer_index": 1, "rationale": "The composition has model-based layers (input rails, detector Layer 4, output rails, primary refusal — each with an evasion surface) and deterministic layers (IronCurtain boundary, harness scope gate — no evasion surface). The detector is the highest-value single layer for indirect-injection defense (catches the bulk cheaply before context insertion) but is not the boundary. The end-to-end bypass rate is the product of the model-layer residuals, floored by the deterministic harness gate. Only the deterministic layers hold under an adaptive adversary because they have no decision boundary to probe." } ] }