🧭 Lesson Overview
Days 7 and 8 gave you the frameworks: ATLAS, NIST AI 100-2, OWASP LLM Top 10. Today you go one level deeper on the attacks that target AI before and during deployment — the attacks that don't look like attacks until it's too late, because they happen in training pipelines, data warehouses, and model registries rather than in network traffic or application code.
The organizing principle for this lesson: every AI attack has a lifecycle position — where in the ML lifecycle it strikes. Understanding that position determines which defenses are possible and who is responsible for implementing them.
Traditional cyberattacks are like poisoning individual glasses — an attacker compromises a specific system, a specific account, or a specific session. The blast radius is bounded by the target. Security teams can hunt for and isolate the compromise.
Attacks on AI training data are like contaminating the water supply. The poison enters upstream, affects everything downstream simultaneously, and may be invisible until well after the harm is done. The "attacker" may be gone from the network before any deployment occurs. Standard security monitoring — which watches for active attacks — cannot see a training data compromise that happened six months ago.
This is why AI data and model attacks require a fundamentally different detection and response strategy: you must monitor what the model produces, not just what enters the network.
Six Attack Categories — Day 9 Coverage
ML Lifecycle Attack Surface Map
| ML Lifecycle Phase | Attacks in This Phase | Who Has Access | ATLAS Tactic |
|---|---|---|---|
| Data Collection & Labeling | Broad data poisoning, label flipping, clean-label poisoning | Data engineers, external data vendors, crowd-sourcing platforms | AML.TA0001 (Resource Dev) → AML.T0019 |
| Model Training | Backdoor/Trojan insertion, model inversion via training data | ML engineers, data scientists, training infrastructure | AML.TA0012 (ML Attack Staging) → AML.T0020 |
| Model Evaluation & Validation | Evaluation set manipulation, canary bypass | ML engineers, quality teams | AML.TA0012 → evasion of evaluation |
| Model Packaging & Registry | Supply chain compromise, model weight tampering, typosquatting | ML engineers, CI/CD pipelines, external model repos | AML.T0048 (Supply Chain Compromise) |
| Deployment & Serving | Adversarial inputs (evasion), prompt injection, model extraction | Any API user, end users, automated systems | AML.T0043, AML.T0051, AML.T0037 |
| Production Monitoring | Silent drift, concept drift, adversarial drift induction | World events, population changes, deliberate manipulation | AML.T0047 (Model Manipulation) |
Day 9 Learning Objectives
- Explain "data as code" — why training data is a control plane, not just stored information.
- Classify data poisoning attacks into the five NIST AI 100-2 sub-categories with examples of each.
- Explain backdoor/Trojan attacks: how triggers are installed, how they survive evaluation, and how they're detected.
- Distinguish the four types of model drift (data, concept, label, prior shift) and select appropriate detection metrics for each.
- Identify MLOps pipeline security threats at each lifecycle phase and the corresponding controls.
- Apply the attack taxonomy to MVCC AI deployment scenarios, identifying lifecycle phase, attacker goal, detection method, and defense.
💡 Module 1 · Data as Code — The Control Plane Reframe
The single most important conceptual shift for AI security is understanding that training data is not an asset to protect — it is a control plane to secure. In traditional software, the control plane is code: the instructions that define system behavior. Compromise the code, and you compromise what the system does. In AI systems, training data IS the control plane — it defines what the model does, how it behaves, where it succeeds, and where it fails.
| Concept | Traditional Software | AI / ML System |
|---|---|---|
| What defines behavior | Code (explicit, inspectable, versionable) | Training data + architecture + training process (partially implicit, partially not directly inspectable) |
| How to change behavior | Modify the code, redeploy | Change training data, retrain, or fine-tune the model |
| Attack on behavior | Exploit a code vulnerability, SQL injection, buffer overflow | Poison the training data — the malicious behavior is baked in before deployment |
| Defense lineage | Code review, static analysis, SAST/DAST, penetration testing | Data provenance, data integrity validation, training audit trails, pre-training bias/backdoor detection |
| Audit trail | Git commit history, code diffs | Data lineage records, training run logs, dataset version hashes |
| Security testing timing | During and after development (test gates) | Before, during, and after training — and continuously in production |
| Blast radius of compromise | Affects the systems where the code runs | Affects every deployment using that training data — including downstream fine-tuned models |
A model's training data is its DNA. Just as DNA determines an organism's traits and vulnerabilities, training data determines a model's capabilities and failure modes. A genetic mutation introduced early in development propagates to every cell in the organism. A training data compromise introduced early in the pipeline propagates to every deployment of the model.
You cannot inspect a person's DNA just by watching them walk. Similarly, you cannot always detect a training data compromise by observing the model's standard outputs — the poisoned behavior may be dormant, activating only when a specific trigger is present.
☠️ Module 2 · Data Poisoning Taxonomy — All Five Sub-Categories
NIST AI 100-2 E2025 categorizes poisoning attacks into five sub-types, each with distinct mechanics, detection difficulty, and defender implications. Understanding the sub-categories matters because each requires a different detection approach and has a different set of responsible parties.
Poisoning Sub-Categories — Flip Cards
Click each card to flip it. Front: attack type and one-liner. Back: how it works, detection method, and MVCC scenario.
Poisoning Sub-Categories — Full Technical Detail
Poisoning Defense Comparison
| Defense | Broad Poisoning | Backdoor/Trojan | Clean-Label | RAG Poisoning | Implementation Level |
|---|---|---|---|---|---|
| Data provenance validation | ✅ | ✅ | ✅ | ✅ | Data pipeline |
| Statistical outlier detection | ✅ | ⚠️ | ❌ | ⚠️ | Pre-training |
| Neural Cleanse / trigger inversion | ❌ | ✅ | ⚠️ | ❌ | Post-training |
| Canary samples in test set | ✅ | ⚠️* | ⚠️ | ❌ | Evaluation |
| RAG corpus integrity validation | ❌ | ❌ | ❌ | ✅ | Retrieval layer |
| Differential privacy in training | ✅ | ⚠️ | ❌ | ❌ | Training process |
| Human review of training examples | ⚠️ | ❌ | ❌ | ⚠️ | Manual QA |
* Canary samples only catch backdoor triggers if the canary includes the trigger pattern — requires trigger knowledge. ⚠️ = partially effective.
🎭 Module 3 · Backdoor & Trojan Attacks — How Dormant Compromises Work
Backdoor attacks are the most operationally sophisticated attacks on AI models because they are designed to be invisible during standard evaluation. A backdoored model performs perfectly on any clean input — passing every evaluation benchmark, achieving high accuracy, behaving exactly as expected in testing. The malicious behavior only manifests when a specific trigger pattern is present in the input.
Backdoored AI models are the sleeper agents of machine learning. During normal operation — day after day, deployment after deployment — they perform their legitimate function flawlessly. The spymaster (attacker) has installed a secret trigger: a specific phrase, pattern, or signal that activates the compromised behavior while leaving everything else intact.
The challenge for defense is identical to the intelligence challenge: you cannot distinguish a perfectly loyal model from a perfectly dormant sleeper without specifically testing for the trigger — and you can only test for triggers you know about. Triggers you don't know exist cannot be tested.
Backdoor Attack Mechanics — Three-Stage Process
Backdoor Variants — Deep Dive
📉 Module 4 · Model Drift — Four Types & Detection Methods
Model drift is the silent degradation of AI system performance over time as the world changes in ways the model was not trained for. Unlike poisoning (which is an attack) or backdoors (which are designed compromises), drift is usually not malicious — it happens naturally as populations shift, behaviors evolve, and ground truth changes. But its consequences can be as severe as an intentional attack: a fraud detection model that drifts becomes an enabler of fraud; a patient risk model that drifts produces wrong risk scores; an at-risk student predictor that drifts flags the wrong students.
According to DataRobot, 73% of failures in production AI are linked to unforeseen shifts in input data relevance. studies of model aging find that most deployed ML models degrade over time if they are not retrained — one widely-cited 2022 analysis put this at roughly 91%. This makes drift detection the most operationally critical monitoring activity for deployed AI systems — and the one most commonly skipped.
The Four Drift Types
Detection Methods by Drift Type
| Drift Type | Statistical Test | Monitoring Metric | Trigger Threshold | Response |
|---|---|---|---|---|
| Data Drift (Covariate Shift) | Population Stability Index (PSI), Kolmogorov-Smirnov test | PSI per feature; feature distribution histograms | PSI > 0.1 (warning), PSI > 0.25 (action) | Feature engineering review; retrain on recent data distribution |
| Concept Drift | ADWIN (ADaptive WINdowing), DDM (Drift Detection Method) | Prediction error rate; actual vs. predicted distribution | Error rate change >X% from baseline (context-dependent) | Retrain with recent labeled data; investigate world-change cause |
| Label Drift | Chi-square test on prediction distribution | Prediction class distribution over time | Distribution shift >15% from training-time base rate | Review labeling assumptions; recalibrate prediction thresholds |
| Prior Probability Shift | Base rate comparison (if ground truth available) | Observed outcome frequency vs. training-time frequency | Base rate change >20% over rolling window | Recalibrate model or update decision thresholds |
The fix: train on both semesters' data; evaluate separately for each semester; configure semester-triggered revalidation; monitor PSI on key features at the start of each term.
🔧 Module 5 · MLOps Pipeline Security — Where the Attacks Actually Land
MLOps (Machine Learning Operations) is the discipline of operationalizing ML models — the CI/CD, data pipelines, experiment tracking, model registries, deployment infrastructure, and monitoring systems that turn research models into production systems. As MLOps has become production-critical infrastructure, it has become a primary attack surface. IBM X-Force research demonstrates that major enterprise MLOps platforms including Azure Machine Learning, BigML, and Google Cloud Vertex AI are vulnerable to adversaries with valid credentials — enabling training data poisoning, model and data theft, and exploitation of production pipelines.
MLOps Security Controls — By Pipeline Phase
MLSecOps vs DevSecOps — The Extension
| DevSecOps Practice | MLSecOps Extension | Why the Extension Is Needed |
|---|---|---|
| SAST/DAST code scanning | ML model scanning (backdoor detection, property testing) | Model weights are not code — static analysis doesn't apply |
| Dependency scanning (SCA) | AIBOM (AI Bill of Materials) + model provenance | ML supply chain includes model weights, training datasets, not just libraries |
| Secret scanning in code | Training data privacy scanning (PII, sensitive content) | Training data is part of the "codebase" — sensitive data in training = sensitive data in model |
| Integration test gates | Behavioral evaluation gates (fairness, robustness, bias) | Functional tests pass even for backdoored or biased models |
| Production monitoring (APM) | AI observability (drift, fairness, prediction quality) | Traditional APM shows green while model accuracy silently degrades |
| Incident response playbook | AI-specific IR (model poisoning, drift discovery, bias incident) | Standard IR playbooks don't have procedures for "training data contamination" |
🔬 Activity A · Attack Taxonomy Lab — Classify Each Attack
For each scenario, identify the correct attack sub-category. Some scenarios describe attacks that look similar — focus on the specific mechanism described, not just the general outcome. Instant feedback explains the classification rationale.
📊 Activity B · Drift Detection Lab — Assign Signals to Drift Types
Drag (or click) each monitoring signal from the pool into the drift type column it best detects. Click a signal to select it, then click the target column. "Check All" reveals which placements are correct.
🛡️ Activity C · Defense Matrix Builder — MVCC AI Deployment
For each MVCC AI deployment attack scenario, select all controls that effectively address that specific attack. Some options are tempting but ineffective for that specific attack type. Check all that apply, then click "Check This Attack."
📉 Activity D · Read the Drift Readout — Hold or Retrain?
Monitoring only helps if you can read it. Below is a 30-day monitoring snapshot for the MVCC enrollment model. For each signal decide whether it is within tolerance or a drift alarm, then make the overall call. Remember: drift is multi-dimensional — a healthy accuracy number can hide input and fairness drift.
| Signal | Reading | Threshold | Your call |
|---|---|---|---|
| Input-feature drift (PSI) | 0.28 | alarm if ≥ 0.25 | |
| Prediction-distribution shift (KS test) | p = 0.002 | alarm if p < 0.05 | |
| Accuracy (labeled 30-day holdout) | 88% | floor ≥ 85% | |
| Null rate in a key input feature | 2% | max ≤ 5% | |
| Demographic-parity gap | 9% | max ≤ 5% |
📝 Assessment Artifact
AI Monitoring Plan — MVCC At-Risk Student Predictor
Design a production monitoring plan for the MVCC at-risk student predictor. Your plan must address all four drift types and the primary poisoning risk for this deployment. Submit as a structured document with these five sections:
- 1. Drift Risk Assessment: For each of the four drift types, assess the likelihood for this specific system (high/medium/low with justification). Which type is highest risk given the semester-boundary deployment pattern?
- 2. Detection Metrics: For each drift type you rated high or medium, specify: the exact statistical test or metric, the monitoring frequency, and the threshold that triggers an alert.
- 3. Poisoning Detection: This model was trained on 3 years of historical LMS and attendance data aggregated from multiple sources. What poisoning sub-category is highest risk, and what one technical control would you implement to detect it?
- 4. MLOps Controls: List 3 pipeline controls you would implement at different lifecycle phases (training, registry, production monitoring) to reduce the attack surface of this system.
- 5. Incident Response Trigger: Write a one-sentence kill-switch trigger condition (in the style of NIST AI RMF MG-2.4) that would cause immediate suspension of this model's predictions pending investigation.
Grading: Daily assignment (25% of course grade). Evaluated on specificity of drift detection metrics, accuracy of poisoning risk assessment, MLOps control lifecycle accuracy, and clarity of the IR trigger.