🧭 Lesson Overview
Days 1–6 built your governance and risk vocabulary. Today you enter the threat landscape — the world as attackers see it. When an adversary targets an AI system, they are not executing a SQL injection or exploiting a CVE. They are targeting something fundamentally different: a system whose behavior emerges from data, whose attack surface includes the training pipeline, the inference API, and the model weights themselves.
Three frameworks give you the vocabulary to think about this systematically. MITRE ATLAS is the adversarial knowledge base — the ATT&CK for AI. NIST AI 100-2 E2025 is the authoritative taxonomy of adversarial machine learning attack types. OWASP LLM Top 10 (2025) is the practitioner threat reference for LLM applications specifically. Each serves a different purpose; all three are necessary for a complete picture.
When automobiles appeared in the 1900s, criminals quickly discovered that the new technology created new crime categories: car theft, road robbery, getaway vehicles. Traditional law enforcement frameworks — which categorized crimes around physical location, weapon type, and victim relationship — didn't have vocabulary for these new patterns. New frameworks had to be developed.
AI systems are the automobile of this era. Traditional security frameworks (ATT&CK, CVE, the CIA triad) don't have vocabulary for "someone manipulated the training data to make the fraud detection system ignore a specific account type." MITRE ATLAS, NIST AI 100-2, and OWASP LLM Top 10 are the new frameworks — built specifically because the old ones run out of vocabulary at the AI attack surface.
Day 7 Learning Objectives
- Explain why traditional threat frameworks (ATT&CK, CVE) are insufficient for AI-specific attack surfaces.
- Compare MITRE ATT&CK and MITRE ATLAS — scope, use cases, and two unique ATLAS tactics not in ATT&CK.
- Navigate the ATLAS matrix: identify tactics, techniques (AML.T codes), and real-world case studies.
- Apply the NIST AI 100-2 E2025 adversarial ML taxonomy — evasion, poisoning, privacy, and misuse attacks — to specific AI deployment scenarios.
- Explain how OWASP LLM Top 10, MITRE ATLAS, and NIST AI 100-2 complement each other and when to reach for each.
- Build a basic AI threat model for a specified AI system using ATLAS tactics as the organizing structure.
- Map real-world AI attack incidents to specific ATLAS techniques and NIST taxonomy categories.
🎯 Module 1 · What Makes AI a Different Kind of Target
When security professionals think about protecting a system, they think about the CIA triad — protecting data confidentiality, system integrity, and service availability. AI systems expose all three — but they also expose something traditional systems don't: the learning mechanism itself as an attack surface.
| Attack Surface | Traditional System | AI System Extension | What ATT&CK Misses |
|---|---|---|---|
| Data | Protect stored data from unauthorized access/modification | Training data defines behavior — poisoning it is a control-plane attack that changes what the system does | Data-as-code: compromised training data = compromised behavior without any code modification |
| Logic/Code | Patch vulnerabilities in executable code | Model weights ARE the logic — and they can be extracted, inverted, or fine-tuned maliciously without touching source code | Model extraction: adversary queries the API enough times to reconstruct a functional copy of the model |
| Inputs | Validate and sanitize user inputs (SQL injection, XSS) | Natural language IS the control plane — instructions and content flow through the same channel; the model cannot reliably distinguish them | Prompt injection: adversarial natural language inputs override system instructions with no syntactic exploit |
| Interface | Exploit bugs in APIs, web apps, network services | The inference API is a new attack vector — an adversary can query it to extract training data, reverse-engineer the model, or fingerprint its decision boundary | Inference API as reconnaissance tool: model inversion, membership inference, and model extraction all operate through legitimate API calls |
| Supply chain | Compromised software dependencies (Log4Shell, SolarWinds) | Foundation models, training datasets, and ML libraries are new supply chain elements — each is an attack vector with unique opacity | Model poisoning via supply chain: a compromised base model or training dataset propagates malicious behaviors to every downstream deployment |
⚔️ Module 2 · MITRE ATT&CK vs. MITRE ATLAS — When to Use Which
Security professionals who know ATT&CK already understand 80% of ATLAS's structure. Both use the same tactic-technique-procedure vocabulary, both are maintained by MITRE, and both are organized as matrices with community-contributed real-world examples. The critical difference is the attack surface each was designed to cover.
Target: Enterprise networks, endpoints, cloud infrastructure, mobile devices, OT/ICS systems
Scope: 14 tactics, 196+ techniques covering traditional cyber attacks — credential theft, lateral movement, command and control, exfiltration
Threat actors: APT groups documented with group profiles (APT28, Lazarus, Cozy Bear)
Best for: SOC threat hunting, red team scoping for traditional infrastructure, incident response, security program maturity assessment
AI gap: No tactics or techniques for training data poisoning, model extraction, adversarial examples, prompt injection, or model inference attacks
Target: AI and machine learning systems — training pipelines, model weights, inference APIs, vector databases, AI agent infrastructure
Scope: 16 tactics, 84 techniques, 56 sub-techniques, 42 case studies, 32 mitigations (v5.1, November 2025)
Two unique tactics NOT in ATT&CK: ML Model Access (AML.TA0004) and ML Attack Staging (AML.TA0012)
Best for: AI red team scoping, AI threat modeling, adversarial ML defense design, AI-specific incident investigation
2025 additions: 14 new agent-focused techniques (Zenity Labs collaboration), covering AI agent memory manipulation, tool poisoning, and agentic C2 channels
The Two Unique ATLAS Tactics — What Makes AI Different
| ATLAS-Unique Tactic | ID | What It Covers | ATT&CK Parallel |
|---|---|---|---|
| ML Model Access | AML.TA0004 | Adversary gains the ability to query, inspect, or manipulate an ML model directly — through public inference APIs, physical access to model files, or supply chain access to training pipelines. Without this, most ML-specific attacks are impossible. | No direct parallel. ATT&CK assumes adversaries access software through standard network/endpoint channels. ML Model Access recognizes that the model itself is a new class of asset with unique access vectors. |
| ML Attack Staging | AML.TA0012 | Adversary prepares their AI-specific attack tools before deployment — crafting adversarial inputs, creating poisoned training examples, engineering backdoor triggers, or building the infrastructure for a model extraction campaign. | Closest to ATT&CK's Resource Development (TA0042) — but ML Attack Staging is specific to the unique preparation work required for adversarial ML attacks, which has no traditional software equivalent. |
🗺️ Module 3 · ATLAS Deep Dive — All 16 Tactics & Key Techniques
The ATLAS matrix organizes adversary actions from left (early attack phases) to right (later phases), exactly like ATT&CK. Each tactic represents an adversary objective; each technique represents a method for achieving that objective. Click any technique in the matrix below to see its description, case study reference, and detection implications.
Interactive ATLAS Tactics Matrix — Click Any Technique
Hover or click any technique to see its description and ATLAS ID. The matrix shows key tactics and representative techniques — not exhaustive.
Key ATLAS Techniques — Expand for Full Detail
The eight most operationally critical ATLAS techniques for enterprise AI security teams. Each includes the official ATLAS description, real-world case study reference, detection approach, and MVCC relevance.
ATLAS Case Studies — What Real Attacks Look Like
📋 Module 4 · NIST AI 100-2 E2025 — The Authoritative AML Taxonomy
NIST published the updated Adversarial Machine Learning taxonomy (NIST AI 100-2 E2025) on March 24, 2025. This is the reference standard for AI security terminology — when regulators, auditors, or contracts reference "adversarial ML attacks," this is the taxonomy they mean. It organizes attacks along two dimensions: what kind of AI system is targeted (Predictive AI or Generative AI) and what phase of the ML lifecycle is attacked (training time or deployment time).
NIST AI 100-2 is the Rosetta Stone of AI security language. ATLAS uses its own technique names (AML.T0051 — Prompt Injection). OWASP uses its own category names (LLM01:2025 — Prompt Injection). Academic papers use yet another vocabulary. NIST AI 100-2 provides standardized definitions that map across all three — it's the common language that lets you compare what ATLAS calls "adversarial data" with what OWASP calls "training data poisoning" with what your vendor calls "data integrity attack." When in doubt about what an attack term means, NIST AI 100-2 is the authoritative definition.
The Two-Dimensional Taxonomy
| Attack Type | Predictive AI (PredAI) | Generative AI (GenAI) | Lifecycle Phase |
|---|---|---|---|
| Evasion | Adversarial examples that cause misclassification. Imperceptible image perturbations, adversarial text. Model still runs correctly — inputs are crafted to fool it. | Jailbreaks, adversarial prompts that bypass content filters, role-play attacks that override safety training | Deployment-time (model already trained) |
| Poisoning | Malicious training examples inserted into training data. Backdoor triggers that cause misclassification only when a specific pattern is present. | RAG poisoning, fine-tuning data manipulation, indirect prompt injection via poisoned retrieved documents | Training-time (before or during model training) |
| Privacy Attacks | Membership inference (was this sample in training data?), model inversion (reconstruct representative features of training data from model outputs), attribute inference | Training data extraction (verbatim memorization), system prompt extraction, conversation history reconstruction | Deployment-time (query the deployed model) |
| Misuse (GenAI-specific) | N/A — traditional PredAI models are not typically used for content generation misuse | CSAM generation, malware code generation, disinformation at scale, targeted harassment, weaponizable synthesis. Using the model for its capabilities, against its intended use. | Deployment-time (using the model as designed, for harmful purposes) |
Adversarial ML Attack Types — Flip Cards: Definition + Detection
Click each card to flip it. Front: attack type and one-sentence definition. Back: how to detect it and the key ATLAS technique it maps to.
AML Attack Types — Full Deep Dive
🔓 Module 5 · OWASP LLM Top 10 (2025) — Practitioner Threat Reference
OWASP Top 10 for LLM Applications (2025) is covered in full depth on Day 8. Today you understand its role in the framework stack and how it relates to ATLAS and NIST AI 100-2. The key distinction: OWASP is an LLM-specific, practitioner-oriented threat catalog — it tells security engineers which threats to test for in LLM applications. ATLAS and NIST AI 100-2 cover the broader AI/ML attack surface.
| Dimension | OWASP LLM Top 10 (2025) | MITRE ATLAS v5.1 | NIST AI 100-2 E2025 |
|---|---|---|---|
| Scope | LLM applications specifically | All AI/ML systems (LLMs + predictive ML) | All AI/ML systems — authoritative definitions |
| Purpose | Developer/AppSec practitioner guide | Threat modeling and red team scoping | Standardized taxonomy and terminology |
| Format | 10 ranked items with attack paths, impacts, mitigations | Tactic-technique matrix with case studies | Conceptual taxonomy with definitions and mitigations |
| Audience | AppSec engineers, LLM developers, security architects | Red teams, threat hunters, AI security architects | Standards bodies, researchers, policymakers, AI governance |
| Updates | Annual community editions (2023, 2025) | Living knowledge base — multiple updates per year | Research-driven (E2023, E2025) |
| Use this for: | Pen testing an LLM application, security requirements for LLM builds | Scoping an AI red team, building a detection program, threat hunting | Writing AI security policy, regulatory compliance, academic research |
OWASP ↔ ATLAS ↔ NIST — Cross-Framework Mapping (Selected)
| OWASP LLM (2025) | ATLAS Technique(s) | NIST AI 100-2 Category |
|---|---|---|
| LLM01:2025 — Prompt Injection | AML.T0051 (LLM Prompt Injection), AML.T0054 (Prompt Injection via Indirect Content) | Evasion (GenAI) — adversarial inputs that bypass model instructions |
| LLM03:2025 — Supply Chain | AML.T0048 (ML Software Dependency Compromise), AML.T0020 (Poison Training Data) | Poisoning — supply chain attack on training pipeline or model components |
| LLM04:2025 — Data and Model Poisoning | AML.T0020 (Poison Training Data), AML.T0019 (Publish Poisoned Datasets) | Poisoning (GenAI) — RAG data manipulation and fine-tuning data attacks |
| LLM06:2025 — Excessive Agency | AML.T0086 (Exfiltration via AI Agent Tool Invocation), AML.T0110 (AI Agent Tool Poisoning) | No direct NIST category — emergent risk specific to agentic GenAI |
| LLM09:2025 — Misinformation | No direct ATLAS technique — societal impact rather than attack technique | Misuse (GenAI) — using the model's generation capability for harmful content |
| LLM10:2025 — Unbounded Consumption | AML.T0029 (Denial of ML Service) | Availability attack — resource exhaustion targeting the model or API |
🗺️ Module 6 · The Framework Stack — Which to Use When
Three frameworks, three different questions. The practical skill is matching the question to the framework — not treating any one as complete.
| Question You're Asking | Reach For | Why |
|---|---|---|
| "What should our AI red team test for in this LLM chatbot?" | OWASP LLM Top 10 (2025) | Ranked, practitioner-oriented, LLM-specific. Gives you the list with attack paths and mitigations. |
| "How do we scope the full AI threat model for this ML pipeline, including training, inference, and supply chain?" | MITRE ATLAS | Covers the full ML lifecycle — not just LLMs. Tactic-technique structure enables systematic threat modeling. |
| "What is the standard definition of 'training data poisoning' and how does it differ from 'backdoor attack'?" | NIST AI 100-2 E2025 | Authoritative taxonomy and terminology. Use for regulatory writing, policy, contracts, and cross-team communication. |
| "What ATT&CK group profile should I use for the adversary modeling part of my AI threat model?" | MITRE ATT&CK (traditional) | For the non-AI parts of the attack chain — how the adversary gets to the AI system in the first place. |
| "What does our AI risk register entry look like for 'prompt injection'?" | All three + NIST AI RMF | OWASP gives the threat; ATLAS gives the technique; NIST AI 100-2 gives the definition; NIST AI RMF MAP gives the risk documentation structure. |
⛓️ Activity A · AI Attack Chain Builder
🔬 Activity B · AI Threat Model Lab — MVCC Chatbot
Answer each threat modeling question. Select the best answer from the options provided. Each question tests a specific ATLAS, NIST AI 100-2, or OWASP framework concept.
🎯 Activity C · Framework Match — Which Framework Answers This?
Flip each card to see which framework (ATLAS, NIST AI 100-2, OWASP LLM Top 10, or ATT&CK + ATLAS) best answers the question or provides the vocabulary for the scenario. After flipping all 10, write down: which framework is most relevant to your current role, and which one gap you would address first in your organization's AI security program.
📝 Assessment Artifact
One-Page AI Threat Model Diagram — MVCC At-Risk Student Predictor
Produce a one-page AI threat model for the MVCC at-risk student predictor using ATLAS as the organizing structure. Your artifact must include:
- System description (2 sentences): What the system does, its data inputs, and its deployment context
- Attack surface inventory: List the five attack surfaces specific to this ML system (training pipeline, inference API, model weights, training data, downstream application)
- Threat chain (4–6 steps): A realistic attack chain from initial reconnaissance through impact, using ATLAS tactic names and at least 3 specific AML.T technique codes
- NIST AI 100-2 classification: For each threat in your chain, classify it as evasion, poisoning, privacy attack, or misuse — with justification
- OWASP mapping: Identify which OWASP LLM Top 10 (2025) items apply to the deployment, with rationale
- Top 3 mitigations: The three highest-priority controls, each mapped to the specific threat they address
Grading: Daily assignment (25% of course grade). Evaluated on accuracy of ATLAS tactic/technique usage, correctness of NIST classification, OWASP mapping logic, and specificity of mitigations.