🧭 Lesson Overview · Day 13 · 4 hoursFrom Principles to Paper — and Paper to Practice
Days 1–12 built the knowledge of AI cybersecurity: frameworks, risk categories, threat landscapes, secure architecture, and vendor governance. Today the course shifts to the documentation layer — the written governance artifacts that make AI security auditable, enforceable, and repeatable. An AI security program that exists only in engineers' heads is not a program; it is a set of tribal customs that disappears when the engineer leaves.
The governance documents covered today — the AI Acceptable Use Policy, the Use Case Register, ISO 42001-aligned management documents, and the AI Incident Response Annex — are the instruments through which everything else in this course becomes operational. They are also the artifacts regulators, auditors, and cyber insurers will ask for first.
⚖️ Anchoring Analogy: The Building Code and the Inspection
A building can be constructed by a skilled architect with excellent judgment who knows every safety principle by heart. But without a building code — a written, publicly known, auditable standard — there is no way for an inspector to verify compliance, no consistent baseline for contractors, and no recourse when something goes wrong. The building code does not replace the architect's judgment. It makes the judgment legible, consistent, and enforceable across every building in the jurisdiction.
AI governance documents are the building code for AI. They make the security team's judgment legible to auditors, consistent across deployments, and enforceable across the organization. A policy that says "use AI responsibly" is not a building code. A policy that specifies which data categories may not enter AI systems, which approvals are required, what happens on violation, and who owns the annual review — that is a building code.
Day 13 Learning Objectives
- 1Explain why governance documents are the operationalizing layer of an AI security program — and why vague policy fails audit.
- 2Construct an AI Acceptable Use Policy that covers the eight essential components and withstands regulatory scrutiny.
- 3Build and populate an AI Use Case Register aligned to NIST AI RMF MAP function and ISO 42001 inventory requirements.
- 4Identify the key ISO 42001 governance documents and their role in an AI Management System (AIMS).
- 5Draft an AI Incident Response Annex that extends the NIST SP 800-61 lifecycle for AI-specific incidents.
- 6Evaluate whether a draft AI governance document would withstand an audit — identifying gaps and remediation language.
Why Governance Documents Fail — and What Good Ones Look Like
Module 1 · 20 minWhy AI Governance Documents Fail — The Four Common Failures
Most organizations that have AI governance documents still fail audits — not because they lack a policy, but because their policy fails one of four common tests. Understanding these failure modes is the prerequisite for building documents that work.
Module 2 · 45 minThe AI Acceptable Use Policy — Eight Essential Components
An AI Acceptable Use Policy (AI AUP) is the foundational governance document for most organizations' AI programs. It defines who may use AI, which tools are approved, what data may be processed, what uses are prohibited, what verification is required for AI outputs, and what happens on violation. In 2026, the absence of an AI AUP is treated by regulators, auditors, and cyber insurers as a governance failure — not a gap.
The eight components below represent the consensus structure drawn from NIST AI RMF GOVERN function requirements, ISO 42001 Clause 6.2 (AI objectives), EU AI Act Article 4 (AI literacy), and current practitioner guidance. Expand each component to see what it must contain and what an auditor expects to find.
What it must contain: An explicit statement of who the policy covers (employees, contractors, vendors, students, volunteers — anyone who accesses organizational systems or data), which AI systems it governs (generative AI, agentic AI, embedded AI features in sanctioned SaaS, personal AI tools used for work tasks, AI provided to students), and which devices are in scope (corporate-managed, BYOD, personal devices accessing organizational data).
The MVCC version: At a community college, scope must explicitly address: faculty using AI tools for instruction (generating course content, grading assistance), students using AI for coursework (a separate academic integrity policy may apply, but the AUP governs institutional data), administrative staff using AI for operations, and vendors whose products include embedded AI features. If the policy doesn't say it covers these populations, it doesn't — and their AI use is ungoverned.
Audit test: An auditor will ask: "Is a faculty member who uses ChatGPT to draft a student communication covered by your AI AUP?" If the answer requires interpretation rather than reference to explicit policy text, the scope is insufficient.
What it must contain: Precise definitions for every term the policy uses whose meaning could be disputed. At minimum: Generative AI, Agentic AI, Shadow AI, Foundation Model, Sanctioned Tool, Prohibited Tool, AI Output, Sensitive Data (mapped to your data classification scheme), Human Review (what constitutes a qualifying review), and Material Modification (when does customizing a tool create new governance obligations).
Why definitions determine enforceability: A policy that says "do not use unsanctioned AI tools" but does not define "unsanctioned" will be argued around by every employee who believes their tool is fine. A policy that says "ChatGPT, Claude, Gemini, and any other generative AI tool not listed in Appendix A is an unsanctioned tool" is enforceable. Ambiguous policies create compliance theater where the document exists but constrains nothing.
The 2026 additions: By 2026, definitions should also include: Agentic AI System (an AI system that can take autonomous actions using tools), Model Context Protocol (MCP) (a standard enabling AI agents to connect to external services), and AI-Embedded Feature (an AI capability added by a SaaS vendor to a product without the deployer making a separate AI procurement decision).
What it must contain: An explicit list (maintained as a living appendix, not embedded in the policy body so it can be updated without full policy revision) of approved AI tools — including tool name, approved version/tier, data categories permitted, approved use cases, and any conditions of use (e.g., "Microsoft 365 Copilot — enterprise tier only; FERPA-protected data prohibited without IT approval"). Anything not on the approved list is unsanctioned.
The three-tier structure (PurpleSec 2026 model):
- Tier 1 — Enterprise Sanctioned: Formally vetted, approved, under data processing agreement, permitted for specified data categories (e.g., Claude for Work enterprise, Microsoft Copilot enterprise tier, internal AI platform)
- Tier 2 — Tolerated with Restrictions: Not formally vetted but not prohibited; may be used for non-sensitive tasks only; review pending (e.g., free-tier consumer AI tools for drafting public content)
- Tier 3 — Prohibited: Explicitly prohibited — tools with no enterprise DPA, tools that train on user data, tools with unacceptable jurisdiction/data residency, tools in sectors with known security issues
Why a living appendix: The AI tool landscape changes faster than any policy can be formally revised. By maintaining the approved list as Appendix A (separately versioned, approved by the AI governance committee rather than requiring full board/president approval), organizations can add or remove tools on a 30-day cycle rather than waiting for the annual policy review.
What it must contain: An explicit matrix of which data categories may and may not be entered into which AI tool tiers. This is the component that most directly addresses the incidents that drove AI AUP adoption: the Samsung engineer who pasted source code into ChatGPT, the marketing associate who uploaded a customer list to a generative AI tool, the finance analyst who shared a confidential earnings report.
| Data Category | Tier 1 Tools | Tier 2 Tools | Tier 3 Tools |
|---|---|---|---|
| FERPA-protected student records | ⚠ With IT approval | ✗ Prohibited | ✗ Prohibited |
| Employee PII (HR records, payroll) | ⚠ With HR + IT approval | ✗ Prohibited | ✗ Prohibited |
| Financial / budget data (non-public) | ⚠ With VP Finance approval | ✗ Prohibited | ✗ Prohibited |
| Proprietary research / IP | ⚠ With PI approval | ✗ Prohibited | ✗ Prohibited |
| Internal operational data (non-sensitive) | ✓ Permitted | ✓ Permitted | ✗ Prohibited |
| Publicly available information | ✓ Permitted | ✓ Permitted | ✗ Prohibited |
The MVCC-specific additions: Community colleges must address FERPA explicitly — student academic records, financial aid data, and enrollment information are FERPA-protected and cannot be entered into any AI system without a school official justification and FERPA-compliant data processing agreement. The policy must also address student work: student submissions, essays, and academic records are student data under FERPA and may not be processed by AI grading tools without appropriate institutional agreements.
What it must contain: An explicit, non-exhaustive list of prohibited AI uses. "Non-exhaustive" is an important qualifier — new prohibited uses will emerge that the policy didn't anticipate, and the policy should include a catch-all ("or any use that violates applicable law, regulation, or institutional policy").
Standard prohibited uses at a community college:
- Using AI to generate academic work submitted by students as their own (governed by Academic Integrity Policy; cross-referenced here)
- Using AI to make or materially influence employment decisions (hiring, performance evaluation, termination) without documented human review and bias audit
- Entering FERPA-protected student data into any AI system not covered by a FERPA-compliant DPA
- Using AI to generate, distribute, or amplify content that is false, misleading, or discriminatory
- Using AI to circumvent institutional security controls, authentication systems, or access restrictions
- Using AI to monitor, surveil, or profile individuals without authorization and applicable legal basis
- Using AI systems classified as prohibited under EU AI Act Article 5 (social scoring, real-time biometric surveillance, subliminal manipulation)
- Deploying agentic AI that can take financial, legal, or data-modifying actions without human approval for each action
The EU AI Act linkage: EU AI Act Article 5 (effective February 2025) prohibits specific AI uses absolutely. If any community college has international students or EU-national faculty, or deploys AI to EU-based persons, these prohibitions apply as legal obligations — not just policy preferences. The AUP should cross-reference Article 5 explicitly.
What it must contain: Explicit requirements for human review and verification of AI outputs before they are used in specific contexts. This is the operationalization of EU AI Act Article 14 (human oversight) at the policy level.
Risk-tiered verification requirements:
| Use Context | Minimum Verification | Documentation Required |
|---|---|---|
| High-stakes decisions (admissions, financial aid, employment) | Qualified human reviews and approves before decision is executed; AI output advisory only | Review log with reviewer name, date, rationale |
| Student-facing educational content | Faculty member reviews for accuracy, bias, and appropriateness before publication | Faculty attestation in LMS or document system |
| External communications (press releases, grant applications) | Department head or designee reviews before submission | Standard approval workflow |
| Internal drafts and operational documents | Author reviews for accuracy before circulation | No additional documentation required |
| Agentic AI actions (sends emails, modifies records, executes transactions) | Human approves each action category; irreversible actions require individual confirmation | Tool call log retained per IR Annex |
What "human review" must not mean: The policy must be specific about what constitutes a qualifying review. Clicking "accept" on an AI-generated document without reading it is not a human review. The policy should state that human review means the reviewer takes intellectual responsibility for the content — they would be able to explain and defend it independently of the AI output.
What it must contain: An explicit definition of shadow AI (any AI tool or feature used for institutional purposes that is not on the sanctioned tools list), and a clear, frictionless pathway for employees to request approval of tools they want to use. The approval pathway is the critical element: a policy that defines shadow AI but provides no pathway for legitimate requests drives usage underground.
The MVCC shadow AI approval pathway:
- Employee identifies a tool they want to use → submits AI Tool Request form (IT portal)
- IT Security reviews: data processing agreement, data residency, training-on-user-data terms (5 business days)
- If approved: tool added to Tier 2 (tolerated) or Tier 1 (sanctioned after DPA review) list
- If denied: employee notified with reason; may request appeal to AI Governance Committee
- Interim use: employee may not use the tool until review is complete; urgent exceptions require VP approval
Enforcement for discovered shadow AI: The policy must specify progressive consequences for discovered shadow AI use — from education and coaching for first-time, low-risk use, to formal disciplinary action for repeated or high-risk violations. Without a graduated enforcement structure, managers face an all-or-nothing enforcement choice that typically results in no enforcement.
What it must contain: Named roles with documented AI governance responsibilities, not just titles. The difference between "the CISO is responsible for AI security" (not specific enough to audit) and "the Director of IT Security maintains the sanctioned tools list, conducts quarterly shadow AI scans, and reports violations to the AI Governance Committee" (auditable).
The MVCC AI Governance Structure:
| Role | AI Governance Responsibilities | Review Cadence |
|---|---|---|
| President / Board | Approves AI AUP; sets AI risk appetite; receives annual AI governance report | Annual |
| AI Governance Committee | Reviews AI use case register; approves high-risk AI deployments; adjudicates shadow AI appeals; reviews incidents | Quarterly |
| Director of IT Security (Policy Owner) | Maintains sanctioned tools list; conducts quarterly shadow AI scans; manages vendor assessments; coordinates AI incidents | Ongoing |
| Legal Counsel | Advises on regulatory compliance; reviews AI DPAs; identifies EU AI Act obligations | On change / on demand |
| Department Heads | Ensure staff complete AI literacy training; enforce AUP within department; report AI incidents | Ongoing |
| All Employees | Complete AI literacy training; follow AUP; report shadow AI and incidents; do not share credentials for AI systems | Annual training |
Review cycle: The policy must specify an annual review cycle at minimum, with triggered reviews on: new regulation effective dates, significant AI capability changes (new model family deployment), major AI incidents, and organizational changes that affect AI risk scope. The review cycle must be documented with evidence (meeting minutes, version control log) — not just stated in the policy.
Module 3 · 30 minThe AI Use Case Register — Your AI Asset Inventory
An AI Use Case Register is the authoritative inventory of every AI system an organization is developing, piloting, or operating — including AI embedded in vendor products. It is the AI-equivalent of a system inventory under NIST CSF or ISO 27001. Without an AI Use Case Register, an organization cannot govern AI it doesn't know about, cannot map its EU AI Act obligations, and cannot conduct a meaningful risk assessment.
🔑 NIST AI RMF MAP Function Connection
The NIST AI RMF's MAP function (covered in Lesson 3) explicitly requires documenting AI system context, capabilities, and limitations before risk assessment. The Use Case Register is the operational artifact that satisfies MAP requirements: MAP 1.1 (context), MAP 2.2 (data), MAP 3.1 (tasks and impacts), and MAP 5.2 (risk factors). An organization with no Use Case Register cannot demonstrate MAP function compliance.
📦 Analogy: The Drug Formulary
A hospital's pharmacy maintains a formulary — an approved list of medications, including their approved indications, dosage requirements, interaction warnings, and prescribing controls. A medication not on the formulary cannot be dispensed without special authorization. The formulary is not just a list: it is a governance instrument that ensures every medication in use is known, approved, and appropriately controlled. The AI Use Case Register is the formulary for AI systems. Every AI system in use must be registered, approved, classified, and reviewed — or it is an ungoverned deployment with unknown risk.
Required Fields — What Every Register Entry Must Capture
Expand each field group to see what information is required and why it matters for governance.
System Name & ID: Unique identifier for the AI system (e.g., MVCC-AI-2026-003). Enables cross-reference from incident reports, vendor contracts, and risk assessments to the register entry.
Lifecycle Status: Pre-deployment / Pilot / Deployed / Retired. Aligned to DHS AI Use Case Inventory 2025 stage categories. Determines which governance controls are active.
EU AI Act Risk Tier: Prohibited / High Risk (Annex III) / GPAI / Limited Risk / Minimal Risk. Determines regulatory obligations. Must be re-evaluated when use case changes.
NIST AI RMF Impact Level: Low / Medium / High / Critical. Drives assessment frequency and control requirements. Based on potential harms to individuals, organization, and society.
AI System Type: Generative / Discriminative / Predictive / Agentic / Recommender / Computer Vision / NLP. Determines which threat categories are most relevant.
Business Owner: Named individual (not a department) accountable for the business use case. Takes responsibility for use case appropriateness, human oversight, and incident escalation. This is not the technical owner.
Technical Owner: Named individual responsible for deployment, maintenance, and technical controls. Accountable for model card maintenance, monitoring, and AIBOM currency.
Vendor / Provider: Legal name of the entity providing the AI system. Links to the vendor risk assessment and contract record. If internally built, this is the development team.
EU AI Act Role for MVCC: Deployer / Provider (if MVCC built or fine-tuned) / Importer. Determines MVCC's legal obligations. Must be reviewed by Legal Counsel for high-risk systems.
Data Categories Processed: Mapped to the institutional data classification scheme. If FERPA-protected: DPA status, FERPA exception applied, data minimization measures. If PII: DPIA reference. This field directly triggers the vendor assessment and contract clause requirements from Lesson 12.
Affected Populations: Who is subject to AI-influenced decisions? Students, employees, members of the public, specific demographic groups? This field informs the bias audit scope and human oversight requirements.
Decision Type: Does the AI make, assist, or inform decisions? Is the decision reversible? High-stakes + irreversible = mandatory human-in-the-loop requirement. This field drives the human oversight control design.
Approved Use Cases: An explicit list of approved uses. Use cases not on this list require AI Governance Committee approval before deployment. Prevents scope creep — the AI tutoring chatbot approved for math assistance should not be answering questions about financial aid eligibility without an amended use case approval.
Deployment Date & Last Review Date: Enables identification of systems that haven't been reviewed against current policy and regulation. An AI system deployed in 2023 that hasn't been reviewed since the EU AI Act's 2025 enforcement date is a compliance gap.
Next Review Date: Set at deployment; frequency determined by risk tier (High risk: quarterly; Medium: semi-annual; Low: annual). Must be tracked by the IT Security team and escalated if missed.
Retirement Trigger Criteria: Explicit conditions under which this system should be retired: model version end-of-life, vendor discontinuation, regulatory prohibition, documented performance degradation below threshold. Prevents zombie AI systems that remain in production past their useful or safe life.
Linked Documents: Cross-references to vendor contract, AIBOM, model card, risk assessment, DPA, and incident history. This cross-reference structure is what transforms a list into an auditable evidence chain.
Live Use Case Register — MVCC Example (Interactive)
The register below shows three pre-populated MVCC AI systems. You can add additional entries using the form below the table. Use this during Activity A to complete the registration of new systems.
| ID | System Name | Status | EU AI Act Tier | Risk | Business Owner | MVCC Role | Data Categories | Next Review |
|---|---|---|---|---|---|---|---|---|
| AI-2026-001 | Math Tutoring Chatbot | Deployed | High Risk | HIGH | Dean, Developmental Ed | Deployer | Student learning data (FERPA) | 2026-09-01 |
| AI-2026-002 | MS 365 Copilot | Pilot | Limited Risk | MED | Director, IT | Deployer | Internal docs, email, Teams (FERPA scope excluded) | 2026-08-01 |
| AI-2026-003 | AI Early Alert (SIS) | Pre-Deployment | High Risk | HIGH | VP, Student Services | Deployer | Academic records, attendance (FERPA) | 2026-07-01 |
Policy Gap Analysis — What's Missing from This AUP?
The text below is an excerpt from MVCC's current draft AI Acceptable Use Policy — written in 2024 and never updated. Your task: identify which of the eight essential AUP components are missing or inadequate and which are present and adequate. Drag or click each component into the correct column.
⚠️ Draft Policy Excerpt — MVCC AI Use Policy (2024)
MVCC employees are encouraged to use AI tools responsibly to enhance their productivity. Staff should exercise good judgment when using AI and avoid sharing confidential information. All AI-generated content must be reviewed before use. The IT Department maintains a list of approved tools. Violations of this policy may result in disciplinary action. This policy was approved by the President's Cabinet in March 2024.
Click each component to assign it:
Debrief (10 min): Which components were hardest to classify? What would it take to bring this 2024 draft up to 2026 standards? The class will draft two remediation sentences for the most critical gap.
ISO 42001 Governance Documents & AI IR Annex
Module 4 · 40 minISO 42001 — The AI Management System Document Architecture
ISO/IEC 42001:2023 is the world's first certifiable AI management system standard — the ISO 27001 analogue for AI. As of a 2025 benchmark, 76% of compliance professionals intend to use ISO 42001 as their AI governance backbone. Understanding its document architecture is increasingly a prerequisite for enterprise AI governance roles.
🏗️ Analogy: The ISO 27001 Model You Already Know
If you've implemented or audited ISO 27001, you know the document hierarchy: the Information Security Policy (strategic direction) → the Statement of Applicability (which Annex A controls apply) → documented procedures and work instructions → records as evidence of implementation. ISO 42001 follows an identical structure — Policy → AIMS Scope → Statement of Applicability (38 Annex A controls) → procedures → records. The content is different; the architecture is the same. This means ISO 27001 practitioners can map their existing document governance skills directly to ISO 42001 implementation.
The ISO 42001 Document Hierarchy — Click Each Level to Expand
What it is: The AI Policy is a high-level statement of organizational commitment to responsible AI — approved by top management, communicated to all personnel, and available to interested parties. It is NOT the AUP (which is operational). The AI Policy sets direction; the AUP implements it.
Required content per ISO 42001 Clause 5.2: The policy must be appropriate to the organization's purpose; include a commitment to satisfy applicable requirements; include a commitment to continual improvement; and provide a framework for setting AI objectives.
At MVCC, the AI Policy would include commitments to: deploy AI in service of student success and institutional mission; protect student and employee data in AI systems; maintain human oversight of AI decisions that materially affect individuals; comply with FERPA, EU AI Act (where applicable), and applicable state AI laws; and continually improve the AI management system.
What it is NOT: It is not a list of prohibited uses (that belongs in the AUP). It is not a technical control specification (that belongs in procedures). It is not a risk assessment (that is a separate artifact). Leadership approval of a one-page AI Policy statement is the starting point for ISO 42001 compliance — the document that authorizes everything below it in the hierarchy.
What it is: A documented statement of the boundaries and applicability of the AI Management System (AIMS) — which organizational units, processes, AI systems, and stakeholder relationships are within the AIMS scope. ISO 42001 Clause 4.3 requires this to be documented.
The AI system inventory: The AIMS scope document links to the AI Use Case Register (Module 3) as the operational inventory of in-scope AI systems. InfosecTrain's 2026 implementation guidance makes this explicit: "You cannot govern what you cannot see — the AI inventory is the foundation of the AIMS." A scope document without an inventory is a boundary without content.
Common scoping decisions for community colleges: Institutions often initially scope to high-risk AI systems only (narrower scope, faster certification path) with a roadmap to expand scope. A community college might scope their AIMS to: AI systems that process student records; AI systems used in instructional delivery; AI systems used in employment decisions. This excludes low-risk productivity AI from the AIMS scope while governance matures — but requires the AUP to cover out-of-scope systems through a separate operational policy.
What it is: A document that lists all 38 ISO 42001 Annex A controls, states whether each control is applicable to the organization, and — for applicable controls — provides evidence of implementation. Directly analogous to the ISO 27001 Statement of Applicability.
The 42 controls are grouped into themes:
- Organizational controls (A.2–A.6): AI policy, roles and responsibilities, risk management, resource management, impact assessment
- AI-specific controls (A.6–A.10): AI system lifecycle, data governance, model training controls, model performance, monitoring and logging
- Human oversight controls (A.8): Human review mechanisms, explainability, user training
- Third-party controls (A.9): Supplier relationships, AIBOM, vendor risk assessment
The EU AI Act alignment: A 2025 ISACA paper documents a direct crosswalk between ISO 42001 Annex A controls and EU AI Act obligations — enabling organizations to use the SoA as simultaneous evidence for both ISO 42001 certification and EU AI Act compliance assessments. This is the "practical pairing" that makes ISO 42001 the preferred governance backbone for organizations with EU exposure.
What they are: The operational documents that specify how ISO 42001 controls are implemented in practice. ISO 42001 requires documented procedures for: AI risk assessment methodology; AI system lifecycle management; AI incident response; AI supplier assessment; model performance monitoring; and internal AIMS audit.
Key procedures for MVCC:
- AI System Intake Procedure: How a new AI system request is received, assessed, approved or rejected, and registered. Includes the data classification step, risk tier assessment, EU AI Act role determination, and vendor assessment initiation trigger.
- AI Risk Assessment Procedure: How AI-specific risks are identified, rated, treated, and documented. Linked to the risk register format and review cadence.
- AI Incident Response Procedure: The AI IR Annex covered in Module 5 below. Extends NIST SP 800-61 for AI-specific incidents.
- AI Shadow AI Discovery Procedure: How unauthorized AI use is detected (CASB logs, network monitoring, employee reporting), documented, investigated, and resolved.
What they are: The documented evidence that procedures were followed and controls are operating — the artifacts an auditor inspects to verify compliance. Without records, every claim in the policy and procedure documents is unverifiable assertion.
Key records for ISO 42001 / EU AI Act compliance:
| Record Type | Content | Retention (ISACA guidance) |
|---|---|---|
| AI Risk Assessment | Risk identification, rating, treatment decisions, residual risk, approver | Life of system + 3 years |
| AI System Approvals | Use case register entries, approval decisions, EU AI Act role determination | Life of system + 3 years |
| Vendor Assessments | Questionnaire responses, AIBOM, contract clause compliance evidence | Duration of relationship + 2 years |
| AI Incident Reports | Detection, analysis, containment, lessons learned, regulatory notifications | 5 years (regulatory exposure) |
| Human Oversight Logs | Evidence of human review before high-stakes AI-assisted decisions | Per decision type (e.g., employment: 7 years) |
| AI Literacy Training Records | Staff completion records; training content version; assessment scores | Duration of employment + 2 years |
| Internal AIMS Audit Results | Audit scope, findings, corrective actions, closure evidence | 5 years |
The ISACA cadence standard: The 2025 ISO 42001 / EU AI Act pairing paper specifies practical record-keeping cadences: quarterly risk reviews at 100% coverage; serious incident reports drafted within 72 hours of detection; and log retention of 180–365 days according to risk tier. These cadences are what turn governance documents from aspirational artifacts into operational evidence.
Module 5 · 40 minThe AI Incident Response Annex — Extending NIST SP 800-61 for AI
The NIST SP 800-61 incident response lifecycle — Preparation → Detection & Analysis → Containment, Eradication & Recovery → Post-Incident Activity — was designed for traditional information security incidents: data breaches, malware infections, unauthorized access. It is correct for AI incidents but incomplete. AI introduces incident types, detection challenges, and response actions that have no parallel in the 800-61 framework.
The AI IR Annex is a supplementary document — it does not replace your existing IR plan, it extends it. Every phase of the 800-61 lifecycle gains new requirements when the incident involves an AI system.
🔑 New Incident Types the Standard IR Plan Doesn't Cover
AI introduces incidents that have no analogue in traditional infosec: Hallucination cascade (model producing systematically false outputs at scale), Model poisoning discovery (finding that training data was tampered with), Prompt injection exfiltration (sensitive data extracted through adversarial inputs), Model drift (performance degradation that crosses a risk threshold), Deepfake impersonation (synthetic media used to impersonate institutional leaders), and Agentic AI runaway (agent takes unauthorized actions beyond its intended scope). Each of these requires detection methods, analysis approaches, and response actions not found in standard IR playbooks.
NIST SP 800-61 Extended for AI — Six Phases with AI Extensions
Click any phase to expand the AI-specific extensions for that lifecycle phase.
Traditional 800-61: Establish IR capability, train IR team, prepare detection tools, define incident categories, establish communication protocols.
AI Extensions:
- Maintain the AI Use Case Register with current deployment status (enables rapid scope identification during incidents)
- Establish model rollback capability for every deployed AI system before deployment — not after an incident occurs
- Define AI-specific incident categories: model behavior anomaly, data poisoning discovery, prompt injection exfiltration, agentic AI scope violation, deepfake impersonation, regulatory notification trigger
- Train IR team on AI-specific forensics: tool call logs, model version history, input/output log analysis, drift metric review
- Establish relationships with foundation model providers (OpenAI, Anthropic) for coordinated incident response on shared infrastructure incidents
- Pre-draft regulatory notification templates for EU AI Act Article 73 (serious incident reporting) and applicable sector regulators
Traditional 800-61: Monitor for security events, triage alerts, confirm incidents, determine scope, characterize attack.
AI Extensions:
- AI-specific detection signals: Model output quality metrics crossing drift thresholds (PSI, ADWIN), unusual input pattern clusters (potential adversarial probing), output content anomalies (systematic hallucination, unexpected data disclosure), tool call log anomalies (agent taking unapproved actions), and user complaint spikes correlated with specific input patterns
- The detection gap: Many AI incidents — including poisoning and backdoor activation — produce no traditional security alerts. Behavioral monitoring of model outputs is the primary detection mechanism, not log-based SIEM rules
- Analysis questions unique to AI incidents: Is this behavior consistent with a known model limitation (in the model card's risk section) or a new threat? Has the model version changed recently? Is the affected population consistent with a targeted attack? Is the behavior triggered by a specific input pattern (backdoor indicator)?
- Severity classification: Determine whether the incident triggers EU AI Act Article 73 serious incident reporting obligation (within 72 hours to the relevant market surveillance authority for high-risk systems)
Traditional 800-61: Isolate affected systems, stop the attack spread, preserve forensic evidence.
AI Extensions:
- Model isolation: Disable or restrict access to the affected AI system without necessarily taking it fully offline — may be possible to implement enhanced input filtering while investigation continues
- Traffic redirection: For API-based AI services, route requests to the prior model version (if rollback capability was established in Preparation) or to a manual process
- Evidence preservation for AI incidents: Capture current model version hash, input/output logs for the period surrounding the incident, tool call logs (for agentic incidents), and model behavioral metrics. Unlike traditional forensics, AI incident evidence is time-sensitive — logs may rotate and behavioral metrics may change as the model continues to operate
- Agentic AI containment: For incidents involving agentic AI, containment must include disabling tool access to prevent further autonomous actions — the equivalent of revoking network access for a compromised endpoint, but applied to the agent's tool permissions
Traditional 800-61: Remove malware, patch vulnerabilities, restore from clean backups, validate recovery.
AI Extensions:
- Model rollback: Restore the prior, known-clean model version. Validate using a clean test set that was not part of the incident investigation. Document the rollback in the AIBOM and model card version history
- Data remediation for poisoning incidents: If training data poisoning is confirmed, eradication requires identifying and removing the poisoned data from the training dataset, retraining the model, and re-running all safety evaluations on the retrained model before redeployment. This is a weeks-to-months process — not an overnight restore
- Prompt injection eradication: For prompt injection incidents, eradication involves patching the input validation layer, updating the system prompt to close the exploitation path, and regression testing the fix against the original attack vector
- Validation before redeployment: AI systems returning from an incident must pass: full safety evaluation, bias re-evaluation (to detect whether poisoning introduced bias), adversarial robustness testing against the incident's attack vector, and AI Governance Committee approval
Traditional 800-61: Lessons learned review, update procedures, improve detection, report to management.
AI Extensions:
- EU AI Act Article 73 notification: Providers of high-risk AI systems must report "serious incidents" to market surveillance authorities. The Annex must define what constitutes a serious incident (death or serious injury; infringement of fundamental rights; loss of critical infrastructure access), the 72-hour notification timeline, and the reporting pathway (EU AI Office for GPAI models; national authorities for high-risk systems)
- AI use case register update: Every AI incident must generate an update to the affected system's Use Case Register entry — documenting the incident date, type, severity, and outcome. This creates the historical risk record that informs future risk assessments
- Model card update: If the incident reveals a previously undocumented limitation, failure mode, or bias pattern, the model card must be updated to document it — and all deployers of that model version must be notified
- Governance program improvement: AI incidents are the highest-value source of improvements to the AI governance program. The lessons-learned process must explicitly ask: what governance control, if present, would have prevented or detected this incident earlier? The answer drives AIMS improvement per ISO 42001 Clause 10
AI Incident Classification Matrix
| Incident Type | Detection Signal | Severity Indicator | Regulatory Trigger | Primary Response |
|---|---|---|---|---|
| Hallucination Cascade | User complaints; output quality metrics; factual accuracy monitoring | Scale × severity of incorrect outputs; affected population size | EU AI Act Art. 73 if high-risk system; sector-specific if medical/legal | Implement enhanced output filtering; notify affected users; model rollback if widespread |
| Prompt Injection Exfiltration | Unusual output content; data classification alerts on AI outputs; SIEM anomaly | Data classification of exfiltrated content; affected individuals count | GDPR Art. 33/34 (data breach); state breach notification laws; EU AI Act Art. 73 | Disable affected endpoint; patch injection vulnerability; forensic log analysis |
| Model Poisoning Discovery | Systematic bias pattern; specific trigger-activated behavior; red team finding | Scope of affected training data; model deployment breadth; decision severity | EU AI Act Art. 73 for high-risk; notify other deployers of same model | Model isolation; forensic analysis of training data; full retrain required |
| Agentic AI Scope Violation | Tool call log anomaly; unauthorized action detected; affected system alert | Irreversibility of actions taken; data/financial impact; authorization bypass | Depends on action type; financial: FinCEN; health: OCR; general: board notification | Disable agent tool access; reverse actions where possible; redesign permission model |
| Model Drift — Threshold Breach | PSI/ADWIN metric crossing threshold; performance monitoring alert | Risk tier of affected system; decision type; affected population | EU AI Act Art. 72 (post-market monitoring) — update monitoring plan | Temporary manual review for affected decisions; model retraining; rollback if severe |
MVCC AI Governance Document Audit — What Would a Regulator Find?
Imagine that MVCC receives an inquiry from the Illinois Attorney General's office following a student complaint that an AI early-alert system incorrectly flagged them as "at-risk" and triggered unsolicited advising outreach, which the student found stigmatizing and inaccurate. The AG's office requests the following documents:
- The AI Acceptable Use Policy governing the system
- The AI Use Case Register entry for the early-alert system
- Evidence of human oversight procedures for AI-generated flags
- The vendor contract, including data processing and AIBOM provisions
- Any AI incident reports related to the system since deployment
Instructions:
- Think (5 min): Based on the governance documents covered today, which of these five document categories is MVCC most likely to have gaps in? Which would a regulator find most problematic? Write your analysis individually.
- Pair (10 min): Compare with a partner. Did you agree on the highest-risk gap? For the document MVCC is most likely to be missing or have inadequate, draft two sentences of remediation language you would want to see.
- Share (10 min): Class share-out. Instructor captures the consensus most-critical gap and the class's proposed remediation language.
- Debrief (5 min): This scenario involves an Illinois state law inquiry — not an EU AI Act enforcement action. What does this tell us about the geographic scope of AI governance obligations? How does a community college's exposure to US state AI law differ from its EU AI Act exposure?
🎓 Instructor Note
The scenario is designed to surface the human oversight documentation gap — the piece most institutions are missing. The student complaint focuses not on whether the AI was wrong (which is a technical question) but on whether there was a process for contesting the AI's output and a human accountable for the decision to contact the student. The class should debate: is human review of every AI early-alert flag operationally feasible? And if not, what governance alternative (tiered review, appeal process, student notification) is both feasible and compliant?
Draft the MVCC AI Acceptable Use Policy — Interactive Builder
Use the interactive builder below to draft MVCC's AI Acceptable Use Policy. The builder guides you through all eight components, prompts you with questions for each section, and generates a formatted policy preview. This is Part 2 of your Mini-Project — the draft you produce here, combined with the vendor risk assessment from Lesson 12, constitutes the complete Mini-Project submission.
MVCC AI Acceptable Use Policy — Draft Builder
Practice — not auto-submitted- All MVCC employees (full-time, part-time, temporary)
- Contractors and consultants with access to MVCC systems
- Students using MVCC-provided AI systems or tools
- Vendors whose products include AI features processing MVCC data
- Volunteers and board members with system access
- Entering FERPA-protected student data into AI tools without an approved DPA
- Making employment decisions (hiring, termination, promotion) based on AI output without documented human review
- Using AI to generate or distribute content that discriminates on protected characteristics
- Using AI to circumvent security controls, authentication systems, or access restrictions
- Deploying agentic AI that can take irreversible actions (financial transactions, data deletion, external communications) without human approval for each action
- Using AI systems that perform functions prohibited by EU AI Act Article 5 (social scoring, real-time biometric surveillance, subliminal manipulation)
- Using AI to generate content that falsely impersonates MVCC officials, faculty, or staff
- Using non-sanctioned AI tools for institutional purposes without completing the tool approval process
- Annually — minimum review regardless of changes
- When a new AI regulation becomes effective that affects MVCC
- Following a significant AI security incident at MVCC or at a peer institution
- When a new high-risk AI system is approved for deployment
- Following an external audit finding related to AI governance
MVCC AI Acceptable Use Policy
Draft generated by AI-103 Day 13 AUP Builder · Review and finalize with Legal Counsel before adoption · Version: Draft-[DATE]
📝 Assessment Artifact · Due: End of ClassAI AUP — Mini-Project Part 2 of 2
📋 Day 13 Assessment Artifact — Mini-Project Completion
AI Acceptable Use Policy — MVCC Draft (500–700 words + structured appendix)
Finalize the AI AUP draft you built in Activity C into a submission-ready policy document. The document must include all eight components as distinct, labeled sections, plus:
- Appendix A — Sanctioned Tools List: Classify at least 5 AI tools in the Tier 1/2/3 structure. For each Tier 1 tool, specify the data categories permitted and any conditions of use.
- Framework alignment statement: For each of the three frameworks covered in this course (NIST AI RMF, ISO 42001, EU AI Act), identify one specific section or article that your policy satisfies — and how. This demonstrates the policy is framework-anchored, not just internally invented.
- Policy gap self-assessment: Identify one component of your draft that you believe is weakest — the section most likely to fail an audit or regulatory inquiry — and explain specifically what additional language or evidence would close the gap.
Combined Mini-Project submission: This AUP (Day 13) + the Vendor Risk Assessment (Day 12) = the complete AI AUP + Vendor Risk Assessment Mini-Project (20% of course grade). Submit both documents together. The Vendor Risk Assessment should cross-reference the AUP's sanctioned tools list for the vendor assessed.
Grading: Evaluated on completeness of all 8 AUP components; specificity and enforceability of policy language (not vague platitudes); accuracy of framework alignment statements; and quality of gap self-assessment.
❓ Day 13 Knowledge Check · 6 Questions · Instant FeedbackTest Your Mastery
Formative check — does not affect your grade. Predicts performance on the Mini-Project.