🧭 Lesson Overview
Day 3 established the foundations: GOVERN set the organizational rules, MAP identified the risks for each specific system. Now comes the work that separates programs that look good from programs that actually work: MEASURE turns MAP's risk inventory into evidence — quantitative and qualitative — and MANAGE turns that evidence into decisions and actions.
This lesson goes deliberately deep. By the time you finish, you will have interacted with every subcategory of both functions, built a complete risk register entry, and made real governance decisions under simulated incident conditions.
MEASURE is the ICU monitor — the array of instruments (heart rate, oxygen saturation, blood pressure, EEG) that turn a patient's condition into readable numbers in real time. The monitor doesn't treat the patient. It makes the patient's condition visible enough to treat correctly. Without the monitor, decisions are guesses. With it, they're grounded in evidence.
MANAGE is the treatment protocol — the structured decision tree the care team follows when a number goes outside its safe range. Which intervention? Which dosage? Who approves? What is the escalation path? MANAGE answers those questions for AI risk: which treatment, which priority, who decides, how do we communicate, and how do we recover if something goes wrong.
MAP told you what to monitor. MEASURE does the monitoring. MANAGE tells you what to do when the monitor alarms.
Day 4 Learning Objectives
- Explain the role and purpose of the MEASURE function and how it differs from traditional software testing.
- Identify all four MEASURE categories and their 22 subcategories by code and purpose.
- Select appropriate metrics and evaluation methods for each NIST trustworthiness characteristic.
- Explain the role of MANAGE and the four risk-treatment options (mitigate, transfer, avoid, accept).
- Identify all four MANAGE categories and their 13 subcategories by code and purpose.
- Complete a risk register entry from MAP through MEASURE into MANAGE for a given AI scenario.
- Apply MANAGE decision-making to simulated AI incident scenarios including the go/no-go gate and the kill-switch trigger.
⚗️ Module 1 · Why AI Measurement is Fundamentally Different
The MEASURE function sits between MAP and MANAGE in the NIST AI RMF. It uses quantitative, qualitative, or mixed-method tools to analyze, assess, benchmark, and monitor AI risk. But it's not traditional software testing — and the differences matter enormously.
| Dimension | Traditional Software Testing | AI MEASURE Function |
|---|---|---|
| Repeatability | Same input → same output. Tests are deterministic. | Same input may produce different outputs. Evaluation must account for stochasticity. |
| Completeness | "Test to completion" — run all tests, check green/red, ship. | No test-to-completion. Evaluation covers distributions of inputs, not exhaustive cases. |
| Timing | Point-in-time gate at release. Pass → ship, done. | Continuous. Performance degrades silently in production due to data drift. |
| Who evaluates | QA team, often the same org that built it. | MEASURE 1.3 requires independent assessors. Separation of duties is mandatory. |
| What's measured | Functional correctness, performance, security. | All seven trustworthiness characteristics — including fairness, explainability, and privacy. |
| Failure mode | Bug: deterministic, reproducible, patchable. | Drift: probabilistic, gradual, may not be reproducible. Requires monitoring, not just patching. |
| Data as variable | Test data is fixed. Prod data is assumed stable. | Production data distribution shifts continuously. Retraining changes model behavior. |
| Human subjects | Not typically applicable. | MEASURE 2.2: evaluations involving humans require subject-protection protocols. |
A doctor doesn't draw blood once at your first appointment and declare you permanently healthy. Blood tests are repeated on schedule, with different panels for different risks, interpreted against reference ranges, and acted on when values drift. A single normal reading at deployment doesn't mean the system will stay healthy. MEASURE is the blood-work schedule for your AI system.
MEASURE: Four Categories at a Glance
| Category | Purpose | Subcategories |
|---|---|---|
| MEASURE 1 | Appropriate methods & metrics are identified and applied — prioritization, documentation, independence | 3 |
| MEASURE 2 | AI systems evaluated for all trustworthy characteristics — TEVV, safety, security, fairness, explainability, privacy, bias, environment | 13 |
| MEASURE 3 | Mechanisms for tracking identified risks over time — evaluation gaps, feedback, continuous improvement | 3 |
| MEASURE 4 | Feedback about efficacy of measurement — internal review, external review, documentation improvement | 3 |
🃏 Module 2 · MEASURE — All 22 Subcategories (Flip to Learn)
Click or tap each card to flip it. The front shows the subcategory code and short name. The back shows what it means and a concrete implementation note. Cover as many as you can before the Module 3 deep dive.
🔬 Module 3 · MEASURE Deep Dive — Expand Each Subcategory
Click any row to expand it. Left column: the official NIST statement + implementation notes. Right column: real example from the MVCC AI context + common failure modes. MS-1 MS-2 MS-3 MS-4 badges indicate which MEASURE category.
MEASURE 1 — Methods & Metrics Selection
MEASURE 2 — Trustworthy Characteristics Evaluation
MEASURE 3 & 4 — Tracking & Feedback
🃏 Module 4 · MANAGE — All 13 Subcategories (Flip to Learn)
MANAGE takes MEASURE's evidence and turns it into decisions. Per NIST AI 100-1 §5.4, after completing the MANAGE function, plans for prioritizing risk and regular monitoring and improvement will be in place. The four MANAGE categories cover: risk prioritization and treatment (MG-1), benefit maximization and harm minimization strategies (MG-2), third-party risk management (MG-3), and response, recovery, and communication (MG-4).
Every security professional knows the four risk treatment options from traditional risk management: Mitigate (implement a control), Transfer (insure it or contract it to a vendor), Avoid (don't do the thing), Accept (document that you're living with it). NIST AI RMF uses exactly the same four options — but applying them to AI requires new thinking. Avoid might mean "don't use AI for this decision." Transfer might mean "require the vendor to indemnify us." Accept must be documented and escalated for any risk above threshold — not just silently assumed.
🔬 Module 5 · MANAGE Deep Dive — Expand Each Subcategory
Click any row to expand it. Left column: official NIST statement + implementation notes. Right column: MVCC example + failure mode to avoid.
MANAGE 1 — Prioritization & Treatment
MANAGE 2 — Benefit Maximization & Harm Minimization
MANAGE 3 & 4 — Third-Party Risk & Response/Recovery
🎯 Activity A · Concept Match — Subcategory to Definition
Click a code on the left, then click its matching definition on the right. Correct pairs lock green. Wrong pairs flash red — try again. Match all 10 to complete.
📋 Activity B · AI Risk Register Builder
Build a complete AI risk register entry for the MVCC At-Risk Student Predictor from Day 3. Each field maps to a specific NIST AI RMF subcategory. Complete all fields, then generate the formatted entry. This is the artifact you'll submit.
⚡ Activity C · Scenario Decision Lab — Apply MEASURE & MANAGE
Each scenario describes a real situation at an AI-deploying organization. Choose the best NIST AI RMF MEASURE or MANAGE action. Instant feedback explains the rationale.
📊 Module · Evaluation & Assurance — Test Before You Trust
MEASURE is where governance meets evidence. Before an AI system is trusted in production, you evaluate it: you pick metrics, set acceptance thresholds in advance, run the tests (including red-team probes), and compare results to the thresholds to make a defensible deploy / no-go decision. "It seemed to work in the demo" is not evaluation.
Setting the bar before you see results prevents the most common assurance failure — moving the goalposts to justify shipping. When Morgan Stanley deployed its internal assistant, it ran formal factuality and hallucination evaluations with human review and expanded only once results cleared the bar. The discipline, not the model, is what made it trustworthy.
A usable AI evaluation plan names four things:
- Metrics — what you measure: robustness (jailbreak/prompt-injection resistance), safety (hallucination rate, PII leakage), performance (accuracy, latency), and fairness where relevant.
- Acceptance thresholds — the pass bar for each metric, set in advance (e.g., "hallucination ≤ 2%," "PII leakage = 0").
- Method & evidence — how you test (benchmark set size, red-team probe count) and what artifact records the result.
- Decision rule — how metric results roll up to deploy / no-go, including which metrics are critical (any failure = no-go).
⚖️ Activity D · "Set the Bar" — Read the Eval, Make the Call
Here is a mock pre-deployment evaluation for the MVCC Enrollment Assistant. For each metric, decide whether the measured result meets its acceptance threshold. Then make the overall deploy decision — treat hallucination and jailbreak resistance as safety-critical (any critical failure = no-go).
| Metric | Measured | Threshold | Your call |
|---|---|---|---|
| Hallucination rate (critical) | 6% | ≤ 2% | |
| Jailbreak resistance (critical) | 91% | ≥ 95% | |
| PII leakage (500 red-team probes) | 0 | = 0 | |
| Answer accuracy (200 benchmark Qs) | 93% | ≥ 90% | |
| P95 latency | 2.1s | ≤ 3s |
NIST AI RMF 1.0 (AI 100-1) — MEASURE; NIST Generative AI Profile (AI 600-1); MITRE ATLAS (adversarial testing); OWASP Top 10 for LLMs (2025) testing guidance.
📝 Assessment Artifact
Complete AI Risk Register Entry — Submitted via Activity B
Submit the completed risk register entry generated in Activity B. All ten fields must be completed; the generated entry is your submission artifact. Additionally, write a 150-word reflection answering: "Given your chosen treatment option, which MANAGE subcategory is most critical to implement first, and why?"
- Risk register entry: All ten fields, logically consistent (treatment must match priority level; residual risk must match treatment choice)
- Reflection: 150 words, references specific MANAGE subcategory code (e.g., MG-2.4), grounded in the scenario
- Connection to Day 3: Your MAP worksheet (Day 3) should have identified this risk. The register entry operationalizes that identification through MEASURE and MANAGE
Grading: Daily assignment category (25% of course grade). Evaluated on subcategory accuracy, logical consistency between fields, and depth of reflection.