Vendor & Supply Chain Risk for AI

Situation: MVCC licenses an AI tutoring platform from a US edtech vendor (Khanmigo-style). MVCC uses it as-is for developmental math students. MVCC does not modify the model, does not add its own branding to the AI system, and does not alter its intended purpose.

Role determination: MVCC is a Deployer. The vendor is the Provider. MVCC's obligations: implement human oversight, train staff on AI literacy, retain logs, monitor for risks, inform students when they are interacting with AI (EU AI Act Art. 50 transparency requirement if system is in scope), and use the system per the provider's instructions for use.

The governance gap: MVCC is obligated to implement human oversight — but can it actually do so if the vendor's platform does not expose the outputs or flagging mechanisms needed? This is a contractual question as much as a technical one. The vendor contract must include provisions for oversight access, audit rights, and incident notification.

Situation: MVCC's IT department fine-tunes an open-source LLM (Llama 3) on internal student success records to predict at-risk students and automatically generate advising interventions. MVCC deploys this system for use by advisors.

Role determination: MVCC is now a Provider under Article 25. By fine-tuning and deploying a purpose-built AI system under MVCC's own authority for a high-risk use case (education AI, Annex III category 3), MVCC has crossed the provider threshold. This means MVCC now carries all provider obligations: risk management system, data governance, technical documentation, conformity assessment, post-market monitoring, and incident reporting — obligations that were designed for commercial AI companies with dedicated compliance teams, not a community college IT department.

The governance lesson: Fine-tuning on internal data for a purpose-specific use case is the most common way a deployer accidentally becomes a provider. Organizations must assess the Article 25 trigger before any fine-tuning project begins.

Situation: A US technology reseller distributes a high-risk AI hiring screening tool built by a German company. The tool is used by US employers. The EU AI Act has extraterritorial reach if the system produces outputs used in the EU — but in this scenario the outputs are used only in the US.

Role determination: This scenario illustrates the extraterritoriality question. If the tool's outputs affect EU persons (e.g., EU citizens applying for jobs), EU AI Act obligations may apply regardless of the jurisdiction of deployment. The US reseller should conduct a jurisdictional assessment and obtain the provider's declaration of conformity as a precaution — especially if any EU persons may interact with the system as job applicants. Even outside the EU, the EU AI Act's documentation standards are becoming a de facto global baseline that enterprise procurement teams require.

Organizations commonly perform the following actions believing they remain deployers. Each of these can trigger Article 25 provider obligations:

Examples: Tableau AI, Power BI Copilot, Salesforce Einstein, SAP AI, Oracle Analytics AI.

Embedded risks: These tools process your most sensitive business data — financial records, customer PII, employee data, strategic plans — and send queries or full data context to foundation model APIs to generate natural-language summaries, anomaly explanations, and forecasts. The data processing agreement you signed for "analytics" may not cover AI model training or cross-tenant data use.

Questions to ask your vendor: Does AI processing occur within your tenant or in a shared environment? Is your data used to improve the vendor's AI model? What data is retained by the AI inference system? Is the AI inference endpoint covered by your existing DPA?

Governance response: Audit all analytics tools for embedded AI features. Require data processing addendum updates that explicitly cover AI inference. Disable AI features in tools processing regulated data (FERPA, HIPAA, GLBA) until the DPA is updated and validated.

Examples: Microsoft 365 Copilot, Google Workspace Gemini, Zoom AI Companion, Slack AI, Notion AI.

Embedded risks: These tools have access to the broadest possible organizational data surface — email, calendar, documents, chat history, meeting transcripts. AI features summarize, draft, and search across all data the user can access. A misconfigured permission model means Copilot can surface data the user shouldn't see — creating an AI-amplified privilege escalation path.

The MVCC example: If MVCC activates Microsoft 365 Copilot without first auditing SharePoint permissions, student FERPA-protected data stored in poorly scoped SharePoint folders becomes discoverable by any faculty member whose Copilot query touches that content. The AI didn't create the permission problem — it made an existing permission problem catastrophically efficient.

Governance response: Audit and remediate data permissions before activating productivity AI features. Configure AI feature scope — most enterprise AI platforms allow restriction to specific data domains. Include AI feature scope in change management review for all major SaaS product updates.

Examples: AI-powered applicant tracking (Workday, iCIMS AI), AI resume screening, student risk prediction, automated advising flags.

Embedded risks: AI in HR and education decisions falls squarely within EU AI Act Annex III high-risk categories: employment and workers management (Annex III, pt. 4) and education and vocational training (Annex III, pt. 3). A vendor who adds AI screening features to their ATS is now providing a high-risk AI system — and the deploying organization becomes a deployer of that high-risk AI system, with all associated obligations.

The bias liability: AI hiring tools have documented disparate impact across protected characteristics. The EEOC and multiple state laws (Illinois, New York City) already require disclosure and bias auditing of AI in hiring. An organization that unknowingly uses an AI-powered feature in their ATS is still liable under these laws.

Governance response: Require vendors providing HR or student systems to identify and document every AI feature that influences decisions about individuals. Conduct bias audits on any AI feature used in high-stakes decisions. Ensure vendor contracts require notification of new AI features before activation.

Examples: AI-powered SIEM (Microsoft Sentinel AI, Splunk AI), endpoint detection, network anomaly detection, AI-assisted vulnerability management.

Embedded risks: Security AI tools process the most sensitive internal data: network traffic, authentication logs, user behavior patterns, incident data. If the vendor's AI model is cloud-hosted, this data leaves your perimeter for inference. A poisoning attack against the security AI could suppress detection of specific attack patterns — creating a blind spot in your security monitoring precisely where an attacker wants one.

Governance response: Require security AI vendors to document data residency for AI inference. Assess whether AI model updates could introduce new blind spots. Maintain a baseline of manual detection rules that do not depend on AI — so that a compromised security AI does not eliminate all detection capability.

Purpose: Requires the vendor to provide an AI Bill of Materials (AIBOM) at contract execution and on every material change to the AI system — enabling the deployer to inventory AI components, track vulnerabilities, and meet regulatory documentation obligations.

Sample language: "Vendor shall provide Customer with a complete AI Bill of Materials (AIBOM) in CycloneDX ML-BOM v1.7 or SPDX 3.0 AI Profile format within 30 days of contract execution and within 15 business days of any material change to the AI system, including model version changes, dataset changes, or infrastructure changes that affect the AI inference pipeline."

Negotiation note: Vendors relying on third-party foundation models (OpenAI, Anthropic, Google) may argue they cannot provide AIBOM fields for the foundation model layer. Accept this limitation but require documentation of the specific model version, API endpoint, and any fine-tuning performed on your organization's data.

Purpose: Requires the vendor to notify the deployer before making material changes to AI systems — including model updates, new AI features, changes to data processing, or changes to the inference provider. Prevents vendors from silently upgrading foundation models or activating new AI capabilities without deployer review.

Sample language: "Vendor shall provide Customer with not less than [30/60] calendar days' prior written notice of any material change to the AI features of the Services, including but not limited to: changes to the underlying AI model or model version; changes to training data or fine-tuning datasets; changes to the AI inference provider or data processing location; and activation of new AI capabilities that process Customer Data."

Negotiation note: Vendors often resist long notice periods for AI changes, arguing that rapid AI development requires faster iteration. A reasonable compromise: 30-day notice for changes affecting regulated data processing; 14-day notice for other AI capability changes; emergency changes with retroactive notification within 72 hours plus a root-cause summary.

Purpose: Grants the deployer the right to audit or commission audits of the vendor's AI system — including third-party bias audits, red team assessments, and security penetration testing of AI components. Also requires vendors to provide evidence of their own AI testing programs.

Sample language: "Vendor shall (a) maintain a program of regular adversarial testing and bias evaluation for the AI systems provided under this Agreement; (b) provide Customer with a summary of testing results annually upon request; and (c) permit Customer or Customer's designated third-party auditor to conduct or commission AI-specific assessments, including bias audits and security assessments of AI inference pipelines, no more than once per year upon [30] days' written notice."

Negotiation note: Most enterprise AI vendors will resist direct audits of their core AI systems (citing IP protection and multi-tenant security). Acceptable compromise: accept SOC 2 Type II reports that include AI-specific trust service criteria, third-party bias audit reports, and the vendor's published penetration testing schedule — while retaining the right to conduct your own assessment of the integration layer that processes your specific data.

Purpose: Defines the vendor's obligations when an AI-specific incident occurs — including model compromise, data poisoning discovery, significant performance degradation, or AI-generated output causing documented harm.

Sample language: "Vendor shall notify Customer within [24/72] hours of discovering any AI-specific security incident, including: (a) evidence of unauthorized access to or modification of AI model artifacts; (b) discovery of data poisoning affecting AI system behavior; (c) AI system outputs that have caused or are likely to cause documented harm to individuals; and (d) regulatory investigations or enforcement actions related to the AI systems provided under this Agreement. Vendor shall provide rollback capability for AI model changes and shall document rollback procedures in the Service Level Agreement."

Negotiation note: The definition of "AI-specific incident" is the critical negotiation point. Vendors prefer to limit notification to traditional security incidents (breaches, unauthorized access). Push for explicit inclusion of model behavior changes, bias discoveries, and regulatory investigations — all of which affect the deployer's governance obligations.

Purpose: Explicitly prohibits the vendor from using the deployer's data — including user queries, uploaded documents, and inferred behavioral signals — to train or improve the vendor's AI models. Prevents the organization's proprietary or regulated data from becoming training data for a shared model.

Sample language: "Vendor shall not use Customer Data, Customer User Data, or any data derived from Customer's use of the Services (including but not limited to: user queries submitted to AI features, documents processed by AI features, and output feedback signals) to train, fine-tune, evaluate, or improve Vendor's AI models, except with Customer's express written consent and subject to Customer's review and approval of the training methodology and data governance documentation."

Negotiation note: This clause is frequently pre-included in enterprise AI contracts at the request of large customers. If the vendor resists, treat this as a Tier 1 red flag — a vendor that insists on the right to train on your data is prioritizing their model improvement over your data governance obligations. For FERPA-protected data, this restriction is not negotiable; it is a legal requirement.