Most of what gets published on AI risk assessment is written for a bank or a hiring department. Bias audits and model cards are fine, but nobody in a GMP facility is going to be asked about those first. An inspector will ask what the AI does, who checks it, what happens to product if it is wrong, and where the evidence lives. That is a different conversation, and in my view it is the one pharma companies should be preparing for.
This guide walks through how to assess risk for an AI system used in a pharmaceutical, biologic, or other regulated manufacturing and quality setting. It leans on the documents an auditor can actually hold you to: the FDA's January 2025 draft guidance on AI for regulatory decision-making, the draft EU GMP Annex 22, ISO/IEC 42001:2023, ICH Q9(R1), and 21 CFR Part 11.
What Is a Pharma AI Risk Assessment?
A pharma AI risk assessment is a documented evaluation of how an AI system could fail, what that failure would do to patient safety, product quality, and data integrity, and which controls bring that risk to an acceptable level. It sits inside your existing quality risk management program, not beside it.
That last point trips people up. Teams often treat AI as exotic and build a separate governance process that nobody in Quality owns. ICH Q9(R1), the 2023 revision of the quality risk management guideline, already gives you the structure: risk identification, analysis, evaluation, control, review. What changes with AI is the list of things that can go wrong and the kind of evidence you need to show the controls work.
Which Regulations and Standards Apply?
No single document governs pharma AI, so the practical answer is a stack. Here is how the main ones compare.
| Source | Status (as of this writing) | What it asks for | Where it bites |
|---|---|---|---|
| FDA, Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products (Docket FDA-2024-D-4689) | Draft guidance, January 2025 (confirm whether a final version has since been issued) | Seven-step, risk-based credibility assessment tied to a defined context of use | Submissions and evidence supporting safety, effectiveness, or quality decisions |
| EU GMP Annex 22, Artificial Intelligence | Draft released for consultation in 2025 (confirm current status) | Controls for AI used in GMP-regulated manufacturing; the consultation draft restricts critical applications to static models with deterministic outputs (cite the section of the current text in any procedure) | GMP manufacturing and quality operations in the EU |
| EU GMP Annex 11 (computerised systems) | Revision circulated alongside Annex 22 in the consultation (confirm current status) | Validation, data integrity, supplier oversight | Any computerised system touching GMP data |
| 21 CFR 211.68 and 21 CFR 211.22 | In force | Controls over computer systems; quality unit responsibility for decisions | US drug cGMP |
| 21 CFR 11.10(a) and 11.10(e) | In force | System validation; secure, time-stamped audit trails | Electronic records and signatures |
| ISO/IEC 42001:2023, clauses 6.1.2, 6.1.4, 8.2 | Published standard | AI risk assessment, AI system impact assessment, operational risk assessment | Voluntary, but a strong framework for an AI management system |
| ICH Q9(R1) | Adopted 2023 | Formal quality risk management with reduced subjectivity | The backbone for how you score and justify risk |
| Regulation (EU) 2024/1689 (EU AI Act), Article 9 | In force since August 2024, with phased application dates that may have shifted; confirm the current schedule | Risk management system for high-risk AI | AI that is a safety component of regulated products, such as some devices. Most pharma manufacturing and quality AI is not high-risk under the Act, so this row will not apply to many GMP use cases |
Draft documents move, and the statuses above were not re-verified on the date this article was last updated, so check the current status before you cite one in a procedure. The principle underneath them has been stable for a while: the more an AI output influences a decision that touches the patient, the more evidence you owe.
Start With the Context of Use
If I could get one habit into every pharma AI project, it would be writing the context of use before anything else. The FDA's January 2025 draft guidance makes it the first step of its credibility framework: define the question the AI addresses, then define the context of use, meaning exactly what the model does and how its output will be used.
A good context-of-use statement is boring and specific. "Uses an ML model to flag likely deviations in batch record review for human QA reviewer follow-up" is one. "Improves quality" is not. The difference matters because every later risk judgment hangs off it. A model that suggests a draft SOP summary and a model that decides whether a batch is released can run on the same technology and need completely different levels of control.
Write down four things:
- What the AI does and what it explicitly does not do.
- Who uses the output and what they do next.
- Whether a human reviews the output before it affects product, records, or patients.
- What data goes in, and where that data comes from.
How Do You Score AI Risk?
The FDA draft guidance assesses model risk using model influence, which is how much the AI output drives the decision relative to other evidence, and decision consequence, which is how bad it would be if the decision were wrong. Note that the guidance is scoped to AI supporting regulatory decision-making for drugs and biologics, not to GMP shop-floor use in general. Applying it to GMP systems is an extension by analogy. I find the pairing useful because it lines up with how QA people already think about severity and detectability.
Here is a simple matrix I use as a starting point. The Low/Medium/High cell values are my own illustration, not FDA's, so adapt them to your own risk procedure.
| Low decision consequence | Medium decision consequence | High decision consequence | |
|---|---|---|---|
| Low model influence (human decides, AI is one input) | Low | Low | Medium |
| Medium model influence | Low | Medium | High |
| High model influence (AI output is the decision) | Medium | High | High |
Examples help. An AI drafting meeting minutes for a management review sits in the top-left corner. An AI that auto-dispositions environmental monitoring excursions sits close to the bottom right, because the output is the decision and the consequence can reach product. A tool that prioritizes which complaints to triage first, with a person reading every complaint, lands in the middle.
The matrix is a prompt for judgment, not a replacement for it. ICH Q9(R1) is explicit that risk-based decisions should be supported by knowledge and that subjectivity should be managed, so write down why each cell was chosen.
What Failure Modes Should You Look For?
Generic AI risk lists are long. For a regulated manufacturing or quality setting, I would narrow the field to these categories and ask for each one what the failure would look like in your process.
Data and Data Integrity Risks
Training data that is incomplete, mislabeled, or drawn from a period when the process behaved differently produces a model that looks fine in testing and drifts in production. Part 11 section 11.10(e) expects audit trails that record operator entries and actions; an AI that edits, summarizes, or generates a record needs the same traceability, including which model version produced which output.
Model Performance and Drift
A model validated on last year's product mix may not hold when you add a new strain, a new supplier of raw material, or a new site. The risk assessment should name the performance metric that matters (false negatives for a defect detector, for example), the acceptance threshold, and the monitoring that tells you when performance slips.
Generative AI and Hallucination
Large language models can produce fluent, confident text that is wrong. In a quality system that is a data integrity problem, because a fabricated citation to an SOP section or an invented root cause could end up in a record. The draft Annex 22 takes a conservative line on generative and dynamic models for critical GMP applications (see the table above), which tells you where European inspectors are likely to start. I wrote separately about why consumer chatbots are a poor fit for quality records, and the short version is that uncontrolled prompts and unversioned models do not survive an audit trail review.
Human Oversight Failures
Automation bias is the quiet one. When an AI is right nearly every time (say 98 times out of 100, a purely illustrative figure), reviewers stop looking hard on the 99th. Your assessment should ask whether the human check is real, meaning the reviewer has the time, the information, and the authority to disagree, or whether it is a signature on a screen.
Supplier and Change Risks
Most pharma AI is bought, not built. If the vendor updates the model quietly, your validated state may change without a change control. Annex 11 and 21 CFR 211.22 both point back to the quality unit remaining responsible, regardless of who wrote the code. Your supplier assessment should cover model versioning, change notification, and access to the performance evidence you need.
Security, Privacy, and Confidentiality
Prompts and uploaded documents can leave your environment. Formulations, batch data, and patient information belong on a list of what must never be sent to an external model unless contractually and technically controlled.
A Step-by-Step Method
This is the sequence I would follow for a new AI use case. It borrows from the FDA's credibility framework, ISO/IEC 42001, and ordinary QRM practice. The FDA draft's seven steps are: (1) define the question of interest, (2) define the context of use, (3) assess the AI model risk, (4) develop a plan to establish model credibility, (5) execute the plan, (6) document the results and any deviations, and (7) determine the adequacy of the model for the context of use. My steps below fold these into a GMP-oriented sequence.
Step 1: Define the context of use and intended outcome
Write the context-of-use statement described earlier, and get Quality, the process owner, and IT to agree on it before anything else is built or bought.
Step 2: Classify the risk
Apply the matrix above, or your own equivalent, and document the reasoning. The class sets how much validation evidence you need.
Step 3: Identify failure modes and hazards
Walk the categories above with the people who actually run the process. A short FMEA-style table works well: failure mode, effect, cause, existing controls, detectability. ISO/IEC 42001 clause 6.1.2 expects an AI risk assessment process, and clause 6.1.4 requires an AI system impact assessment covering potential consequences for individuals and groups and for society (check the exact wording against your copy of the standard), which for pharma means patients first.
A sample row for a deviation-flagging model might look like this:
| Field | Example entry |
|---|---|
| Failure mode | Model fails to flag a batch record entry that is out of specification (false negative) |
| Effect | Deviation goes unreviewed; potentially affected batch proceeds toward release |
| Cause | Training data under-represented a new product strain; drift after a process change |
| Existing controls | QA reviewer still checks every record; model version locked; monthly false-negative check on a sample |
| Detectability | Medium: caught only if the reviewer or the sample check notices |
| Score (illustrative) | Severity High, likelihood Low, detectability Medium: risk Medium; add a drift trigger and a tighter sampling rate |
Step 4: Decide on controls
Controls come in layers: technical (input validation, confidence thresholds, locked model versions), procedural (human review requirements, escalation criteria), and organizational (training, roles, periodic review). Under ISO/IEC 42001 clause 6.1.3 you also need a statement of applicability against Annex A controls if you are pursuing certification.
Step 5: Validate in proportion to risk
The FDA draft guidance calls for a credibility assessment plan with activities matched to model risk. For a low-risk use, that may be a documented fit-for-purpose test against known cases. For a high-risk use, expect representative test data, independence between training and test sets, performance against predefined acceptance criteria, and a clear account of limitations. GAMP 5 Second Edition's emphasis on critical thinking over documentation volume is a good companion here. If you are building the paperwork, AI-assisted validation protocol generation can cut the drafting time, though a qualified person still owns the content.
Step 6: Release under change control
Put the AI system, its model version, and its configuration under the same change control as any other GMP computerised system. Record who approved the move into production and on what evidence.
Step 7: Monitor, review, and re-assess
The FDA framework includes life-cycle maintenance for a reason. Set a review frequency tied to the risk class, define drift triggers that force a re-assessment, and track incidents. ISO/IEC 42001 clause 8.2 is the operational counterpart to clause 6.1.2: it expects AI risk assessments to be performed at planned intervals or when significant changes occur. Feed the results into management review, since ISO/IEC 42001 and ISO 9001:2015 clause 9.3 both expect leadership to see how the system is performing.
What Does a Good Risk Assessment Record Look Like?
Inspectors do not read a risk assessment for style. They read it to see whether you understood the system and whether the controls match what you found. A solid record usually contains:
- The context-of-use statement and system description, including model type and version.
- The risk classification with rationale.
- The failure mode analysis with the controls mapped to each risk.
- Validation or credibility evidence, with acceptance criteria stated before testing.
- Supplier evaluation results where the AI is third-party.
- Monitoring plan, drift triggers, and review schedule.
- Approvals by Quality, as 21 CFR 211.22 makes the quality unit responsible for approving or rejecting what affects product.
If a reviewer can trace one hazard from identification through control through evidence in under five minutes, you are in good shape. If they cannot, the problem is rarely the AI. It is usually that the assessment was written once, filed, and never connected to the system as it runs.
Common Mistakes I See
The first is assessing the vendor instead of the use. A vendor's SOC 2 report says something about their security; it says nothing about whether their model is fit for classifying your deviations.
The second is treating "human in the loop" as a control without testing it. Ask how often reviewers override the AI. If the answer is almost never, you may have a rubber stamp.
The third is skipping the assessment for "just a productivity tool." Drafting help that feeds a controlled document becomes part of the quality system the moment its output is approved. Many of the same lessons that explain why QMS implementations fail apply here: unclear ownership, no change discipline, and training that stops at go-live.
The fourth is waiting for final guidance. The FDA document is a draft and Annex 22 has been out for consultation, but the expectations they describe are consistent with requirements already in force: validated computerised systems, data integrity, and quality unit oversight. Companies that build on those now will not be starting over when the final text arrives.
Where Does This Leave You?
I have come to think the best AI risk assessments in pharma read like good deviation investigations: specific, honest about what is unknown, and clear about what the company will do if it turns out to be wrong. The regulators are asking for the same kind of thinking they have always asked for, applied to a technology that makes it easier to skip the thinking.
So which of your current AI uses has an owner, a context of use, and a monitoring plan today? The ones that do not are where I would start. Then recheck the regulatory statuses above, since they may have moved.
Frequently Asked Questions
Does the FDA require a risk assessment for AI used in pharmaceutical manufacturing?
There is no AI-specific regulation that says so in those words, but existing rules reach the same place. 21 CFR 211.68 requires controls over computer systems, 21 CFR 11.10(a) requires validation of electronic record systems, and 21 CFR 211.22 puts the quality unit in charge of decisions affecting product. The FDA's January 2025 draft guidance (Docket FDA-2024-D-4689) describes a risk-based credibility assessment for AI used to support regulatory decisions about drugs and biologics.
What is the difference between ISO/IEC 42001 and FDA expectations for AI risk?
ISO/IEC 42001:2023 is a voluntary management system standard. Clause 6.1.2 requires an AI risk assessment process and clause 6.1.4 requires an AI system impact assessment. FDA guidance and the GMP regulations are about product quality, patient safety, and data integrity for specific uses. Many companies use ISO/IEC 42001 as the organizational framework and the FDA and GMP expectations as the use-level requirements.
Can generative AI or large language models be used in GMP operations?
It depends on the use and the controls. The consultation draft of EU GMP Annex 22 restricts critical GMP applications to static models with deterministic outputs and treats dynamic and generative models more cautiously; confirm against the current text before quoting it in a procedure. For lower-risk uses with a human reviewing every output, many companies allow them under documented procedures. In both cases the model version, prompts, and outputs need to be controlled and traceable.
How often should an AI risk assessment be reviewed?
Tie the frequency to the risk class and to events. There is no mandated interval. In my practice, a high-risk system is reviewed at least annually and on any model update, data source change, process change, or performance drift trigger. A low-risk system might be reviewed with the periodic review of the computerised system. ICH Q9(R1) expects risk review to be an ongoing activity informed by experience.
Who should own the AI risk assessment?
Quality should own the assessment of any AI that affects GMP decisions or records, with input from the process owner, IT, data science, and the supplier. 21 CFR 211.22 places responsibility for approving or rejecting product-affecting decisions with the quality unit, and that does not change because an algorithm produced the recommendation.
Jared Clark, JD, MBA, PMP, CMQ-OE, CQA, CPGP, RAC, is Principal Consultant at Certify Consulting (https://certify.consulting).
Last updated: 2026-10-04. Regulatory statuses in this article were not re-verified on that date; confirm the current status of each draft before relying on it.
Jared Clark
Principal Consultant, Certify Consulting
Jared Clark is the founder of Certify Consulting, helping organizations achieve and maintain compliance with international standards and regulatory requirements.