AI risk & quality
AI fails silently, gradually, and unpredictably. The PM who understands AI-specific risk signals seniority in every interview and every incident.
Unlike traditional software that fails loudly (exceptions, 500 errors), AI fails silently. A model can degrade for weeks while passing automated checks. These are the six risk categories every AI PM tracks — with likelihood, severity, and the PM-owned signal for each.
Hallucination (confident wrong outputs), distributional shift (production inputs differ from training), adversarial inputs (deliberate manipulation), and model drift (gradual degradation as the world changes). All are normal — the PM job is to design for detection and recovery, not prevention alone.
Correction rate rising, user overrides increasing
Golden test set weekly regression + correction rate dashboard from Day 1.
GPT-4 Bing hallucinated legal citations; users filed complaints before monitoring flagged it. Normal behavior — the PM's job is to design detection, not assume prevention.
Quality in AI products requires four layers — technical performance alone is insufficient. A model with 91% precision that takes 8 seconds is not production-ready. A model with 94% precision whose correction rate is rising is already failing. Define all four layers in the DoD before sprint 1.
Precision ≥ target, latency p99 < SLA, error rate < 0.1%.
Set go/no-go thresholds in the DoD before sprint 1. Review weekly with engineering. Necessary but not sufficient — a model with 91% precision that takes 8 seconds is not production-ready.
Silent Failure Timeline
AI degrades without error codes. A model returning syntactically valid but increasingly wrong outputs will pass automated monitoring for weeks. Correction rate leads accuracy degradation by 2–3 weeks — build it into the production dashboard from Day 1.
Correction rate 3%. No alerts. Everything looks fine.
Correction rate 6%. No monitoring alert yet. Drift is happening.
Accuracy drops 8%. Alert fires. 2-week delay cost trust.
AI incidents follow a predictable pattern — but the response path differs by root cause. The two most common PM mistakes: starting with diagnosis instead of containment, and treating post-mortems as documentation instead of producing DoD changes.
Symptom Router
Select a symptom to see severity level and immediate response path.
6-Step Incident Playbook
Detect
Monitoring alert, correction rate spike, user report, or business metric drop. The earlier you detect, the smaller the blast radius. Silent degradation is hardest — correction rate is your early warning system.
Contain
Increase review threshold, reduce automation level, or disable the AI feature and fall back to rules or human path. The fallback must be designed and tested before launch — not improvised during an incident.
Diagnose
Three root cause categories: model issue (drift, adversarial input) · data issue (stale training, distribution shift) · integration issue (upstream API changed). Each has a different fix path — diagnosing correctly is worth the time.
Fix
Hotfix the prompt (fastest) · adjust confidence threshold · retrain · rollback to previous version · revert to rules-based. Choose based on severity, timeline, and root cause. Retraining takes days — plan for interim measures.
Communicate
What happened, who was affected, what we did to contain it, what we're doing to fix it, when we expect resolution. Communicate early with partial information and update as you learn more — silence is worse than uncertainty.
Post-Mortem
Root cause, contributing factors (what monitoring gap let it go undetected?), DoD improvements (what change would catch this next time?). Post-mortems must produce a change to the model spec or DoD — not just a document.
Risk management frameworks provide a common language for cross-functional teams and a structured audit trail for regulators. Four frameworks matter most for AI PMs — each serves a different purpose and triggers different PM actions.
NIST AI RMF
US federal — voluntary but widely adopted by enterpriseUse the four-function model to structure your risk register: Govern (establish risk culture) → Map (identify risks) → Measure (quantify them) → Manage (mitigate). "Map" is your risk taxonomy. "Manage" is your DoD gates.
EU AI Act
EU law — mandatory for products in EU market, effective 2026Classify your system first. Annex III lists high-risk categories: biometric, employment, credit, law enforcement, critical infrastructure. High-risk = mandatory conformity assessment, technical documentation, and human oversight requirements.
ISO/IEC 42001
International — first ISO management system standard for AIUse as a maturity framework for your org's AI governance practices. Auditable evidence of responsible AI. Increasingly required in enterprise procurement and government contracts — treat it like SOC 2 for AI.
MLOps Risk Controls
Engineering practice — operationalizes risk within the dev pipelineEnsure each MLOps control point has a PM-owned decision: data validation gates, model evaluation thresholds, canary deployment criteria, rollback triggers, drift monitoring alerts. Engineering builds the checks — PM defines what passes.
AI risk is a core competency test in senior PM interviews. The key differentiators: owning all four quality layers, always containing before diagnosing, and treating post-mortems as DoD improvements — not documentation exercises.
How do you manage quality in an AI product where the model's behavior can change over time?
Define quality across all four layers before launch: technical (precision, latency), output (correctness on golden test set), user behavior (correction rate, task completion), and business outcome (automation rate, error cost).
Build monitoring as a product feature, not a post-launch backlog item. Correction rate and override rate are my canaries — they catch model drift 2–3 weeks before accuracy metrics degrade.
Design the fallback path before launch. Every AI feature ships with a pre-tested procedure to increase review threshold or disable the feature entirely. I don't design fallbacks under incident pressure.
Walk me through how you'd handle an AI feature that's performing worse in production.
First, contain — raise the review threshold or reduce automation to stop blast radius from growing. I don't wait for a root cause to contain.
Diagnose in parallel: model issue (drift, adversarial input), data issue (stale training, distribution shift), or integration issue (upstream API changed). Each category has a different fix path.
Fix based on root cause: prompt hotfix (fastest), threshold adjustment, retrain, or rollback. Communicate early with partial information — silence is worse than uncertainty. Run a post-mortem within 2 weeks. I own it. One action item per monitoring gap, closed within 2 sprints.
What AI-specific risks do you track that don't exist in traditional software products?
Traditional software fails loudly — exceptions, 500 errors, null pointer crashes. AI fails silently and gradually. That changes the entire monitoring model.
The six categories I track: model risk (hallucination, drift, adversarial inputs), data risk (training bias, stale data, PII leakage), integration risk (prompt injection, cascading agent failures), operational risk (HITL burnout, reviewer drift), reputational risk (viral failure modes), and regulatory risk (EU AI Act classification, GDPR in training data).
The hardest to catch is operational risk — reviewer fatigue is invisible until accuracy has already degraded.
You're launching an AI hiring tool. What regulatory considerations apply and how do you prepare?
AI hiring tools appear explicitly in EU AI Act Annex III as high-risk. That means mandatory conformity assessment, technical documentation, human oversight requirements, and EU database registration before market placement.
In the US, EEOC guidance applies — disparate impact analysis required. Amazon's recruiting tool is the reference failure: trained on 10 years of male-dominated hires, it systematically downranked women and passed all technical metrics.
My pre-launch checklist: legal sign-off on training data provenance, disaggregated eval by gender/race/age before launch, human review of all rejections (not just a sample), documented algorithmic decision audit trail, and an appeal process — required by EU AI Act for automated decisions affecting employment.
How do you decide where to set the confidence threshold for an AI classifier in production?
The threshold decision is fundamentally about error cost asymmetry. First: what is the cost of a false positive vs. a false negative for this specific product?
For fraud detection: false negative (missed fraud) costs $10k, false positive (blocking a good customer) costs $30 in support plus churn risk. That asymmetry tells me to favor recall over precision.
For medical triage: false negative is catastrophic, so I set a low threshold (more recalls to human) even if it floods the review queue. HITL throughput then becomes a constraint I must solve for separately.
I set the initial threshold in shadow mode — run production traffic against multiple thresholds and compare queue size, error rate, and correction rate. Never set it analytically — always test it empirically at production scale.