AI metrics & evals
The metrics that define AI PM credibility — from precision-recall to LLM eval, guardrails to business KPIs.
Core model quality metrics
Select a metric to see what it measures, when to prioritize it, a real-world example, and what the PM owns. These are the metrics every AI PM interview tests in the first 10 minutes.
Of all predictions labelled positive, how many actually were? High precision = low false positive rate.
When false positives are costly — auto-approving a wrong transaction, firing a false fraud alert, flagging a healthy patient.
The precision target. Not the model architecture that achieves it.
Salesforce Einstein lead scoring: high precision means account managers only call leads that are genuinely likely to close. One wasted call costs $400–$800 in enterprise sales.
Error cost asymmetry — the PM decision framework
Precision vs. recall is not a model decision. It is a business decision. The PM quantifies the cost of each error type and sets the threshold accordingly. Here is the framework across three domains.
LLM-specific evaluation metrics
Classic ML metrics (accuracy, F1) are insufficient for LLM products. These six metrics are now tested in every senior AI PM interview. Select a metric to see the PM ownership layer and real deployment examples.
% of responses containing factually incorrect or fabricated information not grounded in the provided context.
Guardrail — typically set at < 2% for enterprise deployments. Exceeding this halts deployment.
LLM-as-Judge rubric + human spot-check of 100–200 responses per eval cycle. Also tracked in production via user correction signals.
The threshold. Sets the acceptable ceiling and triggers revert if breached post-launch.
Microsoft Copilot for M365: hallucination rate above 3% in early testing delayed general availability by 4 months while RLHF fine-tuning improved grounding.
Classic ML vs. LLM — what changes
- Precision, Recall, F1, AUC-ROC
- Binary or multi-class output
- Deterministic given same input
- Single threshold controls quality
- Accuracy degrades gradually (drift)
- Faithfulness, groundedness, hallucination rate
- Free-text output — open-ended evaluation
- Stochastic — same input gives different outputs
- Guardrail + goal metric split required
- Can fail suddenly on distribution edge cases
Guardrail vs. goal metric
A guardrail metric is a floor — breaching it triggers a halt or rollback, period. A goal metric is a ceiling — what you optimize toward when guardrails are met. They are never traded against each other.
Select an AI product type to see how the metrics split — and what the real threshold values look like.
The rule a PM must never break
Guardrail metrics and goal metrics are never traded against each other in a product review. This sentence will come up in an interview — know it cold.
"Our hallucination rate went up 4% but our task completion rate improved 18% — I think it is worth the trade-off."
"Our hallucination rate exceeded the 2% guardrail. I am halting the rollout and escalating to engineering to diagnose the regression before we resume the A/B test."
The four-phase eval plan
Every AI feature deployment should follow this structure. Candidates who can describe all four phases with exit gates and PM ownership are immediately differentiated from those who only know offline evaluation.
Run the model on a golden test set — 200–500 hand-labeled examples that represent real production inputs. Measure all target metrics against ground truth. This is the first gate: if offline metrics do not meet the launch threshold, do not proceed to shadow mode.
The golden set composition and the minimum quality threshold to advance.
Using only the training set to evaluate. The golden set must be held out and never touched during training.
Run the new model on live traffic in parallel with the production model. Log outputs but do not serve them to users. Compare divergences between the shadow and production models using domain expert review. Only proceed when shadow mode performance matches or exceeds production.
Defining the divergence threshold that triggers a stop. Reviewing expert samples.
Skipping shadow mode when pressure to ship is high. Shadow mode catches distribution shift that offline eval misses — it is not optional for model rollouts.
Expose the new model to a small percentage of live traffic (typically 5–10%). Measure both model metrics and business KPIs in parallel. A/B test accounts for novelty effects and real-world behavior differences that shadow mode cannot capture.
The rollout percentage, the experiment duration, the primary success metric, and the guardrail metrics that trigger auto-rollback.
Running A/B tests too short to reach statistical significance. For low-frequency events (churn prediction), you may need 4–8 weeks.
Monitor the three layers: model metrics (precision, recall, hallucination rate), business metrics (automation rate, correction rate, CSAT), and infrastructure metrics (latency P99, cost per query). Set alert thresholds and automatic rollback triggers. This is the ongoing PM job, not the launch.
Alert thresholds, escalation protocol, and the decision to retrain vs. rollback vs. accept degradation.
Treating launch as the finish line. AI features degrade over time as data distributions shift. Monitoring is not optional.
Metric lifecycle — how goals evolve
The metrics that matter at MVP are not the metrics that matter at scale. Senior AI PMs know which phase they are in and what to optimize for.
- Offline eval passes launch gate
- Shadow mode divergence < 20%
- Internal user NPS > 40
GitHub Copilot early alpha: completion rate on Python measured before public beta.
- Automation rate vs. human baseline
- Time saved per user per week
- Adoption rate > 50% of eligible users
Gmail Smart Reply: expanded from English to 44 languages after proving adoption >60% in EN.
- Cost per query vs. human equivalent
- Model performance vs. new challenger
- Guardrail stability over 6+ months
Salesforce Einstein: shifted focus from accuracy improvement to inference cost reduction after maturity.
Metrics interview Q&A
Five high-signal questions that distinguish AI PMs from traditional PMs. Each has the common trap so you know what to avoid.
What metrics would you use to evaluate a newly launched AI copilot feature for Salesforce account executives?
I would start by separating guardrail metrics from goal metrics — these are not the same category.
Guardrails: hallucination rate < 2%, response latency P99 < 3s, and zero toxicity incidents. These are floors — breaching any of them triggers a rollback regardless of business performance.
Goal metrics: task completion rate as the North Star (what % of user requests does the AI fully resolve without a retry or manual correction?), and suggestion acceptance rate as the leading indicator (do users accept the AI output or immediately edit it?).
Business KPIs: time saved per account executive per week (measured via activity logging), opportunity conversion rate in the A/B cohort vs. control, and cost per assisted interaction.
I would define all of these before sprint 1 and include the threshold values in the PRD. "We measure metrics" is not an interview answer. "We will not ship if hallucination rate exceeds 2% on our golden eval set" is.
Common trap: Listing metrics without distinguishing guardrails from goals, or without naming thresholds. A metric without a threshold is not an evaluation criterion.
How do you explain the precision-recall trade-off to a non-technical stakeholder who wants to increase the automation rate?
I would frame it in terms of the two types of mistakes the model can make, and what each one costs us.
"Right now, the model automatically approves 70% of applications and sends 30% to a human reviewer. If we lower the threshold to get to 85% automation, we catch fewer edge cases — some incorrect approvals will go through without a human check. The question is: how much does one incorrect approval cost us versus how much does one manual review cost us? If an incorrect approval costs us $500 in remediation and a manual review costs $8 in analyst time, the math tells us we need to be very confident before increasing automation. I can show you the exact numbers for each threshold setting."
This turns a machine learning concept into a cost-benefit decision that the stakeholder can evaluate with their own business judgment. The PM provides the trade-off; the stakeholder makes the call.
Common trap: Explaining precision and recall as technical definitions. Stakeholders do not need the formula — they need to understand the business cost of each error type.
Your AI feature has 94% accuracy on the test set but users are complaining it is wrong too often. What is happening and what do you do?
Three likely explanations, and I investigate them in this order.
First: the test set does not represent production distribution. If the test set was collected 18 months ago or from a different user segment, 94% accuracy on it means nothing for today's traffic. I ask data science to run the current model on a fresh production sample and compare metrics.
Second: class imbalance masking failure mode. If 94% of inputs belong to the dominant class and the model learned to predict that class, accuracy is 94% but precision and recall on the minority class are near zero. Users experiencing the minority case experience constant failure. I ask for a breakdown of accuracy by class or segment.
Third: the wrong metric for the use case. Users may be encountering a high false positive or false negative rate in a specific scenario that accuracy averages away. Switching to precision-recall analysis by intent or input type usually surfaces this.
Action: run a disaggregated error analysis on the last 1,000 user complaints. Cluster failure types. The pattern almost always reveals one of these three root causes within 48 hours.
Common trap: Defending the 94% accuracy number. The user complaints are data. The test set metric and the user experience are in conflict — the test set is wrong, not the users.
What is the difference between a guardrail metric and a goal metric, and why does this distinction matter in a product review?
A guardrail metric is a floor — a hard minimum that the feature must not breach. Breaching it triggers a halt or rollback regardless of how well the goal metrics are performing. Examples: hallucination rate above 2%, latency P99 above 3 seconds, bias disparity above 5% across demographic groups.
A goal metric is a ceiling — a metric you are actively trying to improve. Examples: task completion rate, suggestion acceptance rate, cost per query.
The distinction matters in a product review because they should never be traded against each other. You do not get to say "our hallucination rate is 4% but our task completion rate went up 15% so it is worth it." The guardrail is non-negotiable. The goal metric is what you optimize after guardrails are met.
In practice, this means every feature spec has two explicit sections: guardrail thresholds (with rollback triggers) and goal metric targets (with optimization direction). Conflating them leads to shipping features that are technically impressive but violate safety or compliance requirements.
Common trap: Treating all metrics as equally tradeable. This is the most common PM mistake in AI product reviews and signals a candidate who has not shipped a regulated AI product.
How would you set up an evaluation plan for a new RAG-based knowledge assistant being deployed for 5,000 enterprise users?
Four phases, each with a defined exit gate.
Phase 1: Offline eval on a golden set. I work with the domain team to curate 300–500 representative queries with expert-written answers as ground truth. Measure faithfulness (is the answer grounded in source documents?), answer relevancy (does it address the question?), and hallucination rate. Minimum thresholds: faithfulness > 0.88, relevancy > 0.80, hallucination < 1.5%. Below any threshold, do not advance.
Phase 2: Shadow mode for 10 business days. Run the new system on live queries and log responses without serving them. Have 5 domain experts review 50 shadow responses per day. Compare error patterns against the current system.
Phase 3: A/B test with 10% of users for 3 weeks. Primary metric: task resolution rate (user finds answer without escalating to human). Guardrail: auto-rollback if hallucination rate exceeds 2% in production sampling.
Phase 4: Full rollout with continuous monitoring. Weekly review of faithfulness score, override rate, and cost per query. Monthly recheck of golden set eval with any new edge cases added from production failures.
Common trap: Skipping shadow mode or treating the offline eval golden set as the only validation gate. RAG systems fail in production on query distributions that golden sets do not cover.
I'm a PM preparing for an AI metrics interview at a company deploying an LLM-powered customer service assistant. Ask me a metrics question — specifically about guardrail vs. goal metrics or LLM evaluation — then give me feedback on whether my answer distinguishes between metric types, names thresholds, and correctly identifies PM ownership.