Product & team KPIs
Two categories: product KPIs tell you if you are building the right thing. Team KPIs tell you if you are building it well. Know both — and know when each one is lying to you.
AI product metrics operate in three layers. Product metrics without model health monitoring will fail. Model metrics without user interaction tracking will mislead.
Product Layer
- →Adoption Rate
- →Retention Rate
- →NPS / CSAT
- →Churn Rate
- →Time to Value
User-AI Interaction Layer
- →AI output acceptance rate
- →Correction / override rate
- →Escalation rate to human
- →Task completion with AI
- →Return visits to AI feature
Model Health Layer
- →Precision & Recall
- →F1 on golden eval set
- →Inference latency (p95)
- →Hallucination rate
- →Data drift alerts
Leading vs. Lagging
Leading — predict future outcomes
Session frequency declining, output acceptance rate dropping, users reverting to manual workflows. Appear 2–4 weeks before churn becomes visible.
Lagging — confirm past trends
Churn rate, NPS decline, support ticket volume. By the time these fire, users have already decided to leave.
Guardrail Metrics
Guardrails are threshold metrics that block release or trigger a pause regardless of how other metrics look. They are the ethical and reliability floor of AI product work.
Hallucination rate above X% → release blocked
Precision below 90% on sensitive use case → pause
Latency p95 above 2s → not shippable
Fairness gap between groups above 5% → ethics review required
Six product KPIs every AI PM should know — with AI-specific context and the trap that makes each one misleading.
Adoption Rate
Active users ÷ eligible users
Percentage of eligible users actively using the feature. The primary signal of whether users trust and find value in an AI feature.
AI-Specific Context
High accuracy but low adoption almost always means a UX or trust problem — not a model problem. Users resist AI assistance even when the model is right. Investigate adoption segmented by confidence score: are low-confidence outputs suppressing use?
Common Trap
Reporting total users instead of eligible users. An AI feature with 100 users out of 100 eligible is 100% adoption. The same feature with 100 users out of 10,000 eligible is 1% adoption — a crisis, not a success.
HEART Framework
Five delivery KPIs that measure how the team works — not what gets built. Each has an AI-specific pattern to watch for.
Velocity
Avg story points completed per sprint (6-sprint rolling average)
How much work the team reliably completes per sprint. Use for capacity forecasting only — never as a performance target.
AI-Specific Pattern
AI sprints are more variable than traditional software sprints — model experiments have binary outcomes and spike results are unpredictable. Smooth velocity over 6 sprints and flag any sprint with >30% unplanned work as an outlier. AI velocity is a floor, not a ceiling.
Common Trap
Setting a velocity target. This immediately incentivizes story inflation — teams estimate 8 points for a 5-point story to hit the number. The metric becomes noise within two quarters.
Five model health metrics every AI PM must know — including when to use them, the key tradeoff, and the PM role in defining them.
Precision & Recall
Precision = TP ÷ (TP + FP) | Recall = TP ÷ (TP + FN)
When to Use
Classification models — intent detection, content moderation, fraud detection.
Key Tradeoff
Precision and recall pull against each other. Raising the confidence threshold increases precision (fewer false positives) but lowers recall (more false negatives). The right balance depends on which error is more costly.
PM Role
Define the acceptable precision-recall tradeoff before the model is trained. Example: "A missed fraud case costs $500. A false fraud flag costs $10 in customer experience. Recall should be weighted 50x higher than precision."
Common Trap
Using accuracy as the only metric on imbalanced datasets. If 97% of emails are not spam, a model that classifies everything as not-spam has 97% accuracy and 0% usefulness.
How do you measure whether your product is successful?
I separate two distinct categories: product KPIs and delivery KPIs. Product KPIs — adoption, retention, CSAT, NPS, TTV — measure whether we are building something users value. These are defined at requirements stage, before development begins, so everyone agrees on what success looks like. Delivery KPIs — velocity, sprint goal completion, cycle time, defect rate — measure whether the team is working effectively. These are health indicators for the process, not the product. For AI features specifically, I add a third category: model health metrics — precision, recall, latency, drift. These bridge the gap between "the model works in eval" and "the model is delivering value in production."
Common Trap
Treating model performance metrics (F1, accuracy) as product success metrics. A model can hit 92% F1 and still fail in the market if users do not trust or use it. Model performance is a means; adoption and retention are the ends.
Give an example of a leading indicator and a lagging indicator for an AI product. Why does the distinction matter?
For an AI support assistant: Leading indicators (predict future outcomes): session frequency declining, AI output acceptance rate dropping, users switching back to manual methods. These appear 2–4 weeks before churn becomes visible. Lagging indicators (confirm past trends): churn rate rising, NPS declining, support ticket volume increasing. By the time these move, users have already made the decision to leave. The distinction matters because by the time a lagging indicator fires, it is usually too late for intervention. The PM's job is to instrument leading indicators and set alert thresholds — so you are investigating root causes before they become visible in the headline metrics.
Common Trap
Building a dashboard of lagging indicators and calling it "monitoring." A monthly NPS review is not real-time monitoring. Set automated alerts on leading indicators so you find out about drift, output quality drops, or engagement declines within 48 hours, not at the next all-hands.
What unique KPIs would you track for an AI-powered feature beyond standard product metrics?
Three additional layers beyond standard product metrics: Model performance: precision, recall, F1 on the golden eval set. These are the acceptance criteria for launch and the monitoring baseline post-launch. User-AI interaction quality: acceptance rate (did the user use the AI output?), correction rate (how often did they edit it?), escalation rate (how often did they override it entirely?). These tell you whether the model is calibrated to real user needs. Guardrail metrics: hallucination rate, fairness metrics across demographic groups, data drift alerts, latency percentiles. These are threshold metrics — if they breach a defined limit, the feature is paused regardless of adoption numbers.
Common Trap
Skipping the user-AI interaction layer and only tracking model performance and adoption. A model with 95% F1 that users correct 80% of the time is not performing well in production — it is producing technically plausible but contextually wrong outputs. The correction rate is the signal model performance alone will not give you.
How do you handle a situation where optimizing for one KPI negatively impacts another?
KPI conflicts are the norm in AI, not the exception. The standard tension: raising the confidence threshold for predictions increases precision (fewer false positives) but lowers recall (more missed cases). Both move a metric you care about in opposite directions. My framework: define a north star metric — the one that most directly represents user value or business outcome. Then define guardrail metrics — floors and ceilings that cannot be breached. Any optimization of the north star that violates a guardrail is rejected. Example: North star is "intent routing accuracy." Guardrail is "precision ≥ 90% to protect user trust." If improving routing accuracy requires dropping precision below 90%, I do not accept the tradeoff — I find a different architectural path or adjust scope.
Common Trap
Optimizing multiple KPIs simultaneously without a hierarchy. When everything is equally important, nothing is. The north star plus guardrails model forces the team to be explicit about which tradeoffs are acceptable.
Describe a situation where a good-looking KPI masked a deeper product problem.
The classic AI example: high click-through rate on AI recommendations that masked near-zero task completion. Users were clicking the AI suggestions — the metric looked healthy. But we discovered through session recording that users clicked, looked at the recommendation, then immediately returned to their manual workflow. The AI output was interesting but not trustworthy enough to act on. The click-through rate was a vanity metric — it measured curiosity, not utility. The actionable metric was task completion rate with AI assistance: what percentage of users who clicked the recommendation actually used it to complete the task? That number was 12%, which was the crisis the CTR was hiding. The fix: reframe the success metric from CTR to "AI-assisted task completion rate." This immediately surfaced the real problem — model outputs were plausible but contextually wrong for the specific task.
Common Trap
Not having a task completion metric defined before launch. Vanity metrics fill the vacuum when the right metrics are not instrumented. Define the success metrics at the requirements stage, not post-launch when you are scrambling to explain why the product is underperforming.