AI roadmapping
AI roadmaps are not traditional timelines with AI labels — they require probabilistic milestones, evaluation gates, and a structured language for uncertainty.
Implies completion on a specific date. Creates pressure to ship regardless of model quality.
A quality threshold + business outcome + confidence range. Clear stopping condition.
4 Sources of Unpredictability
Select each to understand the timeline impact and PM response.
Data Remediation Sprints
You cannot know the impact of data quality issues until the model trains on them. What looks like 2 weeks of cleanup can become 6. Data confirmation must be a milestone that blocks downstream work — not an assumption baked into engineering estimates.
+1–3 sprints typical
Treat data readiness as the first gate. No data sign-off = no engineering estimate.
AI roadmaps use a confidence tiering system — not uniform dates across all items. The tier reflects how much is known about data readiness, evaluation status, and external dependencies. Select a feature's characteristics to see which horizon it belongs in.
Horizon Classifier
3-Horizon Model
Specific dates + quality thresholds
Data in production, eval passed, shadow mode running
"By Q2 we will achieve X automation rate"
Quarter ranges ("Q3–Q4")
Data exists and validated, eval pending, no known blockers
"We expect to complete this in H2 — details depend on eval results"
Directional only ("next year")
Data gap not resolved, external dependency, future model needed
"On our 12-month horizon, dependent on data platform readiness"
Every AI feature has four mandatory delivery phases, each with a defined quality gate. These phases are roadmap milestones — not engineering details. The most commonly skipped phase (shadow mode) causes the most production incidents.
Offline Eval
Model evaluated against a curated, human-labeled test set. Tests if the model can meet quality thresholds on known examples. The test set must be separate from training data and updated regularly.
Overfitting to the test set. A static golden set that never changes stops being predictive of real-world performance.
Own the golden test set. Define the pass threshold in the DoD before sprint 1. Update it every sprint with new failure cases.
Shadow Mode
Model runs on real production traffic but outputs are not shown to users or acted upon. Validates that eval performance transfers to the production distribution — the most commonly skipped step, and the costliest to skip.
The production gap. Test set performance doesn't transfer — distribution shift in real users is almost always present. Discovery here extends the timeline by 2–6 sprints.
Shadow mode is a mandatory milestone, not optional. Plan 2–4 weeks on the roadmap. Teams that skip it pay in production incidents.
Limited Rollout / A/B
Feature enabled for a subset of users. Validates user behavior metrics — correction rate, task completion, trust score — that shadow mode cannot measure. Also validates business metrics at small scale.
A/B sample size and duration underestimated. Run a power analysis before launch to know when results are statistically significant. Peeking at results early inflates false positive rates.
Define success metrics and minimum A/B duration before starting. No early peeking.
Full Rollout + Monitor
Feature fully enabled. Production monitoring begins: correction rate, override rate, model drift alerts. Monitoring is a product feature — define it in the DoD before launch, not in the post-launch backlog.
Silent degradation. Correction rate rising is the canary — it leads accuracy degradation by 2–3 weeks.
Set alert thresholds before full rollout. Define who gets paged and at what metric threshold.
The same roadmap needs three different languages. Translating between them is a core AI PM skill — not simplification, but intentional reframing to what each audience needs to act on.
Board / CEO
"By Q3 we'll automate 40% of tier-1 support, reducing support cost by $2M annually. We have a tested fallback if we don't hit the quality threshold."
Model architecture, accuracy percentages, technical tradeoffs, eval methodology
Engineering / Data Science
"Target: 90% precision at p99 < 400ms. Shadow mode begins sprint 5. Fallback is rules-based classifier if we miss the threshold by sprint 8."
Business metrics framed as engineering constraints ("the CEO wants 95% by Q2")
End-User Team / Support Ops
"The AI drafts a response — you review and send. You're still in control. When it's wrong, click reject and it improves over time."
Automation rates, model confidence scores, anything that sounds like job replacement
Communicating a Threshold Miss
When the model does not hit its quality threshold by the committed date, the structure is: root cause → options → recommendation. Communicate early — partial information is better than silence.
"We completed shadow mode and hit 83% precision against a 90% target. Root cause: distribution shift in mobile users not represented in training data."
"Three options: (1) ship at 83% with 40% automation and manual review for the remainder — no delay; (2) delay 4 weeks to add mobile training data and retrain; (3) scope to desktop-only at 90% with mobile following in Q4."
"My recommendation: option 2 if quality is a brand requirement, option 1 if business velocity matters more. Your call."
AI roadmapping questions test whether you understand probabilistic milestones, can communicate uncertainty without losing stakeholder trust, and know how to structure delay conversations without losing credibility.
How do you handle pressure to ship an AI feature before it meets the quality threshold?
I quantify the error cost before the conversation. Instead of abstract quality arguments, I calculate: at 81% precision, we produce X wrong decisions per week, costing Y in downstream corrections and Z in user trust erosion. That shifts the trade-off from "dates" to "unit economics."
Then I propose a fallback instead of a delay: "We ship at the current quality level with a higher review threshold — 40% automation instead of 80% — and we have a concrete path to full automation by sprint 8." Shipping with lower automation is a real option. Shipping with a broken user experience is not.
The key is that I had defined the minimum viable quality threshold in the DoD before the sprint started — so "not ready" is an objective state, not my opinion.
What makes an AI roadmap structurally different from a traditional product roadmap?
Three structural differences.
First, milestone format. Traditional: "ship feature by Q2." AI: "achieve 90% precision on production samples, enabling 60% automation, by Q2–Q3." Milestones are quality thresholds met, not calendar events.
Second, confidence tiering. AI roadmaps have three tiers: H1 (committed — data validated, eval passed), H2 (probable — data exists, eval pending), H3 (directional — data gap not resolved). I never put a specific date on an H3 item.
Third, eval gates. Every AI feature has four mandatory phases as roadmap milestones: offline eval, shadow mode, A/B test, full rollout + monitor. Each has a defined pass threshold. Shadow mode especially — the production gap will always surprise you.
How do you communicate roadmap uncertainty to executives who expect specific dates?
Three tools.
First, confidence tiering: I distinguish committed ("Q2 — we've passed shadow mode"), probable ("Q3–Q4 — data is validated but eval pending"), and directional ("next year — dependent on data platform"). This prevents false precision on early-stage work.
Second, eval curves instead of Gantt bars: I show a chart of model quality improving over sprints. This makes the probabilistic nature visible — executives can see we're trending toward the threshold, not just hoping to hit a date.
Third, business outcome framing at every level: the board hears "40% support cost reduction by Q3." Engineering hears "90% precision target at p99 < 400ms." Support ops hears "AI drafts, you review." Same roadmap — three different languages.
The model isn't going to hit its quality threshold by the committed date. How do you communicate this?
Communicate early — partial information is better than silence. I bring three things: what we know, why the estimate changed, and options with quantified trade-offs.
"We completed shadow mode and hit 83% precision against a 90% target. Root cause: distribution shift in mobile users not represented in training data. Three options: (1) ship at 83% with 40% automation and manual review for the remainder — no delay; (2) delay 4 weeks to add mobile training data and retrain; (3) scope to desktop-only at 90% with mobile following in Q4."
I come with a recommendation: option 1 if business velocity is critical, option 2 if quality is a brand requirement. The decision belongs to the stakeholder — I structure the trade-offs.
Walk me through the stages of evaluating and releasing an AI feature.
Four mandatory stages — all roadmap milestones, not engineering details.
Offline eval: test the model against a curated golden test set. I own this test set — I curate failure cases and update it every sprint. Pass threshold is defined in the DoD before sprint 1.
Shadow mode: run on real production traffic but don't surface outputs to users. This catches the production gap — where test set performance doesn't transfer. 2–4 weeks minimum. Many teams skip this and pay in production incidents.
A/B test: feature enabled for a user subset. This is where I validate user behavior metrics (correction rate, task completion) that shadow mode can't measure. I define minimum sample size and duration before starting — no peeking at results early.
Full rollout + monitor: feature fully enabled with production dashboards live. Correction rate and override rate are my canaries for silent degradation.