Agile & Scrum
The operational backbone of AI product delivery — and how the non-deterministic nature of AI changes what each Agile principle actually means in practice.
The hierarchy: Agile is the philosophy. Scrum is the framework. Jira / ADO is the tool. Never conflate them in interviews — interviewers notice.
Individuals over processes
People and communication beat rigid workflows.
A 5-minute sync with a data scientist about distribution shift is more valuable than 50 pages of documentation. In AI, the model's behavior is discovered — it cannot be fully specified. Human judgment and conversation are the adjustment mechanism.
When correction rate starts rising, the right response is not a JIRA ticket — it's a 20-minute model review with the data scientist to understand what changed in production traffic.
Linear sequence. No working software until the end. Change is expensive. Works for construction. Fails for AI because model performance is discovered during development, not before.
Iterative and incremental. Essential for AI — model accuracy and data quality cannot be fully specced upfront. The plan is a hypothesis; evaluation findings update it.
The Scrum Team is three roles: Product Owner (owns the what), Scrum Master (removes blockers, coaches process), Development Team (owns the how). As an AI PM, you act as the Product Owner. Select a ceremony to see the AI PM-specific action at each one.
Owns the "What" — prioritizes backlog to maximize business value
Removes blockers, coaches team on Agile methodology
Cross-functional — owns the "How," commits to sprint goals
Sprint Planning
Sprint start · 2 hoursSelect backlog items and define the sprint goal
Arrive with refined, AC-complete stories. Confirm data readiness and evaluation criteria are locked before the team starts building.
Add a "data readiness gate" before sprint planning: has the training data been validated? Is the eval dataset updated? If no, the story is not sprint-ready regardless of how well the PRD is written.
Three Scrum artifacts — and the AI-specific additions that make each one meaningful for a non-deterministic product.
Product Backlog
The ordered list of everything the product might need. Never complete — evolves alongside the model, data, and market.
Sprint Backlog
The subset of items the team selects for the current sprint plus a plan for delivering them. The team owns this — the PM should not change it mid-sprint.
Increment
The sum of all completed backlog items, meeting the Definition of Done. Potentially shippable every sprint.
AI-Specific Definition of Done
- Model evaluated on held-out test set (separate from training)
- Quality targets met: precision ≥ X%, recall ≥ X%, latency p99 ≤ Xms
- Confidence threshold set and validated on golden test set
- Latency and cost audit completed at expected production load
- Fallback path designed, implemented, and tested in staging
- Human review queue instrumented and queue rate within capacity
- Below-threshold action confirmed: escalate / fallback / queue
- Correction rate baseline recorded (for drift detection)
- Production monitoring dashboard live with alert thresholds set
- Drift detection baseline recorded (input distribution)
- Correction rate alert configured: trigger condition defined, owner named
- Retraining trigger condition documented in model spec
- Model spec updated and signed off by PM + data scientist
- Stakeholder communication sent: what shipped, what it does, what happens when wrong
- Post-mortem template ready (in case of production incident)
- Rollback authority pre-approved: who can reduce automation level
AI changes what each Agile element means in practice. Not the principles — the implementation. These are the five places where traditional Agile assumptions break down when the product is probabilistic.
"Ship the intent classification feature" — a feature delivery statement.
"Achieve 90% precision on the routing classifier golden test set, enabling 60% automation rate" — a quality threshold statement.
"Given [input], when [action], then [output]" — deterministic pass/fail.
"Model achieves ≥ 90% precision on 200-example golden test set. Below-threshold inputs route to human review queue with <4 hour SLA. Fallback tested in staging."
Dependency on another team, design not delivered, API not available.
Model not converging (data/architecture issue), training data not ready, eval dataset not labeled, latency SLA not met, distribution shift detected.
Code reviewed, tests passing, deployed to staging, documentation updated.
All of the above plus: eval on golden test set passed, fallback path tested, monitoring live with alert thresholds set, correction rate baseline recorded.
Relatively predictable — engineering effort can be estimated from past stories.
Highly variable — model training, eval, and shadow mode duration are hard to estimate. Build range estimates, not point estimates.
Agile and Scrum questions test operational fluency — can you run a sprint and adapt when the model doesn't behave as expected? The differentiating moves: naming AI-specific changes to Agile mechanics, and treating evaluation as a mandatory delivery phase.
How does Agile delivery change when you're building an AI feature?
Four structural changes.
Sprint goals become quality thresholds: "achieve 90% precision on the golden test set" instead of "ship intent classification." This makes done objective and testable.
Acceptance criteria specify the failure path: happy path accuracy is necessary but not sufficient. The AC must also define what happens below the confidence threshold and whether the fallback is tested.
Definition of Done adds AI-specific gates: eval on held-out test set, fallback in staging, monitoring live with alert thresholds, correction rate baseline recorded.
Velocity is treated as a range, not a point estimate. Model eval and shadow mode take as long as it takes to hit the threshold. I plan for ranges and communicate that uncertainty explicitly.
What do you do when a story isn't ready for sprint planning?
I pull it out. No story enters a sprint without clear acceptance criteria, resolved data dependencies, and a confirmed eval approach.
For AI stories specifically, readiness has an additional bar: training data validated, eval dataset updated, confidence threshold defined, and fallback path at least designed. If any of those are missing, the story is not ready — even if the PRD is complete.
Ambiguity mid-sprint is the main destroyer of team velocity. An unready AI story is worse than an unready traditional story because the failure mode is silent — the model trains, produces subpar results, and nobody knows whether to retrain, adjust the threshold, or redefine the goal.
How do you define "Definition of Done" for an AI feature?
Four categories.
Technical: model evaluated on held-out test set (separate from training), quality targets met, confidence threshold validated, latency and cost audit completed.
User and HITL: fallback path designed, implemented, and tested in staging. Human review queue instrumented and within reviewer capacity. Below-threshold action confirmed — escalate, fallback, or queue.
Monitoring: production monitoring dashboard live with alert thresholds set. Correction rate alert configured with a named owner. Drift detection baseline recorded. Retraining trigger documented.
Communication: model spec updated, stakeholder communication sent, rollback authority pre-approved — who can reduce automation level without a committee meeting.
Without this DoD, "done" gets defined as "model trained and deployed." That's the beginning of AI PM work, not the end.
What is the difference between Agile, Scrum, and a tool like Jira or ADO?
The hierarchy matters and conflating them in an interview is a clear tell.
Agile is the philosophy — a set of values and principles for how to approach software development under uncertainty. It says nothing about sprints, standups, or backlogs.
Scrum is a framework that implements Agile principles: time-boxed sprints, three roles (Product Owner, Scrum Master, Development Team), five ceremonies (planning, standup, review, retro, refinement), three artifacts (product backlog, sprint backlog, increment).
Jira and ADO are tools for implementing Scrum. They track work items, sprints, and backlogs. They are not Agile. A team can use Jira and be completely un-Agile in their mindset.
How do you approach sprint planning velocity for AI features where the timeline is uncertain?
I plan in ranges, not points, and I communicate that explicitly.
For a sprint containing an AI evaluation phase, I estimate the best case (model hits threshold on first eval run), expected case (2 eval iterations), and worst case (major data issue discovered, shadow mode extended). I build the sprint goal around the expected case and communicate the range to stakeholders before planning starts.
I also separate the sprint goal from the quality threshold. The sprint goal might be "complete shadow mode evaluation" — the quality threshold (90% precision) might take another sprint if the first shadow run reveals distribution shift.
The mistake is treating AI sprint velocity like software velocity — assuming eval takes a fixed time. It takes as long as it takes to hit the threshold. Plan for that explicitly rather than discovering it mid-sprint.