ADO & user stories
The work item hierarchy that turns AI strategy into shipped software. Master story writing for non-deterministic features.
Every AI work item lives somewhere in this hierarchy. Select a level to see how it applies to AI product work.
Epic
Strategic intent — connects AI investment to a business outcome.
AI Example
AI-powered customer self-service that resolves 40% of tier-1 contacts without human intervention.
Business Link
Links to OKR: reduce support cost by $2M annually.
Quality Gate Question
"If we shipped everything under this epic, what measurably changes for the business?"
Hierarchy Depth
AI user stories come in five distinct types. Each has a different author, trigger, and AC pattern.
When to use
Before model training begins
Story Template
Acceptance Criteria Checklist
Common Trap
Skipping data quality AC — "we will check later" causes silent model degradation.
INVEST was designed for deterministic software. Each criterion needs an AI-specific interpretation.
Independent
Stories can be built in any order.
In AI, separate "Data Prep" from "Model Training" from "Inference API." A dependency chain creates sprint-killing blockers when the upstream experiment fails.
Self-Test
"Could another team pick this story up without waiting for us? If not, it is a task, not a story."
Negotiable
AC is a conversation, not a contract.
Model accuracy thresholds should be co-authored with data science. "≥90% precision" may shift to "≥87% with recall ≥95%" once the data is understood. Lock it at sprint start, not before.
Self-Test
"Did PM and DS both sign off on the metric and threshold? If only one did, it is not negotiable — it is a demand."
Valuable
Every story delivers something the user wants.
Experiments that fail are valuable if they close a hypothesis. A spike that proves an approach will not work saves 6 weeks of wasted sprint. "Signal value" counts.
Self-Test
"If this story shipped and the experiment failed, would we still have learned something worth the sprint cost?"
Estimable
The team can assign story points.
If engineers cannot estimate, run a spike first. Data uncertainty and novel algorithms cannot be pointed blind. "We will figure it out in the sprint" is not agile — it is chaos.
Self-Test
"Can the team estimate this in fewer than 15 minutes of discussion? If not, it needs a spike or decomposition."
Small
Fits in one sprint.
If a model training cycle takes 2 weeks, the story is "Data Prep" not "Build model." Splitting by phase (ingest → clean → train → eval → deploy) is correct for AI work.
Self-Test
"Would this story fit in the sprint even if two unexpected bugs appeared on day 3?"
Testable
AC has a binary pass/fail.
"Model is accurate" fails. "Precision ≥92% on golden eval set v4, measured on a held-out 10% split" passes. Every AI story must name the dataset, metric, and threshold.
Self-Test
"Can QA run this test without asking the PM what success looks like?"
Four acceptance criteria patterns every AI story needs. Senior PMs write all four — juniors write only the happy path.
Happy Path
Expected successful outcome for the 80% case.
Why This Pattern Matters for AI
Defines the baseline the model must hit before any other AC is evaluated.
All Four Patterns
How do you write a user story for an AI feature that is not always 100% accurate?
The key is moving from binary output thinking to probabilistic outcome thinking. Instead of "the model returns the correct answer," write: "Precision ≥ 90% on golden eval set v3, with predictions below 0.75 confidence routed to human review." This gives engineering a testable target and QA a binary pass/fail. The story also needs an edge-case AC that defines what happens when confidence is low — fallback behavior, escalation path, or a disclaimer in the UI. The failure to specify this is the most common reason AI stories pass QA but fail in production.
Common Trap
Writing "model accuracy is acceptable" as an AC. Acceptable to whom? Define the metric, dataset, and threshold before sprint start.
How do you apply INVEST principles to an AI backlog?
Each criterion needs an AI-specific lens: "Independent" means separating data prep from model training from deployment — each phase can slip independently without cascading the whole sprint. "Valuable" includes experiment value — a failed model that closes a hypothesis saves future sprint capacity. "Testable" requires naming the metric, dataset version, and threshold explicitly, not leaving it to judgment. The most commonly violated criterion in AI work is "Small." Teams bundle data pipeline, training, eval, and deployment into one story. Split by phase, and use spikes when uncertainty prevents estimation.
Common Trap
Using "Estimable" as an excuse to over-engineer spikes. A spike has a time-box and a binary output: build or kill. Unbounded spikes are just research, and research does not ship.
When would you split an AI user story and what techniques do you use?
Split when a story cannot fit in one sprint, has too much unknown, or conflates multiple stakeholders. The most effective AI-specific technique is splitting by phase: data ingestion → data cleaning → model training → evaluation → deployment. Each phase is independently testable and deliverable. The second technique is confidence tier splitting: ship the high-confidence path first, add mid-confidence HITL in sprint 2, handle low-confidence edge cases in sprint 3. Always run a spike before estimating any story that involves a new model architecture or a dataset the team has not used before.
Common Trap
Splitting stories so small that each one has no independent user value. A story must deliver something a user or the system can observe. If it cannot, it is a task, not a story.
How do you use the ADO hierarchy to communicate AI progress to executives?
Executives see epics and features, not stories. The hierarchy provides the traceability layer. Each epic ties to a business OKR. Each feature is a deliverable capability the exec can demo. Stories are the PM and engineering team's internal granularity — they should never appear in board-level reporting. In AI work, features often look unstable ("feature X is 75% done but not shippable") because model quality gates are binary. Be explicit about which feature has cleared its eval gate and which is still in experiment phase. Frame unshipped AI features as "in eval" not "in progress" — it is more accurate and sets better expectations.
Common Trap
Reporting story completion to executives. "We completed 87% of stories" means nothing if the feature eval gate has not been cleared. Tie all exec comms to feature and epic outcomes.
How do you handle NFRs like model latency or fairness within ADO?
NFRs in AI fall into two categories: story-level and system-level. Story-level NFRs — latency targets, confidence thresholds, specific accuracy for a feature — belong as explicit AC in the relevant story. System-level NFRs — overall p95 latency, fairness across demographic groups, data retention policy — belong as acceptance criteria on the feature or as a shared "quality bar" task linked to the epic. The mistake is making NFRs implicit. "The model should be fast" is not an AC. "p95 inference latency ≤ 500ms under 100 concurrent requests in staging" is an AC. If you cannot test it, it is not a requirement — it is an aspiration.
Common Trap
Putting fairness and bias NFRs in a separate "compliance story" that gets deprioritized. Fairness is a quality gate on the feature, not an optional add-on. It must block release, not trail it.