Scoping AI products
Vague requirements produce vague AI outputs. Scoping is where PM craft translates business intent into a specification a model can be built and evaluated against.
The AI scoping canvas
Seven dimensions every AI feature scope must address. Leaving any one out creates an undefined gap that engineering will fill with their own assumption — which may not be the right product decision.
Business Goal + KPI
What business problem are we solving and how will we measure success? This must be a specific, measurable KPI — not "improve CX" but "reduce first-contact resolution time from 6.2 min to 4.0 min."
Salesforce Einstein: business goal was to reduce manual data entry for sales reps. KPI: time spent on CRM data entry per rep per week. Baseline was 2.1 hours. Target was <45 minutes.
Defining the goal in AI terms ("build a classification model") rather than business terms. If you cannot state the business KPI, you cannot evaluate whether the AI succeeded.
Worked example — customer support ticket router
All 7 canvas dimensions applied to a realistic AI feature.
LLM feature scoping — the 5 dimensions
LLM features require tighter scoping than classical ML because outputs are open-ended and errors are unpredictable. These 5 dimensions are the PM's deliverable before any prompt engineering begins.
The exact input → output transformation in one sentence. Precise enough that a data scientist can start building without a follow-up meeting.
"Summarise documents"
"Given a support transcript up to 3,000 words, produce a 3-sentence summary covering: issue type, resolution status, and customer sentiment."
What goes into the context window — and what does not. Context gaps are a primary cause of hallucination. Define the exact sources and their freshness.
"The model has access to customer information"
"Context includes: email body, account tier, last 3 support interaction notes (max 500 chars each), relevant KB articles (top 3 by BM25 score). Does NOT include billing data or other account information."
Format, length, tone, and constraints. The output contract is what the evaluation rubric scores against. It is also what guardrails enforce.
"A helpful, professional response"
"Plain text, 100–150 words, must include one specific next step for the customer, must not commit to resolution timelines beyond 48 hours, must not mention competitor products."
What happens when the model is uncertain, out-of-scope, or returns a low-confidence output. This is a product feature with its own acceptance criteria.
"Handle errors gracefully"
"If primary intent confidence < 0.75: prepend 'This response may need your review:' and surface to agent with the confidence score visible. If the query is out-of-scope (personal legal advice, medical diagnosis): respond with the approved escalation template."
Rubric, score anchors, acceptance threshold, and the golden test set. Defined before any model work begins. The eval criteria are what make sprint sign-off meaningful.
"The response should be good enough to send"
"5-point rubric: accuracy (does it address the issue?), tone, constraint adherence, and actionability. Acceptance: avg ≥ 4.0 on 300-item golden set. Latency: P99 < 2s. Cost: < $0.015 per draft."
Agent scoping — autonomy levels
Start at the lowest autonomy level that delivers value. Earn trust before expanding scope. The autonomy level is a launch criterion, not a design assumption.
Select a level to see when to use it and a real deployment example.
AI suggests, human decides. No automated action.
First deployment in any domain. High-stakes decisions. Regulatory environments where automated decisions require human sign-off.
Einstein Next Best Action: AI surfaces a recommendation ("Schedule a call"), sales rep clicks to accept or dismiss.
What to define explicitly in agent scope
Common AI scoping failures
Each of these patterns has delayed or killed well-resourced AI features. All are preventable with the right scope document.
"Since it can do X, can it also do Y?" Capability is not scope. Every new task requires its own evaluation criteria, latency budget, and failure handling — treat each capability addition as a new scoping exercise.
An LLM deployed for customer support email classification was asked mid-sprint to also "generate replies." The generation task has a completely different evaluation rubric, output contract, and latency requirement. Adding it mid-sprint invalidated the golden test set and delayed launch by 6 weeks.
Without evaluation criteria defined before build, "done" is negotiated, not measured. Teams ship when they run out of time, not when the model meets quality requirements.
IBM Watson Health initiatives: multiple product teams launched without pre-defined clinical accuracy thresholds. When clinicians tested the models, disagreement on what "accurate enough" meant produced a legal and reputational crisis. The evaluation plan should have been in the scope document.
In agents especially, edge cases cascade. A failed tool call at step 3 of a 7-step agent creates unpredictable downstream state. Every tool must have an explicit failure path in scope.
An HR workflow agent that books interview slots would silently fail when the calendar API returned a 429 (rate limit). The agent continued to the next step — sending confirmation emails — for slots it had not actually booked. Root cause: failure mode for rate-limited API not in scope.
Every AI feature needs an explicit human override mechanism. This is a regulatory requirement for high-risk systems under EU AI Act and a trust-building necessity for any new AI product.
A financial services firm launched an AI loan-eligibility screener without a human review path for edge cases. The first GDPR subject access request revealed automated decision-making without a documented appeal pathway — a compliance violation that paused the product for 3 months.
Bias encoded in training data cannot be retroactively removed at launch. The audit plan must be in the scope document — not the post-launch retrospective.
Amazon's internal hiring tool (retired 2018): trained on 10 years of resume data where the hiring process had historically favored male candidates. The model encoded and amplified that bias. The bias audit that would have caught this was not in the original project scope.
The PM owns the latency SLA and the cost-per-inference budget. Both must be defined in the PRD before model selection. If they are left to engineering, you discover the economics do not work in month 3.
A Workday PM selected GPT-4o for a skills extraction feature based on accuracy alone. At 2M monthly queries, per-token cost was $180k/month — 9x the approved budget. The model had to be replaced post-launch. A cost-per-query requirement in the scope would have forced the right model selection in week 1.
Scoping interview Q&A
Five scenario questions that test whether your scoping instinct is concrete or theoretical.
Walk me through your requirements for an LLM-powered customer service email assistant.
I scope LLM features across five dimensions.
Task definition: the exact input to output transformation. "Given a customer support email body (max 2,000 chars) and a customer account summary, generate a draft reply addressing the primary issue in 3–5 sentences. Tone: professional, empathetic."
Context boundary: the model has access to the email body, account tier, last 3 interaction notes, and relevant knowledge base articles. It does not have access to billing data or other customer accounts.
Output contract: format (plain text, no markdown), max 150 words, must include a specific next step for the customer, must not make pricing claims or commit to resolution timelines outside the approved escalation policy.
Failure handling: if confidence in the primary issue identification is below 0.75, prepend "This may need your review:" and surface to the agent without hiding it as a confident draft.
Evaluation: rubric across four dimensions — accuracy (does it address the actual issue?), tone, constraint adherence, and helpfulness. 5-point scale per dimension. Acceptance threshold: avg score ≥ 4.0 on a golden set of 300 real tickets. Also: latency P99 < 2s, cost per draft < $0.015.
Common trap: Describing capability rather than contract. "The model should understand context" is not a requirement. "The model receives account tier and last 3 notes as context" is a requirement.
A stakeholder wants a fully autonomous agent to handle customer refund decisions end-to-end. How do you respond?
I agree with the destination and push back on the path.
"Full autonomy for refund decisions is the right long-term goal. Here is why we should not start there: a fully autonomous refund agent will make errors we cannot predict in production. When those errors affect customers — wrong refund amount, denied legitimate return — it damages trust in a way that is expensive to repair. The autonomy level should be earned by demonstrated performance, not assumed at launch.
I would scope this in three phases. Phase 1 (L2): AI drafts the refund decision with reasoning, agent reviews and executes with one click. Measure accuracy and agent override rate. Phase 2 (L3): For refund requests below $50 from verified customers, AI executes immediately. Humans review the daily exception queue. Measure override rate and error cost. Phase 3 (L4): If override rate drops below 3% and average error cost is below $2 per decision for 90 days, we remove the routine review gate for the high-confidence segment.
The autonomy level is a product launch criterion, not a design assumption."
Common trap: Refusing to engage with the full autonomy goal. The PM's job is to find the path to the goal within acceptable risk — not to block the ambition.
How do you write acceptance criteria for an LLM feature that needs to be signed off in sprint review?
I write acceptance criteria across four types for LLM features.
Functional AC (Given/When/Then): "Given an email body under 2,000 chars, when the agent generates a draft, then the draft addresses the primary customer issue stated in the email."
Output contract AC: "The generated draft must be 100–150 words, in plain text with no markdown, and must include a specific next step for the customer." These are verifiable programmatically.
Evaluation AC: "The draft must score ≥ 4.0/5.0 on the rubric across the 300-item golden test set before sprint sign-off." This is the gate that says the model is ready — not just that it runs without errors.
Non-functional AC: "P99 response latency < 2 seconds. Cost per draft < $0.015. Available in 99.9% of requests."
Without the evaluation AC, "done" in sprint review means "it works." With the evaluation AC, "done" means "it is good enough to ship."
Common trap: Writing only functional AC and leaving evaluation to post-sprint QA. If the evaluation threshold is not in the AC, the sprint is "done" when the demo works — not when the model meets the quality bar.
How do you define the toolset for an AI agent to ensure safety and prevent unintended actions?
I define the toolset as an explicit allowlist — not "access to CRM" but a specific set of operations with defined permissions and constraints.
For each tool I specify: what it can call, what parameters it can pass, what it cannot do, and the error handling contract if the tool call fails.
For a sales agent with CRM access: "read_contact(contact_id) — read only, returns contact name, email, account tier, and last 5 interactions. Cannot access billing history, cannot read other contacts. create_task(contact_id, title, due_date) — creates task under the assigned rep, cannot assign to other reps. No delete operations of any kind."
The step budget is also part of the tool definition — the maximum number of tool calls per task execution. Uncapped agents that enter reasoning loops are a cost risk and a quality risk. I define: max 8 tool calls per task session, escalate to human if not resolved within budget.
This allowlist goes in the engineering specification, reviewed by security before any agent code is written.
Common trap: Granting broad access ("CRM access") and trusting the agent to self-limit. Agents will use whatever they can access, and broad access is a safety incident waiting to happen.
Describe a time you had to significantly de-scope an AI feature during development. What was the trigger and what was the outcome?
Situation: we were scoping an intent classification model for customer support tickets across 20 categories. Six weeks into development, the data science team surfaced that 14 of the 20 categories had fewer than 800 labeled examples — well below the 3,000 minimum for reliable classification.
Action: I pulled the data distribution analysis into a stakeholder review rather than waiting for the sprint demo. I presented three options: wait 8 weeks and label additional examples for all 20 categories, launch with 6 high-confidence categories and expand quarterly, or abandon the multi-class approach and use a binary triage (escalate vs. self-serve) with a simpler model.
I recommended option 2 with a specific expansion roadmap: launch with the 6 categories representing 71% of ticket volume, define the labeling pipeline for the remaining 14 categories, and commit to adding two categories per sprint cycle based on data readiness, not schedule.
Outcome: the feature launched on the original date with the 6-category scope. It achieved 91% classification accuracy and reduced routing errors by 58% for the categories it covered. Twelve categories have been added in the 6 months since. If we had held for all 20, we would have launched 5 months late with worse accuracy across the board.
Common trap: Treating de-scoping as a failure. In AI, de-scoping based on data evidence is product judgment — it is the PM catching a risk before engineering bakes it in.
I'm preparing for an AI PM interview. Give me an AI feature to scope — something like an LLM-powered writing assistant or an AI document classifier. Then ask me to walk you through all 5 scoping dimensions (task definition, context boundary, output contract, failure handling, evaluation criteria). Give me feedback on whether my scope is specific enough to build against.