Data & evaluation
Data is the product. Model quality cannot exceed data quality — and evaluation is how you know whether you're ready to ship. Both are PM responsibilities.
Four data questions before sprint 1
The ceiling rule: model quality cannot exceed data quality. Every sprint on the model is wasted if the data is wrong. Answer these four questions with data engineering before any model work begins.
Exact input format, required fields, expected volume per day, and freshness requirement (real-time vs. batch). Write this as a data spec alongside the model spec — not after the first sprint review.
PM owns: The data spec. Not the model architecture.
Internal, third-party, or user-generated? Who is the data owner and what access approval is required? Surface ownership questions in discovery — not when the data engineer is blocked mid-sprint.
PM owns: The discovery gate: "Data access approved" must be checked before sprint 1.
Who labels it, what is the inter-annotator agreement target, how long does annotation take, what does it cost per example? Most AI business cases underestimate labeling cost by 3–5x.
PM owns: The labeling budget and the IAA quality threshold.
Training on historical data that does not match future inputs is a primary AI failure mode. Does the training set reflect the distribution of inputs — including edge cases and underrepresented subgroups — the model will actually see at launch?
PM owns: The requirement to validate distribution match. Shadow mode is the tool.
Data flywheel — design for it from day 1
The best AI products generate their own training data through use. Design the feedback loop before the first sprint: which user actions signal quality? Which corrections can be captured and routed back to training?
No historical data? Build a rule-based baseline → use its outputs as silver labels for a seed training set → replace the rule-based system once model confidence exceeds it. The rule-based system is data collection, not the product.
Stripe Radar v1: manual fraud rules generated labeled data. The ML model replaced those rules once it outperformed them on held-out data.
Design UI actions that implicitly signal quality: accepts, dismisses, edits, retries. Each is a label. Capture them systematically and route them to the training pipeline.
Gmail Smart Reply: users sending or dismissing suggestions generated millions of daily implicit labels — no manual annotation required after launch.
Acceptable for edge case augmentation — not as a substitute for real-world distribution data. Synthetic data improves coverage of rare cases; it cannot replace the distributional signal of real user interactions.
Medical imaging AI: synthetic augmentation (rotations, brightness adjustments) on rare pathology images improved recall on underrepresented conditions by 12%.
Five data quality dimensions
Select a dimension to see how to evaluate it, what the PM action is, and a real example of what happens when it is ignored.
Volume
The number of labeled examples available for training. Volume determines model capacity — there is a floor below which no architecture performs reliably.
General rule: 1,000+ examples to beat a strong baseline; 10,000+ for reliable production models; 1M+ for fine-tuning large language models.
Confirm volume with data engineering in discovery week 1. If volume is below threshold, budget a labeling sprint before any model work. Do not let labeling be implicit in the sprint estimate.
GitHub Copilot: trained on 50M+ public code repositories. Fine-tuned on code completion completions across 8+ languages. Volume was the primary enabler — not architecture.
Risk if ignored: Most AI business cases underestimate labeling cost by 3–5x. 10,000 examples at $2/label = $20,000 + 8 weeks of annotation management.
Data quality checklist — discovery gate
All five must be confirmed green before sprint 1. Missing any one creates an undefined risk that will surface as a sprint blocker.
Volume: sufficient labeled examples confirmed by data science — with volume estimate by class/segment.
Label quality: inter-annotator agreement (IAA) target set and baseline measured on a 100-item sample.
Recency: data freshness confirmed — training distribution matches expected production distribution.
Distribution: disaggregated analysis planned — accuracy will be measured by input subgroup, not overall only.
Access & compliance: legal and privacy sign-off on data use for training obtained before annotation begins.
The evaluation hierarchy
Evaluation is a PM deliverable — not a data science afterthought. Select an eval level to see when to use it, what it catches, and what the PM owns.
All five levels are required before launch. They are not alternatives — they are sequential gates.
Test format adherence, schema validity, output constraints, and boundary conditions. Fast and cheap — run in CI pipeline. Catch regressions but do not assess quality.
Output contract test for a summarization model: "Response must be 50–150 words, plain text, no markdown, must not include phrases from the banned list." Fails if the model returns a 300-word response or a JSON object.
The output contract that defines what the tests verify. Without a defined output contract, unit tests cannot be written.
The golden test set — PM deliverable
The golden test set is the PM's most important eval artifact. Its composition determines what "good enough" means. These rules are non-negotiable.
Planning for iteration cycles
AI development does not converge in one sprint. Budget 3–5 iteration cycles before production as scheduled work — not contingency. Each cycle follows the same pattern.
Labeling strategies when budget is constrained
Active learning
Use the model to identify the examples it is least confident about. Prioritize those for human labeling. The same annotation budget produces 2–3x more model improvement than random sampling.
Most modern annotation platforms (Scale AI, Labelbox) support active learning queuing natively.
Weak supervision
Write simple rules and heuristics that programmatically label large volumes at low accuracy. Combine multiple weak sources into a probabilistic label good enough to bootstrap training. (Snorkel framework)
Best for high-volume classification tasks where rule-based logic partially captures the task.
Rule-based bootstrap
Build a rule-based classifier first. Use its outputs as silver labels. Supplement with human-labeled gold examples where the rule-based system is uncertain. The rules are data collection, not the product.
Used by Stripe for early Radar fraud rules, GitHub for early Copilot training signal design.
Data & eval interview Q&A
Five scenario questions that test whether your data and eval instincts are concrete or theoretical.
How do you know when a model is ready to ship?
Three gates must all be green.
First: the model meets the pre-defined acceptance threshold on the golden test set — the threshold we defined in the model spec before any training began. If we did not define a threshold before building, we are negotiating what "good enough" means under schedule pressure, which always produces the wrong answer.
Second: it passes the eval hierarchy. Automated unit tests for regressions, model-graded evals at scale, human eval on the golden set, and shadow mode against production-like inputs. All four layers must clear, not just the headline accuracy number.
Third: the error cost test. Even at threshold, I ask: what is the business cost of residual errors at the intended automation level? If a 12% error rate at L2 automation means one human reviewer catches and corrects those errors — and that is cheaper than the value the automation creates — we ship. If the error cost at the intended automation level exceeds the value, I drop the automation level to L1 rather than lower the quality threshold.
Common trap: Using test set accuracy as the only gate. The test set may not represent production distribution, and accuracy hides subgroup failures. Launch readiness requires all three gates.
Your model has 92% accuracy on the test set but users report it is wrong far more often than expected. What is happening and what do you do?
Three candidate explanations, investigated in this order.
First: test set distribution mismatch. If the test set was collected at a different time, from a different user population, or with a different labeling policy, 92% accuracy on the test set tells me nothing about production. I ask data science to run the current model on a fresh production sample from the last 30 days and compare metrics. If accuracy on the production sample is 72%, the test set was the problem.
Second: class imbalance. If the dominant class represents 91% of inputs and the model learned to predict it, the 92% accuracy hides near-zero performance on minority classes. Users in the minority case experience constant failure. I request accuracy broken down by category and input type.
Third: the quality metric is wrong. Users may define "correct" differently from the eval rubric. "92% accuracy" may mean 92% of outputs matched the label — but users may care about relevance, completeness, or confidence rather than label match. I run a 50-user qualitative review to understand what "wrong" means from their perspective.
Action: disaggregated error analysis on the last 1,000 user-reported failures. Cluster by input type, category, and confidence score. The root cause is usually visible within 48 hours.
Common trap: Defending the 92% test set number. User complaints are data. The test set and the user experience are in conflict — the test set is wrong, not the users.
How do you plan AI development timelines and communicate progress to stakeholders?
I plan in phases, not sprints.
Phase 1: data readiness (2–4 weeks). This is the work most stakeholders do not see and most PMs skip, which is why AI projects run late. Data audit, labeling pipeline, access approval, and baseline measurement. This phase determines whether the rest of the plan is realistic.
Phase 2: iteration cycles (3–5 sprints). Each cycle: build → eval on golden set → identify failure modes → improve → re-eval. I do not commit to a ship date. I commit to a milestone: "We will have an eval result by this date. If we are at threshold, we ship. If not, we will have the go/no-go conversation with this specific plan to close the gap."
I communicate progress via eval scores on the golden set — a number that moves week to week — rather than feature completion percentages. "We are at 84% on our 88% target, the primary failure mode is edge cases in the German language locale, and we expect to close the gap in 2 sprints" is more credible and more honest than "we are 70% complete."
Stakeholders who have worked with me prefer this model. They know what they are getting.
Common trap: Committing to a fixed ship date before the data readiness phase is complete. Ship dates for AI features are milestones with exit criteria, not calendar commitments.
How do you handle training-serving skew — when production inputs differ from training data?
Training-serving skew is one of the top AI deployment failure modes and it is almost always preventable.
Prevention: I require that the data preprocessing pipeline for training and the data preprocessing pipeline for serving are the same code, not similar code. Different pipelines diverge silently — a column renamed, a normalization step added — and the model sees inputs it was never trained on. I make this a deployment gate in the PRD.
Detection in pre-launch: shadow mode runs the new model on production traffic and compares input feature distributions against the training set. If key features (input length, category distribution, timestamp patterns) drift by more than 15% from the training baseline, we investigate before cutover.
Detection in production: I define monitoring dashboards that track key input feature distributions continuously. Alert thresholds: if a feature distribution shifts by more than 20% from the 30-day baseline, it triggers an investigation. This is the earliest signal of concept drift — catching it in feature distributions rather than waiting for model metric degradation.
The PM owns: the alert thresholds and the escalation protocol. Engineering owns the monitoring infrastructure.
Common trap: Treating training-serving skew as an engineering concern. It is a PM concern because it surfaces in production as quality degradation that users notice before monitoring does.
How do you approach building a labeled dataset when budget and time are constrained?
Three strategies, applied in order of cost-efficiency.
First: active learning. Instead of labeling a random sample, use the model to identify the examples it is most uncertain about and prioritize those for human labeling. The same annotation budget produces 2–3x more model improvement because every labeled example teaches the model something new. Most modern annotation tools support this natively.
Second: weak supervision (Snorkel-style). Write labeling functions — simple rules, heuristics, or existing noisy labels from legacy systems — that programmatically label large volumes of data at low accuracy. Combine multiple weak sources into a probabilistic label that is often good enough to bootstrap training. Accuracy is not perfect but the volume makes it useful.
Third: bootstrap from a rule-based system. Build a simple rule-based classifier first. Use its outputs as silver labels for a high-volume, low-confidence training set. Supplement with human-labeled gold examples for the cases where the rule-based system is uncertain. The rule-based system is not the final product — it is the data collection strategy.
In all cases: prioritize quality over quantity for the human-labeled portion. 500 clean, high-agreement examples outperform 5,000 noisy ones for most classification tasks.
Common trap: Defaulting to random sampling for annotation. This is the least efficient use of annotation budget. Active learning and weak supervision are not advanced techniques — they are standard practice for resource-constrained labeling projects.
I'm preparing for an AI PM interview. Give me a data and evaluation scenario — for example, I've just been assigned to a churn prediction model and need to plan the data strategy and evaluation framework. Ask me to walk through: data quality assessment, labeling strategy, eval design, acceptance threshold, and how I communicate progress to stakeholders. Then give me feedback.