AI PM Ready

Data & evaluation

Data is the product. Model quality cannot exceed data quality — and evaluation is how you know whether you're ready to ship. Both are PM responsibilities.

Four data questions before sprint 1

The ceiling rule: model quality cannot exceed data quality. Every sprint on the model is wasted if the data is wrong. Answer these four questions with data engineering before any model work begins.

1.What data does the model need?

Exact input format, required fields, expected volume per day, and freshness requirement (real-time vs. batch). Write this as a data spec alongside the model spec — not after the first sprint review.

PM owns: The data spec. Not the model architecture.

2.Where does it come from and who owns it?

Internal, third-party, or user-generated? Who is the data owner and what access approval is required? Surface ownership questions in discovery — not when the data engineer is blocked mid-sprint.

PM owns: The discovery gate: "Data access approved" must be checked before sprint 1.

3.Is it labeled — at what cost and quality?

Who labels it, what is the inter-annotator agreement target, how long does annotation take, what does it cost per example? Most AI business cases underestimate labeling cost by 3–5x.

PM owns: The labeling budget and the IAA quality threshold.

4.Does it represent what the model will see in production?

Training on historical data that does not match future inputs is a primary AI failure mode. Does the training set reflect the distribution of inputs — including edge cases and underrepresented subgroups — the model will actually see at launch?

PM owns: The requirement to validate distribution match. Shadow mode is the tool.

Data flywheel — design for it from day 1

The best AI products generate their own training data through use. Design the feedback loop before the first sprint: which user actions signal quality? Which corrections can be captured and routed back to training?

Cold start

No historical data? Build a rule-based baseline → use its outputs as silver labels for a seed training set → replace the rule-based system once model confidence exceeds it. The rule-based system is data collection, not the product.

Stripe Radar v1: manual fraud rules generated labeled data. The ML model replaced those rules once it outperformed them on held-out data.

Implicit feedback

Design UI actions that implicitly signal quality: accepts, dismisses, edits, retries. Each is a label. Capture them systematically and route them to the training pipeline.

Gmail Smart Reply: users sending or dismissing suggestions generated millions of daily implicit labels — no manual annotation required after launch.

Synthetic data

Acceptable for edge case augmentation — not as a substitute for real-world distribution data. Synthetic data improves coverage of rare cases; it cannot replace the distributional signal of real user interactions.

Medical imaging AI: synthetic augmentation (rotations, brightness adjustments) on rare pathology images improved recall on underrepresented conditions by 12%.

Was this page useful?