AI agents
From reactive chatbots to autonomous systems. What every AI PM must own: trust radius, tool governance, eval metrics, and when not to build an agent at all.
Avoid when: When the task has more than 2 steps or requires external tool calls.
Avoid when: When the path through steps varies based on real-time conditions.
Avoid when: When a deterministic workflow can do the job — agents cost 10x more and fail more often.
User: "Book me on the cheapest NYC→London flight tomorrow."
The orchestrator receives the goal and must plan: what tools exist, what order to call them, what confirmation is needed before any action executes.
You define the goal format and ambiguity handling. What happens if "tomorrow" is ambiguous across timezones? That is a PM spec decision, not an engineering edge case.
Agent drafts an email and surfaces it for review. You click Send. The agent cannot send without your approval.
Financial actions, external communications, and any operation that cannot be undone.
The Trust Radius — the most important mental model for agent PMs: Every agent has a radius of actions it can take autonomously. PMs define that radius — not engineers. Too small and the agent delivers no value. Too large and you get catastrophic failures. Every agent feature ships with a trust radius doc: what it can do freely, what requires confirmation, what is permanently blocked.
Act as a travel booking agent. You have access to: search_flights, book_flight, cancel_booking, send_email. I want to change my NYC-London flight to tomorrow. Walk me through every decision you make and every tool call, including what could go wrong at each step and what confirmation you'd ask for.
I'm a PM at a fintech company. My agent can read transactions and send payment reminders. Write the trust radius document for this agent: what it can do autonomously, what needs human approval, and what is permanently prohibited. Be specific about each action type.
I want to automate customer onboarding. Steps: verify ID, check credit score, assign account manager, send welcome email. Should I build an agent or a deterministic workflow? Walk me through the exact decision criteria and give me your recommendation.
Agents are for unknown step paths and multi-tool reasoning. Default to the simpler option — a deterministic workflow is cheaper, more reliable, and easier to debug.
You cannot write acceptance criteria like a traditional feature. Define failure modes and stopping conditions, not just success paths.
Classify every action: runs freely, requires confirmation, or permanently blocked. This document belongs in the PRD, not the engineering design doc.
An agent that completes 2 of 4 steps and fails is worse than one that never started. Write rollback specs for every write and irreversible action before engineering begins.
| Type | Risk | Examples | Approval required |
|---|---|---|---|
| Read | Low | search_flights, get_calendar, read_doc, query_db | No confirmation needed |
| Write | Medium | send_email, update_crm, create_ticket, post_message | L2 confirmation recommended |
| Irreversible | High | book_flight, delete_record, process_payment, deploy_code | Mandatory human gate + rollback spec required |
Agents given 40+ tools call irrelevant ones constantly. Tool proliferation degrades planning quality. The tool manifest — what tools exist and what they can do — is a PM-owned product constraint, not an engineering default.
MCP (Anthropic, Nov 2024) is an open protocol that standardizes how AI models discover and call external tools. By early 2025, Zed, Replit, Sourcegraph, and Cursor had adopted it. It is not magic — it is JSON-RPC 2.0 with a schema discovery layer. The "USB port" analogy understates what PMs need to understand about the trust model.
The Host manages trust boundaries — which agents can access which servers, with what permissions. The PM owns these decisions before engineering builds the integration.
Callable functions. Each tool is a trust decision — what action can the agent actually execute? Highest risk.
Read-only data: files, DB rows, API responses. Still a data governance question — what can be read and by whom.
Pre-built templates that guide agent behavior. Controlled by the server, not the end user. Lowest risk primitive.
MCP security is a PM responsibility. Every MCP server your agent connects to is a trust boundary. Before approving any integration: What data can this server read? What actions can it execute? What is the authentication model? Who audits tool call logs? These are product requirements to spec — not security team sign-offs to collect at the end.
I'm designing an AI agent for customer support. Suggest 5 tools it needs (with function signatures), then classify each as Read, Write, or Irreversible. For each Write/Irreversible tool, specify the exact confirmation mechanism and rollback approach.
My agent will connect to an MCP server that gives it read and write access to our company Postgres database. As a security-conscious PM, what are the specific questions I need to answer before approving this? Include attack vectors like prompt injection and data exfiltration.
Using the principle of least privilege, design the permission model for a customer support agent. It can see order history, send emails, and initiate refunds up to $50. Define exact confirmation gates and what happens if a refund exceeds $50.
What tools an agent can access is a product decision with security, cost, and reliability implications. Don't let it default to whatever engineering adds.
MCP standardizes tool discovery and calls. It does not handle authentication, sandboxing, or error recovery. Those are your product requirements to spec.
Well-scoped agents with 5–8 tools outperform agents with 40. Constrain the tool set as a product requirement before the engineering design doc is written.
Every tool call must log: tool name, inputs, output, timestamp, and agent reasoning. This is the audit trail for every downstream incident. Non-negotiable.
"Users can search and filter products"
"Agent achieves [outcome] within [max steps] at [max cost per run] with [acceptable error rate]"
Feature completeness, latency, uptime
Task completion rate, step efficiency, cost-per-task, error recovery rate, hallucinated tool call rate
Empty state, error state, loading state
Loop detected, tool call failed, context window exhausted, irreversible action attempted, task partially complete
Feature flag, A/B test
Shadow mode → L1 (suggest only) → L2 (confirm before act) → L3 (notify after act). Each phase gated on eval metrics.
| Metric | What it measures | Alert threshold |
|---|---|---|
| Task completion rate | Successfully completed tasks / total tasks initiated | <80% in production → investigate before scaling |
| Step efficiency | Actual steps / optimal steps for task type | >2x optimal consistently → prompt or tool schema issue |
| Cost per task | Total token cost / completed tasks | >$0.50 for consumer use cases → routing or caching needed |
| Error recovery rate | Tasks recovered after first tool failure / total failures | <60% recovery → tool design or error handling gap |
| Hallucinated tool calls | Calls with malformed params or invalid tool names | >5% → model needs better tool schemas or few-shot examples |
A 10-step agent using Claude Sonnet 4.6 at ~3K input + 600 output tokens per step costs approximately $0.15 per run. At 500K runs/month that is $75,000 — a number that belongs in the PRD, not the post-launch finance review.
Use a fast/cheap model (Gemini Flash, Haiku) for planning steps. Reserve the frontier model for execution steps where quality matters most.
Set a hard max step count in the PRD. Agent that exceeds the budget escalates to a human — it does not loop indefinitely.
Cache read tool responses within the same session. Avoid re-fetching identical search results or database queries on subsequent reasoning steps.
Debugging a production agent is a PM skill. When task completion drops from 80% in staging to 45% in production: first check distribution shift (real prompts are messier than test prompts). Then check tool error rates by tool type. Then check context window exhaustion on long tasks. Only after ruling all of these out should you ask engineering to change the model or prompt.
Write the 'Failure modes and stopping conditions' section of a PRD for a customer support agent. For each failure mode: what triggers it, what the agent does, what the user sees, what gets logged, and what the recovery path is.
My agent uses Claude Sonnet 4.6. Average task: 8 steps, 3K input + 800 output tokens per step. We expect 500K tasks per month. Calculate total monthly cost, then show me 3 approaches to cut it by 60% without degrading task quality.
Design a 6-metric monitoring dashboard for a PM running a customer support agent in production. For each metric: what it measures, the alert threshold, and the first investigation step when the alert fires.
Add to every agent PRD: max step count, token budget, stopping conditions, fallback behavior, tool manifest, and rollback specs. These are PM deliverables, not engineering concerns.
Define task completion rate, step efficiency, and cost-per-task targets before a line of agent code is written. Retrofitting evals after launch means you are flying blind.
Every agent feature needs a cost model at requirements stage: budget per session, per user, per month. Surprises here become executive-level incidents.
Thought → Tool Called → Result logs are required for every production agent. For debugging, compliance, and user trust. An agent without trace logs is not shippable.