AI PM Ready
Back to the Digest

JioStar's AI Discovery at Scale: Lessons on Evals and Production from OpenAI

OpenAI BlogAI & Agentic AI3 min readImportantOriginal source ↗

JioStar, with 500 million users, is leveraging AI for discovery and has shared key lessons on evaluation (evals) and production deployment. The insights from their Chief Architect, Vijay Seshadri, focus on practical application and risk mitigation in large-scale AI systems. This information is crucial for PMs aiming to implement AI features effectively in high-user-volume products.

What happened

JioStar's Chief Architect, Vijay Seshadri, shared insights on AI discovery at a 500-million-user scale in a discussion with OpenAI. The key takeaways revolve around the practical application of AI, including lessons learned from implementing evaluations (evals) and deploying AI into production environments. This real-world experience from a large-scale platform offers valuable guidance for AI product managers.

A scenario you might face

A mid-market e-commerce company is planning to introduce AI-driven personalization features to its platform, which serves several million users. The product manager needs to understand how to effectively test and deploy these features to ensure they enhance user experience without introducing unforeseen issues. Learning from JioStar's experience at scale can help them refine their evaluation strategy and anticipate potential production challenges, guiding their roadmap for AI integration.

What changes in your practice

In your next product review meeting, propose a dedicated discussion on the evaluation strategy for your upcoming AI feature, specifically asking how it will be tested at scale before a full rollout, drawing parallels from JioStar's reported experience.

Questions for your team

  1. 01What specific metrics does JioStar use to measure the success of AI-driven discovery features at scale?
  2. 02How does JioStar approach the A/B testing and canary release of new AI models or features to its 500M user base?
  3. 03What guardrails and safety mechanisms does JioStar have in place to mitigate risks associated with AI-generated content or recommendations?
  4. 04What was the process for iterating on their AI models based on production feedback and evaluation results?

Practise this

Interviewers ask about stories like this one. Rehearse your answer in a timed mock interview.

Start a mock interview →
All stories
Was this page useful?