What it actually takes to build frontier models | Tulsee Doshi (Head of Product for Gemini)

Gemini's product chief on why unpredictable models make evals the job

Frontier models behave unpredictably, so building with them at Google starts with testing what they actually do. Doshi argues PMs need daily hands-on use, strong opinions and their own evals. Products should be built ahead of model capability yet loose enough to retest as models change every few weeks.

Takeaways

• Models are grown, so testing them is everything. Doshi compares training to Snape's potions class, where ingredients go in and some surprising magic comes out. Because behaviour cannot be predicted in advance, product work centers on discovering what a model actually does through hands-on use and evals.
• Google hires PMs with opinions on models. Doshi wants PMs who play with models daily, stress test them on things they care about and can build evals. Agility matters because the space changes every two weeks. Lenny notes builders are rediscovering PMs as easy building produces unwanted products.
• Start evals tiny, then make them holistic. Begin with about five prompts or a mini eval of roughly 20, using tasks you care about to find where models are close to good. Full evals must also cover judge quality, the production stack and serving, since all shape what users experience.
• Build ahead of models, but stay flexible. Doshi's 60-20-20 rule has 60% of a product working on today's models. She warns against rigid systems and urges regular retesting, echoing Boris Cherny's advice to remove skills occasionally because newer models may handle those tasks themselves.
• Google competes across the full stack. Google spans hardware to products serving billions, with roughly 14 products past a billion users by Doshi's estimate. It aims for models along a cost-to-quality curve, from Gemini 4 Argonne at the frontier to GenMedia, open Gemma and audio models.
• A model has its own UX. A model behaviour team turns user feedback into specs, rubrics and reward modeling. Doshi also argues for new interaction paradigms, citing NotebookLM and Flow, AI Mode for billions of search users and Antigravity for power users.
• Prototypes and data are the new PRD. The PM role now means adding substance rather than writing requirements. Doshi says burnout comes from churn and unenjoyable work, and hopes AI strips out grunt work. Google will pull all PMs out of their jobs for a two-day November workshop.

Lenny's challenges on Google's position

• Lenny raised the debate over Google lagging Anthropic and OpenAI. Doshi said Google does want to win the frontier conversation.
• Asked why Google has no standout assistant using Gmail and Calendar data, Doshi said the team is working on amazing things but there is more work to do.