Lenny's Podcast: Product | Career | Growth
00:01:15 · Tulsee Doshi, Lenny Rachitsky
[00:01:15] S0: Thanks. I'm excited to be here. [00:01:17] S1: You are head of product for Google's Frontier models, which includes Gemini, which people know. But there's also a bunch of other models that people may not be familiar with. Share the other models that are under your umbrella. [00:01:30] S0: Yeah, so we have a set of models we kind of call our GenMedia models. These are our models that focus on video editing and generation, image editing and generation, if you've used Nano Banana, Lyria, our music model. We also have Gemma, our suite of open models. And then one set of models that actually falls under our Gemini umbrella, but I feel like we don't talk about as much as our audio models. So dialogue, audio understanding, text to speech, translation. All a part of the portfolio of how we build FredShare models. [00:02:03] S1: Okay, so I want to spend a little time on models. Everyone sees models launch. There's like a model launch every week, it feels like. You have a really unique lens in being able to see these models from research to pre-training, post-training launch. What's something that you think people maybe misunderstand about how models are created or something that may surprise people about how models are made?
00:02:26 · Tulsee Doshi
[00:02:26] S0: Yeah, it's a good question. What I'll say, especially coming from a product management lens is one thing that I think, you know, when you typically talk about product management for anything, including a model, you talk about what are the goals you have for a model, right? What are the capabilities you want to set? What are the outcomes you want to drive? What goals do you want to hit on a set of evals? And often when I talk to customers or even internal users, you know, they'll ask me, okay, what are the goals that you're setting? And they'll sort of expect that we can Really make sure that the model will deliver on those goals in a very traditional product sense. And I think for me, what is often been surprising but also exciting about model development and being a PM for the model is you should do that and you should have an intentional view, but the model isn't predictable. And so it's kind of like, like Snape's potions class, right? So you can put all the ingredients in and you should be thoughtful about what you expect. But you also have to expect that there will be a little bit of magic and that that magic is going to be surprising and that magic is going to be unique. And so your job as a PM is not just to try to shape what goes into the model, but it is also to look at what comes out the other side and really identify where does the model shine, where does it actually look unique in ways that surprise you, and then be able to take that and shape that into something that is amazing for users.
00:03:55 · Lenny Rachitsky
[00:03:55] S1: It's so interesting that it's just like this code that we have written that we feed data and we need to, like we don't exactly know what's going to happen and how it works. I recently had an interpretability research person on the podcast from Anthropic and we talked a lot about this, this idea that models are grown, not built. And just to kind of reflect back what you're saying, there's these goals that researchers have. For this next model, we're going to get better at writing. We're going to try to be funnier. We're going to try to become better engineers. And you're saying it's unpredictable exactly what ends up at the end. They put in these ideas. They change the data that it's trained on. Is there anything more there about just this kind of like unexpected, I don't know, process of training?
00:04:36 · Tulsee Doshi
[00:04:36] S0: You know, of course, there's brilliant research that goes into these models. And so many aspects of it are predictable in the sense that we are consistently monitoring these runs. We have researchers who are tracking them, who are identifying when to intervene, who are seeing different metrics climb up as the run progresses. And so we are making it as much of a science as possible. But I think for me, what is really amazing as a PM is you also need to really understand the vibes of the model on the other side of it, too. So, yes, you can understand that a model has improved on a certain capability. But where does that capability really shine through? How does that showcase that downstream in a product? How do you feel communicating with the model in terms of its behavior and persona? And there's a number of these aspects that we definitely build into training. But also as we understand the model and as we evaluate it and as we test it, we always learn more. We always learn something new, I think, about what the model can do.
00:05:29 · Lenny Rachitsky
Ad[00:05:29] S1: This episode is brought to you by this season's presenting sponsor, WorkOS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay and hundreds of other winning companies have in common? They're all powered by WorkOS if you're building for enterprises. You have felt the pain of integrating single sign-on, SCIM, RBAC, and audit logs. WorkOS turns those deal blockers into drop-in APIs built for B2B SaaS. And now there's a new kind of user that needs access to your app, Agents. AI agents need to register, authenticate, and get permissions, just like a person. WorkOS handles it all. AgentAuth to give agents scope, revocable identities, agent registration with AuthMD, and Airlock to judge every agent action and allow, deny, or escalate to a human. I've noticed that every startup that I'm an investor in that expands upmarket ends up using WorkOS because they're the best. Go to WorkOS.com to make your app enterprise ready today. I want to talk a little bit about Google's place in the AI race. It feels like a lot of people feel that Google lags behind the top two labs right now, Anthropic and OpenAI. And Google has these moments of being the best and then it kind of feels like it slows down a bit. I'm curious what you can share about just Google's strategy in the AI race and what maybe people don't understand on the inside of how you all think about this stuff.
00:06:54 · Tulsee Doshi
[00:06:54] S0: First of all, hopefully by the time this conversation goes out, you will be using Gemini 4 Argonne, which is exciting. And so I really do hope we're having a conversation about a frontier AI model that folks are excited to use. And I think Gemini 4 speaks a little bit to our strategy, right? Because it really is our push to continue to build a frontier lab, a frontier set of models, a frontier set of capabilities on our path. I think if you look at Google's strategy, it really is full stack, right? So one of the things that I think is powerful about Google is all the way from hardware to end products that serve billions of users. I think the stat is like we have 14 products that are above a billion users. I should check that number, but I think that's the stat. And so if you think about that, we actually go all the way from the hardware to then the modeling to then this kind of end user application. And our goal ultimately is to not just build amazing frontier experiences, which is definitely a part of the goal, but it is also to build models across this Pareto curve of cost to quality that can serve different users at different points in their lifestyle, whether that be a search user who is coming for Information, right? Whether it is an engineer who is trying to build the next amazing thing, whether it is for a cyber defender who is trying to protect a critical piece of infrastructure. I think our goal is to make sure that we have models that serve that range of use cases and that we're actually building with feedback from these users.