Lenny's Podcast: Product | Career | Growth
00:00:00 · Tulsee Doshi, Lenny Rachitsky
[00:00:00] S0: The model isn't predictable. It's kind of like Snape's potions class. You can put all the ingredients in, but you also have to expect that there will be a little bit of magic, and that magic is going to be surprising. [00:00:10] S1: As you're hiring PMs these days, is there anything you look for more and more? [00:00:15] S0: You're looking for PMs who understand the vibes of our models or other models. I want PMs who have an opinion, who are playing with these models every day. You ideally want to stress test the model on things that you are excited about. [00:00:28] S1: A lot of people feel that Google lags behind the top two labs right now. [00:00:32] S0: I actually think we do want to win the frontier conversation. I don't think we should say that Google is not participating in the frontier. [00:00:39] S1: You have this framework around how far ahead to build. You call it a 60-20-20 rule. [00:00:44] S0: Traditional startup wisdom. You're just like, build an MVP that works. But I think you want to actually build features that aren't necessarily always working today. [00:00:52] S1: It feels like every AI assistant launching these days is taking all this juicy Google data, my Gmail, Google Calendar, my Docs. I'm curious just why Google hasn't launched something awesome yet. [00:01:03] S0: The team is working on some amazing things, but I think you are right that there is more work we need to do here on this front. [00:01:12] S1: Tulsee, thank you so much for being here. Welcome to the podcast. [00:01:15] S0: Thanks. I'm excited to be here.
00:01:15 · Tulsee Doshi, Lenny Rachitsky
[00:01:15] S0: Thanks. I'm excited to be here. [00:01:17] S1: You are head of product for Google's Frontier models, which includes Gemini, which people know. But there's also a bunch of other models that people may not be familiar with. Share the other models that are under your umbrella. [00:01:30] S0: Yeah, so we have a set of models we kind of call our GenMedia models. These are our models that focus on video editing and generation, image editing and generation, if you've used Nano Banana, Lyria, our music model. We also have Gemma, our suite of open models. And then one set of models that actually falls under our Gemini umbrella, but I feel like we don't talk about as much as our audio models. So dialogue, audio understanding, text to speech, translation. All a part of the portfolio of how we build FredShare models. [00:02:03] S1: Okay, so I want to spend a little time on models. Everyone sees models launch. There's like a model launch every week, it feels like. You have a really unique lens in being able to see these models from research to pre-training, post-training launch. What's something that you think people maybe misunderstand about how models are created or something that may surprise people about how models are made?
00:02:26 · Tulsee Doshi
[00:02:26] S0: Yeah, it's a good question. What I'll say, especially coming from a product management lens is one thing that I think, you know, when you typically talk about product management for anything, including a model, you talk about what are the goals you have for a model, right? What are the capabilities you want to set? What are the outcomes you want to drive? What goals do you want to hit on a set of evals? And often when I talk to customers or even internal users, you know, they'll ask me, okay, what are the goals that you're setting? And they'll sort of expect that we can Really make sure that the model will deliver on those goals in a very traditional product sense. And I think for me, what is often been surprising but also exciting about model development and being a PM for the model is you should do that and you should have an intentional view, but the model isn't predictable. And so it's kind of like, like Snape's potions class, right? So you can put all the ingredients in and you should be thoughtful about what you expect. But you also have to expect that there will be a little bit of magic and that that magic is going to be surprising and that magic is going to be unique. And so your job as a PM is not just to try to shape what goes into the model, but it is also to look at what comes out the other side and really identify where does the model shine, where does it actually look unique in ways that surprise you, and then be able to take that and shape that into something that is amazing for users.
00:03:55 · Lenny Rachitsky
[00:03:55] S1: It's so interesting that it's just like this code that we have written that we feed data and we need to, like we don't exactly know what's going to happen and how it works. I recently had an interpretability research person on the podcast from Anthropic and we talked a lot about this, this idea that models are grown, not built. And just to kind of reflect back what you're saying, there's these goals that researchers have. For this next model, we're going to get better at writing. We're going to try to be funnier. We're going to try to become better engineers. And you're saying it's unpredictable exactly what ends up at the end. They put in these ideas. They change the data that it's trained on. Is there anything more there about just this kind of like unexpected, I don't know, process of training?
00:04:36 · Tulsee Doshi
[00:04:36] S0: You know, of course, there's brilliant research that goes into these models. And so many aspects of it are predictable in the sense that we are consistently monitoring these runs. We have researchers who are tracking them, who are identifying when to intervene, who are seeing different metrics climb up as the run progresses. And so we are making it as much of a science as possible. But I think for me, what is really amazing as a PM is you also need to really understand the vibes of the model on the other side of it, too. So, yes, you can understand that a model has improved on a certain capability. But where does that capability really shine through? How does that showcase that downstream in a product? How do you feel communicating with the model in terms of its behavior and persona? And there's a number of these aspects that we definitely build into training. But also as we understand the model and as we evaluate it and as we test it, we always learn more. We always learn something new, I think, about what the model can do.
00:05:29 · Lenny Rachitsky
Ad[00:05:29] S1: This episode is brought to you by this season's presenting sponsor, WorkOS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay and hundreds of other winning companies have in common? They're all powered by WorkOS if you're building for enterprises. You have felt the pain of integrating single sign-on, SCIM, RBAC, and audit logs. WorkOS turns those deal blockers into drop-in APIs built for B2B SaaS. And now there's a new kind of user that needs access to your app, Agents. AI agents need to register, authenticate, and get permissions, just like a person. WorkOS handles it all. AgentAuth to give agents scope, revocable identities, agent registration with AuthMD, and Airlock to judge every agent action and allow, deny, or escalate to a human. I've noticed that every startup that I'm an investor in that expands upmarket ends up using WorkOS because they're the best. Go to WorkOS.com to make your app enterprise ready today. I want to talk a little bit about Google's place in the AI race. It feels like a lot of people feel that Google lags behind the top two labs right now, Anthropic and OpenAI. And Google has these moments of being the best and then it kind of feels like it slows down a bit. I'm curious what you can share about just Google's strategy in the AI race and what maybe people don't understand on the inside of how you all think about this stuff.