Lenny's Podcast: Product | Career | Growth
00:03:55 · Lenny Rachitsky
[00:03:55] S1: It's so interesting that it's just like this code that we have written that we feed data and we need to, like we don't exactly know what's going to happen and how it works. I recently had an interpretability research person on the podcast from Anthropic and we talked a lot about this, this idea that models are grown, not built. And just to kind of reflect back what you're saying, there's these goals that researchers have. For this next model, we're going to get better at writing. We're going to try to be funnier. We're going to try to become better engineers. And you're saying it's unpredictable exactly what ends up at the end. They put in these ideas. They change the data that it's trained on. Is there anything more there about just this kind of like unexpected, I don't know, process of training?
00:04:36 · Tulsee Doshi
[00:04:36] S0: You know, of course, there's brilliant research that goes into these models. And so many aspects of it are predictable in the sense that we are consistently monitoring these runs. We have researchers who are tracking them, who are identifying when to intervene, who are seeing different metrics climb up as the run progresses. And so we are making it as much of a science as possible. But I think for me, what is really amazing as a PM is you also need to really understand the vibes of the model on the other side of it, too. So, yes, you can understand that a model has improved on a certain capability. But where does that capability really shine through? How does that showcase that downstream in a product? How do you feel communicating with the model in terms of its behavior and persona? And there's a number of these aspects that we definitely build into training. But also as we understand the model and as we evaluate it and as we test it, we always learn more. We always learn something new, I think, about what the model can do.
00:05:29 · Lenny Rachitsky
Ad[00:05:29] S1: This episode is brought to you by this season's presenting sponsor, WorkOS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay and hundreds of other winning companies have in common? They're all powered by WorkOS if you're building for enterprises. You have felt the pain of integrating single sign-on, SCIM, RBAC, and audit logs. WorkOS turns those deal blockers into drop-in APIs built for B2B SaaS. And now there's a new kind of user that needs access to your app, Agents. AI agents need to register, authenticate, and get permissions, just like a person. WorkOS handles it all. AgentAuth to give agents scope, revocable identities, agent registration with AuthMD, and Airlock to judge every agent action and allow, deny, or escalate to a human. I've noticed that every startup that I'm an investor in that expands upmarket ends up using WorkOS because they're the best. Go to WorkOS.com to make your app enterprise ready today. I want to talk a little bit about Google's place in the AI race. It feels like a lot of people feel that Google lags behind the top two labs right now, Anthropic and OpenAI. And Google has these moments of being the best and then it kind of feels like it slows down a bit. I'm curious what you can share about just Google's strategy in the AI race and what maybe people don't understand on the inside of how you all think about this stuff.
00:06:54 · Tulsee Doshi
[00:06:54] S0: First of all, hopefully by the time this conversation goes out, you will be using Gemini 4 Argonne, which is exciting. And so I really do hope we're having a conversation about a frontier AI model that folks are excited to use. And I think Gemini 4 speaks a little bit to our strategy, right? Because it really is our push to continue to build a frontier lab, a frontier set of models, a frontier set of capabilities on our path. I think if you look at Google's strategy, it really is full stack, right? So one of the things that I think is powerful about Google is all the way from hardware to end products that serve billions of users. I think the stat is like we have 14 products that are above a billion users. I should check that number, but I think that's the stat. And so if you think about that, we actually go all the way from the hardware to then the modeling to then this kind of end user application. And our goal ultimately is to not just build amazing frontier experiences, which is definitely a part of the goal, but it is also to build models across this Pareto curve of cost to quality that can serve different users at different points in their lifestyle, whether that be a search user who is coming for Information, right? Whether it is an engineer who is trying to build the next amazing thing, whether it is for a cyber defender who is trying to protect a critical piece of infrastructure. I think our goal is to make sure that we have models that serve that range of use cases and that we're actually building with feedback from these users.
00:08:30 · Lenny Rachitsky, Tulsee Doshi
[00:08:30] S1: So what I'm hearing is there's so many vectors where Google can win. There's the most efficient kind of Pareto Frontier. For token costs, there's the smartest model. There's also just making your products better. For example, AI mode, AI overviews. And so it's not that you need to be the best at the most advanced AI model that people can use, like Chatbot, basically. You can win in all these other areas. But it sounds like you're also trying to win and have the most state-of-the-art model [00:08:58] S0: I actually think we do want to win the frontier conversation, right? I don't think we should say that Google is not participating in the frontier. And I think we are building better and better models. And I think Gemini for Argonne definitely is at the frontier in the place that we are at. But I think one thing that is amazing is being able to build a breadth of experiences and build a diversity of experiences for users. And I think for Google, Our goal ultimately is to build amazing experiences that actually make people's lives better, right? And that is, I think, ultimately the goal, whatever part of the frontier you're building for. [00:09:42] S1: Google products definitely make my life better. It's just like amazing how much of your life is within a Google ecosystem, like not even the AI stuff, but just Google Calendar, Gmail, all these things, you're just like Google Docs. So it's happening. [00:09:55] S0: There you go.
00:09:55 · Tulsee Doshi, Lenny Rachitsky
[00:09:55] S0: There you go. [00:09:56] S1: There we go. I want to talk about one other broad AI question that I'm going to dig into more product specific stuff. A big topic of conversation these days is AI safety and this kind of fear that AI is going to harm humans. It's going to cause damage as it gets smarter and smarter. You have a really interesting lens on this because you led product for the responsible AI team at Google for six years. This is what you did right before this role? And so you've been thinking about this for a long time, a lot longer than most people. I'm curious just to get your take on this debate. Just, I don't know, are you optimistic we will figure this out? What do you think people maybe don't understand about this debate around AI safety?