Lenny's Podcast: Product | Career | Growth
00:04:36 · Tulsee Doshi
[00:04:36] S0: You know, of course, there's brilliant research that goes into these models. And so many aspects of it are predictable in the sense that we are consistently monitoring these runs. We have researchers who are tracking them, who are identifying when to intervene, who are seeing different metrics climb up as the run progresses. And so we are making it as much of a science as possible. But I think for me, what is really amazing as a PM is you also need to really understand the vibes of the model on the other side of it, too. So, yes, you can understand that a model has improved on a certain capability. But where does that capability really shine through? How does that showcase that downstream in a product? How do you feel communicating with the model in terms of its behavior and persona? And there's a number of these aspects that we definitely build into training. But also as we understand the model and as we evaluate it and as we test it, we always learn more. We always learn something new, I think, about what the model can do.
00:05:29 · Lenny Rachitsky
Ad[00:05:29] S1: This episode is brought to you by this season's presenting sponsor, WorkOS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay and hundreds of other winning companies have in common? They're all powered by WorkOS if you're building for enterprises. You have felt the pain of integrating single sign-on, SCIM, RBAC, and audit logs. WorkOS turns those deal blockers into drop-in APIs built for B2B SaaS. And now there's a new kind of user that needs access to your app, Agents. AI agents need to register, authenticate, and get permissions, just like a person. WorkOS handles it all. AgentAuth to give agents scope, revocable identities, agent registration with AuthMD, and Airlock to judge every agent action and allow, deny, or escalate to a human. I've noticed that every startup that I'm an investor in that expands upmarket ends up using WorkOS because they're the best. Go to WorkOS.com to make your app enterprise ready today. I want to talk a little bit about Google's place in the AI race. It feels like a lot of people feel that Google lags behind the top two labs right now, Anthropic and OpenAI. And Google has these moments of being the best and then it kind of feels like it slows down a bit. I'm curious what you can share about just Google's strategy in the AI race and what maybe people don't understand on the inside of how you all think about this stuff.
00:06:54 · Tulsee Doshi
[00:06:54] S0: First of all, hopefully by the time this conversation goes out, you will be using Gemini 4 Argonne, which is exciting. And so I really do hope we're having a conversation about a frontier AI model that folks are excited to use. And I think Gemini 4 speaks a little bit to our strategy, right? Because it really is our push to continue to build a frontier lab, a frontier set of models, a frontier set of capabilities on our path. I think if you look at Google's strategy, it really is full stack, right? So one of the things that I think is powerful about Google is all the way from hardware to end products that serve billions of users. I think the stat is like we have 14 products that are above a billion users. I should check that number, but I think that's the stat. And so if you think about that, we actually go all the way from the hardware to then the modeling to then this kind of end user application. And our goal ultimately is to not just build amazing frontier experiences, which is definitely a part of the goal, but it is also to build models across this Pareto curve of cost to quality that can serve different users at different points in their lifestyle, whether that be a search user who is coming for Information, right? Whether it is an engineer who is trying to build the next amazing thing, whether it is for a cyber defender who is trying to protect a critical piece of infrastructure. I think our goal is to make sure that we have models that serve that range of use cases and that we're actually building with feedback from these users.
00:08:30 · Lenny Rachitsky, Tulsee Doshi
[00:08:30] S1: So what I'm hearing is there's so many vectors where Google can win. There's the most efficient kind of Pareto Frontier. For token costs, there's the smartest model. There's also just making your products better. For example, AI mode, AI overviews. And so it's not that you need to be the best at the most advanced AI model that people can use, like Chatbot, basically. You can win in all these other areas. But it sounds like you're also trying to win and have the most state-of-the-art model [00:08:58] S0: I actually think we do want to win the frontier conversation, right? I don't think we should say that Google is not participating in the frontier. And I think we are building better and better models. And I think Gemini for Argonne definitely is at the frontier in the place that we are at. But I think one thing that is amazing is being able to build a breadth of experiences and build a diversity of experiences for users. And I think for Google, Our goal ultimately is to build amazing experiences that actually make people's lives better, right? And that is, I think, ultimately the goal, whatever part of the frontier you're building for. [00:09:42] S1: Google products definitely make my life better. It's just like amazing how much of your life is within a Google ecosystem, like not even the AI stuff, but just Google Calendar, Gmail, all these things, you're just like Google Docs. So it's happening. [00:09:55] S0: There you go.
00:09:55 · Tulsee Doshi, Lenny Rachitsky
[00:09:55] S0: There you go. [00:09:56] S1: There we go. I want to talk about one other broad AI question that I'm going to dig into more product specific stuff. A big topic of conversation these days is AI safety and this kind of fear that AI is going to harm humans. It's going to cause damage as it gets smarter and smarter. You have a really interesting lens on this because you led product for the responsible AI team at Google for six years. This is what you did right before this role? And so you've been thinking about this for a long time, a lot longer than most people. I'm curious just to get your take on this debate. Just, I don't know, are you optimistic we will figure this out? What do you think people maybe don't understand about this debate around AI safety?
00:10:36 · Tulsee Doshi
[00:10:36] S0: Our safety teams here are incredible. And it's been incredible to see the intentionality that they've been putting in all the way from, you know, Initial trainings of the model to how we deploy them. And I think safety is embedded in that process. Right. And we need to be. I think the part that is important, actually, as we talk about how do you deploy models safely is actually thinking through what are the right sort of mitigations and structures around these models. Right. So, for example, one of the things that's really important is thinking about monitoring. Right. And so for us actually deploying monitors that can monitor the model's chain of thought and actually identify whether the model is misaligned with its directed actions or whether it is aligned and then being able to intervene appropriately. That's actually the kind of mitigation that gives us more confidence in deploying these models, but also allows us to better understand where we might see misalignment and better actually improve the models over time. And so I think Things like monitoring give me more confidence in how we're putting our models out. I think the other thing that we don't talk about enough when we talk about this debate is also the role that AI can play in hardening our systems, right? And so one of the things that we're doing with Gemini 4 is actively partnering with a number of cyber defenders, including actually just even partnering with Wiz, that is now a Google company, to really think through How can we actually identify and patch vulnerabilities in systems across the globe? Right. We actually have this interesting case study. Wiz has this scan for good program where they're actually like actively working with systems across the world to identify, you know, potential issues. And they worked with a hospital system to identify a vulnerability and then patch it. And it's that kind of thing that I think if we can actually leverage these tools to do more of, we can actually make sure that our infrastructure and our systems are robust. Right. Which is also, I think, really critical as this whole space grows.