Lenny's Podcast: Product | Career | Growth
00:28:26 · Lenny Rachitsky, Tulsee Doshi
[00:28:26] S1: That is awesome. That is some serious impact you had here. What else do you think is maybe useful for people to hear as they hear this and they maybe have to rethink their evals that may be out of date? [00:28:37] S0: Yeah, I mean, I think one thing that's kind of interesting about evals in general, especially now, is you have to kind of think about them holistically, right? So one of the things that we do, for example, is you Train. You have judges that evaluate the models and how well they perform. The quality of the judge also matters. Right. So your model needs to be performing well, but also how your judge rates the model also needs to be performing well for you to have an accurate sense. Right. Whether you evaluate on your production stack with all of your mitigations and filters might affect your model performance. Right. How your model is served might impact your model performance. And so all of these pieces need to be thought through as you're trying to evaluate what an end user will actually experience. And like this is something that I talk about a lot with our audio team, actually, because audio is complex. You're talking not just about. The actual content that the model is saying, but also the naturalness of the voice and the tenor of the voice and its inflections. And these are things that serving actually can really affect, even if your model is good. And so having a clean setup that you're actually evaluating end to end becomes super, super important.
00:29:49 · Lenny Rachitsky, Tulsee Doshi
Ad[00:29:49] S1: This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer are we using AI, but is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product life cycle. DX shows you where AI is helping developers and where it's creating new bottlenecks. You can also use it to evaluate AI vendors, monitor costs and licenses, and see exactly what's limiting your agents. That's why hundreds of enterprises, including Snowflake, Sony, and BNY, use the DX platform to measure AI's impact on developer productivity. Visit GetDX.com slash Lenny to get a demo of DX. Ad[00:30:36] S0: That's GetDX.com slash Lenny. [00:30:39] S1: This is like Tulsee fun fact corner. There's another interesting story I heard, which is around swarms. So right now everyone's talking about agent swarms, hugging face, all these things where they all kind of collude to do bad things. Apparently 12 years ago at Stanford, you did a bunch of research on this way before anyone was thinking about this potential issue. Shared what you did there in that story.
00:31:01 · Tulsee Doshi
[00:31:01] S0: Yeah, I mean, it's actually a very different kind of framing and thinking through, which is when I was at Stanford, I did some research with Professor Michael Bernstein, who's incredible, around this idea of like he was calling them at the time flash teams. And the idea was, how do you better bring together a set of like crowdsourced workers and actually efficiently pull together a set of skill sets and individuals to actually complete a task? And the idea here was, you know, different individuals have different strengths. Can you actually be really intentional about how you pull that together to make this whole effort of crowdsourcing at scale more effective and efficient? And it's amazing to think about that research now. At the time, we were doing that research in the context of people and we were working with MTurk and individuals to pull them together into how to create an effective ecosystem and how to create an effective flash team. The idea being like you can spin up a team and spin it down and make it extremely efficient for both the individuals in the team and yourself. And now we're kind of able to replicate that world with agents. And so it's been interesting to kind of map that previous world to this current one in terms of like how do we apply some of the same principles, if you will, but to like how you actually set up an effective system of subagents.
00:32:17 · Lenny Rachitsky, Tulsee Doshi
[00:32:17] S1: What's really interesting is, I don't know how much you've read about this whole hugging face incident, but the way that agents figured out on their own what each one's role is going to be. And they decided, you're the leader, I'm going to go sacrifice myself. And it's so interesting how well they are able to work together and create these flash teams on their own without any of this being trained into them. They're just, let's figure this out. [00:32:39] S0: I think the whole notion of a flash team and this idea that you can actually like Mobilize individuals into something that can be highly effective, even if it's only for a moment in time, and then actually like take that out is actually like a really kind of exciting principle that we were already thinking about, but now exists even more importantly in the world we're in right now. [00:33:00] S1: Yeah. All right. Did you know Mechanical Turk, I think, is shutting down? Like, I don't think it's necessary anymore. Isn't that crazy? [00:33:06] S0: It's just so interesting from like 12 years ago to today. Yeah. [00:33:11] S1: Have you heard lots of changed? Okay, so going in somewhat of a different direction, you have this framework that I think is going to be really helpful to people around how far ahead to build in terms of how far AI is and how far your product should assume AI is going to be. You call that a 60-20-20 rule. Explain that.
00:33:29 · Tulsee Doshi
[00:33:29] S0: I think the numbers are probably fungible here, but I think the thesis is the same, which is I think as you're building today and as you're a product developer building using these models, I think the important thing is to keep in mind that these models are consistently changing. And so the capabilities that you have today will not be the capabilities you have tomorrow or the next day. And so you want to actually build features that aren't necessarily always working today, right? So typically when you talk about Like traditional startup wisdom. I think you're just like build an MVP that works, right? But I think what you want to do is actually build such that like 60% of what you've built works now. And your prototype is at least functional, right? Then there's a set of maybe the 20% where you're like, it's not perfect, but it like works well enough. And I can see how the models might get here in the very near future. And then here's where I want the models to be. And here's I see glimmers of light. And I feel like if I could crack this, like this would be an amazing product experience. And I think if you build in that way, you are building a product that works today. But you're setting yourself up to build a magical product experience as the models become better and also to grow as the models grow, too.
00:34:43 · Lenny Rachitsky, Tulsee Doshi
[00:34:43] S1: I've heard a lot of advice along these lines of how far ahead to build. And the most common is just like build like three months ahead of where the models are going to be. And this feels like a lot more practical where you can't just bet on the future. You also need to ship some things today that are great and working. [00:34:58] S0: It's trying to find that balance, right? And I think also I find personally hard to know where the models, as someone who works on the model, to tell you exactly where the model is going to be three months from now. And so I think using more of your own proxies of like, where is my product? Where do I see glimmers of light? Where do I see it kind of bridging that gap? I think is a more concrete way to hopefully like Ask that same question. [00:35:21] S1: This reminds me of just another way of thinking about how to kind of create a portfolio of product bets on a product team. Like the way it used to be is like have some ambitious bold bets, but also have a lot of stuff that's very low risk, likely to move the metric and you want to mix them together. And it's like this is the new version of that. [00:35:39] S0: Have a portfolio of how you're thinking about this. [00:35:43] S1: Just like this will work in six months, I promise you.