Lenny's Podcast: Product | Career | Growth
00:29:49 · Lenny Rachitsky, Tulsee Doshi
Ad[00:29:49] S1: This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer are we using AI, but is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product life cycle. DX shows you where AI is helping developers and where it's creating new bottlenecks. You can also use it to evaluate AI vendors, monitor costs and licenses, and see exactly what's limiting your agents. That's why hundreds of enterprises, including Snowflake, Sony, and BNY, use the DX platform to measure AI's impact on developer productivity. Visit GetDX.com slash Lenny to get a demo of DX. Ad[00:30:36] S0: That's GetDX.com slash Lenny. [00:30:39] S1: This is like Tulsee fun fact corner. There's another interesting story I heard, which is around swarms. So right now everyone's talking about agent swarms, hugging face, all these things where they all kind of collude to do bad things. Apparently 12 years ago at Stanford, you did a bunch of research on this way before anyone was thinking about this potential issue. Shared what you did there in that story.
00:31:01 · Tulsee Doshi
[00:31:01] S0: Yeah, I mean, it's actually a very different kind of framing and thinking through, which is when I was at Stanford, I did some research with Professor Michael Bernstein, who's incredible, around this idea of like he was calling them at the time flash teams. And the idea was, how do you better bring together a set of like crowdsourced workers and actually efficiently pull together a set of skill sets and individuals to actually complete a task? And the idea here was, you know, different individuals have different strengths. Can you actually be really intentional about how you pull that together to make this whole effort of crowdsourcing at scale more effective and efficient? And it's amazing to think about that research now. At the time, we were doing that research in the context of people and we were working with MTurk and individuals to pull them together into how to create an effective ecosystem and how to create an effective flash team. The idea being like you can spin up a team and spin it down and make it extremely efficient for both the individuals in the team and yourself. And now we're kind of able to replicate that world with agents. And so it's been interesting to kind of map that previous world to this current one in terms of like how do we apply some of the same principles, if you will, but to like how you actually set up an effective system of subagents.
00:32:17 · Lenny Rachitsky, Tulsee Doshi
[00:32:17] S1: What's really interesting is, I don't know how much you've read about this whole hugging face incident, but the way that agents figured out on their own what each one's role is going to be. And they decided, you're the leader, I'm going to go sacrifice myself. And it's so interesting how well they are able to work together and create these flash teams on their own without any of this being trained into them. They're just, let's figure this out. [00:32:39] S0: I think the whole notion of a flash team and this idea that you can actually like Mobilize individuals into something that can be highly effective, even if it's only for a moment in time, and then actually like take that out is actually like a really kind of exciting principle that we were already thinking about, but now exists even more importantly in the world we're in right now. [00:33:00] S1: Yeah. All right. Did you know Mechanical Turk, I think, is shutting down? Like, I don't think it's necessary anymore. Isn't that crazy? [00:33:06] S0: It's just so interesting from like 12 years ago to today. Yeah. [00:33:11] S1: Have you heard lots of changed? Okay, so going in somewhat of a different direction, you have this framework that I think is going to be really helpful to people around how far ahead to build in terms of how far AI is and how far your product should assume AI is going to be. You call that a 60-20-20 rule. Explain that.
00:33:29 · Tulsee Doshi
[00:33:29] S0: I think the numbers are probably fungible here, but I think the thesis is the same, which is I think as you're building today and as you're a product developer building using these models, I think the important thing is to keep in mind that these models are consistently changing. And so the capabilities that you have today will not be the capabilities you have tomorrow or the next day. And so you want to actually build features that aren't necessarily always working today, right? So typically when you talk about Like traditional startup wisdom. I think you're just like build an MVP that works, right? But I think what you want to do is actually build such that like 60% of what you've built works now. And your prototype is at least functional, right? Then there's a set of maybe the 20% where you're like, it's not perfect, but it like works well enough. And I can see how the models might get here in the very near future. And then here's where I want the models to be. And here's I see glimmers of light. And I feel like if I could crack this, like this would be an amazing product experience. And I think if you build in that way, you are building a product that works today. But you're setting yourself up to build a magical product experience as the models become better and also to grow as the models grow, too.
00:34:43 · Lenny Rachitsky, Tulsee Doshi
[00:34:43] S1: I've heard a lot of advice along these lines of how far ahead to build. And the most common is just like build like three months ahead of where the models are going to be. And this feels like a lot more practical where you can't just bet on the future. You also need to ship some things today that are great and working. [00:34:58] S0: It's trying to find that balance, right? And I think also I find personally hard to know where the models, as someone who works on the model, to tell you exactly where the model is going to be three months from now. And so I think using more of your own proxies of like, where is my product? Where do I see glimmers of light? Where do I see it kind of bridging that gap? I think is a more concrete way to hopefully like Ask that same question. [00:35:21] S1: This reminds me of just another way of thinking about how to kind of create a portfolio of product bets on a product team. Like the way it used to be is like have some ambitious bold bets, but also have a lot of stuff that's very low risk, likely to move the metric and you want to mix them together. And it's like this is the new version of that. [00:35:39] S0: Have a portfolio of how you're thinking about this. [00:35:43] S1: Just like this will work in six months, I promise you.
00:35:43 · Lenny Rachitsky, Tulsee Doshi
[00:35:43] S1: Just like this will work in six months, I promise you. [00:35:47] S0: The other thing that's interesting, too, is like I think thinking about how you make sure that your product is agile to the model getting better. I think one thing we've also seen with customers sometimes is like they've built a very rigid system around, for example, where the model was. And now the model has gotten so much better. Right. So how do you go from where the model was to like being adaptable to kind of consistent continuing to iterate it with, you know. [00:36:11] S1: Yeah, the common kind of advice I hear is that just the model is going to eat your harness for breakfast, that all these things you built, all these optimizations are just going to, they will go away because the model gets smarter and they kind of have to take all these things away. There's some advice that I think Boris Cherny was sharing of just like get rid of all your skills once in a while and just see what happens because often it's now as good as all these things you've built into it. [00:36:32] S0: Yeah, or just like consistently retest, right? Because you might be even hampering what you're getting out of the model, out of a new model, because you're trying to force fit it into your existing structure.