Lenny's Podcast: Product | Career | Growth
00:32:17 · Lenny Rachitsky, Tulsee Doshi
[00:32:17] S1: What's really interesting is, I don't know how much you've read about this whole hugging face incident, but the way that agents figured out on their own what each one's role is going to be. And they decided, you're the leader, I'm going to go sacrifice myself. And it's so interesting how well they are able to work together and create these flash teams on their own without any of this being trained into them. They're just, let's figure this out. [00:32:39] S0: I think the whole notion of a flash team and this idea that you can actually like Mobilize individuals into something that can be highly effective, even if it's only for a moment in time, and then actually like take that out is actually like a really kind of exciting principle that we were already thinking about, but now exists even more importantly in the world we're in right now. [00:33:00] S1: Yeah. All right. Did you know Mechanical Turk, I think, is shutting down? Like, I don't think it's necessary anymore. Isn't that crazy? [00:33:06] S0: It's just so interesting from like 12 years ago to today. Yeah. [00:33:11] S1: Have you heard lots of changed? Okay, so going in somewhat of a different direction, you have this framework that I think is going to be really helpful to people around how far ahead to build in terms of how far AI is and how far your product should assume AI is going to be. You call that a 60-20-20 rule. Explain that.
00:33:29 · Tulsee Doshi
[00:33:29] S0: I think the numbers are probably fungible here, but I think the thesis is the same, which is I think as you're building today and as you're a product developer building using these models, I think the important thing is to keep in mind that these models are consistently changing. And so the capabilities that you have today will not be the capabilities you have tomorrow or the next day. And so you want to actually build features that aren't necessarily always working today, right? So typically when you talk about Like traditional startup wisdom. I think you're just like build an MVP that works, right? But I think what you want to do is actually build such that like 60% of what you've built works now. And your prototype is at least functional, right? Then there's a set of maybe the 20% where you're like, it's not perfect, but it like works well enough. And I can see how the models might get here in the very near future. And then here's where I want the models to be. And here's I see glimmers of light. And I feel like if I could crack this, like this would be an amazing product experience. And I think if you build in that way, you are building a product that works today. But you're setting yourself up to build a magical product experience as the models become better and also to grow as the models grow, too.
00:34:43 · Lenny Rachitsky, Tulsee Doshi
[00:34:43] S1: I've heard a lot of advice along these lines of how far ahead to build. And the most common is just like build like three months ahead of where the models are going to be. And this feels like a lot more practical where you can't just bet on the future. You also need to ship some things today that are great and working. [00:34:58] S0: It's trying to find that balance, right? And I think also I find personally hard to know where the models, as someone who works on the model, to tell you exactly where the model is going to be three months from now. And so I think using more of your own proxies of like, where is my product? Where do I see glimmers of light? Where do I see it kind of bridging that gap? I think is a more concrete way to hopefully like Ask that same question. [00:35:21] S1: This reminds me of just another way of thinking about how to kind of create a portfolio of product bets on a product team. Like the way it used to be is like have some ambitious bold bets, but also have a lot of stuff that's very low risk, likely to move the metric and you want to mix them together. And it's like this is the new version of that. [00:35:39] S0: Have a portfolio of how you're thinking about this. [00:35:43] S1: Just like this will work in six months, I promise you.
00:35:43 · Lenny Rachitsky, Tulsee Doshi
[00:35:43] S1: Just like this will work in six months, I promise you. [00:35:47] S0: The other thing that's interesting, too, is like I think thinking about how you make sure that your product is agile to the model getting better. I think one thing we've also seen with customers sometimes is like they've built a very rigid system around, for example, where the model was. And now the model has gotten so much better. Right. So how do you go from where the model was to like being adaptable to kind of consistent continuing to iterate it with, you know. [00:36:11] S1: Yeah, the common kind of advice I hear is that just the model is going to eat your harness for breakfast, that all these things you built, all these optimizations are just going to, they will go away because the model gets smarter and they kind of have to take all these things away. There's some advice that I think Boris Cherny was sharing of just like get rid of all your skills once in a while and just see what happens because often it's now as good as all these things you've built into it. [00:36:32] S0: Yeah, or just like consistently retest, right? Because you might be even hampering what you're getting out of the model, out of a new model, because you're trying to force fit it into your existing structure.
00:36:32 · Tulsee Doshi, Lenny Rachitsky
[00:36:32] S0: Yeah, or just like consistently retest, right? Because you might be even hampering what you're getting out of the model, out of a new model, because you're trying to force fit it into your existing structure. [00:36:43] S1: Let me ask you a completely unrelated question, just because I think this might be helpful for folks to understand. There's this term harness that people often hear, this idea of there's a model, there's a harness that sits on top of the model. What's the simplest way to understand what a harness is? [00:36:56] S0: Yeah, I mean, I think of a harness as like the wrapper around the model essentially, right? So think of it as like, what are the system instructions, the tools, the skills, the context that the model has access to and how can it leverage those things, right? And so what you want to basically do is give the model an environment in which to operate. And that's really what your harness builds out. [00:37:17] S1: So essentially it's a system prompt plus tools that have access to do things. And then whatever other context, other data [00:37:25] S0: information that it has access to and skills, right? Like all of these things kind of come together to build out your harness.
00:37:25 · Tulsee Doshi, Lenny Rachitsky
[00:37:25] S0: information that it has access to and skills, right? Like all of these things kind of come together to build out your harness. [00:37:31] S1: Awesome. Okay. That is very helpful. So one other thing I wanted to get your thoughts on, there's a lot of change, obviously. A lot of people are trying to figure out how to thrive in this new world. How are you upskilling and helping your team get better and kind of understand what they need to be good at in this new world? What are just ways you're helping your team develop and kind of Survive and thrive. Survive and thrive. Yeah.