Lenny's Podcast: Product | Career | Growth
00:34:43 · Lenny Rachitsky, Tulsee Doshi
[00:34:43] S1: I've heard a lot of advice along these lines of how far ahead to build. And the most common is just like build like three months ahead of where the models are going to be. And this feels like a lot more practical where you can't just bet on the future. You also need to ship some things today that are great and working. [00:34:58] S0: It's trying to find that balance, right? And I think also I find personally hard to know where the models, as someone who works on the model, to tell you exactly where the model is going to be three months from now. And so I think using more of your own proxies of like, where is my product? Where do I see glimmers of light? Where do I see it kind of bridging that gap? I think is a more concrete way to hopefully like Ask that same question. [00:35:21] S1: This reminds me of just another way of thinking about how to kind of create a portfolio of product bets on a product team. Like the way it used to be is like have some ambitious bold bets, but also have a lot of stuff that's very low risk, likely to move the metric and you want to mix them together. And it's like this is the new version of that. [00:35:39] S0: Have a portfolio of how you're thinking about this. [00:35:43] S1: Just like this will work in six months, I promise you.
00:35:43 · Lenny Rachitsky, Tulsee Doshi
[00:35:43] S1: Just like this will work in six months, I promise you. [00:35:47] S0: The other thing that's interesting, too, is like I think thinking about how you make sure that your product is agile to the model getting better. I think one thing we've also seen with customers sometimes is like they've built a very rigid system around, for example, where the model was. And now the model has gotten so much better. Right. So how do you go from where the model was to like being adaptable to kind of consistent continuing to iterate it with, you know. [00:36:11] S1: Yeah, the common kind of advice I hear is that just the model is going to eat your harness for breakfast, that all these things you built, all these optimizations are just going to, they will go away because the model gets smarter and they kind of have to take all these things away. There's some advice that I think Boris Cherny was sharing of just like get rid of all your skills once in a while and just see what happens because often it's now as good as all these things you've built into it. [00:36:32] S0: Yeah, or just like consistently retest, right? Because you might be even hampering what you're getting out of the model, out of a new model, because you're trying to force fit it into your existing structure.
00:36:32 · Tulsee Doshi, Lenny Rachitsky
[00:36:32] S0: Yeah, or just like consistently retest, right? Because you might be even hampering what you're getting out of the model, out of a new model, because you're trying to force fit it into your existing structure. [00:36:43] S1: Let me ask you a completely unrelated question, just because I think this might be helpful for folks to understand. There's this term harness that people often hear, this idea of there's a model, there's a harness that sits on top of the model. What's the simplest way to understand what a harness is? [00:36:56] S0: Yeah, I mean, I think of a harness as like the wrapper around the model essentially, right? So think of it as like, what are the system instructions, the tools, the skills, the context that the model has access to and how can it leverage those things, right? And so what you want to basically do is give the model an environment in which to operate. And that's really what your harness builds out. [00:37:17] S1: So essentially it's a system prompt plus tools that have access to do things. And then whatever other context, other data [00:37:25] S0: information that it has access to and skills, right? Like all of these things kind of come together to build out your harness.
00:37:25 · Tulsee Doshi, Lenny Rachitsky
[00:37:25] S0: information that it has access to and skills, right? Like all of these things kind of come together to build out your harness. [00:37:31] S1: Awesome. Okay. That is very helpful. So one other thing I wanted to get your thoughts on, there's a lot of change, obviously. A lot of people are trying to figure out how to thrive in this new world. How are you upskilling and helping your team get better and kind of understand what they need to be good at in this new world? What are just ways you're helping your team develop and kind of Survive and thrive. Survive and thrive. Yeah.
00:37:58 · Tulsee Doshi, Lenny Rachitsky
[00:37:58] S0: I mean, honestly, I feel like my team is helping my team in a lot of this, actually. I think when you look at the PMs on my team, you see some who have really leaned in and built a lot of skill in leveraging our internal tools and they're individuals who play around with the models a lot. And they actually are, I think, amazing assets to the rest of my team because they build skills that others can use. And one thing we've been starting to do more of is like, how do you share a skill that you've built that is giving you some value in your day to day job with others, right? So that we can actually build out an entire repository that helps the PM team. PMs who have gotten stronger and more confident in eyeballing data and really looking at it and understanding longer traces and trajectories. How can we run workshops with other PMs so that we kind of all collectively build more of that confidence? And so for me, a lot of it is about confidence building. How do we build everyone's confidence in the tools that we have and their ability to leverage them effectively? And I think that's something that we're trying to build more and more across the team. [00:39:00] S1: So going one level deeper there, what is it you do? So you said workshops, there's like PMs that are good at a thing, teach a workshop to the rest of the PMs. Is there anything else just like very practically that you do to help people get better at these things?
00:39:00 · Lenny Rachitsky, Tulsee Doshi
[00:39:00] S1: So going one level deeper there, what is it you do? So you said workshops, there's like PMs that are good at a thing, teach a workshop to the rest of the PMs. Is there anything else just like very practically that you do to help people get better at these things? [00:39:12] S0: The biggest thing I would say actually that is needed to get better at these things, and I say this knowing that I am starting to do this, but we are not quite there yet, is actually giving people the time and space. I think when you're just sprinting constantly, it is hard to create the space to actually take a step back and make sure you know the tools well. But once you do, then it accelerates you 10x on the other side. And so a lot of it is about pushing the team to say, hey, could we leverage JetSki, our internal tool, to do this? Right. And hey, I have a skill that I use to do this. Do you want to try? Right. And actually building that kind of set of skills, if you will. And actually, I think sometimes the most practical thing to do is like actually build a skill to do something. And when I say skill, I don't mean like the human skill. I mean, like the technical skill to do it. And then that skill gets shared to others. That builds more confidence. That then leads to someone else building another skill or improving that skill. And now you have this like iteration flywheel, which I think is good. [00:40:15] S1: Does Google still do the 20% thing? Because that feels like a really good place to spend time on this stuff. Is that still a thing or less so these days?