Lenny's Podcast: Product | Career | Growth
00:21:44 · Tulsee Doshi
[00:21:44] S0: Yeah, I mean, I think more and more so we are looking for PMs who, you For lack of a better way of putting it, understand the vibes of our models or other models. I want PMs who have an opinion, who are playing with these models every day, who have their own use cases for how they judge whether a model is really good, who play with and stress test these capabilities themselves, who are capable of diving in and looking at data and getting scrappy and rolling up their sleeves. And so I think it's both PMs who are trying to build that intuition and As well as who are technical enough in the sense of being willing to kind of like, like I said, roll up sleeves, go in, really understand the data, build an eval, right? Actually sit down and pull those pieces together. I think that's really important. The other thing I'll say is I think PMs who are agile and adaptive are like also really important. I think the space is changing every two weeks. A roadmap you wrote two months ago. You're going to be looking at it again and rewriting it. Right. And so I think PMs who are constantly keeping an eye out on the environment, on the market landscape and actually consistently reinterrogating, like, what are we doing and why are we doing it? As opposed to assuming that there was a fixed kind of plan of action.
00:23:05 · Lenny Rachitsky
[00:23:05] S1: This is awesome. OK, so vibes, evals, adaptive. I want to poke at these a little bit. So with evals, say someone doesn't really know how to write an eval, how to approach writing an eval, their first eval. What are some tips to start getting better at the skill writing evals?
00:23:20 · Tulsee Doshi
[00:23:20] S0: Yeah. And to be honest, we're still figuring that out internally with our PMs as well. And I would say like we see some PMs who are stronger at this than others. And so we're kind of working to upskill our whole team here. I think for me, a big thing is start with even just five prompts, right? Like start, you don't have to start large. I think we often think about like, oh, I need to build some sort of massive, large scale evaluation or benchmark. I think often like even just a mini evaluation where you get to like 20 prompts that can really stress test a set of capabilities where you can see signal between your model and another model or model A and model B. I think that already is so powerful. And that's an unlock because it doesn't require you to think as critically about all of the pieces. Right. Like, start simple. Start with even just yourself, like you as the Raider, you looking at a prompt, you trying to decide if the results in the response is good, especially when you're talking about these like long, multi-step, multi-turn trajectories. Just even being able to identify how you're thinking about good is really hard, actually. And so even getting that set of good examples will already give you a sense of, oh, this is what I'm trying to get to. It'll give yourself clarity. And then you figure out how to scale that up, I think.
00:24:37 · Lenny Rachitsky
[00:24:37] S1: Awesome. Okay. That is really good advice. There's this debate, I don't know, six months ago on Twitter around evals versus vibes. And it's interesting. Your advice is both actually a really important structured evals of rubrics and also getting good at getting the vibes. So on that vibes piece, there's something I struggle with, to be honest, when I get a new model, I never know exactly what to do with it to understand what it's gotten better at. So I'm just like, hello, let's try some things I've done in the past. This is cool. This is great. Maybe better. What are some tips you have for folks that want to get better at this very specific skill of understanding where a model has gone better than the vibes and how they've shifted maybe from a previous model?
00:25:14 · Tulsee Doshi
[00:25:14] S0: There isn't honestly a one right answer to this because you ideally want to stress test the model on things that you are excited about, right? That's where you're going to also see magic in the model, I think. And so I actually see that different people have very, very different notions of this, right? So you'll see, for example, on Twitter, a lot of people trying the kind of like, 3D animations and more visual type aesthetics, because that's one way to kind of visually gauge a sense of how models are operating. You'll see others throw models at writing, whether that be creative writing or writing essays or synthesizing information. You'll se others throw it at just really hard financial analysis. And so I think part of it is like finding the right task where you feel like today certain models are already showing signs of life and showing light, but they're not perfect under your definition of perfect. And that gives you some malleability to see different models handle the problem differently. And that already gives you, I think, some insight into how the models perform.
00:26:16 · Lenny Rachitsky
[00:26:16] S1: This is great advice. So what I'm hearing is stick to things you actually care about and watch the models evolve on how good they are at this thing. Could be writing a PRD for you, could be writing a strategy doc, could be generating graphics. Okay. I have a couple of eval stories that one of your colleagues shared that are fun facts about you. Oh, oh boy. So this is Eli Collins, VP product. He's a colleague of yours at Google DeepMind. Okay. So one is you worked on something called the Fitzpatrick scale and the monk skin tone. And this was a big shift in how people in how models worked around and evals were a core part of this. Share this story. What this work was.