Lenny's Podcast: Product | Career | Growth
00:23:20 · Tulsee Doshi
[00:23:20] S0: Yeah. And to be honest, we're still figuring that out internally with our PMs as well. And I would say like we see some PMs who are stronger at this than others. And so we're kind of working to upskill our whole team here. I think for me, a big thing is start with even just five prompts, right? Like start, you don't have to start large. I think we often think about like, oh, I need to build some sort of massive, large scale evaluation or benchmark. I think often like even just a mini evaluation where you get to like 20 prompts that can really stress test a set of capabilities where you can see signal between your model and another model or model A and model B. I think that already is so powerful. And that's an unlock because it doesn't require you to think as critically about all of the pieces. Right. Like, start simple. Start with even just yourself, like you as the Raider, you looking at a prompt, you trying to decide if the results in the response is good, especially when you're talking about these like long, multi-step, multi-turn trajectories. Just even being able to identify how you're thinking about good is really hard, actually. And so even getting that set of good examples will already give you a sense of, oh, this is what I'm trying to get to. It'll give yourself clarity. And then you figure out how to scale that up, I think.
00:24:37 · Lenny Rachitsky
[00:24:37] S1: Awesome. Okay. That is really good advice. There's this debate, I don't know, six months ago on Twitter around evals versus vibes. And it's interesting. Your advice is both actually a really important structured evals of rubrics and also getting good at getting the vibes. So on that vibes piece, there's something I struggle with, to be honest, when I get a new model, I never know exactly what to do with it to understand what it's gotten better at. So I'm just like, hello, let's try some things I've done in the past. This is cool. This is great. Maybe better. What are some tips you have for folks that want to get better at this very specific skill of understanding where a model has gone better than the vibes and how they've shifted maybe from a previous model?
00:25:14 · Tulsee Doshi
[00:25:14] S0: There isn't honestly a one right answer to this because you ideally want to stress test the model on things that you are excited about, right? That's where you're going to also see magic in the model, I think. And so I actually see that different people have very, very different notions of this, right? So you'll see, for example, on Twitter, a lot of people trying the kind of like, 3D animations and more visual type aesthetics, because that's one way to kind of visually gauge a sense of how models are operating. You'll see others throw models at writing, whether that be creative writing or writing essays or synthesizing information. You'll se others throw it at just really hard financial analysis. And so I think part of it is like finding the right task where you feel like today certain models are already showing signs of life and showing light, but they're not perfect under your definition of perfect. And that gives you some malleability to see different models handle the problem differently. And that already gives you, I think, some insight into how the models perform.
00:26:16 · Lenny Rachitsky
[00:26:16] S1: This is great advice. So what I'm hearing is stick to things you actually care about and watch the models evolve on how good they are at this thing. Could be writing a PRD for you, could be writing a strategy doc, could be generating graphics. Okay. I have a couple of eval stories that one of your colleagues shared that are fun facts about you. Oh, oh boy. So this is Eli Collins, VP product. He's a colleague of yours at Google DeepMind. Okay. So one is you worked on something called the Fitzpatrick scale and the monk skin tone. And this was a big shift in how people in how models worked around and evals were a core part of this. Share this story. What this work was.
00:26:55 · Tulsee Doshi
[00:26:55] S0: Yeah. So I like like you noted earlier, I spent six years in responsible AI and a big part of that time we were looking at skin tone and skin tone representation in our models, in our products. And one of the things we realized is that early on there was a scale that individuals were using the Fitzpatrick scale, which was six skin tones. And that was how we were evaluating the effectiveness of product. And Realistically, six skin tones cannot capture the diversity of skin tones in the world. Not only that, it was actually built for an entirely different purpose. It was built for like health care purposes. And so it actually skewed extremely light skin tone, given the problems that it was trying to solve. And so it actually not only was only six skin tones, it was also not even the six like correct skin tones if you're trying to represent like a global population. And so we worked with a professor at Harvard, Professor Monk. He had been from like a more anthropology, sociology perspective, looking at characterizing a set of skin tones. And so he had a scale of 10 skin tones that he had built through a number of user interviews and interviews. Research that he had been doing over years. And we pulled that set of skin tones into our evaluations and actually redid a number of the ways that we evaluated our products and our models to reflect that 10 skin tone scale and actually use that to improve things like our efforts in the pixel camera, like some of our image generation to be more intentional about how skin tones are represented.
00:28:26 · Lenny Rachitsky, Tulsee Doshi
[00:28:26] S1: That is awesome. That is some serious impact you had here. What else do you think is maybe useful for people to hear as they hear this and they maybe have to rethink their evals that may be out of date? [00:28:37] S0: Yeah, I mean, I think one thing that's kind of interesting about evals in general, especially now, is you have to kind of think about them holistically, right? So one of the things that we do, for example, is you Train. You have judges that evaluate the models and how well they perform. The quality of the judge also matters. Right. So your model needs to be performing well, but also how your judge rates the model also needs to be performing well for you to have an accurate sense. Right. Whether you evaluate on your production stack with all of your mitigations and filters might affect your model performance. Right. How your model is served might impact your model performance. And so all of these pieces need to be thought through as you're trying to evaluate what an end user will actually experience. And like this is something that I talk about a lot with our audio team, actually, because audio is complex. You're talking not just about. The actual content that the model is saying, but also the naturalness of the voice and the tenor of the voice and its inflections. And these are things that serving actually can really affect, even if your model is good. And so having a clean setup that you're actually evaluating end to end becomes super, super important.