Lenny's Podcast: Product | Career | Growth
00:23:05 · Lenny Rachitsky
[00:23:05] S1: This is awesome. OK, so vibes, evals, adaptive. I want to poke at these a little bit. So with evals, say someone doesn't really know how to write an eval, how to approach writing an eval, their first eval. What are some tips to start getting better at the skill writing evals?
00:23:20 · Tulsee Doshi
[00:23:20] S0: Yeah. And to be honest, we're still figuring that out internally with our PMs as well. And I would say like we see some PMs who are stronger at this than others. And so we're kind of working to upskill our whole team here. I think for me, a big thing is start with even just five prompts, right? Like start, you don't have to start large. I think we often think about like, oh, I need to build some sort of massive, large scale evaluation or benchmark. I think often like even just a mini evaluation where you get to like 20 prompts that can really stress test a set of capabilities where you can see signal between your model and another model or model A and model B. I think that already is so powerful. And that's an unlock because it doesn't require you to think as critically about all of the pieces. Right. Like, start simple. Start with even just yourself, like you as the Raider, you looking at a prompt, you trying to decide if the results in the response is good, especially when you're talking about these like long, multi-step, multi-turn trajectories. Just even being able to identify how you're thinking about good is really hard, actually. And so even getting that set of good examples will already give you a sense of, oh, this is what I'm trying to get to. It'll give yourself clarity. And then you figure out how to scale that up, I think.
00:24:37 · Lenny Rachitsky
[00:24:37] S1: Awesome. Okay. That is really good advice. There's this debate, I don't know, six months ago on Twitter around evals versus vibes. And it's interesting. Your advice is both actually a really important structured evals of rubrics and also getting good at getting the vibes. So on that vibes piece, there's something I struggle with, to be honest, when I get a new model, I never know exactly what to do with it to understand what it's gotten better at. So I'm just like, hello, let's try some things I've done in the past. This is cool. This is great. Maybe better. What are some tips you have for folks that want to get better at this very specific skill of understanding where a model has gone better than the vibes and how they've shifted maybe from a previous model?
00:25:14 · Tulsee Doshi
[00:25:14] S0: There isn't honestly a one right answer to this because you ideally want to stress test the model on things that you are excited about, right? That's where you're going to also see magic in the model, I think. And so I actually see that different people have very, very different notions of this, right? So you'll see, for example, on Twitter, a lot of people trying the kind of like, 3D animations and more visual type aesthetics, because that's one way to kind of visually gauge a sense of how models are operating. You'll see others throw models at writing, whether that be creative writing or writing essays or synthesizing information. You'll se others throw it at just really hard financial analysis. And so I think part of it is like finding the right task where you feel like today certain models are already showing signs of life and showing light, but they're not perfect under your definition of perfect. And that gives you some malleability to see different models handle the problem differently. And that already gives you, I think, some insight into how the models perform.
00:26:16 · Lenny Rachitsky
[00:26:16] S1: This is great advice. So what I'm hearing is stick to things you actually care about and watch the models evolve on how good they are at this thing. Could be writing a PRD for you, could be writing a strategy doc, could be generating graphics. Okay. I have a couple of eval stories that one of your colleagues shared that are fun facts about you. Oh, oh boy. So this is Eli Collins, VP product. He's a colleague of yours at Google DeepMind. Okay. So one is you worked on something called the Fitzpatrick scale and the monk skin tone. And this was a big shift in how people in how models worked around and evals were a core part of this. Share this story. What this work was.
00:26:55 · Tulsee Doshi
[00:26:55] S0: Yeah. So I like like you noted earlier, I spent six years in responsible AI and a big part of that time we were looking at skin tone and skin tone representation in our models, in our products. And one of the things we realized is that early on there was a scale that individuals were using the Fitzpatrick scale, which was six skin tones. And that was how we were evaluating the effectiveness of product. And Realistically, six skin tones cannot capture the diversity of skin tones in the world. Not only that, it was actually built for an entirely different purpose. It was built for like health care purposes. And so it actually skewed extremely light skin tone, given the problems that it was trying to solve. And so it actually not only was only six skin tones, it was also not even the six like correct skin tones if you're trying to represent like a global population. And so we worked with a professor at Harvard, Professor Monk. He had been from like a more anthropology, sociology perspective, looking at characterizing a set of skin tones. And so he had a scale of 10 skin tones that he had built through a number of user interviews and interviews. Research that he had been doing over years. And we pulled that set of skin tones into our evaluations and actually redid a number of the ways that we evaluated our products and our models to reflect that 10 skin tone scale and actually use that to improve things like our efforts in the pixel camera, like some of our image generation to be more intentional about how skin tones are represented.