Lenny's Podcast: Product | Career | Growth
00:26:16 · Lenny Rachitsky
[00:26:16] S1: This is great advice. So what I'm hearing is stick to things you actually care about and watch the models evolve on how good they are at this thing. Could be writing a PRD for you, could be writing a strategy doc, could be generating graphics. Okay. I have a couple of eval stories that one of your colleagues shared that are fun facts about you. Oh, oh boy. So this is Eli Collins, VP product. He's a colleague of yours at Google DeepMind. Okay. So one is you worked on something called the Fitzpatrick scale and the monk skin tone. And this was a big shift in how people in how models worked around and evals were a core part of this. Share this story. What this work was.
00:26:55 · Tulsee Doshi
[00:26:55] S0: Yeah. So I like like you noted earlier, I spent six years in responsible AI and a big part of that time we were looking at skin tone and skin tone representation in our models, in our products. And one of the things we realized is that early on there was a scale that individuals were using the Fitzpatrick scale, which was six skin tones. And that was how we were evaluating the effectiveness of product. And Realistically, six skin tones cannot capture the diversity of skin tones in the world. Not only that, it was actually built for an entirely different purpose. It was built for like health care purposes. And so it actually skewed extremely light skin tone, given the problems that it was trying to solve. And so it actually not only was only six skin tones, it was also not even the six like correct skin tones if you're trying to represent like a global population. And so we worked with a professor at Harvard, Professor Monk. He had been from like a more anthropology, sociology perspective, looking at characterizing a set of skin tones. And so he had a scale of 10 skin tones that he had built through a number of user interviews and interviews. Research that he had been doing over years. And we pulled that set of skin tones into our evaluations and actually redid a number of the ways that we evaluated our products and our models to reflect that 10 skin tone scale and actually use that to improve things like our efforts in the pixel camera, like some of our image generation to be more intentional about how skin tones are represented.
00:28:26 · Lenny Rachitsky, Tulsee Doshi
[00:28:26] S1: That is awesome. That is some serious impact you had here. What else do you think is maybe useful for people to hear as they hear this and they maybe have to rethink their evals that may be out of date? [00:28:37] S0: Yeah, I mean, I think one thing that's kind of interesting about evals in general, especially now, is you have to kind of think about them holistically, right? So one of the things that we do, for example, is you Train. You have judges that evaluate the models and how well they perform. The quality of the judge also matters. Right. So your model needs to be performing well, but also how your judge rates the model also needs to be performing well for you to have an accurate sense. Right. Whether you evaluate on your production stack with all of your mitigations and filters might affect your model performance. Right. How your model is served might impact your model performance. And so all of these pieces need to be thought through as you're trying to evaluate what an end user will actually experience. And like this is something that I talk about a lot with our audio team, actually, because audio is complex. You're talking not just about. The actual content that the model is saying, but also the naturalness of the voice and the tenor of the voice and its inflections. And these are things that serving actually can really affect, even if your model is good. And so having a clean setup that you're actually evaluating end to end becomes super, super important.
00:29:49 · Lenny Rachitsky, Tulsee Doshi
Ad[00:29:49] S1: This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer are we using AI, but is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product life cycle. DX shows you where AI is helping developers and where it's creating new bottlenecks. You can also use it to evaluate AI vendors, monitor costs and licenses, and see exactly what's limiting your agents. That's why hundreds of enterprises, including Snowflake, Sony, and BNY, use the DX platform to measure AI's impact on developer productivity. Visit GetDX.com slash Lenny to get a demo of DX. Ad[00:30:36] S0: That's GetDX.com slash Lenny. [00:30:39] S1: This is like Tulsee fun fact corner. There's another interesting story I heard, which is around swarms. So right now everyone's talking about agent swarms, hugging face, all these things where they all kind of collude to do bad things. Apparently 12 years ago at Stanford, you did a bunch of research on this way before anyone was thinking about this potential issue. Shared what you did there in that story.
00:31:01 · Tulsee Doshi
[00:31:01] S0: Yeah, I mean, it's actually a very different kind of framing and thinking through, which is when I was at Stanford, I did some research with Professor Michael Bernstein, who's incredible, around this idea of like he was calling them at the time flash teams. And the idea was, how do you better bring together a set of like crowdsourced workers and actually efficiently pull together a set of skill sets and individuals to actually complete a task? And the idea here was, you know, different individuals have different strengths. Can you actually be really intentional about how you pull that together to make this whole effort of crowdsourcing at scale more effective and efficient? And it's amazing to think about that research now. At the time, we were doing that research in the context of people and we were working with MTurk and individuals to pull them together into how to create an effective ecosystem and how to create an effective flash team. The idea being like you can spin up a team and spin it down and make it extremely efficient for both the individuals in the team and yourself. And now we're kind of able to replicate that world with agents. And so it's been interesting to kind of map that previous world to this current one in terms of like how do we apply some of the same principles, if you will, but to like how you actually set up an effective system of subagents.
00:32:17 · Lenny Rachitsky, Tulsee Doshi
[00:32:17] S1: What's really interesting is, I don't know how much you've read about this whole hugging face incident, but the way that agents figured out on their own what each one's role is going to be. And they decided, you're the leader, I'm going to go sacrifice myself. And it's so interesting how well they are able to work together and create these flash teams on their own without any of this being trained into them. They're just, let's figure this out. [00:32:39] S0: I think the whole notion of a flash team and this idea that you can actually like Mobilize individuals into something that can be highly effective, even if it's only for a moment in time, and then actually like take that out is actually like a really kind of exciting principle that we were already thinking about, but now exists even more importantly in the world we're in right now. [00:33:00] S1: Yeah. All right. Did you know Mechanical Turk, I think, is shutting down? Like, I don't think it's necessary anymore. Isn't that crazy? [00:33:06] S0: It's just so interesting from like 12 years ago to today. Yeah. [00:33:11] S1: Have you heard lots of changed? Okay, so going in somewhat of a different direction, you have this framework that I think is going to be really helpful to people around how far ahead to build in terms of how far AI is and how far your product should assume AI is going to be. You call that a 60-20-20 rule. Explain that.