Unsupervised Learning with Jacob Effron
00:15:39 · S2, Brendan Foody
[00:15:39] S2: Do you think there'll always be demand for, I guess, humans in the data labeling process? There's obviously more and more that can be done with these models, or big model gets really good at one task and then can train smaller model. How do you see that evolving over time? [00:15:54] S1: Yeah. The way I think about it is that so long as there are things in the economy that humans can do that models can't do, we will need to create evals or our own environments so that models can learn how to do those things. So I think there's certain domains where that's just going to get solved sooner than others, right? Like within math or even many parts of code, like you don't need that much data. It's super verifiable. The models will solve those problems. But then there's other domains that are like much much more open ended. What makes a good founder when we're assessing them? Right? Or or, you know, most... Honestly, like a large chunk of knowledge work domains, maybe a majority of them are these like open ended problems that are really difficult to verify and understand what good looks like. And you just need to get all of that understanding that the models don't have into the models. And that's why I expect like orders of magnitude increase in the human data and evaluations market over time.
00:16:49 · Jacob Effron, Brendan Foody
[00:16:49] S0: If I understand correctly, you guys... You know, clearly, think one of the initial arbitrages you... And and what inspired the company is you have these great coders that are all around the world, and like, you know, they're not getting access to some of these jobs, and obviously, that ended up being really important for coding Obviously, you've expanded into other areas as well, like what if... Coding, again, it's a perfect RL use case, it's probably also really perfect for evaluation. What have you had to change or improve as you've gone into some of these fuzzier domains and recruiting people in those areas? [00:17:16] S1: Yeah. I think that leaning on a lot of the heuristics of what a human would do manually is probably a good way to do it. So for example, if you want to automate being a consultant, how do you assess consultants that can help to do that, giving them a case study? Maybe that's specific to their background, maybe it's like Well, [00:17:33] S0: maybe a silly question, but like, you guys are all probably great coders, and so I imagine you know how to evaluate coders. Like, you're starting to get a doctor on the platform, like Yeah. How do you even know how to... What the heuristics are for humans?
00:17:33 · Jacob Effron, Brendan Foody
[00:17:33] S0: maybe a silly question, but like, you guys are all probably great coders, and so I imagine you know how to evaluate coders. Like, you're starting to get a doctor on the platform, like Yeah. How do you even know how to... What the heuristics are for humans? [00:17:43] S1: Well, I think the point you're getting at is really interesting, which is that as you start to get into domains beyond the machine learning team's capabilities, they need to have these experts. We need to have doctors that are helping us create our doctor assessments and our evals for what makes a good doctor, as well as a bunch of other domains. And similarly, it's what the researchers need to do with all of their technology, right? When we were all going on LIMSYS, it was super easy to look at the high school level physics and say what problem was right or which one was slightly better. But when it's like PhD level chemistry and the researcher doesn't have a PhD in chemistry, it's really hard to understand what's going on to interpret these evals to figure out how we can improve them. And so I think that that's the other big shift to your question earlier around evaluations is is that both for assessing our talent as well as the the way the researchers assess models, it's just gonna be this much more collaborative process and working with people to help trip up the model and improve capabilities. I've I've heard you
00:17:43 · Brendan Foody, Jacob Effron
[00:17:43] S1: Well, I think the point you're getting at is really interesting, which is that as you start to get into domains beyond the machine learning team's capabilities, they need to have these experts. We need to have doctors that are helping us create our doctor assessments and our evals for what makes a good doctor, as well as a bunch of other domains. And similarly, it's what the researchers need to do with all of their technology, right? When we were all going on LIMSYS, it was super easy to look at the high school level physics and say what problem was right or which one was slightly better. But when it's like PhD level chemistry and the researcher doesn't have a PhD in chemistry, it's really hard to understand what's going on to interpret these evals to figure out how we can improve them. And so I think that that's the other big shift to your question earlier around evaluations is is that both for assessing our talent as well as the the way the researchers assess models, it's just gonna be this much more collaborative process and working with people to help trip up the model and improve capabilities. I've I've heard you [00:18:40] S0: talk before about how this kind of like short term data label contract work is like, it's kind of the perfect initial market for what you've done, and there's a massive amount of demand, and it's kind of this wedge to eventually doing just end to end labor markets. I'd love to just hear you riff a little bit on what's the sequencing of the company look like from here toward that vision?
00:18:56 · Brendan Foody
[00:18:56] S1: Yeah. Well, I wrote our secret master plan that goes over this a little bit, but the way I think about it is we have, like, the reason that marketplaces are generally hard to build is that they're very network effect intensive. And so the thing that makes them defensible also makes them hard to build. And so it's important right now that we're very focused on, like, drilling this wedge of huge amounts of demand that we have to expand the network effects, grow the marketplace and focus on that right now. But then we're also starting to see a lot of demand for hiring high volumes of contractors from our existing customers of big tech companies where they might need hundreds of data scientists or software engineers or whatever the role is for a particular domain outside of human data, which is really the exact same kind of request. It's just a little bit more of a legacy market where you'd be going up against like the Accenture's or delights of the world historically. So leaning into that as like the second main focus, and then expanding to all sorts of full time hiring. But but one of the key things is that like the lifetime of the business, we've been doing all of these. Like even the first year of the business had nothing to do with human data. It was just like hiring contractors for our friends and for ourselves, many of which became full time employees. And so it's it's much more continuous, and there's a lot of things that unify them and that we know that all companies want more candidates. They wanna be able to hire them more quickly and they want confidence that they'll perform well. And so if we just measure those things and improve them over time, that'll position us for every stage of the business.
00:20:22 · Jacob Effron
[00:20:22] S0: Yeah. Was there a moment that it was like obvious to you to lean into the human data side that it was just like so abundantly clear, this is where to...