Unsupervised Learning with Jacob Effron
00:09:45 · Jacob Effron, Brendan Foody
[00:09:45] S0: pretty good at predicting how well these candidates will do. 27 doesn't matter that that's explainable, or like these models just have like, you know, in a black box, like, yeah, this person's gonna be good, and this person's not. [00:09:54] S1: Yeah. I think it does matter that it's explainable for sort of two reasons. First is for customers to like understand and trust those those claims, right? Like building trust and all all the reasoning chains. And then the second is obviously making sure that the models are selecting people for the right reasons. The reasons that they they should be considering. And so it's beneficial, but I I think like the end state of the economy is probably just that like, it'll be, you know, some sort of API or interaction where people want work done or they they need some level of human involvement, and just a confidence interval on how that well that person will perform on the job, and there's far less of the, intermediation that humans play in the process. [00:10:37] S0: Yeah. It's like an interim trust milestone on the way on the way there. Exactly. No. It makes a ton of sense. And then obviously, today in in kind of the first or or one of the areas we have a lot of fit on the data labeling side, there's kind of these clean feedback loops of like, how bad you could even score, like how accurately and you probably have multiple people looking at the same pieces of data. Talk about some of the challenges maybe in translating this to maybe more vague domains of human work.
00:10:37 · Jacob Effron, Brendan Foody, S2
[00:10:37] S0: Yeah. It's like an interim trust milestone on the way on the way there. Exactly. No. It makes a ton of sense. And then obviously, today in in kind of the first or or one of the areas we have a lot of fit on the data labeling side, there's kind of these clean feedback loops of like, how bad you could even score, like how accurately and you probably have multiple people looking at the same pieces of data. Talk about some of the challenges maybe in translating this to maybe more vague domains of human work. [00:10:59] S1: Totally. I I mean Like venture capital. Yeah. [00:11:02] S0: Wait wait fifteen years, and then you get your feedback loop. [00:11:06] S2: Yeah.
00:11:06 · S2, Brendan Foody, Jacob Effron
[00:11:06] S2: Yeah. [00:11:06] S1: One way I think about it is if you have a 100 people that are all doing the same job, it's very easy to stack rank them. Versus if you have a 100 people doing a very different job, right, like founders, right? Like they're all working on something that's nuanced in one way or another. It's very difficult to like pattern match. Like what is the thing that they said or the thing that we learned that actually translated to the outcome because there's just like so many confounding variables in in the equation. And so I think that it's gonna be like relatively easy for the like larger pipelines of roles like if you're hiring 20 account executives, right? Stack ranking all of them, learning from those signals. And then the models are starting to be able to learn from these, you know, much more complicated things where everyone's working on something else. Like we're doing a ranking of a bunch of the the Teal fellows, and and that's that's like a fun case, but it it definitely is more challenging, and relies more underlying reasoning capabilities of the models. [00:12:03] S0: Maybe just talk through what are some of the challenges that emerge in doing that?
00:12:03 · Jacob Effron, Brendan Foody
[00:12:03] S0: Maybe just talk through what are some of the challenges that emerge in doing that? [00:12:06] S1: Yeah, well, it's basically that oftentimes there's a lot of things that aren't in model context, and so models struggle to learn from that, and people forget to add it to model context. So maybe it's like, I heard my friend said this good thing about using this company's product, Or these things that might not be making their way in, making sure that all references are added, all of the interpersonal stuff that humans might pick up on. So we found that actually, oftentimes just making sure the requisite data is in model context is majority of the problem. [00:12:41] S0: Yeah, I guess in the future maybe we're just recording every conversation with our smart glasses. Yeah, yeah. Easy enough to feed into the problem. [00:12:47] S1: Bridgewater had it right all along. Yeah. Exactly. Exactly. Is that where we're headed? [00:12:50] S0: Is it just like gonna be, you know, Bridgewater at scale?
00:12:50 · Jacob Effron, Brendan Foody, S2
[00:12:50] S0: Is it just like gonna be, you know, Bridgewater at scale? [00:12:53] S1: We we'll see. I mean, think of course a lot of companies will be adverse to to that, and I think there will be regulatory reasons and illegal reasons people don't wanna do that. But I I also think there's just gonna be better processes for how models help get this information in context. Right? Like maybe it's AI doing an exit interview of the manager and the people on the teams to help better understand what was going on because all the people have so many details in their head around this that we we just need to get into the models for them to be able to make these superhuman predictions. [00:13:23] S2: Yeah. There are certainly more and more both founders and, you know, all kinds of people that are bringing AI to their meetings, and so I think a lot of those meetings and interactions will be recorded for AI to learn from. [00:13:35] S1: Totally. [00:13:35] S2: I think that'll be interesting. [00:13:36] S0: We need you to take our transcripts and stack rank us against [00:13:39] S1: each Only [00:13:41] S0: if I come up on that. [00:13:43] S2: What do you think of the data labeling landscape today? How do you see the different players differentiating from each other? It seemed like scale was really in a position to run away, but then now there's been a bunch of new players in the landscape. How think do about that world?
00:13:57 · Brendan Foody
[00:13:57] S1: Yeah. I I think like the key thing that most people don't understand in the like data annotation and evaluation landscape is just the shift in the market and how dramatically different it is from what it used to be two years ago. Because when ChatGPT came out, it was... The models like weren't that good. It was easy to trip them up. They were making mistakes left and right. Like, even like a high school student, or or like college undergrad could do a lot of completions or evals to help improve the models in this crowd sourcing fashion where they run these huge pipelines to get hundreds of thousands of pieces of SFT or RLHF. SFT being inputs outputs, RLHF being choosing between a bunch of different, like, preference options like you would see in ChatGPT. But as the models got really good, that crowdsourcing model started to break because you needed really high quality people that would work directly with researchers to help them understand why is this model doing well? Why is it not doing well? How can we, you know, create this really complicated data that helps to trip up the model and actually reflects like the real world things that we want to automate. And so our platform of finding exceptional people that you would want to work with was perfectly positioned for that, and that we can hire these really high quality people super quickly. And that caused us to take off and and have all of the traction working with the big labs. And I think that trend will continue and that like the companies that are stuck in these like super high volume crowdsourcing pipelines, are probably gonna... Are certainly gonna see a lot of churn. And it's going to be the new players that understand the direction the market is headed, and lean into really high quality talent underpinning it that are going to continue taking a lot of market share.