Unsupervised Learning with Jacob Effron
00:05:09 · S2, Brendan Foody
[00:05:09] S2: Maybe for our listeners too, on the Merkor side, you guys obviously have a bunch of uses of AI and screening candidates going through resumes. Can you talk through some of the different use cases that you have for AI and then what the stack looks like that you guys are building on today? [00:05:23] S1: Yeah, think a good heuristic is just thinking about all the things that humans would do manually, creating evals over those and seeing how we can automate them. So similar to how a human would review resume, conduct an interview, and then rank people or decide who should be hired. We automate all of those processes with LMs, and so we have evals for how accurately are we parsing the resume, how accurately are we, you know, scoring different parts of the resume, how accurately are we asking questions in an interview, evaluating that interview, and then passing that all into model context along with the references or every other kind of data that we have on a candidate to make the end prediction around how well they'll perform. [00:05:58] S2: Is it mostly off the shelf models, and you're curating the evaluation and context around them? [00:06:04] S1: Yeah, there's a lot of off the shelf models for more basic things, but particularly for the hardest problem of making the end evaluation of a candidate is where the post training comes in. Yeah. And learning from all of the data we get from our customers of who's doing well, for what reasons, how can we learn from those signals to make better predictions around who we should be hiring in the future.
00:06:04 · Brendan Foody, S2
[00:06:04] S1: Yeah, there's a lot of off the shelf models for more basic things, but particularly for the hardest problem of making the end evaluation of a candidate is where the post training comes in. Yeah. And learning from all of the data we get from our customers of who's doing well, for what reasons, how can we learn from those signals to make better predictions around who we should be hiring in the future. [00:06:25] S2: Have you learned anything about anything surprising about those signals of something that the AI found where you thought, you know, maybe this isn't how I would thought about it or humans would have thought about it. [00:06:35] S1: Yeah. There's there's all sorts of things. I think that one of the key benefits of AI is that it's able to just go way more in-depth about, like, everything about a candidate, and it's able to pick up on all the small details that humans sometimes miss or like, you know, the the vibe check sort of skips over because people already have their mind made up on a candidate. And so there's all sorts of like little resume signals if people have demonstrated extreme interest in a particular area where they're just doing it for fun, as you would anticipate all the way to like different signals of whether someone studied abroad in a country that is where they're doing the end job. They might communicate better and be more conducive in a work environment. So there's lots of those little things that come up and are very specific to projects and customers.
00:06:35 · Brendan Foody, S2
[00:06:35] S1: Yeah. There's there's all sorts of things. I think that one of the key benefits of AI is that it's able to just go way more in-depth about, like, everything about a candidate, and it's able to pick up on all the small details that humans sometimes miss or like, you know, the the vibe check sort of skips over because people already have their mind made up on a candidate. And so there's all sorts of like little resume signals if people have demonstrated extreme interest in a particular area where they're just doing it for fun, as you would anticipate all the way to like different signals of whether someone studied abroad in a country that is where they're doing the end job. They might communicate better and be more conducive in a work environment. So there's lots of those little things that come up and are very specific to projects and customers. [00:07:26] S2: Are there certain things that you see will always be done by people? You were talking about the multimodal stuff, but I guess how do you see AI and human interviewers working together versus a world where it just goes all AI assessment?
00:07:26 · S2, Brendan Foody, Jacob Effron
[00:07:26] S2: Are there certain things that you see will always be done by people? You were talking about the multimodal stuff, but I guess how do you see AI and human interviewers working together versus a world where it just goes all AI assessment? [00:07:41] S1: At a simplistic level, the hiring process involves assessing candidates and selling candidates. And the assessment part, I think, is going to soon get so good from LLMs that it'll be foolish to think we know better. Right? Uh-huh. Will people just take the recommendation because it'll have proof that it is performing so much better on the eval, on the end outcome that customers care about. Where humans, I think, will still continue to play a really large role in the selling process of this person that we're going be working with and spending time with. And I think about it as enabling human recruiters and hiring managers to spend all of their time on the candidates they want to hire rather than all these interviews of people that they don't end up wanting to hire. And so really, yeah, unlocking them to, you know, help people better understand the role, better understand the people that they're gonna be working with, and all the things that they should be excited about. [00:08:33] S2: Yeah. I love that. Will, will people start gaming the assessment? Like, is that something that you've seen at all? I guess the LMs are picking up on certain things if you put in [00:08:41] S0: this They all decided they studied abroad in the
00:08:41 · Jacob Effron, S2, Brendan Foody
[00:08:41] S0: this They all decided they studied abroad in the [00:08:43] S2: They all studied abroad in the, you know, in the place where they're recruiting for. That's [00:08:47] S1: Yeah. Yeah. It's why sometimes you have to be a little bit secretive about the signals. Right? But yeah, I mean, we we have so many things where we we deal this with this as every large hiring process does. And so I think the key is ensuring that assessments are relatively dynamic, either like the problem that they're working on is changed frequently or that you're asking them super in-depth questions about a particular part of their background. Because there's so much in the way of talent assessment that becomes possible when the models are able to do immense preparation for an interview. Right? Like when I'm like doing a first interview of an executive candidate, like maybe sometimes I'll have references on them, but most of time I look at their LinkedIn profile for a couple of minutes. I have like, some preliminary notes, but imagine if I could go like listen to a podcast that they were on. Right? Go read through like blog posts that they've written, all of the papers that they might have done during their PhD, and ask about those things. Right? You can get way more in-depth and nuanced in a way that's very hard to game. Obviously, have these models that are
00:08:47 · Brendan Foody, Jacob Effron
[00:08:47] S1: Yeah. Yeah. It's why sometimes you have to be a little bit secretive about the signals. Right? But yeah, I mean, we we have so many things where we we deal this with this as every large hiring process does. And so I think the key is ensuring that assessments are relatively dynamic, either like the problem that they're working on is changed frequently or that you're asking them super in-depth questions about a particular part of their background. Because there's so much in the way of talent assessment that becomes possible when the models are able to do immense preparation for an interview. Right? Like when I'm like doing a first interview of an executive candidate, like maybe sometimes I'll have references on them, but most of time I look at their LinkedIn profile for a couple of minutes. I have like, some preliminary notes, but imagine if I could go like listen to a podcast that they were on. Right? Go read through like blog posts that they've written, all of the papers that they might have done during their PhD, and ask about those things. Right? You can get way more in-depth and nuanced in a way that's very hard to game. Obviously, have these models that are [00:09:45] S0: pretty good at predicting how well these candidates will do. 27 doesn't matter that that's explainable, or like these models just have like, you know, in a black box, like, yeah, this person's gonna be good, and this person's not.