"Turpentine VC" | Venture Capital and Investing
00:08:11 · Usman Hanif, Brendan Foody
[00:08:11] S1: Yeah. And so I guess touching on the why now a bit, would you point that to technology, which is sort of the advancements in AI recently allow you to do some of these operations a lot more efficiently and cost effectively? [00:08:26] S2: Hugely. Yeah. I mean, language models are getting extremely intelligent, especially when fine tuned on the right data sets. And that was just previously impossible. Like the idea of an agent conducting an interview of a human two years ago was completely impossible. Right? And now agents are able to not only conduct human level interviews, but conduct interviews that are way better. It's as if a human had prepared for twenty four hours before their interview, right, reviewing all of their GitHub repositories, thinking about all of the details of projects that they have on their resume and asking this in-depth, thoughtful interview about someone's background, giving each person the chance to be considered. And that was, you know, wasn't even close to possible before we had super powerful foundations.
00:08:26 · Brendan Foody, Usman Hanif
[00:08:26] S2: Hugely. Yeah. I mean, language models are getting extremely intelligent, especially when fine tuned on the right data sets. And that was just previously impossible. Like the idea of an agent conducting an interview of a human two years ago was completely impossible. Right? And now agents are able to not only conduct human level interviews, but conduct interviews that are way better. It's as if a human had prepared for twenty four hours before their interview, right, reviewing all of their GitHub repositories, thinking about all of the details of projects that they have on their resume and asking this in-depth, thoughtful interview about someone's background, giving each person the chance to be considered. And that was, you know, wasn't even close to possible before we had super powerful foundations. [00:09:16] S1: Yeah. It sounds like you've sort of offloaded a lot of the vetting process from humans to technology, specifically AI. We have a pretty technical audience. I think it'd be really fun to dive into some of that. On the side of the candidate, you know, they're often, as you mentioned, talking or always talking to the AI interviewer when they first sign up. There's also a resume analyzer tool I believe you guys use, plus a system that you mentioned earlier that can even predict candidate performance. I'd love to just start diving into all of that. You know? How did... You know, what is the technology behind that? How did you start building this? How do you even tackle this problem of teaching AI how to vet a candidate?
00:09:58 · Brendan Foody
[00:09:58] S2: Yeah. Yeah. I mean, it's it's a really challenging problem because humans struggle with it significantly. Right? So when you're creating evaluations for model performance, the the baseline in question that that you're trying to optimize for is very challenging. I would say, like, the first thing that's really interesting about the product is the AI interviewer, where being able to, you know, scalably do this volume of traffic requires a lot of pre processing to, you know, achieve the results that we want to as well as agentic behavior during the interview to be thoughtful about what the next questions are going to be. And then the other thing that's really fascinating are all of the models around predicting which candidates will do well, when we're surfacing candidates. The rough framework that we use for this is that there's two dimensions to how good a fit a candidate is for a role, there's the relevancy of that individual, and then there's the excellence of the individual, how competent they are. But these dimensions are like deeply intertwined. And so we do a lot of work with not only leveraging language models, fine tuning language models on huge amounts of proprietary data, but also on training, like learn to rank models from the ground up, and doing all sorts of fine tuned embeddings to surface the best candidates for customers.
00:11:19 · Usman Hanif
[00:11:19] S1: Yeah. In terms of, like, quantifying or even trying to qualify excellence, how do you sort of put that? I mean, you're mentioning how you're fine tuning models. What data are you looking at? How are you training these models? Anything that you can speak on there would be great.
00:11:34 · Brendan Foody
[00:11:34] S2: Yeah, there's two kinds of evaluations or KPIs that we go on. The first are KPIs around human preference, where we have internal members of the team that will create evaluations for like which candidates we think are best for roles for what reasons. And then we'll map to allowing models to predict that in the same way that humans can, considering, you know, all sorts leveraging language models and various other models to predict based on all of the attributes of an individual ranging from features of their GitHub contributions to their facial expression during an interview, who is likely passionate and likely perform well in that given role. The other kind of, like, evaluation KPI that we go on, and the more important one, in particular, as we scale, is around customer satisfaction, where we collect the reviews of all of the candidates and the experiences that customers are having on them, and then train models that can predict for a given candidate, what is the likely experience that they're going to have. So imagine like a transformer model that could intake all of the performance reviews of candidates for given roles and then can predict what is the most likely next performance review that this candidate is going to have, based on all of the data about their background, the role that they're being considered for and all the data that we find tuned up.
00:12:59 · Usman Hanif
[00:12:59] S1: I say, wow. I I mean, I think it's really cool. You mentioned you guys are even taking into account facial expression during these AI interviews. I suppose a human does the same thing. Right? And so, you know, touching on that a little bit more since it sounds pretty interesting, is this a matter of sort of collecting all this data, whether it be facial expression plus what they said and then feeding it to an LLM and letting it reason? Or how are you guys approaching this?