"Turpentine VC" | Venture Capital and Investing
00:08:26 · Brendan Foody, Usman Hanif
[00:08:26] S2: Hugely. Yeah. I mean, language models are getting extremely intelligent, especially when fine tuned on the right data sets. And that was just previously impossible. Like the idea of an agent conducting an interview of a human two years ago was completely impossible. Right? And now agents are able to not only conduct human level interviews, but conduct interviews that are way better. It's as if a human had prepared for twenty four hours before their interview, right, reviewing all of their GitHub repositories, thinking about all of the details of projects that they have on their resume and asking this in-depth, thoughtful interview about someone's background, giving each person the chance to be considered. And that was, you know, wasn't even close to possible before we had super powerful foundations. [00:09:16] S1: Yeah. It sounds like you've sort of offloaded a lot of the vetting process from humans to technology, specifically AI. We have a pretty technical audience. I think it'd be really fun to dive into some of that. On the side of the candidate, you know, they're often, as you mentioned, talking or always talking to the AI interviewer when they first sign up. There's also a resume analyzer tool I believe you guys use, plus a system that you mentioned earlier that can even predict candidate performance. I'd love to just start diving into all of that. You know? How did... You know, what is the technology behind that? How did you start building this? How do you even tackle this problem of teaching AI how to vet a candidate?
00:09:58 · Brendan Foody
[00:09:58] S2: Yeah. Yeah. I mean, it's it's a really challenging problem because humans struggle with it significantly. Right? So when you're creating evaluations for model performance, the the baseline in question that that you're trying to optimize for is very challenging. I would say, like, the first thing that's really interesting about the product is the AI interviewer, where being able to, you know, scalably do this volume of traffic requires a lot of pre processing to, you know, achieve the results that we want to as well as agentic behavior during the interview to be thoughtful about what the next questions are going to be. And then the other thing that's really fascinating are all of the models around predicting which candidates will do well, when we're surfacing candidates. The rough framework that we use for this is that there's two dimensions to how good a fit a candidate is for a role, there's the relevancy of that individual, and then there's the excellence of the individual, how competent they are. But these dimensions are like deeply intertwined. And so we do a lot of work with not only leveraging language models, fine tuning language models on huge amounts of proprietary data, but also on training, like learn to rank models from the ground up, and doing all sorts of fine tuned embeddings to surface the best candidates for customers.
00:11:19 · Usman Hanif
[00:11:19] S1: Yeah. In terms of, like, quantifying or even trying to qualify excellence, how do you sort of put that? I mean, you're mentioning how you're fine tuning models. What data are you looking at? How are you training these models? Anything that you can speak on there would be great.
00:11:34 · Brendan Foody
[00:11:34] S2: Yeah, there's two kinds of evaluations or KPIs that we go on. The first are KPIs around human preference, where we have internal members of the team that will create evaluations for like which candidates we think are best for roles for what reasons. And then we'll map to allowing models to predict that in the same way that humans can, considering, you know, all sorts leveraging language models and various other models to predict based on all of the attributes of an individual ranging from features of their GitHub contributions to their facial expression during an interview, who is likely passionate and likely perform well in that given role. The other kind of, like, evaluation KPI that we go on, and the more important one, in particular, as we scale, is around customer satisfaction, where we collect the reviews of all of the candidates and the experiences that customers are having on them, and then train models that can predict for a given candidate, what is the likely experience that they're going to have. So imagine like a transformer model that could intake all of the performance reviews of candidates for given roles and then can predict what is the most likely next performance review that this candidate is going to have, based on all of the data about their background, the role that they're being considered for and all the data that we find tuned up.
00:12:59 · Usman Hanif
[00:12:59] S1: I say, wow. I I mean, I think it's really cool. You mentioned you guys are even taking into account facial expression during these AI interviews. I suppose a human does the same thing. Right? And so, you know, touching on that a little bit more since it sounds pretty interesting, is this a matter of sort of collecting all this data, whether it be facial expression plus what they said and then feeding it to an LLM and letting it reason? Or how are you guys approaching this?
00:13:24 · Brendan Foody, Usman Hanif
[00:13:24] S2: Yeah. So we started out with just textual data, like just the transcript of the interview. And then we expanded to also do visual stuff. But I will say that we're just starting to dip our toes into a lot of the more complicated visual stuff just because models are only now beginning to have really precise, you know, ability to extract features from these interviews. But I find it to be such like an interesting technical problem, because it's something that humans struggle immensely with. And models are actually superhuman. Like MIT released a paper that they were able to predict the heart rate of an individual and an interview with a reasonably high degree of competence based on the recording of that interview. And I believe that there's all of these subtle nuances and features of the heart rate of the individual, the facial expressions, have the tone, all of these, you know, little nuances that actually go a long way in predicting that person's attitude, their passion, their competence, and all these different characteristics and how they'll perform for a role. And so our focus now is in digging in beyond just what they're saying to all the other characteristics of that individual and how all of these subtle things that are often overlooked can can be priced in reasonably to predict their performance in a given task or job. [00:14:47] S1: And and have you seen any promise with that, like being able to take their facial expression or, you know, their voice inflection, whatever it may be? Is that actually a pretty reasonable predictor?