"Turpentine VC" | Venture Capital and Investing
00:14:58 · Brendan Foody
[00:14:58] S2: So we found that audio is certainly something that's really, really important and has way more signal than people's transcript. However, the visual evaluation is just starting to take we found some signals and visual evaluation. There's interesting things like the blurriness of someone's camera actually contains some signal. Wow. Yeah. And, and there's like little features like that. We're really just dipping our toes into the facial expression stuff. But I believe that as we get more data about this, that there's going to be a lot of signals that emerge. One thing we talked about a lot internally is that like, this fascinating problem of predicting human ability is extremely high dimensional, right? There are so many things that go into what make you good at a given task or job, everything from, you know, all the different tangentially relevant experiences to your motivation to the manager in that job, whether they're like incentivizing you and you want to do a good job. And because the problem is so high dimensional, similar to predicting the next token and a transformer model, it requires an immense amount of data to actually help you meaningfully extract the right features. And so one of the reasons that we're so excited about the business and all of the progress that we're making, is that it allows us to operate at a much larger scale with a lot more data, where we're actually able to make sense of these, you know, small features that we previously weren't able to. And so as the performance data expands, it makes all that possible.
00:16:37 · Usman Hanif, Brendan Foody
[00:16:37] S1: Yeah, absolutely. And just to probe a bit more here, just because I find this technology to be so exciting. You know, when when you're collecting this information, you know, now are you taking into account more, say, multimodal models, vision language models? Is is this where you guys are headed now if you're expanding beyond just the textual space? [00:16:56] S2: Completely. Yeah. So that's why the frontier models that can do good multimodal evaluation are extremely compelling to us. And we've been using vision models for a while and are starting to, you know, explore a lot of what's possible with audio models as well as more sophisticated features with vision models. There also are other kinds of custom models we run experiments with, like the facial expression stuff that we were previously discussing as an example of that and beyond beyond just language models. But there's there's all sorts of models that go into extracting good features. And then it's about, you know, mapping those features to the right performance metrics that we wanna optimize for.
00:16:56 · Brendan Foody, Usman Hanif
[00:16:56] S2: Completely. Yeah. So that's why the frontier models that can do good multimodal evaluation are extremely compelling to us. And we've been using vision models for a while and are starting to, you know, explore a lot of what's possible with audio models as well as more sophisticated features with vision models. There also are other kinds of custom models we run experiments with, like the facial expression stuff that we were previously discussing as an example of that and beyond beyond just language models. But there's there's all sorts of models that go into extracting good features. And then it's about, you know, mapping those features to the right performance metrics that we wanna optimize for. [00:17:38] S1: Yeah. And and I'm curious in the early days. I mean, you know, the sort of the... You know, it it makes complete sense that once you get a bit more data on one of the candidates, say they've done a couple jobs through the the platform, the reviews are pretty valuable. But when it comes to that first almost, you know, zero shot, one shot approach of sort of classifying this candidate, how did you guys sort of test this? Was it just like, pray that this works and and hopefully the the the company like it? Or how did you go about that?
00:17:38 · Usman Hanif, Brendan Foody
[00:17:38] S1: Yeah. And and I'm curious in the early days. I mean, you know, the sort of the... You know, it it makes complete sense that once you get a bit more data on one of the candidates, say they've done a couple jobs through the the platform, the reviews are pretty valuable. But when it comes to that first almost, you know, zero shot, one shot approach of sort of classifying this candidate, how did you guys sort of test this? Was it just like, pray that this works and and hopefully the the the company like it? Or how did you go about that? [00:18:10] S2: It started with a lot of doing things that don't scale and was, you know, of doing human processes and think how we can replace them, where, you know, my co founders and I have done thousands, not exaggerating ourselves. And we obviously would be able to reach the scale that the models are at now for us at hundreds of 1000s of interviews. But you gain an immense amount of appreciation for like all the different ways to optimize an interview experience, and the importance of automating all of those processes. And so it started with, you know, manually reviewing resumes, and then figuring out how we can automate at least the shortlisting, even if not the final review. And then I remember for a time, we automated conducting the interview, but not yet evaluating the interview because it would actually be way faster. We could just like watch interviews on two x speed and then, you know, and and stop an interview if someone wasn't good. And then and then pushing towards complete automation, which is which is where we're
00:19:09 · Erik Torenberg, Usman Hanif
[00:19:09] S0: at today. Hey. We'll continue our interview in a moment after a word from our sponsors. Hey, everyone. Eric here. In this environment, founders need to become profitable faster and do more with smaller teams, especially when it comes to engineering. That's why Sean Lenehan started Squad, a specialized global talent firm for top engineers that will seamlessly integrate with your org. Squad offers rigorously vetted top 1% talent that will actually work hard for you every day. Their engineers work in your time zone, follow your processes, and use your tools. Squad has front end engineers excelling in TypeScript, React, and Next. Js ready to onboard to your team today. For back end, Squad engineers are experts at Node. Js, Python, Java, and a range of other languages and frameworks. While it may cost more than the freelancer on Upwork billing you for forty hours but working only two, Squad offers premium quality at a fraction of the typical cost without the headache of assessing for skills and culture fit. Squad takes care of sourcing, legal compliance, and local HR for global talent. Increase your velocity without amping up burn. Visit chewsquad.com and mention turpentine to skip the wait list. [00:20:18] S1: I see. What would you say was the most challenging part when it came to building this technology? I mean, it could either be one of these specific features, whether it be the AI interviewer or the resume analyzer, or maybe it was just at a broader level, maybe data collection, like what was really hard to get right for this to work?
00:20:18 · Usman Hanif, Brendan Foody
[00:20:18] S1: I see. What would you say was the most challenging part when it came to building this technology? I mean, it could either be one of these specific features, whether it be the AI interviewer or the resume analyzer, or maybe it was just at a broader level, maybe data collection, like what was really hard to get right for this to work? [00:20:38] S2: Yeah, I think the largest challenge, which will continue to be the largest challenge for probably a long, long time in the business is in predicting who's going to perform well, for what reasons. It's just like such a challenging problem, because humans aren't very good at it. And there's so many problems within that of collecting the right data, data is like super subjective, and you want to be very thoughtful about, you know, minimizing bias and having peer review on all of that data and having experts that have deep expertise in the domains that the person is discussing, all the way to model architectures to, you know, different approaches and training them. And just optimizing that model to be as predictive as possible is, I think, the most interesting, as well as the most challenging problem in the business.