Unsupervised Learning with Jacob Effron
00:00:00 · Jacob Effron, Brendan Foody
[00:00:00] S0: Brandon Futi is the cofounder and CEO of Merkor, a company building the infrastructure for AI native labor markets. Merkor's platform is already being used to label data, screen talent, predict performance, and evaluate both human and AI candidates. It's a really interesting company at the intersection of recruiting, evals, and core to improving foundation models. Brandon's team recently raised a $100,000,000, and they're working with some of the most sophisticated companies in AI today. Our conversation today hit a lot of interesting things, including what role humans will play in labor in the future. We talked about the types of data labeling that really matter to improve models going forward. Brennan reflected on Merkur's rapid ascent and some of the key decisions he made, and we also hit on where AI does and doesn't work in the hiring process today. All in all, a really interesting conversation. I think you'll really enjoy it. Before we get to the episode, I just have one plug. So many of you have rated the show on Spotify or Apple, and thank you so much for that. If you're enjoying this, please consider leaving a rating on the show on either platform. Ratings help us grow, which help us to continue bringing on the best guests and keep the bar really high for the conversations. Now, here's Brendan Futi. Well, thanks so much for for coming on the podcast. Really appreciate it. [00:01:05] S1: Yeah. Thank you so much for having me on. I'm a big fan, so excited to chat.
00:01:05 · Brendan Foody, Jacob Effron
[00:01:05] S1: Yeah. Thank you so much for having me on. I'm a big fan, so excited to chat. [00:01:09] S0: Yeah. Excited to to have you here. I figured we'd start at like the the highest level place, which is for our listeners, I'd love if you just contextualize like, where are we today? What's the state of AI evaluating talent? Like what works? What doesn't? What's going on? [00:01:23] S1: I'm amazed at how good it is. Like I think that everything that a human is able to evaluate over text, the models are close to superhuman at. Whether it be the transcripts of someone's interview, the assessments that they're filling out in a written way, or even the signals on their resume. And it's a fascinating dichotomy because so little of that has actually been distributed in the economy. Right? And so there's just this huge greenfield associated with doing that, and it's one of the things we're really excited about working on and building out. [00:01:54] S0: Yeah. Were things that didn't work pre reasoning models that... Maybe talk about the last six months. I guess these models have gotten better. What's finally started to work for you guys?
00:01:54 · Jacob Effron, Brendan Foody
[00:01:54] S0: Yeah. Were things that didn't work pre reasoning models that... Maybe talk about the last six months. I guess these models have gotten better. What's finally started to work for you guys? [00:02:03] S1: Yeah. I remember back in end of March of twenty twenty or 2023 when GBD4 came out, and we were like build our first prototype of AI interviewer, and nothing worked. It was like the model would hallucinate every two or three questions and all of that. And so, it's just been riding this incredible tailwind over time. And I think the reasoning models are just obviously the knowledge and the models improved a lot in sort of the first year. And then the reasoning models have made them much better at particularly handling a lot of context, figuring out what matters, what to focus on, etcetera. It's been really cool. Still, there's multimodal things that the models aren't as good at just because it historically hasn't been as much of a focus of the labs and it's a lot harder to do RL with, but we're excited about that being added soon. Yeah. What are [00:02:52] S0: the milestones that met? You're like, I can't wait till the model can do x or y or...
00:02:52 · Jacob Effron, Brendan Foody
[00:02:52] S0: the milestones that met? You're like, I can't wait till the model can do x or y or... [00:02:56] S1: Yeah. There's a handful of things. There are certain things that humans are very good at, like this vibe check of whether I would enjoy working with this this person, whether this person is passionate and, like, they seem really genuine about what they're saying, that the model... It's really hard. Right? It's hard for the best humans, let alone models. And so really excited about that and building evals out for a good chunk of it. But when whenever I read through like the reasoning chains of the models, I'm like in trying to decipher things that are eval, I'm always thinking like, wow, the the model seems a lot more reasonable than like our... Than than whatever research on our team is creating the eval, right? And so it's it's really incredible how fast they've improved. And I think everyone obviously is seeing everything working in in code, but but we're just in the early innings of of a lot of other domains that are taking off in an incredible way. [00:03:49] S0: And obviously, seems like a big part of what you're doing is basically coming up with evals for humans, [00:03:53] S1: and how [00:03:53] S0: good they'll be at jobs. Obviously, we have all these people creating AI employees now, it's like, agents are gonna do this, or you'll have an AI agent doing this set of tasks that an employee would do. Do you guys play into this at all?
00:03:53 · Jacob Effron, Brendan Foody
[00:03:53] S0: good they'll be at jobs. Obviously, we have all these people creating AI employees now, it's like, agents are gonna do this, or you'll have an AI agent doing this set of tasks that an employee would do. Do you guys play into this at all? [00:04:04] S1: Absolutely. So, I mean, we do a huge chunk of this. Maybe giving a little bit of the backstory of the company. The reason we started is that we felt like there were incredibly talented people all around the world that weren't getting opportunities. And the primary reason is that labor markets are very fragmented and that a candidate somewhere else in the world, maybe it's remotely in The US or another country, was only applying to a handful of jobs. The company in San Francisco is considering a fraction of percent of people because there's this matching problem that they're solving manually. And through applying LMs, could solve this matching problem so that we could build this global unified labor market that every candidate applies to and every company hires from. But then we realized that there is this huge take off in hiring people associated with these new knowledge work roles and evaluating LLMs. And so now we hire all sorts of experts for the top AI labs that use our technology to help facilitate that, both for creating evals to evaluate our experts, as well as to evaluate the models and all of these agents that you're discussing.
00:04:04 · Brendan Foody, S2
[00:04:04] S1: Absolutely. So, I mean, we do a huge chunk of this. Maybe giving a little bit of the backstory of the company. The reason we started is that we felt like there were incredibly talented people all around the world that weren't getting opportunities. And the primary reason is that labor markets are very fragmented and that a candidate somewhere else in the world, maybe it's remotely in The US or another country, was only applying to a handful of jobs. The company in San Francisco is considering a fraction of percent of people because there's this matching problem that they're solving manually. And through applying LMs, could solve this matching problem so that we could build this global unified labor market that every candidate applies to and every company hires from. But then we realized that there is this huge take off in hiring people associated with these new knowledge work roles and evaluating LLMs. And so now we hire all sorts of experts for the top AI labs that use our technology to help facilitate that, both for creating evals to evaluate our experts, as well as to evaluate the models and all of these agents that you're discussing. [00:05:09] S2: Maybe for our listeners too, on the Merkor side, you guys obviously have a bunch of uses of AI and screening candidates going through resumes. Can you talk through some of the different use cases that you have for AI and then what the stack looks like that you guys are building on today?