Unsupervised Learning with Jacob Effron
00:20:27 · Brendan Foody
[00:20:27] S1: Yeah. I remember it was always on college. So I mean, the background of the business is I met my co founders when we were 14 in high school. We were all this feature debate team together. They were like winning all the tournaments that I wasn't as good as them, but I was building companies. And then we started hiring people internationally at the IITs in India. Like we partnered with IT Croc course code club and we were amazed that there were these smart people as you're mentioning that weren't getting jobs and we felt like we could hire them to build projects. Our friends wanted to pay us to hire them. We could take a small fee. So we hustled a lot, boot strap that to a million dollar revenue run rate. We'd profited 80 k after paying ourselves before dropping out, which I was like very proud of. But the parents still weren't satisfied with that, of course, not till we raised money. But to your question, in August 2023, one of our customers introed us to the co founders of xAI while they were still working out at the Tesla office. He said, Merkur has these really smart engineers in India that are phenomenal, like math and coding. And then the xAI co founder the next day got or one of them or two of them got on a call with me and our team and we're just like really excited and then two days later they had us into the Tesla office to meet with the entire XAI co founding team except for Elon. It was right before one of their meetings with Ilan. We were still in college, right? This is insane. And we were just like, wow, why do they want what we've built so badly? And it's because there was this change that was happening so fast in the market that no one else had realized yet. Right? And now, of course, we've scaled that up and are talking about it because we have critical mass of the market share, but that was the point. But they weren't ready for human data yet, and so it wasn't until, call it six months later that we started working with a lot of the Frontier Labs and really scaling up the business.
00:22:17 · Jacob Effron, Brendan Foody
[00:22:17] S0: You could see the tidal wave coming. [00:22:18] S1: Yeah, I think one thing I've realized over time in founders looking for product market fit is that people try to force things too much sometimes. It's like you need to just look for the signs of the market where it's like, wow, there's like gold to be found and just like drill after that. Because if it's hard to get an initial sale, then it's gonna be hard to scale up the process. You need to rather look at what are the really strong pain points where the wealthiest companies will pay whatever it takes. Right? And just sniff those out, and then lean into them. [00:22:52] S0: I guess if you've expanded beyond coding, maybe to go back to the doctor example, because I struck by when you were describing it, one... In some sense is eval ing what a good doctor is, is actually what you're eventually gonna bring these people to the model companies, they're gonna figure out like, is this the reasoning process that a good doctor would use? What are you actually doing when you're working with someone to do evals?
00:22:52 · Jacob Effron, Brendan Foody
[00:22:52] S0: I guess if you've expanded beyond coding, maybe to go back to the doctor example, because I struck by when you were describing it, one... In some sense is eval ing what a good doctor is, is actually what you're eventually gonna bring these people to the model companies, they're gonna figure out like, is this the reasoning process that a good doctor would use? What are you actually doing when you're working with someone to do evals? [00:23:12] S1: Yeah, I think that one of the key things that humans are better... A lot better at right now is like learning over time from the instructions from the training from all the feedback and so excuse me looking for these proxies that people have demonstrated of like they're, you know, asking the right questions, about the problem. They're going about thinking about it in the right way. They have signals in their background that indicate they've been in these high performing environments where people are obviously learning significantly over time, and all of those translating to them finding ways to trip up the model and improve capabilities. Do you guys use your [00:23:48] S0: own product today, and like how does it get used in your own hiring process?
00:23:48 · Jacob Effron, Brendan Foody, S2
[00:23:48] S0: own product today, and like how does it get used in your own hiring process? [00:23:51] S1: Absolutely. We use it for every role except for executive roles. I mean, we still have the listing for executive roles, but most of our executives like I I would take the first interview rather than sending them straight to the AI interview for the selling reason, not the vetting reason. Yeah, I mean, it's extremely effective. In fact, we've found that in many cases, it's like the most predictive signal. I think one thing people underestimate hiring processes is that humans have this very strong bias toward thinking that they're right in this vibes based assessment. [00:24:21] S0: Hiring is like the original vibe, everything. Definitely [00:24:24] S2: do not suffer from that.
00:24:24 · S2, Brendan Foody
[00:24:24] S2: do not suffer from that. [00:24:27] S1: Yeah, and it's like let's ground everything in the performance data of who's doing well on the job. I I remember actually like we so we have this role we're hiring for strategic project leads. And we used to have a human case study before like the strategic project lead on-site to and on-site is like working with us for a day to see how they would do on various parts of the job and figure out who to hire. And then we switched over to fully an AI process before the on-site, and the conversion went up on the on sites. And so it's like through using the AI interviewer, I just speak a lot more objective about the comparisons, Having it standardized throughout, you know, everyone who's applying to the role rather than just mixed across three different interviewers, it was allowing us to have a lot better conversion. [00:25:07] S2: What about on the eval side? Are you guys using kind of a bunch of people that you source for your own evals? Do you do a lot of that internally? [00:25:14] S1: Yeah, we use a lot of people, or we work with a lot of people from our marketplace to create our own evals, and so it's a similar process to what we go through with our customers. And of course, still need the researchers involved with those people and understanding what are the reasons that the model's making mistakes, how can we create our air taxonomy, how our post training data reflect that air taxonomy and he'll climb on the eval. But it's all the same processes and people.
00:25:14 · Brendan Foody, Jacob Effron
[00:25:14] S1: Yeah, we use a lot of people, or we work with a lot of people from our marketplace to create our own evals, and so it's a similar process to what we go through with our customers. And of course, still need the researchers involved with those people and understanding what are the reasons that the model's making mistakes, how can we create our air taxonomy, how our post training data reflect that air taxonomy and he'll climb on the eval. But it's all the same processes and people. [00:25:41] S0: Obviously, you talked a little bit about using multimodal capabilities to determine passion and other things like, what other things are you thinking about with incorporating video and other things that are futuristic for platform. [00:25:54] S1: Yeah. One thing that I think about a lot is what role RL will have in the sort of like timeline to improve video capabilities, and that RL is really good at these search problems. Yeah. And video is just like a huge amount of tokens. That's why like models struggle with it. And so it's like, in many ways, the search problem of like, how do we look for the signal where that person was really excited about the particular thing or, if they cheated on the interviewer, like what other things where we could find in multimodal context. And so I think a lot about how we can effectively create the right data to get the model to pay attention to those, as well as a lot of what the Frontier Labs are doing improve those base capabilities.