Ep 68: CEO of Mercor Brendan Foody on Evals Replacing Knowledge Work, AI x Hiring Today & the Future of Data Labeling

AI now judges talent, and Mercor is betting on it

Mercor is building the evaluation layer for work. AI can now judge people over text as well as humans can, which began with hiring and moved into expert data for frontier labs, where crowdsourced labeling no longer works. Foody sees the same capability extending to contractors and full-time roles.

Takeaways

• AI now judges text-based work near superhuman. Foody says models can assess anything a human can judge over text, including interviews, written assessments and resumes. Mercor's first GPT-4 interviewer hallucinated every two or three questions, but reasoning models fixed context handling. Multimodal evaluation and vibe checks on passion still lag.
• Crowdsourced data labeling is breaking down. Once models got strong, labs needed high-quality experts working directly with researchers, not pipelines of hundreds of thousands of crowdsourced SFT or RLHF items. Foody expects churn at high-volume vendors, while human data and evals grow by orders of magnitude.
• Mercor automates hiring end to end. Resume parsing, scoring, interview questions and ranking each have their own evals. Off-the-shelf models handle basics, while post-training on customer outcomes drives the final prediction. Foody sees the end state as an API giving a confidence interval on performance.
• Chase real pain points, not forced fit. A customer introduced Mercor to xAI's cofounders in August 2023, and frontier lab work began about six months later. Foody advises finding strong pain points where rich customers will pay. His earlier bootstrapped business reached a $1m run rate.
• Productivity gains may increase hiring. Foody says demand to build 100 or 1,000 times more software is elastic. The most useful skills are fast learning and daily work with AI. He calls what humans do in five or ten years the biggest open question and faults regulators for ignoring economic disruption.
• Mercor kept dilution low to fund incentives. Preemptive Series A and B rounds held dilution to about 5%, building a war chest for referral incentives and post-training data. Mercor recently raised $100m.
• OpenAI stays a product company. Foody thinks a general API cannot support many players. Trading firms will eventually strike large partnerships with frontier labs despite being less advanced in post-training. He calls evals underhyped and SFT data overhyped, and says SFT and RLHF budgets are falling.

Mercor's expansion path

1. Use human data and evals work for top AI labs as the wedge.
2. Move to contractor hiring for big tech, competing with firms such as Accenture, and then to full-time hiring.