The Frontier with Brendan Foody
00:16:15 · Jonathan Frankle
[00:16:15] S0: It's kind of, I think it's a bit of a bell curve, but I wouldn't underestimate how fast enterprises are getting savvy about this. I met with like the C-suite of a major healthcare company a few months ago. People were gonna ask, like, what is AI? What is an agent? They were grilling me on the finer points of evals and judges. Like the CFO and like the CEO were grilling me on the finer points of judges. Okay, this is like it's getting there. It's getting there. I think we've been through enough of these big tech transitions in the past decade or two. You know, cloud, mobile, like, I don't know, crypto, say what you will about that, but you know, felt like a tech transition for a minute. You know, even just the web, that I think any company leader who's survived this long and has grown this far knows that when you see a tech trend coming, learn as much as you can about it and prepare yourself and your company for it. So I see people understanding the concept. The commitment to really do it and do it well enough that you can build a flywheel is harder. I met with a Databricks customer this week that has actually annotated 12,000 examples. Um, they're in a medically sensitive application where they have to be really careful. They have 70-something judges that they've built and calibrated. It's insanely impressive. Like I was blown away when I heard this. And I've been chatting with them for a couple of years. Initially, it was talking to them about the importance of judges two years ago. Yeah. And now they are the most judge-pilled, eval-pilled people in the world. And it's extraordinary. They're like the level of rigor is something beyond basically anything I've ever seen anywhere, even if you paid somebody to do it. But they need to because they're in an area where using AI related to medicine would be very risky otherwise. So I'm seeing them as a leading indicator of where things are going. And once you have that, the amount of value you start to get out of it is huge. I'm also seeing kind of this debate between online and offline evals. Like how much do you do eval that is, you know, a static set of examples that you can constantly run against, your kind of unit tests. And how much are you just inspecting traces every day?
00:18:00 · Jonathan Frankle, Brendan Foody
And how do you get a pipeline from inspecting traces back into your offline eval, kind of so that, you know, when you see something weird, you add a unit test to it? I don't think any of these processes have really been figured out yet. We're kind of, I know we've talked about this, like metaphorically, we're in the 1960s of software for AI and you know, we're still using punch cards. I can't wait for us to discover the object-oriented programming of what AI looks like. [00:18:22] S1: Well, and we've worked with so many customers that have the thumbs up, thumbs down flywheels that they try to initially use for eval, but obviously the challenge with that is even if you have thumbs down signal, that doesn't give you the full understanding of what was the problem with that, what could have been different that would have made that a correct solution, and how do you effectively hill climb that in a iterative way? And so that ties so much into how do we effectively understand the full problem space of what's possible, what's good, what's bad, and measure that in a reliable way, which ties into a lot of the importance of offline evals. But maybe going off of that, I imagine that in the enterprise, you've seen so many examples of enterprises that are sophisticated, like you're describing, but also a lot of examples of enterprises that aren't quite there yet. What have been your largest takeaways on what makes a good eval? When you use the word rigor, what does that mean in the enterprise context of these evals actually being useful?
00:19:16 · Jonathan Frankle
[00:19:16] S0: So I'll give you the like grand sweeping cosmic version of this and the highly concrete version of this. The cosmic version is like imagine Waymo cars. We let 16-year-olds on the road if they can drive in a circle in parallel park and not, you know, crash the car, basically. Would we let a Waymo on the road if it could pass that test? And like, you know, we let 16-year-olds literally go and drive on any highway in the country if they can like drive in a circle and turn the car around properly without crashing. It's kind of crazy. We would never let a Waymo loose on public highways if it could pass that test. Um, and you know, I some people say, well, we're unfair to the AI. And I look at it the other way as like, with AI, you can scale something to an arbitrarily large scale. You know, Waymo is limited only by the number of cars they can manufacture, but otherwise, one software change scales everywhere at once. And with AI, you know, we've kind of built our entire world around human strengths and weaknesses and the fact that humans are limited. You know, we live in meat space, not cyberspace, as they would say in the 90s, but AI is not. So the way that we measure AI has to have a degree of rigor like a Waymo car, you know, where you have four different LIDAR sensors and, you know, annotation going on over the course of a decade to measure every ounce of driving in a way that would have been unimaginable a decade ago. And, you know, that's not about measuring AI driving. That's about measuring driving. And, you know, if I had a 16-year-old child someday and they were learning how to drive, if driving is still a thing at that point, I would want them to drive a Waymo type car that has all those sensors and is giving me and them constant feedback and measuring their driving. The driving test should be how do you do over the course of several days driving in a car with that many sensors? So I think with AI, the risk increases if you're going to deploy AI widely. But it also gives us this opportunity to measure work with a degree of rigor that goes beyond a six-month performance review where somebody says, Yeah, I'm feeling this way about your work. Like I don't think anyone believes a performance review is particularly rigorous at any job in the same way the driving test is not rigorous. So, what does that look like for any kind of work?
00:21:11 · Jonathan Frankle
It starts off with one example. A Waymo car starts off with one little driving course, probably on some Google campus somewhere, where there are some street signs and some traffic lights and some fake pedestrians that, you know, some fake people who like carefully walk in front of the car and we see if the car stops or not. What does that look like for you? It could be one example where you have one spreadsheet or one question you just keep asking, and you have a couple of golden answers, and you see if the output matches according to some mediocre LLM judge. But one example is a great place to start. And then two examples, and then three examples, whatever you would vibe test. I think whenever you see anyone use a new model, everybody's got their favorite question, whether it's the unicorn drawing or, you know, how many R's are there in Strawberry or what have you. I know that whenever a new model comes out, I have my favorite questions I ask. Internally at Databricks in our agents when the new model is there. And those become your e-vals. And then you start looking at like where do things go wrong? Where did somebody else complain? It's kind of vibe testing transforms into evals over time. And then eventually you can get systematic once you've deployed something at scale. You know, Waymo now, like, you know, as they run into these weird corner cases, they issue a recall because it's a real product. They add that to their eval suite and they collect a lot more data about that situation. I think that's how you bootstrap it.
00:22:26 · Brendan Foody, Jonathan Frankle
[00:22:26] S1: When these enterprises put a lot of time and effort into creating a robust eval, have you seen cases where they're comfortable sharing that with foundation model labs? Or in most cases, are you finding that they're extremely sensitive to that? [00:22:39] S0: I think they're extremely sensitive to that. And I'm gonna choose my words carefully here. But if you're a company, your data is basically your lifeblood. Like it's some sense, you know, and data is both the literal data you have and kind of the human capital, the data that's invested in that human capital. If you're building an LLM, data is also your lifeblood, but it's your lifeblood in the sense that you would like more of it. And so it's a very scary thing if you're an enterprise where you have your crown jewels, yeah, and you know that there's a company that both really wants to help you, but also really, really needs more data. That's a complicated proposition. So I don't know, being at Databricks, it's been nice to kind of be Switzerland in this conversation, where in some sense we're the place where the data lives, we're the place where the models arrive, and we can kind of intermediate a little bit to make it possible to measure agents and give some feedback without, you know, having this weird relationship where we would like the data for ourselves.
00:23:35 · Brendan Foody
[00:23:35] S1: And we're finding that, of course, this is the source of the real-to-sim gap in so many ways. If every enterprise had robust evals that they could share with the labs, we would solve the real-to-sim gap very quickly. But because they don't, there needs to be this huge workforce of contractors that we scale out to help Frontier Labs approximate that distribution throughout the entire economy. And of course, there is a gap on public benchmarks even today, as much as uh we're trying to make a lot of effort to resolve that. How do you think about public benchmarks today? Which parts of them are useful, which parts of them aren't? And what are what are the largest gaps that uh we as an industry need to solve?