The Frontier with Brendan Foody
00:24:15 · Jonathan Frankle
[00:24:15] S0: I mean, it's easy to bash public benchmarks. Nobody has ever liked a public benchmark. Like I think I've never, in my, you know, very long decade of having been an AI, which I guess makes me ancient at this point. I've never seen a public benchmark where someone said, Oh, that benchmark is amazing and clearly measures the real world. And people definitely aren't overfitting to that benchmark. I mean, CFAR 10, everybody hated, but that's also 32 by 32 images, where if you look at those images, you can't tell that that's an airplane or not. Um, ImageNet has like, I think it's 10% of the categories in ImageNet are just dog breeds. And then there's one category for cats. Like you can tell the people who created ImageNet and how they feel about that, but it's not a realistic benchmark. But the ImageNet benchmark alone, I think, is one of the most important contributions to the fact that we have all of this, you know, AI today. That benchmark showed that deep learning was valuable and led to a huge amount of investment over five years. And even though we totally beat the crap out of that benchmark to the point of uselessness, it's hugely advanced the field. So I think we can live in this in-between where new benchmarks challenge us and new public benchmarks are important because they give us a shared measuring stick. And we tend to bend them out of shape pretty quickly these days. And it's easy to wonder whether people are, you know, bench maxing or people are, you know, literally training on the benchmark data. And that's a risk. So they're useful to the field, they're useful to advancing science. That doesn't mean they're perfect, and they tend to have bias built into them of first of all, what is possible to build in the academic world. And second, what the academic world thinks is important for AI. For example, there's a lot of emphasis on coding right now, which is understandable. Coding is really popular. I don't think we're putting enough emphasis on kind of knowledge-related tasks or kind of knowledge synthesis tasks. That's very Databricks pilled. So take that, Freddy Will, and very enterprise pilled. You know, I think that's true.
00:25:56 · Brendan Foody, Jonathan Frankle
[00:25:56] S1: I mean, like software engineering is something like 1% of the economy, right? And in Silicon Valley, we feel like it's everything. [00:26:02] S0: Well, come to New York. Come get a bagel with me and you can see the rest of the economy. And I'm sure we can look out this window right now and we can see only tech companies. But I think there's also like even a bias in the RL environment world toward coding-related environments. The idea that a Docker container is sufficient to contain an environment is a very coding-pilled way to look at the world. When I look at the world from a Databricks or enterprise pill perspective, many of the environments I look at are tens of petabytes. Go put that in a Docker container. [00:26:29] S1: Yeah.
00:26:29 · Jonathan Frankle
[00:26:29] S0: And go make that something that you can easily modify and reset. Like that's actually a new systems problem that needs to be solved at scale. How do you allow an AI agent to work in a 10 petabyte environment, make some changes, and immediately revert that so it doesn't break prod? And how do you do that without literally breaking prod because the environment you're working in is also your production environment? That's something very different than I think the academic world is focused on. And it's not a bad thing. These are just different perspectives, different problems. But I think the thing that's always missing in any academic task, you know, going back to time immemorial in computer science is scale. The story goes that when Google came into existence, it basically killed the field of information retrieval in academia for a decade. Because why would you do information retrieval on a nice synthetic task in academia when you literally have the whole internet? It's just a different scale of problem. And we're watching that happen in academia again. Like look at the number of faculty, yours truly included in some sense, who either never arrived in academia, sorry to everybody at Harvard, or who, you know, have left academia recently because I'm sorry, but working on 16 H100s is just not the same as working at cursor training incredible models or working at Databricks training models or what have you. It's the same thing with benchmarks. We're missing that scale. There's no academic benchmark that works on a tens of petabytes data set, nor can there be.
00:27:44 · Brendan Foody, Jonathan Frankle
[00:27:44] S1: What are the public benchmarks that you find most useful today? [00:27:46] S0: That's a good question. I think when like increasingly at Databricks, we're not really working off of any public benchmarks. It's all in-house annotated benchmarks and usually on our own data because our data is real and messy. And we can at least, it's data we have full access to. We don't have to ask anybody's permission. We don't have to create a synthetic data set. Honestly, I mean, things like Browse Comp Plus have been really useful. Things like Stark have been really useful. Like a lot of retrieval benchmarks are good. They're contrived. Like you can read the questions in Browse Comp Plus, and you're not going to say this is a real search anybody would do, but they do challenge some capability of a model in a meaningful way. [00:28:19] S1: Totally. [00:28:20] S0: We've released our own benchmarks at this point. Like the Office QA benchmark that we've worked really hard on is a parsing and reasoning benchmark. Because it turns out that PDF parsing is still an unsolved problem, even in this day and age. Shocking. And you know, the the fun part, we actually just released it this week, was that it's a benchmark on treasury documents. And we actually worked with the Treasury and got, you know, Treasury documents going back to 1789 to try to parse. And good frickin' luck parsing benchmarks in 1789. We have a copy of like Alexander Hamilton's paycheck, which is just hilarious. [00:28:50] S1: Jonathan, is that actually in the user distribution?
00:28:50 · Brendan Foody, Jonathan Frankle
[00:28:50] S1: Jonathan, is that actually in the user distribution? [00:28:52] S0: Um, some Databricks customers are actually hundreds of years old and actually have some data like this. Okay. It's really cool. But like I was chatting with a customer that has jet engine schematics going back to before there were jet engines because they developed some of the first ones. And that data is there. And people want to ask questions about it. So, again, not a realistic benchmark in a sense that nobody, at least I don't think any Databricks customers are trying to figure out how much Alexander Hamilton was getting paid in 1790. But it's a really challenging benchmark that pushes on some important capabilities. And it's one we could package cleanly enough that like now OpenAI includes it in every new model release as a benchmark that they use. And we just released a new version of it and we're continuing to iterate. Even that is tiny compared to the stuff we're doing internally, where we do have tens of petabytes of workspaces and the environments are dynamic and the answer may actually change day to day. [00:29:38] S1: Another part of benchmarking that we've talked about is governance, which is obviously becoming extremely important to enterprises as they trust the models with more tools, more data, more access. How are you seeing those evals evolve? Do most enterprises also have evals for governance yet alongside these evals that they're curating for accuracy? Or uh is that still an earlier process that many of them are just beginning to embark on?
00:30:06 · Jonathan Frankle
[00:30:06] S0: I don't think anyone's even figured out how to measure this yet. I think it's worth defining governance because it's a like if you want to talk about an enterprise-pilled word that the average AI person doesn't even think about. What access should a model have? What access should anyone have? And this feels like a really simple question initially. Like, you should have access to this code, but not that code. You should have access to this information, but not that financial information, or these Google Docs you have permission to, but not those Google Docs. And then when you get to a company with a thousand people, let alone 10,000 people, the permissions get so complicated. And then also think about I think for anyone out there, the number of times you've made a Google Doc world readable to everybody at your company and have just sent it to a couple of people, but you didn't want to have to keep updating the permissions list, or you've made a document, worst of all, world editable because you didn't want to have to keep updating the list. This is the kind of thing that like you probably have access to hundreds of Google Docs that you have never looked at and never thought about. Agents tend to find that stuff. And so the question of what permissions an agent should have, agents do very creative, very strange things when you give them permissions. Things that can make a ton of sense when you really think about it, but are kind of inexplicable in the moment that you wouldn't have predicted. And so when you think about an enterprise, there are probably hundreds of thousands or millions of individual permissions that you need to keep track of. Even just doing that for humans is complicated. And there are all sorts of solutions. We have our own at Databricks, but lots of governance solutions just to track that globally across all of your different kinds of data, both within Databricks and beyond. Now apply that to agents. You're going to give an agent permission to go and pursue a task. A lot of people say, like, oh, just give the agents your permissions and you're responsible for the agent's actions. Um, the agent is gonna find that doc that you would never have found. Agents work at scale in ways that humans don't. You don't have time to go looking through all the Google Docs you technically have permission to.