The Frontier with Brendan Foody
00:09:36 · Brendan Foody
[00:09:36] S1: Talk a little bit more about what your data day looks like, what key initiatives that Databricks you're owning, especially within the enterprise, um, as we think through what are the bottlenecks to AI becoming useful and the economy and enterprise deployment.
00:09:50 · Jonathan Frankle
[00:09:50] S0: Yeah, I feel like running a research team is such a weird job, especially if you do it at a company where you're really thinking about the customers. On the one hand, my day job is to like I frame it lots of different ways to the team depending on my mood and what we're trying to do and what I want to sell them on. One version is that we're the risk budget for the company. If you're the CEO of a company like Databricks or, you know, any reasonably sized company, you need someone who's willing to go out there and just take a lot of risks, do the crazy things. And, you know, whenever you look at just your team, you say, I need a portfolio of risks, I need to like do some conservative things so I deliver value and do some high-risk things, but we aren't a portfolio as a team. We are literally the high risk budget for a massive company. And so if we're not taking risks and trying new things, we're not serving our purpose within the organization. And so our job is to fail. And that means failing on some research projects, trying to do some reinforcement learning at scale on a task that's too hard and failing at it. That means trying to develop some new products that just don't work because the technology is not there yet. And then trying again next year and the technology is there, and this has happened a couple of times where we've taken two or three cracks at the same problem over the course of two or three years. And like there's a thing this year that started working that someone had been trying, I think since 2022. And finally the models got good enough that this project actually started working in a really big way and has been game-changing for the team and the product. Well, the other way I look at this is that my job is to spend a lot of time with our customers. Just hang out with a lot of enterprises that have AI needs, figure out what problems they're getting stuck on, and build entirely new technology to solve those problems. It's not just giving them a product or an interface or a way to do a thing like the plumbing, it is inventing a new way of solving that problem through our expertise in machine learning. And that's a lot of fun. Like there aren't that many research teams in the world where you can say, I'm a scientist and I spend, you know, a quarter of my time with customers.
00:11:35 · Jonathan Frankle, Brendan Foody
The way I frame it to our team at least is the customers are the evals. Like the one true measure of our success in the world is is there somebody at one of our customers who actually adopted it and actually liked it and is paying to use it and maybe got promoted for using it and we made them look really good? That's the eval. Like Suitebench is great, Suitebench does not matter at all. Birdbench is great, Birdbench doesn't matter at all. Are people using our models? [00:11:57] S1: Yeah. And you have a very unique vantage point in both leading a research team as well as spending a lot of time with customers and understanding what things are actually working about the models in production and what things aren't. What do you see as the largest bottleneck to 10xing enterprise adoption and making these models actually useful on the true eval of the customer?
00:12:18 · Jonathan Frankle, Brendan Foody
[00:12:18] S0: Very simply put, it's evals. It's measurement. You know, there's a lot more to say about that, but the bottom line is I think we have plenty of intelligence. I think that, you know, I the line I draw in the sand is like, I think Claude Sonnet 3.5, which also feels like a very quaint statement to make that makes me sound ancient. Some people don't remember anything before Claude 4 at this point. But back in my day, we had Sonnet 3.5, and that was an awesome upgrade. And from then on, it's kind of felt like the models are good enough at a lot of really complex enterprise tasks and knowledge tasks much of the time. The problem isn't the model being smarter. You can take the world's smartest person in the world and sit them down at a desk and say, do work. That's an underspecified task. So I think the real challenge is describing what you want AI to do and knowing whether AI did it. That's an eval task by and large. If we get better at that, I think we will unlock huge amounts of AI value for enterprises. If we don't get better at that, you could give me a model that is 10x smarter than Mythos and the world's smartest person without knowing what they're supposed to do and what it means to do a good job is still going to fail. [00:13:19] S1: And of course, there's sort of two processes that the enterprise would go through first creating the eval, and then second, hill climbing it. How much do you see the hill climbing process involving automated prompt optimization with JEPA or manual prompt optimization versus custom model training to hill climb those evals?
00:13:37 · Jonathan Frankle
[00:13:37] S0: It's a good question. The way I think about it is kind of metaphorically when it comes to software. Usually, if you're gonna create a prototype, you just throw it together in Python. It's not gonna be fast, it's not gonna be great, it's not gonna be robust, but you need to demonstrate that it's the prototype is good enough that it's worth bothering to actually build a high quality version of it. Or you make a wireframe in Figma. That's where you start. You don't go and build out a whole user interface and make a perfect product, write it in Rust, and then say, like, is anybody gonna buy this? Make a quick prototype and see if anybody likes it. And I think this is true even more so with agents, where we're at the early stages of knowing what a good agent is and when an agent is worth building. First, just pull up your favorite foundation model, do a little prompt engineering, maybe build one or two quick eval examples or kind of vibe checking examples, iterate on it a bit and be like, is this even possible? If I throw the world's best model at this and spend a lot of time coaching it, can I get it through this task successfully? Like prove that it's worth it. And then from there, show that people are like people actually like this, show that you can make it good. That's where evals come in, obviously. And that's where you know actually testing it with users comes in. And it's okay if it's expensive initially. This is still a proof of concept. Time is much more valuable. If you can fail fast, if you can determine that this is working, sure, spend the money. I think Cursor is kind of in some sense a giant example of this. They built an incredible product that was very popular on that. And once you have commitment to a use case, you know that something's going to be around for a while. That's a great time to train a custom model. Where, you know, I think of this as like switching from Python to Rust. You're really committed. Nobody should write their first prototype in Rust, I think. Um I don't know, maybe some people really, really like Rust. But when you're really committed to something, yeah, you need to make it efficient and make it scalable. Write it in Rust. And it's the same thing. Do some custom RL on a model. But there are lots of stages in between where, you know, there are a lot of steps in the ladder. Custom proptimization is awesome.
00:15:17 · Jonathan Frankle, Brendan Foody
Run Jeppa. Use Jeppa against an eval set that you've built. That'll get you something better than you could probably tune by hand or at least faster, and you'll get some more value out of it. Figure out whether you can use a smaller model for it to make it more efficient and more scalable. But at the end of the day, once you've committed to something at large scale, you know, custom models can shave off 5x or more of your cost. And, you know, we're seeing Cursor talk about this. I think Harvey just mentioned something about this recently. And I'm sure there are lots of other companies that haven't talked about it yet. I know at our summit this week, there are a bunch of slides that had custom models showing up all over the place. And, you know, I won't say any more than is in those slides, but you can imagine, you know, that's because these use cases are so popular that we know they're going to be around for a while and it's worth committing. [00:15:58] S1: Yeah. You and I are probably two of the most eval-pilled people in the world, but I imagine that a lot of enterprises aren't there yet. Uh, how are you seeing that journey develop? Like, are enterprises actually building robust evals, or do you think that's more so something that they need to do over the next 12 months?