The Frontier with Brendan Foody
00:07:53 · Jonathan Frankle
[00:07:53] S0: And thank God that Obby didn't listen to me. Um, and one of my favorite things about running a research team is the people who fail to listen to me do something, get it to the point where there's some value there, and then I'm like, okay, we'll pivot the whole team. He went and built out a multi-node training stack on his own, built out the initial version of our what became our LLM pre-training stack on its own. And this was in like summer of 22. So pre-Chat GPT, he just he felt like something was coming. And it ended up working well enough that we pivoted the whole team, and that became our product, custom LLMs. Um, and you know, less than a year later, we had trained the first open source reproduction of Llama, which again sounds really quaint. 7 billion parameter model on 1 trillion tokens. Really hard thing to do back then. Now an intern project for many people. But you know, back then it was a full team ordeal on a really scary, this is gonna be sound like a huge cluster, 256 A100s. That was so expensive and so scary. Why did we get that? Why are we burning all this cash? Legend speaks that Naveen Rao and Ali God seemed that at the first Cerebral Valley. I have no idea if this is actually true or if this is just what legend speaks of, because these things tend to accelerate, but they met, they hit it off and became part of Databricks and got to go big. Um, it turns out that, you know, when you're a custom training company, you really need data. And when you're a data company, you really need custom training. And to Databricks' credit, the annual conference for Databricks, which is why I'm even out here, is called the Data and AI Summit. And it sounds like a very 2026 pilled name. I think the name was changed from the Spark Summit to the Data and AI Summit in like 2018. There was, I think the Databricks founders understood at that time that AI is what you do with data. Yeah. And, you know, AI doesn't work without data. Now we call these environments and agents, but the two go together.
00:09:36 · Brendan Foody
[00:09:36] S1: Talk a little bit more about what your data day looks like, what key initiatives that Databricks you're owning, especially within the enterprise, um, as we think through what are the bottlenecks to AI becoming useful and the economy and enterprise deployment.
00:09:50 · Jonathan Frankle
[00:09:50] S0: Yeah, I feel like running a research team is such a weird job, especially if you do it at a company where you're really thinking about the customers. On the one hand, my day job is to like I frame it lots of different ways to the team depending on my mood and what we're trying to do and what I want to sell them on. One version is that we're the risk budget for the company. If you're the CEO of a company like Databricks or, you know, any reasonably sized company, you need someone who's willing to go out there and just take a lot of risks, do the crazy things. And, you know, whenever you look at just your team, you say, I need a portfolio of risks, I need to like do some conservative things so I deliver value and do some high-risk things, but we aren't a portfolio as a team. We are literally the high risk budget for a massive company. And so if we're not taking risks and trying new things, we're not serving our purpose within the organization. And so our job is to fail. And that means failing on some research projects, trying to do some reinforcement learning at scale on a task that's too hard and failing at it. That means trying to develop some new products that just don't work because the technology is not there yet. And then trying again next year and the technology is there, and this has happened a couple of times where we've taken two or three cracks at the same problem over the course of two or three years. And like there's a thing this year that started working that someone had been trying, I think since 2022. And finally the models got good enough that this project actually started working in a really big way and has been game-changing for the team and the product. Well, the other way I look at this is that my job is to spend a lot of time with our customers. Just hang out with a lot of enterprises that have AI needs, figure out what problems they're getting stuck on, and build entirely new technology to solve those problems. It's not just giving them a product or an interface or a way to do a thing like the plumbing, it is inventing a new way of solving that problem through our expertise in machine learning. And that's a lot of fun. Like there aren't that many research teams in the world where you can say, I'm a scientist and I spend, you know, a quarter of my time with customers.
00:11:35 · Jonathan Frankle, Brendan Foody
The way I frame it to our team at least is the customers are the evals. Like the one true measure of our success in the world is is there somebody at one of our customers who actually adopted it and actually liked it and is paying to use it and maybe got promoted for using it and we made them look really good? That's the eval. Like Suitebench is great, Suitebench does not matter at all. Birdbench is great, Birdbench doesn't matter at all. Are people using our models? [00:11:57] S1: Yeah. And you have a very unique vantage point in both leading a research team as well as spending a lot of time with customers and understanding what things are actually working about the models in production and what things aren't. What do you see as the largest bottleneck to 10xing enterprise adoption and making these models actually useful on the true eval of the customer?
00:12:18 · Jonathan Frankle, Brendan Foody
[00:12:18] S0: Very simply put, it's evals. It's measurement. You know, there's a lot more to say about that, but the bottom line is I think we have plenty of intelligence. I think that, you know, I the line I draw in the sand is like, I think Claude Sonnet 3.5, which also feels like a very quaint statement to make that makes me sound ancient. Some people don't remember anything before Claude 4 at this point. But back in my day, we had Sonnet 3.5, and that was an awesome upgrade. And from then on, it's kind of felt like the models are good enough at a lot of really complex enterprise tasks and knowledge tasks much of the time. The problem isn't the model being smarter. You can take the world's smartest person in the world and sit them down at a desk and say, do work. That's an underspecified task. So I think the real challenge is describing what you want AI to do and knowing whether AI did it. That's an eval task by and large. If we get better at that, I think we will unlock huge amounts of AI value for enterprises. If we don't get better at that, you could give me a model that is 10x smarter than Mythos and the world's smartest person without knowing what they're supposed to do and what it means to do a good job is still going to fail. [00:13:19] S1: And of course, there's sort of two processes that the enterprise would go through first creating the eval, and then second, hill climbing it. How much do you see the hill climbing process involving automated prompt optimization with JEPA or manual prompt optimization versus custom model training to hill climb those evals?
00:13:37 · Jonathan Frankle
[00:13:37] S0: It's a good question. The way I think about it is kind of metaphorically when it comes to software. Usually, if you're gonna create a prototype, you just throw it together in Python. It's not gonna be fast, it's not gonna be great, it's not gonna be robust, but you need to demonstrate that it's the prototype is good enough that it's worth bothering to actually build a high quality version of it. Or you make a wireframe in Figma. That's where you start. You don't go and build out a whole user interface and make a perfect product, write it in Rust, and then say, like, is anybody gonna buy this? Make a quick prototype and see if anybody likes it. And I think this is true even more so with agents, where we're at the early stages of knowing what a good agent is and when an agent is worth building. First, just pull up your favorite foundation model, do a little prompt engineering, maybe build one or two quick eval examples or kind of vibe checking examples, iterate on it a bit and be like, is this even possible? If I throw the world's best model at this and spend a lot of time coaching it, can I get it through this task successfully? Like prove that it's worth it. And then from there, show that people are like people actually like this, show that you can make it good. That's where evals come in, obviously. And that's where you know actually testing it with users comes in. And it's okay if it's expensive initially. This is still a proof of concept. Time is much more valuable. If you can fail fast, if you can determine that this is working, sure, spend the money. I think Cursor is kind of in some sense a giant example of this. They built an incredible product that was very popular on that. And once you have commitment to a use case, you know that something's going to be around for a while. That's a great time to train a custom model. Where, you know, I think of this as like switching from Python to Rust. You're really committed. Nobody should write their first prototype in Rust, I think. Um I don't know, maybe some people really, really like Rust. But when you're really committed to something, yeah, you need to make it efficient and make it scalable. Write it in Rust. And it's the same thing. Do some custom RL on a model. But there are lots of stages in between where, you know, there are a lot of steps in the ladder. Custom proptimization is awesome.