The Frontier with Brendan Foody
00:03:18 · Jonathan Frankle
[00:03:18] S0: We're not proving that the model behaves well. We're measuring its behavior. In 2018, that was a really controversial thing to say. If you go and look at the lottery ticket hypothesis papers, like iClear reviews, because they're all public, the scores were 9, 9, 5, and the five was there's no proof of this. And I remember when I was on the faculty market, this is just this is stuck with me forever. Someone during an interview asked me, You've done all this empirical work. Do you plan to do anything principled? What this person meant was principled is synonymous in computer science with theoretical and provable. I was really insulted because like empiricism's principled. We can do empirical science in a principled way. That was the state of the field in 2018. So the work itself was about understanding the training dynamics of models. And I ended up, you know, having some kind of interesting, curious findings that were popular at the time. I don't think they really stood the test of time in terms of being that useful, quite frankly. But it was cool at the time. And I think really the lasting impact was empiricism is a lens into understanding how to use neural networks that allows us to go further than theory will allow us to go. We can ask bigger questions that we just can't prove or describe in a mathematical way. And today, I think we all just take that for granted. Like, is anyone doing math about deep learning? Like any of Mercore's customers doing like proofs to try to understand if their model is good? Mercore wouldn't exist if you could prove this.
00:04:37 · Brendan Foody
[00:04:37] S1: Yeah, it all has come down to empirics. And that was obviously an incredibly impactful contribution to the field. And talk a little bit more about what brought you from your PhD to Mosaic and the early days of that team and why you joined, how you got started, what you worked on.
00:04:53 · Jonathan Frankle
[00:04:53] S0: Yeah. I mean, what brought me to Mosaic was a cold email from my, you know, the person who became my CEO saying, nice paper, want to do a startup on it? It's like, who the hell are you? And why are you cold emailing me? I don't know. If you have an MIT email address as a PhD student, you get a lot of weird emails from weird people asking weird things. And I was just like, okay, another crazy person. And actually, there was this really wonderful friend and lab mate, Jesse, who I happened to be running into that day. And I was like, I got this weird email. And he's like, You don't know who that is? You've never heard of a startup before? Like, you're an idiot. Um, you should really talk to this guy. I was gonna just ignore it. So I'm really glad I opened the email. And, you know, that turned into mosaic. Um, you know, Naveen Rao reached out and said, Hey, we should do a startup. And I was like, that's dumb. The lottery ticket paper is not that useful and it's never gonna be valuable in the world. And I still think that was right. But we evolved that into something about training efficiency and trying to look for empirical ways to this is gonna sound really quaint, but changing the training algorithm to get better quality in the model more efficiently and being okay with that, even though it was a different algorithm, which today we call doing deep learning. But back then that was a very controversial thing because you're gonna get something different than you would have gotten otherwise. And people were really nervous about that. So it was a controversial thing at the time to say, yeah, we'll train your model, but it'll be trained slightly differently than you planned. We'll get you to the same quality on the eval, and that's all that matters. That was a scary thing for people, which today sounds so hilarious.
00:06:14 · Brendan Foody
[00:06:14] S1: Well, and I definitely feel like in the arc of the industry over the last few years, it felt like initially there was a big push towards custom models and efficiency. And then the frontier models improved so quickly that all of the inference really went towards the frontier models, I think, for most of the last 12 months. But it feels like as costs are ballooning so dramatically, we will naturally converge more towards more and more of those costs, going towards custom, smaller, more efficient models. And a lot of that uh work and the continuation of it that you're doing at Databricks will be really, really important and impactful as that transition accelerates. Um, so maybe share more about uh the acquisition and how you all got in touch with the Databricks team, what that was like and what you're now working on and spending your time on at Databricks.
00:07:00 · Jonathan Frankle, Brendan Foody
[00:07:00] S0: Yeah. I mean, Mosaic went on this crazy journey where we were trying to train custom vision models for a while because who didn't want ResNet 50 as an image classifier? That was, you know, such a hot thing back then. Um, and who didn't want a segmentation model that was, you know, a custom, like, I don't even remember the name of the model at this point, or an object detection model, like a mascar CNN, like all these things that sound like they're from a history book at this point because they're three whole years old or five years old. And we were doing a lot of that. And then someone on my team said to me at some point, maybe people are gonna want more than eight GPUs, and they're gonna want to train these things, these large language models, something that's like maybe a billion parameters. It's like, Obby, nobody's gonna need more than eight GPUs, and nobody's ever gonna need to train, you know, a model with a billion parameters. 25 million is already a lot. Like you're 25 million. [00:07:46] S1: Wow. [00:07:47] S0: Yeah, those. I mean, this is again, I've this was 2022. [00:07:50] S1: Yeah. [00:07:51] S0: This wasn't that long ago. This was four years. [00:07:53] S1: Yeah.
00:07:53 · Jonathan Frankle
[00:07:53] S0: And thank God that Obby didn't listen to me. Um, and one of my favorite things about running a research team is the people who fail to listen to me do something, get it to the point where there's some value there, and then I'm like, okay, we'll pivot the whole team. He went and built out a multi-node training stack on his own, built out the initial version of our what became our LLM pre-training stack on its own. And this was in like summer of 22. So pre-Chat GPT, he just he felt like something was coming. And it ended up working well enough that we pivoted the whole team, and that became our product, custom LLMs. Um, and you know, less than a year later, we had trained the first open source reproduction of Llama, which again sounds really quaint. 7 billion parameter model on 1 trillion tokens. Really hard thing to do back then. Now an intern project for many people. But you know, back then it was a full team ordeal on a really scary, this is gonna be sound like a huge cluster, 256 A100s. That was so expensive and so scary. Why did we get that? Why are we burning all this cash? Legend speaks that Naveen Rao and Ali God seemed that at the first Cerebral Valley. I have no idea if this is actually true or if this is just what legend speaks of, because these things tend to accelerate, but they met, they hit it off and became part of Databricks and got to go big. Um, it turns out that, you know, when you're a custom training company, you really need data. And when you're a data company, you really need custom training. And to Databricks' credit, the annual conference for Databricks, which is why I'm even out here, is called the Data and AI Summit. And it sounds like a very 2026 pilled name. I think the name was changed from the Spark Summit to the Data and AI Summit in like 2018. There was, I think the Databricks founders understood at that time that AI is what you do with data. Yeah. And, you know, AI doesn't work without data. Now we call these environments and agents, but the two go together.