The Frontier with Brendan Foody
00:03:08 · Jonathan Frankle, Brendan Foody
Or are we going to just have to measure and empirically find our way through by looking at the properties of models? And we're here today to talk about evals, like it's very clear which one won. [00:03:17] S1: Yeah.
00:03:18 · Jonathan Frankle
[00:03:18] S0: We're not proving that the model behaves well. We're measuring its behavior. In 2018, that was a really controversial thing to say. If you go and look at the lottery ticket hypothesis papers, like iClear reviews, because they're all public, the scores were 9, 9, 5, and the five was there's no proof of this. And I remember when I was on the faculty market, this is just this is stuck with me forever. Someone during an interview asked me, You've done all this empirical work. Do you plan to do anything principled? What this person meant was principled is synonymous in computer science with theoretical and provable. I was really insulted because like empiricism's principled. We can do empirical science in a principled way. That was the state of the field in 2018. So the work itself was about understanding the training dynamics of models. And I ended up, you know, having some kind of interesting, curious findings that were popular at the time. I don't think they really stood the test of time in terms of being that useful, quite frankly. But it was cool at the time. And I think really the lasting impact was empiricism is a lens into understanding how to use neural networks that allows us to go further than theory will allow us to go. We can ask bigger questions that we just can't prove or describe in a mathematical way. And today, I think we all just take that for granted. Like, is anyone doing math about deep learning? Like any of Mercore's customers doing like proofs to try to understand if their model is good? Mercore wouldn't exist if you could prove this.
00:04:37 · Brendan Foody
[00:04:37] S1: Yeah, it all has come down to empirics. And that was obviously an incredibly impactful contribution to the field. And talk a little bit more about what brought you from your PhD to Mosaic and the early days of that team and why you joined, how you got started, what you worked on.
00:04:53 · Jonathan Frankle
[00:04:53] S0: Yeah. I mean, what brought me to Mosaic was a cold email from my, you know, the person who became my CEO saying, nice paper, want to do a startup on it? It's like, who the hell are you? And why are you cold emailing me? I don't know. If you have an MIT email address as a PhD student, you get a lot of weird emails from weird people asking weird things. And I was just like, okay, another crazy person. And actually, there was this really wonderful friend and lab mate, Jesse, who I happened to be running into that day. And I was like, I got this weird email. And he's like, You don't know who that is? You've never heard of a startup before? Like, you're an idiot. Um, you should really talk to this guy. I was gonna just ignore it. So I'm really glad I opened the email. And, you know, that turned into mosaic. Um, you know, Naveen Rao reached out and said, Hey, we should do a startup. And I was like, that's dumb. The lottery ticket paper is not that useful and it's never gonna be valuable in the world. And I still think that was right. But we evolved that into something about training efficiency and trying to look for empirical ways to this is gonna sound really quaint, but changing the training algorithm to get better quality in the model more efficiently and being okay with that, even though it was a different algorithm, which today we call doing deep learning. But back then that was a very controversial thing because you're gonna get something different than you would have gotten otherwise. And people were really nervous about that. So it was a controversial thing at the time to say, yeah, we'll train your model, but it'll be trained slightly differently than you planned. We'll get you to the same quality on the eval, and that's all that matters. That was a scary thing for people, which today sounds so hilarious.
00:06:14 · Brendan Foody
[00:06:14] S1: Well, and I definitely feel like in the arc of the industry over the last few years, it felt like initially there was a big push towards custom models and efficiency. And then the frontier models improved so quickly that all of the inference really went towards the frontier models, I think, for most of the last 12 months. But it feels like as costs are ballooning so dramatically, we will naturally converge more towards more and more of those costs, going towards custom, smaller, more efficient models. And a lot of that uh work and the continuation of it that you're doing at Databricks will be really, really important and impactful as that transition accelerates. Um, so maybe share more about uh the acquisition and how you all got in touch with the Databricks team, what that was like and what you're now working on and spending your time on at Databricks.
00:07:00 · Jonathan Frankle, Brendan Foody
[00:07:00] S0: Yeah. I mean, Mosaic went on this crazy journey where we were trying to train custom vision models for a while because who didn't want ResNet 50 as an image classifier? That was, you know, such a hot thing back then. Um, and who didn't want a segmentation model that was, you know, a custom, like, I don't even remember the name of the model at this point, or an object detection model, like a mascar CNN, like all these things that sound like they're from a history book at this point because they're three whole years old or five years old. And we were doing a lot of that. And then someone on my team said to me at some point, maybe people are gonna want more than eight GPUs, and they're gonna want to train these things, these large language models, something that's like maybe a billion parameters. It's like, Obby, nobody's gonna need more than eight GPUs, and nobody's ever gonna need to train, you know, a model with a billion parameters. 25 million is already a lot. Like you're 25 million. [00:07:46] S1: Wow. [00:07:47] S0: Yeah, those. I mean, this is again, I've this was 2022. [00:07:50] S1: Yeah. [00:07:51] S0: This wasn't that long ago. This was four years. [00:07:53] S1: Yeah.