Latent Space: The AI Engineer Podcast
00:35:15 · Shawn Wang, Jamin Ball
[00:35:15] S1: Oh, yeah. [00:35:15] S0: Which, you know, I don't... I don't know. Personal belief, I I think we're gonna be compute constrained for for far longer than than the world thinks. [00:35:24] S1: Yeah. I I mean, I'd be curious if what the what the markets price the world as thinking and then doing the work on, like, what do we actually think and have... The delta is probably... You know, could be three years, could be thirty years. Yeah. And Yeah. Yeah. Yeah. And that's a very material thing to figure out. [00:35:40] S0: Yeah. Of course. Of course.
00:35:40 · Jamin Ball, Shawn Wang
[00:35:40] S0: Yeah. Of course. Of course. [00:35:41] S1: I I think one thing to guide people towards is... Well, there's all this political stuff. And, honestly, I will say, Meta does a lot of work, and I think increasingly, OpenAI does a lot of work in, like, well, here's how we we make sure we're a positive, presence in the communities that we're in. Right? Yes. You you do need to take care of that. And... But I think, like, coming back to, again, research, relevance, ROI, and what it means about the industry, this profit move here, I I I do wonder sometimes, and this is a thing that has come up in our other podcast, particularly tying in the research side, is how much left is there to improve. Right? So was this kind of like a onetime upgrade of our inference stack, where where you... Which you noted was, like, something like a seven x improvement in, you know, inference efficiency. Yeah. Because, like, you can't... Like, probably, like, there's no improvements forever until the end of time in our inference stack. Yeah. Like, unless we fundamentally change the chip, we probably we probably started out pretty unoptimized. Mhmm. And we are now a lot more optimized than we were two years ago. Yeah. Okay. Well, there's a limit to how optimized you can get. Mhmm. Just like physical, like, jewels moving around and Mhmm. Bits and bytes moving around. So that that is my question in terms of, like, the the... How much efficiency there's left to squeeze. Yeah. And maybe there... It might cap out, you know, in terms of, like, that that year on year growth.
00:37:05 · Jamin Ball, Shawn Wang
[00:37:05] S0: Yeah. Well, look. I I think I I have a chart I'll get to in a second, but I think this is a debate that's existed for quite some time. Yes. Right? You had, you know, the... I don't remember when this was. Two years ago? You know, like, the ilea moment of, like, hey. The data and the Internet is a fossil fuel. We've we've exhausted it. There's no more data left to train on, and, we're hitting a plateau. Yeah. That was years ago. Then we hit inference time reasoning. Average time compute and reasoning models, you hit a new paradigm. But I think maybe embedded in your question is, was the gap between the frontier and the leading open weight models, is that shrinking? Like, is is the gap that the frontier Yeah. [00:37:45] S1: I don't mind the the gap. It's not something I talk about. [00:37:47] S0: It's... But I think it's interesting because, like, what... The real way to think about or to answer that question is not the the gap between the best leading open weight models today and the frontier models today, because the reality is is OpenAI and Anthropic, they have far more powerful models that they've developed that they just haven't released yet. Like, this is a chart that Mark Chen published from OpenAI, actually, switching gears from Anthropic to OpenAI. [00:38:10] S1: He's been on the pod... Cooking pod. Oh, nice. Nice. We made some soup for researchers.
00:38:10 · Shawn Wang, Jamin Ball
[00:38:10] S1: He's been on the pod... Cooking pod. Oh, nice. Nice. We made some soup for researchers. [00:38:15] S0: Amazing. Amazing. That's... I I love that pod. But, you know, when they saw Navier Stokes, that big math problem, you know, Mark came out and said, hey. Here's a model. It it may be dark blue. Light blue is harder to see, but, like, the top bar is an internal model we used to solve that problem. The blue bar is Astra. Hey. Look at how much more powerful the internal model is to what's publicly out there, which on some ways that like could scare you a little bit. Right? Like, hey. These labs have even more powerful models. So if you think about what's the gap between open weight and Frontier today, I think it's actually far wider than people expect because there are just far better models at the Frontier that haven't been publicly released yet, either whether that's safety, testing, alignment, some combination of everything, you know, unclear. But I do think you're still seeing the frontier accelerate even if it may appear that what's released is plateauing. I'm not even sure if it it does look like what's released has been plateauing. [00:39:16] S1: No. No. No. No one's saying it is plateauing today. It's it's it's... We're all discussing could it plateau in the future. Yeah. Right? Because that that is a pretty important thing to figure out [00:39:25] S0: Mhmm.
00:39:25 · Jamin Ball, Shawn Wang
[00:39:25] S0: Mhmm. [00:39:25] S1: For your investing Yeah. Which is part of the the deal here. Totally. Yeah. So so that is a very well phrased point. I thought you were going a different area where you're gonna argue about bitter lesson and you didn't. So kudos on that. And, like, I I think that's a very, very fair point that I I think we're, we've got still every generation... Model generation, it still shows quantum leaves. Mhmm. This is only in math. Mhmm. But it only has a pretty good track record of generalizing beyond just math Yeah. Or or coding. Yeah. Right? Totally. And so I I I do think that is. My point was a little bit more as on the financial analyst hat on Yeah. Which is you had this one time seven x improvement in revenue efficiency or whatever. And, like, it's not like they raised prices. Yeah. Yeah. Right? So like, where is it coming from? Oh, here's another good one. They didn't raise prices. [00:40:14] S0: Here... Oh, I think I have a chart. [00:40:15] S1: So you know what I'm saying? Right? Yeah. They have a 100... They have 100
00:40:15 · Shawn Wang, Jamin Ball
[00:40:15] S1: So you know what I'm saying? Right? Yeah. They have a 100... They have 100 [00:40:19] S0: Yeah. It's actually opposite. This is an awesome chart. Yes. This shows the cost to compete They're charging less. Yes. Exactly. The the cost They're charging less and making more money? The cost to complete a task with each model family. And so what you see is with whether it's Fable or Opus, and then you have Fable five one and Opus 5.5, you see about a 25% to 40% reduction in cost to complete the same task with each successive model family. And so it's kind of like, yes, the costs are going the costs are going way down, Their compute efficiency is going way up. And I think a lot of that just demonstrates, I mean, again, it's gonna sound a little like VC cliche, but like Jevons paradox. Oh my know. I know. [00:41:03] S1: I know. You can't have [00:41:05] S0: a podcast on AI without that coming [00:41:07] S1: up. Yeah. Know. It is... This is the appropriate time to bring it up. Yeah. Yes. [00:41:10] S0: And as as things get... You know, you go back to like the early days of the Internet. Like, many bytes were traveling across the Internet in 2000? How many bytes are traveling across the Internet today? Like many, many, many, many orders of magnitude more. And guess what? The price to transfer bytes over the network has dropped many... I mean, anyway. [00:41:29] S1: It's a switch meter. [00:41:30] S0: It's and Q. Price has gone way down. [00:41:34] S1: Yeah.