Latent Space: The AI Engineer Podcast
00:32:56 · Jamin Ball, Shawn Wang
[00:32:56] S0: Of course. [00:32:58] S1: I think, you know, I I would I would just tell people, like, if you think this is bad now, wait till you see the OpenAI as one. Yeah. And then, like, it's not bad. It's... Like, that's the whole point. It's like they're actually really trying to get compute from every source they can get Mhmm. And, like, OpenAI is buying TPUs. Yeah. Yeah. Like, there was a cold day in hell that never happened. [00:33:16] S0: TPUs. Right? They have their own chip initiative, Jalapeno, which has pretty rave reviews. Uh-huh. Anthropic staffed up a compute team to build their own chips. Obviously, the hyperscalers are building their own chips. It's I mean A [00:33:28] S1: mad dash. [00:33:29] S0: It's a it's a mad dash. [00:33:30] S1: Only, you know, only winners are TSMC, NVIDIA, and ASML Yeah. Whatever, in the supply chain stack. [00:33:37] S0: Totally. And then I think, look, it's it's getting harder and harder, especially in The United States. You have a lot of pushback. I mean, you have all kinds of pushback. Data centers are too noisy. Don't want them in my neighborhood. They add a lot of air pollution. The water. Don't want it. It's gonna... The water. [00:33:49] S1: Oh my god. The
00:33:50 · Jamin Ball
[00:33:50] S0: water. The water. It's... Which is... It uses too much water, which it doesn't. It's the electricity. It uses a lot of electricity. It's gonna increase my electricity bill if I live in close proximity. Or it's just the simple AI is going to kill us and I don't want to enable that, which again, you have all kinds of pushback. It's become hyper hyper political, but I think the unfortunate thing is it has become a lot harder. You do see a lot more pushback. Midterms are coming up. Right? Who who kind of like takes control in, like, the house and the senate? Like, could have a big impact on regulation or lack thereof around data center and intelligence center build outs. And the good thing is that it's getting harder and harder to build these data centers. Behind the meter is harder and harder. And so you do have a natural governor on the build out, and you're... We're... I think we're a lot less likely to overbuild too quickly that leads to a supply glut. Right? Kind of the the Internet bubble analogy of the early two thousands where we built way too much fiber. There wasn't as much demand for the Internet. There was a lot of dark fiber. Right? Fiber that had no bits and bytes traveling over it. We have no dark GPUs today. So And it's it's probably a good thing that there's some rate limiter. It will stop us from entering a huge bubble, but it's also bad because we need... You know, the industry needs more compute, especially as these models get more and more powerful. You know, the security concerns go through the roof. You also need a lot more... Like, the defenders need more compute. The defenders need more capabilities
00:35:15 · Shawn Wang, Jamin Ball
[00:35:15] S1: Oh, yeah. [00:35:15] S0: Which, you know, I don't... I don't know. Personal belief, I I think we're gonna be compute constrained for for far longer than than the world thinks. [00:35:24] S1: Yeah. I I mean, I'd be curious if what the what the markets price the world as thinking and then doing the work on, like, what do we actually think and have... The delta is probably... You know, could be three years, could be thirty years. Yeah. And Yeah. Yeah. Yeah. And that's a very material thing to figure out. [00:35:40] S0: Yeah. Of course. Of course.
00:35:40 · Jamin Ball, Shawn Wang
[00:35:40] S0: Yeah. Of course. Of course. [00:35:41] S1: I I think one thing to guide people towards is... Well, there's all this political stuff. And, honestly, I will say, Meta does a lot of work, and I think increasingly, OpenAI does a lot of work in, like, well, here's how we we make sure we're a positive, presence in the communities that we're in. Right? Yes. You you do need to take care of that. And... But I think, like, coming back to, again, research, relevance, ROI, and what it means about the industry, this profit move here, I I I do wonder sometimes, and this is a thing that has come up in our other podcast, particularly tying in the research side, is how much left is there to improve. Right? So was this kind of like a onetime upgrade of our inference stack, where where you... Which you noted was, like, something like a seven x improvement in, you know, inference efficiency. Yeah. Because, like, you can't... Like, probably, like, there's no improvements forever until the end of time in our inference stack. Yeah. Like, unless we fundamentally change the chip, we probably we probably started out pretty unoptimized. Mhmm. And we are now a lot more optimized than we were two years ago. Yeah. Okay. Well, there's a limit to how optimized you can get. Mhmm. Just like physical, like, jewels moving around and Mhmm. Bits and bytes moving around. So that that is my question in terms of, like, the the... How much efficiency there's left to squeeze. Yeah. And maybe there... It might cap out, you know, in terms of, like, that that year on year growth.
00:37:05 · Jamin Ball, Shawn Wang
[00:37:05] S0: Yeah. Well, look. I I think I I have a chart I'll get to in a second, but I think this is a debate that's existed for quite some time. Yes. Right? You had, you know, the... I don't remember when this was. Two years ago? You know, like, the ilea moment of, like, hey. The data and the Internet is a fossil fuel. We've we've exhausted it. There's no more data left to train on, and, we're hitting a plateau. Yeah. That was years ago. Then we hit inference time reasoning. Average time compute and reasoning models, you hit a new paradigm. But I think maybe embedded in your question is, was the gap between the frontier and the leading open weight models, is that shrinking? Like, is is the gap that the frontier Yeah. [00:37:45] S1: I don't mind the the gap. It's not something I talk about. [00:37:47] S0: It's... But I think it's interesting because, like, what... The real way to think about or to answer that question is not the the gap between the best leading open weight models today and the frontier models today, because the reality is is OpenAI and Anthropic, they have far more powerful models that they've developed that they just haven't released yet. Like, this is a chart that Mark Chen published from OpenAI, actually, switching gears from Anthropic to OpenAI. [00:38:10] S1: He's been on the pod... Cooking pod. Oh, nice. Nice. We made some soup for researchers.
00:38:10 · Shawn Wang, Jamin Ball
[00:38:10] S1: He's been on the pod... Cooking pod. Oh, nice. Nice. We made some soup for researchers. [00:38:15] S0: Amazing. Amazing. That's... I I love that pod. But, you know, when they saw Navier Stokes, that big math problem, you know, Mark came out and said, hey. Here's a model. It it may be dark blue. Light blue is harder to see, but, like, the top bar is an internal model we used to solve that problem. The blue bar is Astra. Hey. Look at how much more powerful the internal model is to what's publicly out there, which on some ways that like could scare you a little bit. Right? Like, hey. These labs have even more powerful models. So if you think about what's the gap between open weight and Frontier today, I think it's actually far wider than people expect because there are just far better models at the Frontier that haven't been publicly released yet, either whether that's safety, testing, alignment, some combination of everything, you know, unclear. But I do think you're still seeing the frontier accelerate even if it may appear that what's released is plateauing. I'm not even sure if it it does look like what's released has been plateauing. [00:39:16] S1: No. No. No. No one's saying it is plateauing today. It's it's it's... We're all discussing could it plateau in the future. Yeah. Right? Because that that is a pretty important thing to figure out [00:39:25] S0: Mhmm.