Latent Space: The AI Engineer Podcast
00:31:06 · Shawn Wang, Jamin Ball
[00:31:06] S1: Same quarter matched. [00:31:07] S0: Exactly. So even as... What you're seeing is even as they've shifted more of their compute to non revenue generating use cases, they've still scaled their compute margin to well over 50%. What you're... So what you're seeing is they're getting hyper efficient at the compute they bring online. The compute they decide to use for revenue generating use cases is becoming... It's become seven x more efficient over the last year. I think you have to ask yourself, will they continue to ramp compute and capacity? I think the answer is yes. I think if you ask anyone at Anthropic, again, this is all speculation. They probably say, we wish we started doing it sooner. We wish we were more [00:31:46] S1: They're trying to commit to more compute. That's that's [00:31:48] S0: that's right. [00:31:49] S1: If you don't talk to any of these guys, you come away with, like, oh my god. They're... You know, they... These guys are absolutely crazy and being fiscally irresponsible. Yeah. But I think if you have any confidence or if you see any... If you see up close the the demand for intelligence that people are snapping up and immediately consuming whatever you give them Mhmm. Then, like, yes, actually, you're trying to get more compute. Yeah. That... That's... The the game is not to reduce the commitments. It is actually to get more commitments. Yeah. They they would have a trillion if they could.
00:31:49 · Shawn Wang, Jamin Ball
[00:31:49] S1: If you don't talk to any of these guys, you come away with, like, oh my god. They're... You know, they... These guys are absolutely crazy and being fiscally irresponsible. Yeah. But I think if you have any confidence or if you see any... If you see up close the the demand for intelligence that people are snapping up and immediately consuming whatever you give them Mhmm. Then, like, yes, actually, you're trying to get more compute. Yeah. That... That's... The the game is not to reduce the commitments. It is actually to get more commitments. Yeah. They they would have a trillion if they could. [00:32:17] S0: Yeah. I don't think I've spoken to anyone in the AI industry, certainly not anyone with the big labs who has said, yeah, we're, you know, I feel [00:32:23] S1: We're good. We're the compute we have. [00:32:25] S0: It's almost always... And and I think a underrated factor on compute demands in the future will be compute that's used for safety, alignment, and verification. Like, I think more and more compute is going to go towards those functions, whether that's on research Yeah. Or just on verification and alignment that happens at inference time, that will be an increased compute demand for all of [00:32:50] S1: these things. Probably not gonna break those out, you know, but, like, it'll it'll be, like, maybe 5% now and [00:32:55] S0: Yeah. Of course. [00:32:55] S1: 20% in future. [00:32:56] S0: Of course.
00:32:56 · Jamin Ball, Shawn Wang
[00:32:56] S0: Of course. [00:32:58] S1: I think, you know, I I would I would just tell people, like, if you think this is bad now, wait till you see the OpenAI as one. Yeah. And then, like, it's not bad. It's... Like, that's the whole point. It's like they're actually really trying to get compute from every source they can get Mhmm. And, like, OpenAI is buying TPUs. Yeah. Yeah. Like, there was a cold day in hell that never happened. [00:33:16] S0: TPUs. Right? They have their own chip initiative, Jalapeno, which has pretty rave reviews. Uh-huh. Anthropic staffed up a compute team to build their own chips. Obviously, the hyperscalers are building their own chips. It's I mean A [00:33:28] S1: mad dash. [00:33:29] S0: It's a it's a mad dash. [00:33:30] S1: Only, you know, only winners are TSMC, NVIDIA, and ASML Yeah. Whatever, in the supply chain stack. [00:33:37] S0: Totally. And then I think, look, it's it's getting harder and harder, especially in The United States. You have a lot of pushback. I mean, you have all kinds of pushback. Data centers are too noisy. Don't want them in my neighborhood. They add a lot of air pollution. The water. Don't want it. It's gonna... The water. [00:33:49] S1: Oh my god. The
00:33:50 · Jamin Ball
[00:33:50] S0: water. The water. It's... Which is... It uses too much water, which it doesn't. It's the electricity. It uses a lot of electricity. It's gonna increase my electricity bill if I live in close proximity. Or it's just the simple AI is going to kill us and I don't want to enable that, which again, you have all kinds of pushback. It's become hyper hyper political, but I think the unfortunate thing is it has become a lot harder. You do see a lot more pushback. Midterms are coming up. Right? Who who kind of like takes control in, like, the house and the senate? Like, could have a big impact on regulation or lack thereof around data center and intelligence center build outs. And the good thing is that it's getting harder and harder to build these data centers. Behind the meter is harder and harder. And so you do have a natural governor on the build out, and you're... We're... I think we're a lot less likely to overbuild too quickly that leads to a supply glut. Right? Kind of the the Internet bubble analogy of the early two thousands where we built way too much fiber. There wasn't as much demand for the Internet. There was a lot of dark fiber. Right? Fiber that had no bits and bytes traveling over it. We have no dark GPUs today. So And it's it's probably a good thing that there's some rate limiter. It will stop us from entering a huge bubble, but it's also bad because we need... You know, the industry needs more compute, especially as these models get more and more powerful. You know, the security concerns go through the roof. You also need a lot more... Like, the defenders need more compute. The defenders need more capabilities
00:35:15 · Shawn Wang, Jamin Ball
[00:35:15] S1: Oh, yeah. [00:35:15] S0: Which, you know, I don't... I don't know. Personal belief, I I think we're gonna be compute constrained for for far longer than than the world thinks. [00:35:24] S1: Yeah. I I mean, I'd be curious if what the what the markets price the world as thinking and then doing the work on, like, what do we actually think and have... The delta is probably... You know, could be three years, could be thirty years. Yeah. And Yeah. Yeah. Yeah. And that's a very material thing to figure out. [00:35:40] S0: Yeah. Of course. Of course.
00:35:40 · Jamin Ball, Shawn Wang
[00:35:40] S0: Yeah. Of course. Of course. [00:35:41] S1: I I think one thing to guide people towards is... Well, there's all this political stuff. And, honestly, I will say, Meta does a lot of work, and I think increasingly, OpenAI does a lot of work in, like, well, here's how we we make sure we're a positive, presence in the communities that we're in. Right? Yes. You you do need to take care of that. And... But I think, like, coming back to, again, research, relevance, ROI, and what it means about the industry, this profit move here, I I I do wonder sometimes, and this is a thing that has come up in our other podcast, particularly tying in the research side, is how much left is there to improve. Right? So was this kind of like a onetime upgrade of our inference stack, where where you... Which you noted was, like, something like a seven x improvement in, you know, inference efficiency. Yeah. Because, like, you can't... Like, probably, like, there's no improvements forever until the end of time in our inference stack. Yeah. Like, unless we fundamentally change the chip, we probably we probably started out pretty unoptimized. Mhmm. And we are now a lot more optimized than we were two years ago. Yeah. Okay. Well, there's a limit to how optimized you can get. Mhmm. Just like physical, like, jewels moving around and Mhmm. Bits and bytes moving around. So that that is my question in terms of, like, the the... How much efficiency there's left to squeeze. Yeah. And maybe there... It might cap out, you know, in terms of, like, that that year on year growth.