Latent Space: The AI Engineer Podcast
01:28:19 · Jamin Ball, Shawn Wang
[01:28:19] S0: I don't know. Okay. That's interesting to me. [01:28:23] S1: Because I I think Anthropic OpenAI was ahead. Correct. Yeah. Right? By by a lot. Yeah. Exactly. But I think Anthropic slows the gap a lot. And OpenAI at at least started... Like, Stargate was a plan... The first stage in a plan to get to 30. Mhmm. 30 gigawatts. Mhmm. Mhmm. And so Like, twenty thirty. Yeah. So... So I was like, okay. Well, that's I I haven't heard that plan from Anthropic. [01:28:40] S0: But again, like, let's let's hit... Let's do some fun math really quick on the fly. Let's see if [01:28:44] S1: I can do it.
01:28:44 · Jamin Ball
[01:28:44] S0: Oh. Getting tired. OpenAI, let's say... I think when they came out with that, it was maybe a year ago, and they were what? Like, one and a half gigawatts, something like that. So whatever. Round that down to zero for just the sake of the argument. Let's say open and anthropic, each one go from zero to 30 over five years. So you're gonna add 60 gigawatts of AI capacity, just the two labs. What's a good assumption for their total capacity? I don't know, 40, 50%. Let's say it's 40%. And... Well, I don't know. Now we're getting too tough in math in my head, but let's say that gets you to, I don't know, 110, 120 gigawatts of total capacity brought online over the next five years. Let's say a 100, just so the math is easier. And let's say it costs $50,000,000,000 to build out a gigawatt. That's $5,000,000,000,000. $5,000,000,000,000 over the next five years. Right? And so call it a trillion a year. You're you're talking about a nontrivial percent of GDP, you know, real GDP. That's like 5% plus or minus of GDP. Like, that is a massive Bonkers. Massive build out. You know? Rivals, the real world build out. I think I've seen some charts somewhere that kind of compare those expected numbers, when you're measuring things in whole number percents of GDP, that's huge. And so like, the credit markets, debt markets need to be super, super, super deep and super, super open for that to happen. You need global build out. You know, it's kind of funny. I had a... You know, semi analysis number for next year is like 50, and I think Brad at the All In Summit said, oh, maybe it's lower, and I think there was some trolling going on back and forth. And I think the end result of that was like, hey, maybe the 50 gigawatt number, I don't want to speak for anyone, was more of like what's constructed. Brad was more of what's constructed and live generating tokens because there can be a gap in what's constructed and ready to go versus what's actually live and generating tokens. But I think it's like a real debate in the world. You know? And then you could see a pretty wide range of outcomes. You know? Do we really add 50 gigawatts of capacity next year as an industry? Or is it 20 or 30? Who knows? I just know everyone you talk to in the supply chain is saying, We have orders or indications for orders that are well over 100% of our capacity. Yeah.
01:31:05 · Jamin Ball, Shawn Wang
And, like, I don't have line of sight to that slowing down for the next few years. [01:31:10] S1: Yeah. And Yeah. And and the limit, like, LAN, Shell Power, all these, but also the silicon, like, straight up Totally. Like, compute memory, every every part of the supply chain. [01:31:19] S0: Totally. [01:31:20] S1: That's the work. That's why 78,000 is so powerful right now. [01:31:22] S0: I mean, look at these recent Kimi models. Kimi K three. You know what's memory? You need gigabytes of memory just to serve that. You're not running that on your laptop. You know, you need like a rack to serve one model. Yeah. Again, are just the big models. Right? Like you're also gonna have tons of small models, but I know the big models are getting bigger, which only require more memory and more compute to run. [01:31:44] S1: I used to say they were capping out. I used to say like the max size is one to two, and then, like, we would just optimize from there. Yeah. I've been wrong. [01:31:50] S0: What's what's interesting is it does feel like the open weight models are capping more around, a 100,000,000,000 active parameters. [01:31:55] S1: That's right. Yeah. And It's mostly, like, an inference speed thing. Totally. But, like, I [01:31:59] S0: think Totally.
01:31:59 · Jamin Ball, Shawn Wang
[01:31:59] S0: think Totally. [01:32:00] S1: It's actually really interestingly tied to the generations of GPUs that they have. Yeah. Right? Exactly. You you, like, wait for the Vera Rubins to to roll out and, like, whatever China has. Yeah. Exactly. And and... Well, then then they'll get it. You know? Yeah. Totally. Then... So it it... They're really memory constrained at the inference layer, and that actually shapes the model architecture, which is, like, kinda cool. Totally. Oh, so that is Anthropic. Yes. Powerhouse tour. Congrats. I don't know how you felt coming into this, but you you did great. Oh, this is fun. Yeah. And, like, obviously, more to do and more to analyze, but I just definitely wanted on the podcast to do this Yeah. Because I I know you. [01:32:34] S0: Yeah. Of course. That's been [01:32:35] S1: And, like, when else We've [01:32:37] S0: been meaning do this for a while. Else are we gonna get this make it happen. [01:32:39] S1: Yeah. Exactly. Exactly. Any other commentary when you don't spend time on, you know, the labs? What are you what are spending time on?
01:32:39 · Shawn Wang, Jamin Ball
[01:32:39] S1: Yeah. Exactly. Exactly. Any other commentary when you don't spend time on, you know, the labs? What are you what are spending time on? [01:32:45] S0: Oh, man. It's... Again, in a world that's highly unknown and unknowable and uncertain, I kind of fall back to what are my core beliefs that I do have conviction in, and that is the world's gonna consume more models, world's gonna consume more power, world's gonna consume more compute, networking, and storage, we're and gonna need to build a lot more data centers to serve the intelligence we're building. Those are kinda like my six core truths. And so, like, a lot of what I'm looking for are what are businesses that are a direct derivative of one of those one of those core beliefs that I have. [01:33:13] S1: Yeah. And any any other expressions of those core beliefs that you've done recently? [01:33:17] S0: Done recently. Yeah. Certainly. There's a chip company still in stealth that's super exciting. Maybe we can talk about later. K. There's, you know, power company, American terawatt. They wanna build a new grid, an industrial grid to serve industrial offtake. Big, huge, bold vision. Yeah. We talked about Elorean and glimpse and kind of the visual thinking. [01:33:39] S1: Anything on the compute market side? So effectively, like, slicing out compute in order to increase the efficiency. Trading futures trading futures on compute almost. [01:33:49] S0: I haven't... I I haven't spent as much time there. Yeah. I haven't spent as much time there. [01:33:53] S1: You know, we spent some time with Anj who's
01:33:53 · Shawn Wang, Jamin Ball
[01:33:53] S1: You know, we spent some time with Anj who's [01:33:56] S0: Okay. [01:33:56] S1: Doing National Compute and and AmpGrid and Yeah. And SF Compute and all those those guys. In in New York, we launched a compute trading track where we're we're... Like, mostly it's just the increased utilization. It's the broader goal. Yeah. But that's how you can magically create a whole bunch more compute, which is better use the ones we already have. [01:34:13] S0: Totally. [01:34:13] S1: And so, know, if the MFU or whatever choice of utilization is at 30 to 40% right now, taking up to 60%, you've doubled the number of GPUs you have. [01:34:23] S0: Yeah. Totally. [01:34:24] S1: And so I [01:34:25] S0: I mean, I think there's gonna be so much cool innovation. It's why, like, this is the most exciting time to be in tech, to be investing, to be around all these startups. Because, like, you're just... You're seeing innovation at the silicon layer. You're seeing innovation at the capital markets layer. You're seeing innovation at the product. I mean, like, it's like every... Creative [01:34:42] S1: destruction everywhere. I noticed you're not doing much on the application layer. [01:34:46] S0: You know, look, I I would [01:34:47] S1: say agent layer.