Latent Space: The AI Engineer Podcast
01:31:05 · Jamin Ball, Shawn Wang
And, like, I don't have line of sight to that slowing down for the next few years. [01:31:10] S1: Yeah. And Yeah. And and the limit, like, LAN, Shell Power, all these, but also the silicon, like, straight up Totally. Like, compute memory, every every part of the supply chain. [01:31:19] S0: Totally. [01:31:20] S1: That's the work. That's why 78,000 is so powerful right now. [01:31:22] S0: I mean, look at these recent Kimi models. Kimi K three. You know what's memory? You need gigabytes of memory just to serve that. You're not running that on your laptop. You know, you need like a rack to serve one model. Yeah. Again, are just the big models. Right? Like you're also gonna have tons of small models, but I know the big models are getting bigger, which only require more memory and more compute to run. [01:31:44] S1: I used to say they were capping out. I used to say like the max size is one to two, and then, like, we would just optimize from there. Yeah. I've been wrong. [01:31:50] S0: What's what's interesting is it does feel like the open weight models are capping more around, a 100,000,000,000 active parameters. [01:31:55] S1: That's right. Yeah. And It's mostly, like, an inference speed thing. Totally. But, like, I [01:31:59] S0: think Totally.
01:31:59 · Jamin Ball, Shawn Wang
[01:31:59] S0: think Totally. [01:32:00] S1: It's actually really interestingly tied to the generations of GPUs that they have. Yeah. Right? Exactly. You you, like, wait for the Vera Rubins to to roll out and, like, whatever China has. Yeah. Exactly. And and... Well, then then they'll get it. You know? Yeah. Totally. Then... So it it... They're really memory constrained at the inference layer, and that actually shapes the model architecture, which is, like, kinda cool. Totally. Oh, so that is Anthropic. Yes. Powerhouse tour. Congrats. I don't know how you felt coming into this, but you you did great. Oh, this is fun. Yeah. And, like, obviously, more to do and more to analyze, but I just definitely wanted on the podcast to do this Yeah. Because I I know you. [01:32:34] S0: Yeah. Of course. That's been [01:32:35] S1: And, like, when else We've [01:32:37] S0: been meaning do this for a while. Else are we gonna get this make it happen. [01:32:39] S1: Yeah. Exactly. Exactly. Any other commentary when you don't spend time on, you know, the labs? What are you what are spending time on?
01:32:39 · Shawn Wang, Jamin Ball
[01:32:39] S1: Yeah. Exactly. Exactly. Any other commentary when you don't spend time on, you know, the labs? What are you what are spending time on? [01:32:45] S0: Oh, man. It's... Again, in a world that's highly unknown and unknowable and uncertain, I kind of fall back to what are my core beliefs that I do have conviction in, and that is the world's gonna consume more models, world's gonna consume more power, world's gonna consume more compute, networking, and storage, we're and gonna need to build a lot more data centers to serve the intelligence we're building. Those are kinda like my six core truths. And so, like, a lot of what I'm looking for are what are businesses that are a direct derivative of one of those one of those core beliefs that I have. [01:33:13] S1: Yeah. And any any other expressions of those core beliefs that you've done recently? [01:33:17] S0: Done recently. Yeah. Certainly. There's a chip company still in stealth that's super exciting. Maybe we can talk about later. K. There's, you know, power company, American terawatt. They wanna build a new grid, an industrial grid to serve industrial offtake. Big, huge, bold vision. Yeah. We talked about Elorean and glimpse and kind of the visual thinking. [01:33:39] S1: Anything on the compute market side? So effectively, like, slicing out compute in order to increase the efficiency. Trading futures trading futures on compute almost. [01:33:49] S0: I haven't... I I haven't spent as much time there. Yeah. I haven't spent as much time there. [01:33:53] S1: You know, we spent some time with Anj who's
01:33:53 · Shawn Wang, Jamin Ball
[01:33:53] S1: You know, we spent some time with Anj who's [01:33:56] S0: Okay. [01:33:56] S1: Doing National Compute and and AmpGrid and Yeah. And SF Compute and all those those guys. In in New York, we launched a compute trading track where we're we're... Like, mostly it's just the increased utilization. It's the broader goal. Yeah. But that's how you can magically create a whole bunch more compute, which is better use the ones we already have. [01:34:13] S0: Totally. [01:34:13] S1: And so, know, if the MFU or whatever choice of utilization is at 30 to 40% right now, taking up to 60%, you've doubled the number of GPUs you have. [01:34:23] S0: Yeah. Totally. [01:34:24] S1: And so I [01:34:25] S0: I mean, I think there's gonna be so much cool innovation. It's why, like, this is the most exciting time to be in tech, to be investing, to be around all these startups. Because, like, you're just... You're seeing innovation at the silicon layer. You're seeing innovation at the capital markets layer. You're seeing innovation at the product. I mean, like, it's like every... Creative [01:34:42] S1: destruction everywhere. I noticed you're not doing much on the application layer. [01:34:46] S0: You know, look, I I would [01:34:47] S1: say agent layer.
01:34:47 · Shawn Wang, Jamin Ball
[01:34:47] S1: say agent layer. [01:34:48] S0: No. Sir sir... Certainly are. Again, like, I I I... I mean, I'm I'm I'm speaking more personally than necessarily on the the Altimeter behalf. Wanna believe there's a strong product research or combo. There's certainly a lot of great teams, and we think, you know, early on in market cycles, you know, we kind of use these like triangle analogies where you have compute or infra, like like hardware infra, software infra, software apps. And in a mature market, the biggest TAM is the apps. And the smallest TAM is hardware. Early on in cycles, it's inverted. And the biggest markets are at the both top or bottom of the triangle, more of the physical hardware, and then the infrastructure, and then the apps. And eventually the triangles invert, and so, hey, at some point in the future, I think the apps and the agents will be the biggest part of the TAM. We're just early on, and there's lots of opportunity in the other parts of
01:35:50 · Shawn Wang, Jamin Ball
[01:35:50] S1: Objectively, that is correct. Yeah. I I I do feel like... And and and by the way, you can see various versions of this in all of computer history. Yeah. Right down to, like, the Western Micro... West... Like, the... Like, when hard disks were were were getting built out or whatever. Anyway, so I I I don't think those are the best examples. Western Digital? Western Digital. Right? Yeah. Yeah. Yeah. Anyway, so they'll... There... Yeah. I I I think that is true, but, like, to me, domain specificity is very interesting. And to me, the app layer, the agent layer is supposed to capture that. Yeah. They... They're just like... The biggest outcome here is cursor Yeah. Right now. Yeah. Yeah. Yeah. And that's 60,000,000,000. We're Yeah. Here talking about $2,000,000,000,000. [01:36:27] S0: Yeah. Exactly. [01:36:28] S1: So you're very different. [01:36:29] S0: Yeah. Totally. Totally. I think I also just, like, skew a little bit more infra than... You know, just personal bias. [01:36:33] S1: Yeah. Yeah. We... We're we're infra guys at heart. [01:36:35] S0: Yeah. Exactly. [01:36:36] S1: Okay. Great. My closing question was gonna be, Friedrich, Brad's story. I mean, Brad's a character. Yeah. And you've you've worked with him for like, I don't know, three, four years now. Six years. Six years. Yeah. Okay. So so Call of Judgment ends. Okay. Alright. [01:36:50] S0: Is Right around. Yeah. Yeah. [01:36:51] S1: I I feel like I'm getting my time... Either my timeline is wrong or I met you way earlier than I thought I met you.