The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:43:19 · Harry Stebbings, Chase Lochmiller
[00:43:19] S1: I agree with you. My question to you is we see these swarms of rogue agents doing things that are nefarious. We see whole industries that are being put to question. My girlfriend's a lawyer. Six months ago, she didn't use the legora. Now she hasn't written a document in five months. There is just such uncertainty across everything. We have RSI that could [00:43:36] S0: come and and change absolutely everything. So, sure, I get the comment that there's uncertainty around what the future of work will look like in certain ways, but I also feel like it's it's actually not uncertain that AI is gonna be a critical underlying engine for work and productivity that is gonna shift things quite a bit... Quite dramatically. And so it's more about that there is certainty that AI is establishing capabilities that are at human level or greater than human level in terms of overall productive capacity. So sure. Is that gonna change things? Yes. That is gonna change things. So so there's uncertainty in the change, but not... To me, it's certain that there will be change. Maybe that's a different way of putting it. [00:44:18] S1: Can I ask, we we mentioned the managed inference being a great part of your business? [00:44:22] S0: Mhmm. [00:44:22] S1: Gavin Baker, one of your investors, has said that the lowest cost producer of intelligence wins. When you think about that, like, what's the unit? Is it, like, dollars per token, per training run? What is that unit fee?
00:44:35 · Chase Lochmiller
[00:44:35] S0: Dollars per token is a is a good measure in terms of this, but it's it's not necessarily... You know, there is token efficiency, so, you know, just not all tokens are created equal. So it's not like every single token has the same, you know, unit of value, but I think it is a good indicator. I do think that, you know, there are different things that folks optimize for in terms of utilization, whether it's if throughput, so the amount of tokens per second you're able to generate for for for a very specific model use case, as well as time to first token, so latency, time to first token, time to last token, you know, how quickly you're able to actually serve this. And a lot of that comes down to it's a confluence of of different things. Right? So there's, you know, what is the cost that go into that? Obviously, by being vertically integrated and owning the infrastructure, you're able to stack a lot of margin across the process. So it gives you more flexibility in terms of how you actually, you know, bill for that. Additionally, I think utilization of the GPU itself is a critical aspect of being able to deliver tokens ultra efficiently and shine on metrics like latency and throughput. And so how does that actually manifest? Well, you know, the GPU is actually the most valuable thing in the entire data center, right? It is like the most expensive thing. So it's the thing you want to make sure you're keeping busy. If it's sitting idle, that's just money burning, right? If you think about that from that perspective, I think one of the key aspects is actually the management of memory and the management of the, you know, of the KV cache specifically. So the KV cache is the key value cache. It's really like, you know, you can kind of think about, you know, when tokens are fed into a a large neural network, you can compute the output tokens by basically running this feed forward process and running all these matrix multiplications that takes time. You may also already know the answer of that output token before you put the... So if you... If you've actually put put those same tokens in before, you actually know the answers and you can look them up in this look up table. The KBCash can get quite large and so it can expand well beyond the amount of, you know, memory you have on chip in the HBM.
00:46:39 · Chase Lochmiller, Harry Stebbings
And so actually being able to manage that both across multiple different GPUs, HBM layers, as well as system memory, DRAM, as as well as things like NVMe. So any any sort of, you know, solid state drive that you have on the system, and then sort of expanding that into the object storage layer, of your cluster. And being able to really, really efficiently move data around is actually a critical aspect to being able to serve inference and be able to keep those GPUs busy so that you're able to really shine on those metrics of latency and throughput. [00:47:08] S1: What does no one see about this layer of the value segment that you provide that everyone should see? What do we not know? [00:47:15] S0: Well, I think the big aspect is that not all inference providers are created equal. I mean, you have like great open source projects like SGLANG and VLLM. But, you know, there are ways to... You know, those are very generalized or generalizable. [00:47:31] S1: When you look at like an inference provider like Fireworks. [00:47:33] S0: Fireworks is great. They've done an incredible, incredibly great job. [00:47:36] S1: To what extent do they compete with you and cannibalize your managed inference business?
00:47:36 · Harry Stebbings, Chase Lochmiller
[00:47:36] S1: To what extent do they compete with you and cannibalize your managed inference business? [00:47:41] S0: Look, I think there are multiple layers across the stack that Crusoe competes with. What do they have that you don't by being specialized? I think there are certain workloads that, you know, Fireworks, you know, does an incredibly great job serving. I think, you know, the user experience, I think, is something that is unique across all of these different inference platforms that, you know, people may like more on on one than the other. There's a suite of different tools that, you know, people are... You know, there... There's a whole bunch of different features that people basically provide across, you [00:48:12] S1: know, the producers. You're, I wish I was them because then we could do this.
00:48:12 · Harry Stebbings, Chase Lochmiller
[00:48:12] S1: know, the producers. You're, I wish I was them because then we could do this. [00:48:17] S0: I sort of come back to this notion of, you know, the oil and gas analogy analogy where where the... If you look at the value chain, there are different places where you can compete and there are different places where you can collaborate. When Exxon is pumping wells at the wellhead, they may compete with another upstream oil and gas company to to, you know, most efficiently drill and operate wells. But they may sell to those same customers if they're building a midstream pipeline to transport oil from the wellhead to the refinery. They may provide capacity in that pipeline to other folks that they're competing with. And so, again, I I think it comes down to this perspective that, like, I just view there are so many more ways for us to work with folks across the AI infrastructure stack than compete with them. [00:49:01] S1: Can I just ask you, when you look at, like, open versus closed, given the purview that you have from the managed inference perspective, what do you see about distribution of dollars of tokens across open versus closed given that unique perspective you have?
00:49:01 · Harry Stebbings, Chase Lochmiller
[00:49:01] S1: Can I just ask you, when you look at, like, open versus closed, given the purview that you have from the managed inference perspective, what do you see about distribution of dollars of tokens across open versus closed given that unique perspective you have? [00:49:15] S0: I do think that people are spending more money on closed source frontier models than they are on open source, but they are generating more tokens on open source than closed source. We think open source is actually gonna be very important, very valuable. I think it's important for... From a data sovereignty standpoint, folks wanting to own their own models, own their own intelligence. I think there's far more private data that's untapped that can actually lead to massive performance gains. [00:49:42] S1: Do you think we will exist in a world where many companies, both mid market and large enterprise, will have their own models with their own data, and that will actually cannibalize a lot of frontier model business? [00:49:53] S0: I think it's possible, but I do think there's always gonna be demand for the frontier. Like the frontier is [00:49:58] S1: I'm not saying there's not, but you're looking at your Harvey's and your Ram's and your... You name it. And you're an investor in McCall, we're an investor in Fireworks. And both of those enable ads. They propagate that as a marketing message for sure. And I'm just wondering, is that true?