Venture with Grace
00:04:48 · Debajyoti Datta
[00:04:48] S1: Oh, For us, inference at scale, this context for real time conversation is super relevant. Imagine you're talking to our agent and our agent is saying a word, that, when you ask how are you doing today and the agent says, hey, am... And there's a just a pause above six hundred milliseconds. Humans don't talk like that. And now you are suddenly worried over phone. Have you disconnected the call? Is this person still there? And you've lost trust, because now it doesn't sound natural enough. So the thing that most people miss about voice latency is that the p ninety ninth latency matters. If you have hundreds of turns in a conversation, every single time you'll hit your p 99 in every call. And you have to be very precise because you cannot sound weird. We do not suddenly pause. We do not have sudden inflections in our speech. So it has to sound natural. It cannot be over excited about a topic or under excited. So this is why real time voice conversations is so tricky.
00:05:51 · Aaron Jacobson
[00:05:51] S2: Yeah. When I'm looking at, real time, I think it is a game of latency. And, for me, when I'm looking at companies or advising our companies, it's really all about what is your underlying inference stack look like. If you look in the world of inference, we're starting to see a separation from like general purpose GPUs, which are, you know, good at a lot of things, but not great at everything. And you're starting to see this splintering where folks are now, you know, basically shifting into dedicated GPUs for, for different parts of the inference chain. And the reason folks are doing that is in order to speed up inference, so you can actually drive latency much faster. Because if you separate to, basically, if you separate from pre code, from pre fill into decode, you can now optimize the hell out of the decode layer to try to drive as many tokens as possible. And a lot of latency is there because you have this k v catch, which is basically the met... Basically, you know, dependent on the memory. You can only load so much of that into your, into your chip. So you gotta optimize the hell out of the hell out of that in order to get as much data in and out of the GPU as fast fast as possible. As you get more users, your k v... You're in the the context window expands, the k v cache is expanding, that problem just starts to get really, really, really hard in terms of actually delivering on that real time, inference at low latency. And so there's a ton of opportunities to optimize at the hardware layer, the software layer. That's why I think these two companies are super interesting because they have to really optimize all the way down the chain in order to deliver on that latency for their end users.
00:07:18 · Grace Gong
[00:07:18] S0: A question for Lance. So you're an investor of Groq together, Cohere, many others. So how are these AI company leverage edge and faster compute?
00:07:30 · Lance Matthews
[00:07:30] S3: Yeah. For sure. I mean, Groq, that was the name of the game. I mean, a lot of what we're talking about here is speed and like does speed matter? And like if you know, think of your own experiences operating on any website, seconds, milliseconds matter in terms of how much attention you're paying to something, how excited, how much you well liked, you like the product itself, and so speed at the end of the day matters. Like I recently posted on LinkedIn a conversation I had with Sunny, the COO of Groq, and it was you know one of our first conversations and, I tried to get really cute and I was like, hey there's a bunch of multi agent reasoning in which these processes are gonna do complex calculations over time like like does being fast actually matter? And he had a really good retort which was like, open up your iPhone like do you like it more or less if it takes longer to boot up? And I was like, that's a pretty good point. I definitely like it way less. And so speed matters and specialization matters and so to the point that Aaron was just making is like, now we're in this world of specialization. The specialized chips, specialized hardware, model, all to decrease that piece around speed. Right? User satisfaction and sometimes it's mission critical, like in healthcare situations, and sometimes it's just a much better performance like what Alberto is building in terms of real time models in which you want an extremely smooth enjoyable experience with, with the model itself or with the game itself or with, you know, whatever you're kind of building.
00:08:51 · Grace Gong
[00:08:51] S0: Aaron and Lance, you guys are investors. So on our last investor panel, people were saying inference is actually under invested. Do you guys agree or no?
00:09:03 · Aaron Jacobson
[00:09:03] S2: Yeah. I actually do agree just because I think it's gonna be the world's largest market we've ever seen. Right? I mean, inference is ultimately, knowledge work. And as you think about the amount of times that I use AI every day, and what it actually takes to deliver that kind of experience to, you know, the 9,000,000,000 people worldwide, it's just gonna be a massive infrastructure build out. I mean, are talking about, you know, trillions of dollars of CapEx that needs needs to be spent. And if you think about the amount of money that went to the ground to build, fiber to get Internet access, you know, it took ten years, right, to get Internet access to the world. But to actually deliver AI real time, you know, super intelligence to everyone, it's gonna take far more invested into power, into GPUs, than I think we can ever even imagine. So I do actually think it is under invested. And there's still a lot of opportunity. So not only is under invested in terms of dollars that are gonna go in in order to come out the other side that produces real value, but there's just so much opportunity to continue to drive down the cost of inference and optimize it. And then we also have to basically figure out how we're gonna solve for the biggest issue, which is power. Power and politics I think are honestly the two biggest issues today in inference. But at least in power, we need a ton more, investment. Not only the power itself, but how do we actually shift, the basically the the cost per token, to drive it down. And currently today, if you look at how the models operate, they're multiple of orders magnitude greater than the human brain. The human brain is like one of the most efficient compute engines in the world. And so I think ultimately opportunity to drive inference to try to use low, power to be as power efficient as the human brain, and we're nowhere there near there yet. And so that's the opportunity, and that's why there's a lot more, dollars and, challenges to be solved in the world of inference.