Venture with Grace
00:03:13 · Grace Gong, Alberto Taiuti
[00:03:13] S0: I want to start with Alberto since like you mentioned about like real time data and real time model. So like I guess like, what is it different from the traditional model building experience? [00:03:25] S4: Yeah. So when you're doing real time, one of the underappreciated aspects of it is that, latency matters a lot. And when latency matters, there are both physics based and software based constraints that oftentimes are overlooked and people don't optimize for them. So... But if you take real time and and the latency seriously, then a lot of your decisions around what hardware you choose, where you choose it, how you build the whole stack become very important and you also are going to do it very differently from what has been done so far with respect to say, text generation, image generation, and video generation. So I suppose that in terms of inference, effectively, whole stack looks looks pretty different from what other companies have built today. For example, a Together AI or a file even. And so you have to make certain decisions that maximize, sorry, minimize latency actually instead of maximizing throughput. And as always in technology, if you pull if you pull a string one way, you lose some advantages on the other end. So you have to pick and choose what what you're doing. And in this case, it would be you maximize you minimize latency so that... But you don't have maybe the maximum throughput, and that's a trade off you make.
00:04:48 · Debajyoti Datta
[00:04:48] S1: Oh, For us, inference at scale, this context for real time conversation is super relevant. Imagine you're talking to our agent and our agent is saying a word, that, when you ask how are you doing today and the agent says, hey, am... And there's a just a pause above six hundred milliseconds. Humans don't talk like that. And now you are suddenly worried over phone. Have you disconnected the call? Is this person still there? And you've lost trust, because now it doesn't sound natural enough. So the thing that most people miss about voice latency is that the p ninety ninth latency matters. If you have hundreds of turns in a conversation, every single time you'll hit your p 99 in every call. And you have to be very precise because you cannot sound weird. We do not suddenly pause. We do not have sudden inflections in our speech. So it has to sound natural. It cannot be over excited about a topic or under excited. So this is why real time voice conversations is so tricky.
00:05:51 · Aaron Jacobson
[00:05:51] S2: Yeah. When I'm looking at, real time, I think it is a game of latency. And, for me, when I'm looking at companies or advising our companies, it's really all about what is your underlying inference stack look like. If you look in the world of inference, we're starting to see a separation from like general purpose GPUs, which are, you know, good at a lot of things, but not great at everything. And you're starting to see this splintering where folks are now, you know, basically shifting into dedicated GPUs for, for different parts of the inference chain. And the reason folks are doing that is in order to speed up inference, so you can actually drive latency much faster. Because if you separate to, basically, if you separate from pre code, from pre fill into decode, you can now optimize the hell out of the decode layer to try to drive as many tokens as possible. And a lot of latency is there because you have this k v catch, which is basically the met... Basically, you know, dependent on the memory. You can only load so much of that into your, into your chip. So you gotta optimize the hell out of the hell out of that in order to get as much data in and out of the GPU as fast fast as possible. As you get more users, your k v... You're in the the context window expands, the k v cache is expanding, that problem just starts to get really, really, really hard in terms of actually delivering on that real time, inference at low latency. And so there's a ton of opportunities to optimize at the hardware layer, the software layer. That's why I think these two companies are super interesting because they have to really optimize all the way down the chain in order to deliver on that latency for their end users.
00:07:18 · Grace Gong
[00:07:18] S0: A question for Lance. So you're an investor of Groq together, Cohere, many others. So how are these AI company leverage edge and faster compute?
00:07:30 · Lance Matthews
[00:07:30] S3: Yeah. For sure. I mean, Groq, that was the name of the game. I mean, a lot of what we're talking about here is speed and like does speed matter? And like if you know, think of your own experiences operating on any website, seconds, milliseconds matter in terms of how much attention you're paying to something, how excited, how much you well liked, you like the product itself, and so speed at the end of the day matters. Like I recently posted on LinkedIn a conversation I had with Sunny, the COO of Groq, and it was you know one of our first conversations and, I tried to get really cute and I was like, hey there's a bunch of multi agent reasoning in which these processes are gonna do complex calculations over time like like does being fast actually matter? And he had a really good retort which was like, open up your iPhone like do you like it more or less if it takes longer to boot up? And I was like, that's a pretty good point. I definitely like it way less. And so speed matters and specialization matters and so to the point that Aaron was just making is like, now we're in this world of specialization. The specialized chips, specialized hardware, model, all to decrease that piece around speed. Right? User satisfaction and sometimes it's mission critical, like in healthcare situations, and sometimes it's just a much better performance like what Alberto is building in terms of real time models in which you want an extremely smooth enjoyable experience with, with the model itself or with the game itself or with, you know, whatever you're kind of building.
00:08:51 · Grace Gong
[00:08:51] S0: Aaron and Lance, you guys are investors. So on our last investor panel, people were saying inference is actually under invested. Do you guys agree or no?