Venture with Grace
00:00:55 · Aaron Jacobson, Alberto Taiuti, Grace Gong, Debajyoti Datta, Lance Matthews
[00:00:55] S2: Aaron Jacobson, partner at NEA, global venture capital firm with about 30,000,000,000 of assets under management. Also one of the oldest firms been around almost fifty years. I lead our investing in AI infrastructure, cyber security, developer tools. I have been fortunate to work with companies like Databricks, Together, Factory, Perplexity, which I'm gonna speak to in terms of what we're seeing, in terms of inference at scale. [00:01:20] S4: And I am Alberto. I am the CEO and co founder of Reactor, and we're building the inference platform for world models and real time video AI. Thank you for having me. [00:01:30] S0: I wanna start with a broader questions for the audience. So what does inference at scale meant for you? [00:01:40] S1: I can go first. So for us, this is a patient facing agent that calls patients. So, it is, ASR model, about 30 large language models that is running on hundreds of b 300 nodes, thousands of b 300 nodes, and then a TTS. So for me, inference at scale means all these thirty thirty five models need to work every single time under four hundred milliseconds for a smooth user experience, and we do billions of interactions a day. [00:02:13] S3: Yeah. I mean, think I'll take a little bit higher level, but I think it's just effectively AI working for people when they need it. You know what I mean? At the end of the day, it's it's you know, putting out a prompt to get an answer and sort of solving something sort of immediately for you. So, at the end the day, that's what I think inference is.
00:02:13 · Lance Matthews, Aaron Jacobson, Alberto Taiuti, Grace Gong
[00:02:13] S3: Yeah. I mean, think I'll take a little bit higher level, but I think it's just effectively AI working for people when they need it. You know what I mean? At the end of the day, it's it's you know, putting out a prompt to get an answer and sort of solving something sort of immediately for you. So, at the end the day, that's what I think inference is. [00:02:27] S2: Yeah. I see it as, inference scales all the optimization. So it's how do you get the capacity you need, but also how do you drive, the best token per dollar per watt across software, hardware, power? [00:02:44] S4: I think in our case, it's a little different. I think it is about really democratizing access to real time AI. And when it... I say democratizing because I think if you do inference at scale, then you're implying that you want to reach the widest amount of people in the world. And if that's your goal, then effectively you're trying to democratize access to something. And so I think doing that means really unlocking more benefit for more people and that's one of our goals as well. [00:03:13] S0: I want to start with Alberto since like you mentioned about like real time data and real time model. So like I guess like, what is it different from the traditional model building experience?
00:03:13 · Grace Gong, Alberto Taiuti
[00:03:13] S0: I want to start with Alberto since like you mentioned about like real time data and real time model. So like I guess like, what is it different from the traditional model building experience? [00:03:25] S4: Yeah. So when you're doing real time, one of the underappreciated aspects of it is that, latency matters a lot. And when latency matters, there are both physics based and software based constraints that oftentimes are overlooked and people don't optimize for them. So... But if you take real time and and the latency seriously, then a lot of your decisions around what hardware you choose, where you choose it, how you build the whole stack become very important and you also are going to do it very differently from what has been done so far with respect to say, text generation, image generation, and video generation. So I suppose that in terms of inference, effectively, whole stack looks looks pretty different from what other companies have built today. For example, a Together AI or a file even. And so you have to make certain decisions that maximize, sorry, minimize latency actually instead of maximizing throughput. And as always in technology, if you pull if you pull a string one way, you lose some advantages on the other end. So you have to pick and choose what what you're doing. And in this case, it would be you maximize you minimize latency so that... But you don't have maybe the maximum throughput, and that's a trade off you make.
00:04:48 · Debajyoti Datta
[00:04:48] S1: Oh, For us, inference at scale, this context for real time conversation is super relevant. Imagine you're talking to our agent and our agent is saying a word, that, when you ask how are you doing today and the agent says, hey, am... And there's a just a pause above six hundred milliseconds. Humans don't talk like that. And now you are suddenly worried over phone. Have you disconnected the call? Is this person still there? And you've lost trust, because now it doesn't sound natural enough. So the thing that most people miss about voice latency is that the p ninety ninth latency matters. If you have hundreds of turns in a conversation, every single time you'll hit your p 99 in every call. And you have to be very precise because you cannot sound weird. We do not suddenly pause. We do not have sudden inflections in our speech. So it has to sound natural. It cannot be over excited about a topic or under excited. So this is why real time voice conversations is so tricky.
00:05:51 · Aaron Jacobson
[00:05:51] S2: Yeah. When I'm looking at, real time, I think it is a game of latency. And, for me, when I'm looking at companies or advising our companies, it's really all about what is your underlying inference stack look like. If you look in the world of inference, we're starting to see a separation from like general purpose GPUs, which are, you know, good at a lot of things, but not great at everything. And you're starting to see this splintering where folks are now, you know, basically shifting into dedicated GPUs for, for different parts of the inference chain. And the reason folks are doing that is in order to speed up inference, so you can actually drive latency much faster. Because if you separate to, basically, if you separate from pre code, from pre fill into decode, you can now optimize the hell out of the decode layer to try to drive as many tokens as possible. And a lot of latency is there because you have this k v catch, which is basically the met... Basically, you know, dependent on the memory. You can only load so much of that into your, into your chip. So you gotta optimize the hell out of the hell out of that in order to get as much data in and out of the GPU as fast fast as possible. As you get more users, your k v... You're in the the context window expands, the k v cache is expanding, that problem just starts to get really, really, really hard in terms of actually delivering on that real time, inference at low latency. And so there's a ton of opportunities to optimize at the hardware layer, the software layer. That's why I think these two companies are super interesting because they have to really optimize all the way down the chain in order to deliver on that latency for their end users.
00:07:18 · Grace Gong
[00:07:18] S0: A question for Lance. So you're an investor of Groq together, Cohere, many others. So how are these AI company leverage edge and faster compute?