Venture with Grace
00:07:30 · Lance Matthews
[00:07:30] S3: Yeah. For sure. I mean, Groq, that was the name of the game. I mean, a lot of what we're talking about here is speed and like does speed matter? And like if you know, think of your own experiences operating on any website, seconds, milliseconds matter in terms of how much attention you're paying to something, how excited, how much you well liked, you like the product itself, and so speed at the end of the day matters. Like I recently posted on LinkedIn a conversation I had with Sunny, the COO of Groq, and it was you know one of our first conversations and, I tried to get really cute and I was like, hey there's a bunch of multi agent reasoning in which these processes are gonna do complex calculations over time like like does being fast actually matter? And he had a really good retort which was like, open up your iPhone like do you like it more or less if it takes longer to boot up? And I was like, that's a pretty good point. I definitely like it way less. And so speed matters and specialization matters and so to the point that Aaron was just making is like, now we're in this world of specialization. The specialized chips, specialized hardware, model, all to decrease that piece around speed. Right? User satisfaction and sometimes it's mission critical, like in healthcare situations, and sometimes it's just a much better performance like what Alberto is building in terms of real time models in which you want an extremely smooth enjoyable experience with, with the model itself or with the game itself or with, you know, whatever you're kind of building.
00:08:51 · Grace Gong
[00:08:51] S0: Aaron and Lance, you guys are investors. So on our last investor panel, people were saying inference is actually under invested. Do you guys agree or no?
00:09:03 · Aaron Jacobson
[00:09:03] S2: Yeah. I actually do agree just because I think it's gonna be the world's largest market we've ever seen. Right? I mean, inference is ultimately, knowledge work. And as you think about the amount of times that I use AI every day, and what it actually takes to deliver that kind of experience to, you know, the 9,000,000,000 people worldwide, it's just gonna be a massive infrastructure build out. I mean, are talking about, you know, trillions of dollars of CapEx that needs needs to be spent. And if you think about the amount of money that went to the ground to build, fiber to get Internet access, you know, it took ten years, right, to get Internet access to the world. But to actually deliver AI real time, you know, super intelligence to everyone, it's gonna take far more invested into power, into GPUs, than I think we can ever even imagine. So I do actually think it is under invested. And there's still a lot of opportunity. So not only is under invested in terms of dollars that are gonna go in in order to come out the other side that produces real value, but there's just so much opportunity to continue to drive down the cost of inference and optimize it. And then we also have to basically figure out how we're gonna solve for the biggest issue, which is power. Power and politics I think are honestly the two biggest issues today in inference. But at least in power, we need a ton more, investment. Not only the power itself, but how do we actually shift, the basically the the cost per token, to drive it down. And currently today, if you look at how the models operate, they're multiple of orders magnitude greater than the human brain. The human brain is like one of the most efficient compute engines in the world. And so I think ultimately opportunity to drive inference to try to use low, power to be as power efficient as the human brain, and we're nowhere there near there yet. And so that's the opportunity, and that's why there's a lot more, dollars and, challenges to be solved in the world of inference.
00:10:57 · Lance Matthews
[00:10:57] S3: Yeah. Just quickly, I mean, I think we're in the kind of still monolithic compute era, like, I was a history major back in the day at school, and like, this kind of makes me think of railroads, and like, not all... Every data center can be the size of Manhattan, like, that's just not gonna work. And so thus you need inference at scale, you need small distributed data centers that are providing this for people. And like the the railroad perspective, you know, one railroad stop would completely change the town and there was a bunch of investment that would kind of go and occur that, very transformational of course, but you know what's also very transformational was when there was roads built throughout these towns and commerce was able to occur and businesses in and around it. And so just using that analogy like quite loosely, that to me is inference in which this is like impacting real people doing real things to complete their job in a really efficient way. I think I think once you have inference at scale, or it has become sort of more of the norm, I think you could even potentially see sort of a change what we're seeing now on the on the politics side and all of the public response, which again like is literally just like the railroads. Like, you can... I don't know. Imagine these people standing with pitchforks and like an old western, you know, being upset about a data center kind of coming in. So... Yeah.
00:12:07 · Grace Gong
[00:12:07] S0: Debaj and Alberto, you guys are the builders in the, in the room. So I wanna talk about, like, what is some constraints in your particular sector that you're facing today? Since, you know, Debaj is in health care and then Alberto is in, visual or media space?
00:12:24 · Debajyoti Datta, Grace Gong
[00:12:24] S1: So, I'll, say a very quick incident that actually happened, which is, we have a a big health research institute that, had about 1,700 patients that were lost to follow-up. That basically means his health systems try to reach them out through multiple ways, phone calls, messages, emails, even physical letters, bots, and they failed. We took over this use case and we managed to reach 200 such folks. And then about one hundred and eighty of them got their lung CT scans. And nine of them, if they did not get the scans, they would have been back for with their stage four cancer diagnosis. This is a use case that just happened because of all these GPUs. So GPUs are so critical not just for simple ad clicks. Like straight up health and safety of the population, the entire states and societies. We are... Obviously, like, GPU shortage is a very easy answer, to say, but, like, I think GPU is in the right place, right data center that follows, if it's a HIPAA regulation, it has to be within the Continental US. And so all these weird nuances matter. And there's GPUs also need to be closest to the... How your infrastructure works so that you again maintain the latency. So I think this holistic infrastructure, which I think all these other investors and, touched on that, I think is a more critically missing thought piece. [00:13:58] S0: I wanna end with a hot take round. So in two years, what share of inference runs somewhere else other than hyperscalers?