Unsupervised Learning with Jacob Effron
00:12:45 · Erik Bernhardsson
[00:12:45] S1: I I think we're still pretty early with inference. I I think, you know, a lot of stuff is like... It's it's like one of those, like, very empirical sciences. Like, I think people are trying different things and seeing what works. I mean I mean, over time, like, people have kinda figured out a lot of stuff. You know? Take something like disaggregate it prefilled. Right? Like, it's a very nonobvious idea. Like, you you sort of, you know, do the prefill on on on a set of notes and then you copy the data over RDMA and do the decode and kinda swap. There's some, like, very non obvious things, but, we've invested a lot in those types of things. The KV caching, there's a lot of complexity around, like, KV caching. Of course, like running more efficient kernels, that's something people do a lot. There's a lot of these things that, you know, we we have a team of, I think, 20, 30 people right now who are extremely focused on just like LM performance optimization. And, you know, it takes a lot of work. You have to stay up with with research, you have to read papers, you have to look at all the software coming out all the time and try try and and see what works, what doesn't work. It's still kind of an empirical science. Over time, I think, you know, patterns will emerge. I think also the models themselves will change very quickly. Like, now we have, like, mixture of experts and more complex things over time. Like, who knows? Like, maybe we'll swap out the transformers for something else and, you know, and that would change how, like, kvcaching works. Like, there's all these things, like, that could, you know, change overnight if someone figures out, like, a very different paradigm for all these things. And maybe they will, maybe they won't. I don't know. There's also obviously a lot of stuff around quantization and and and other things. So so, yeah, it's it's it's it's a wild world out there. But but, yeah, we're we're we're we're playing around with all these things.
00:14:20 · Jacob Effron, Anne Maris
[00:14:20] S0: But it feels like there's so many folks that are working on, you know, different parts of of of improving inference. And, obviously, what's, I think, been really interesting in the last two, three years is a lot of those optimizations and improvements have been in open source, and for the broader community, maybe shared via research. As a CEO, how do you think about the level of... The kinds of problems you want your team to go solve versus it's like, hey, we're gonna benefit from just the entire community pushing this space Hi. [00:14:46] S3: I'm Anne Maris, and I'm the producer of Unsupervised Learning. Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your listenership, and see you next episode. [00:15:02] S0: As like a CEO, how do you think about the level of... The kinds of problems you want your team to go solve versus it's like, we're gonna benefit from just the entire community pushing this space
00:15:02 · Jacob Effron, Erik Bernhardsson
[00:15:02] S0: As like a CEO, how do you think about the level of... The kinds of problems you want your team to go solve versus it's like, we're gonna benefit from just the entire community pushing this space [00:15:12] S1: forward? We actually love open source because I think a lot of our alpha is is in the proprietary side, in the infrastructure layer, like all this cool stuff around auto scaling for instance. I think unlike other inference providers, we've actually leaned very heavily into open source. We work very closely with both the VLM team and the SGLang team. We contribute very actively to both those repos and do a lot of optimizations with them. And and so I I think this may be a little bit of a different approach than than maybe other inference providers. Of course, there's gonna be, you know, certain things that, you know, we we keep internal, but but almost everything we we push to to open source. And I and I think it's, you know, it's because, like, we have so much alpha in the infrastructure anyway. Like, we might as well just, you know, work with the amazing open source engines that are out there that maybe were... You know, they weren't state of the art, like, you know, a few years ago, but now they are. And and and that's an incredible thing that, like, we'd love to leverage, frankly.
00:15:12 · Erik Bernhardsson, Patrick Chase
[00:15:12] S1: forward? We actually love open source because I think a lot of our alpha is is in the proprietary side, in the infrastructure layer, like all this cool stuff around auto scaling for instance. I think unlike other inference providers, we've actually leaned very heavily into open source. We work very closely with both the VLM team and the SGLang team. We contribute very actively to both those repos and do a lot of optimizations with them. And and so I I think this may be a little bit of a different approach than than maybe other inference providers. Of course, there's gonna be, you know, certain things that, you know, we we keep internal, but but almost everything we we push to to open source. And I and I think it's, you know, it's because, like, we have so much alpha in the infrastructure anyway. Like, we might as well just, you know, work with the amazing open source engines that are out there that maybe were... You know, they weren't state of the art, like, you know, a few years ago, but now they are. And and and that's an incredible thing that, like, we'd love to leverage, frankly. [00:16:07] S2: You know, you were talking about heterogeneous compute and the the different places where you're getting compute from and also how the, you know, kind of inference stack is changing. How important are, you know, the long tail of chips to you guys, the non NVIDIA chips and and things like that as you think about your supply or kind of what the... What your serving stack looks like over time?
00:16:29 · Erik Bernhardsson
[00:16:29] S1: I I mean, I think NVIDIA is still like the king here. Like, they they have the best chips. There's way more demand for NVIDIA versus anything else. Like, you know, we're we're, you know, curious about some of the other chips. Like, you know, we made some experiments, like, taking a look at it. But but NVIDIA definitely has the best chips. But, you know, like, looking at AMD, Tranium, TPUs, etcetera, like, starting to be, like, pretty good. So so we're always curious about that. And then then, of course, there's there's a lot of, like, older chips too. And and I I think the the challenge has been, like, for a lot of the LMs, like, you're kinda constantly pushing, like, you know, how many parameters can you fit. So so, typically, latest and greatest LMs, like, they will always require, you know, a a full node, maybe even in the future, even multiple nodes. Who knows? So, like, eight b three hundreds. Like, they will require the latest chip and, like, eight of them because that's how many you can fit on a node. But but there's plenty of models outside of that, like older LMs, smaller ones that have... You know, they're they're optimized for performance might be fewer parameters. And then outside language, there's there's a lot of stuff around, voice and and, you know, audio transcription or speech synthesis or or, like, you know, pharma, like, bio stuff. Like, a lot of that stuff actually... They don't require, like, the latest and greatest GPUs. Like, they actually get more cost efficient serving on on hoppers or or even m pairs or, like, you know, different types of, like, l 40 x or RTX Pro 6,000, stuff like that. So we actually... I mean, we are ramping up, like, very, very aggressively on on Blackwalls, primarily GBs now. But we we see actually, like, plenty of demand also in the older chips, which I think is quite exciting. There's just so much stuff going on. People forget about the hoppers, but, like, the... They... They're our most common type right now still at modal, like, h one hundreds. We run some very, like, large production use cases based on hoppers.
00:18:19 · Jacob Effron
[00:18:19] S0: How efficient, like, is model to chip matching today, like, know, in the industry at large? Do you feel like most people are using the, like, optimal, you know, chip or or, you know, from a price perspective for what they're what they're working, or is there still a lot of room for improvement there?