No Priors: Artificial Intelligence | Technology | Startups
00:35:04 · Sarah Guo, Noam Brown
[00:35:04] S1: Okay. That's very principle of you, which is like, yes, routing is fine, but it's all subject to the same budget question. Yep. Right. If you put it on the same scaler, then you can make an optimal decision, and I think maybe I win. [00:35:15] S0: Mhmm. I I I don't even know necessarily that I I would believe that the routing does better, but then there's still a question of, is it gonna do significantly better? Is it very fragile? Is it reflective of real world use cases compared to benchmarks? Because, like, one issue you could run into is that you could optimize for certain benchmarks with the routing, and then show, like, oh, yeah. We see there's big improvement on these benchmarks. But in real world use cases, it actually ends up not being a significant improvement. So I I would say, at the very least, like, I would say, you wanna control for test on compute, and then you also wanna have all the same skepticism about benchmarks that you would normally have. [00:35:50] S1: Awesome. Noam, thanks so much, and and for being on the mission, for, breaking us out of this false equilibrium. [00:35:57] S0: Yeah. It's great to be back. [00:36:00] S1: Find us on Twitter nopriorspod. Subscribe to our YouTube channel if you wanna see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at nopriors.com.