No Priors: Artificial Intelligence | Technology | Startups
00:00:00 · Noam Brown, Sarah Guo
[00:00:00] S0: With GBT three, you couldn't scale test time compute. Like, if you gave it a budget of $10,000,000 and said, okay. Well, let's see what GBT three can do. It really can't do that much. The precarious frameworks and responsible scaling policies, they don't really account for the amount of test time compute. They just say, okay. Well, what's the capability of the model? The problem is we're in a world now where the capability of the model is a function of how much money you put into it, basically. If you give it a budget of $10,000, it can do a lot more than what it can do with a budget of $10. Give it a budget of $10,000,000, it can do even At what budget should you evaluate these models? The policies that exist today don't really address that question. [00:00:43] S1: Hi, listeners. I'm Sarah Gore, and welcome back to No Priors. Today, I'm here with Noam Brown, one of our godfathers of AI reasoning. We talk about the broken state of evaluations, very large scale test time compute, how he thinks about recursive self improvement, and what's next on the horizon for competition at the frontier. Welcome. Noam, I'm so excited to have you back. [00:01:05] S0: That's great to be back. Yeah. [00:01:06] S1: You are our first guest. I'm very proud of my taste in friends and researchers for the pod, given, you know, how important, you know, inference time scaling has become to the industry. You should be proud too, having actually pioneered [00:01:21] S0: it. Played apart, yeah, among many others.
00:01:21 · Noam Brown, Sarah Guo
[00:01:21] S0: it. Played apart, yeah, among many others. [00:01:24] S1: You just wrote this essay that really resonated about large scale test time compute and why the industry is not evaluating these models as robustly as it should be. What was the motivation for it?
00:01:37 · Noam Brown
[00:01:37] S0: Yeah. The motivation was we released 5.5, and the initial reaction was kinda skepticism that it was a substantially better model. This to be fair, that only lasted for a few hours before people had some time to play around with it and and try it out themselves, and they saw that it was actually substantially better. But I think a lot of the skepticism came from the benchmark grid that was published. Basically, whenever a new model is released, there's this benchmark grid where they show all these different benchmarks on on the x axis and then the performance of different models on the y axis, and you can just, like, compare different models. It's like a single number for a model on a single benchmark. And if you look on paper at the difference between, like, five point five and five point four or or other models, it wasn't it was an improvement, but it wasn't a huge improvement. It was only a few percentage points on some benchmarks. So people looked at that, and they were skeptical that it was actually a better model. Once they played around with it, the story changed. I think the reason why it doesn't show up as so much better on the benchmarks is because the benchmarks are being presented the benchmark results are being presented in the wrong way. They're not controlling for the amount of test on compute that is being used on that benchmark question. It turned out that 5.5 is just much more efficient with its thinking. If you run it at max settings, 5.4 is thinking for a lot longer. It takes longer to get back a response than 5.5. And once you control for the amount of thinking time, actually, you can see that 5.5 is a substantial jump over 5.4. That is, I think, people's day to day experience with it. And then when I mention this to people, the reaction the typical question I get is like, okay. Well, why not just have 5.5 think for as long as 5.4? And the question is like, well, how long should they think for? Typically, the response I get is, well, until the performance plateaus. Right? This is at some point where the performance on the benchmark is gonna plateau, and you just evaluate to that point. The thing is, the point at which it plateaus is actually really far out these days.
00:03:28 · Noam Brown, Sarah Guo
I mean, if it's true in g p d three land back in 2022, models couldn't really think productively for that long, and so you could just run them until they plateau. It's not that far away. But what we're seeing today with the modern models is that 5.5 and other models can think for if you scaffold them reasonably well, can think for weeks even, before having performance plateau on some of these benchmarks. And so the point at which they plateau is simply too far out to reasonably test. [00:03:52] S1: We all need to actually reinforce either, like, a patience limit or a budget limit from a token perspective now, and that wasn't true a few years ago. [00:04:00] S0: Exactly. And so I think the proper way to and so my claim is the proper way to evaluate the models now is you either have some kind of budget for the benchmark, whether it's tokens or cost or time or whatever, or you plot the performance as a function of the amount of test time compute that's going into the model. And then it becomes much more clear how to compare the performance between these different models. [00:04:20] S1: Given the model evaluation cycle and the fact that performance does not asymptote for many tasks over quite a long period of time, what do you do about that issue, the fact that some of the evals that you would want to run are both beyond the scope of budget or time that's reasonable given the current model release cycle?
00:04:20 · Sarah Guo, Noam Brown
[00:04:20] S1: Given the model evaluation cycle and the fact that performance does not asymptote for many tasks over quite a long period of time, what do you do about that issue, the fact that some of the evals that you would want to run are both beyond the scope of budget or time that's reasonable given the current model release cycle? [00:04:40] S0: I mean, think for things like cyber, we've seen, and actually the AISI in their evaluations has shown that the models continue to improve at a 100,000,000 tokens. You know, if you run them for a 100,000,000 tokens, they're still improving at beyond that point. Mhmm. And that can take a very long time to run. But you also do see that, like, the performance is is it's not just, like, a discontinuous jump. It's actually, like, you can see the slope of improvement over those 100,000,000 tokens. And so you you could probably do some kind of evaluation up to a certain budget and then just say, okay. Well, this is what we project the performance to look like. And I think this this there hasn't been a lot of research on this yet. I actually think this would be a great paper to publish if there's any academics out there looking for something to research. Can you predict what the performance looks like at an inference budget of, let's say, $10,000, only using inference budgets up to 10 or 100 or 10 or $100? [00:05:34] S1: So maybe an orthogonal question for you. Do you think users are systematically, like, not thinking long enough with their models about problems? [00:05:42] S0: What do you mean by not thinking long enough?
00:05:42 · Noam Brown, Sarah Guo
[00:05:42] S0: What do you mean by not thinking long enough? [00:05:44] S1: If you can build an agent or control the amount of test time compute being used, like, there's what is done by the model itself, and there's what the user can do. Do you think that, you know, the industry is using test time compute an optimal amount, way undershooting it, or it's, you know, it's a problem in the models where they just need to be able to do that thinking faster? [00:06:08] S0: I I think it depends on the problem. I think this idea that the models, you just let them think for a week or whatever, then they respond. It's it sounds nice, and, yes, the benchmarks look great, but it's not very practical when working because, like, okay, you ask the model a question, then you sit there for a week waiting for it to come back to you. Mhmm. I think what people have have found most effective is to kind of, like, iterate quickly with the models. And so the thinking time, I think, needs to be flexible. When it makes sense to respond quickly to the user, it should respond quickly. And then when it makes sense to think for a long time and the user wants it to think for a long time, then it makes sense to think for a long time. I think people have been striking the right balance given what they have to deal with right now.