No Priors: Artificial Intelligence | Technology | Startups
00:12:23 · Sarah Guo, Noam Brown
[00:12:23] S1: Mhmm. [00:12:24] S0: But the question is, okay. Well, how do you evaluate whether the model is capable of that? And they have, like, various protocols about, like, how they do these evaluations. But a lot of these frameworks were developed around the era of ChatGPT, either before or after, when test time compute scaling was not really as much of a thing. Mhmm. And it made sense. Like, with GPT three, you couldn't scale test time compute. Like, if you gave it a budget of $10,000,000 and said, okay. Well, let's see what g p t three can do, it really can't do that much more than what you could do with, like, $10 or $1. The preparedness frameworks and responsible scaling policies, they don't really account for the amount of test time compute. They just say, okay. Well, what's the capability of the model? The problem is we're in a world now where the capability of the model is a function of how much money you put into it, basically. [00:13:08] S1: Mhmm. [00:13:08] S0: If you give it a budget of $10,000, it can do a lot more than what it can do with a budget of $10. If you give it a budget of $10,000,000, it could do even more. And so at what budget should you evaluate these models? The policies that exist today don't really address that question. [00:13:25] S1: Mhmm.
00:13:25 · Sarah Guo, Noam Brown
[00:13:25] S1: Mhmm. [00:13:25] S0: Some do some do better than others, but for the most part, this is not really a factor that's being heavily considered. Now whether it should be released anyway, I don't I don't wanna wade into this question. I think there's, you know, there's arguments on both sides. But I think the important thing to recognize is that this is a question that is not being we're we're just kind of, like, you know, pretending that this issue doesn't exist, and I think it's important to just, you know, one way or the other account for it. [00:13:50] S1: Yeah. It was the mirror image of the capability question of if the models can continue to do more and more without asymptoting on some tasks at very large budgets, then they should also be able to do so for tasks we don't want them to do as a society. Right? And so testing for that and what budget is allocated, it also seems out of sync from the model release cycle itself. Right? There's been this acceleration of, you know, you get a new model every sometimes few days and weeks at this point versus six months. And you have a line in the essay where you say, like, the the only way to truly evaluate an agent on some very long running task might be to run it for a year, and that's gonna be true of both, like, useful and negative tasks. Right? And so how do you think about that versus the model release cycle? [00:14:44] S0: Yeah. This this is also an interesting dynamic where, basically, as the models have become stronger, they've they're more they're better able to operate over longer horizons.
00:14:44 · Noam Brown, Sarah Guo
[00:14:44] S0: Yeah. This this is also an interesting dynamic where, basically, as the models have become stronger, they've they're more they're better able to operate over longer horizons. [00:14:53] S1: Mhmm. [00:14:54] S0: So, again, with g p d three, if you wanted to run it for, you know, a week, there's really not much you could do to scaffold it into something useful that could actually run for a week. But we're seeing now with the most recent models that you can actually scaffold, for example, 5.5 into doing a series of experiments that can run for weeks, for months. [00:15:12] S1: Have you given your poker solver task, like, infinite budget yet? [00:15:19] S0: I haven't really scaffolded something together where I just tell it, like, okay, just run this for for weeks. I think I could it I I Until it has [00:15:27] S1: some types.
00:15:28 · Noam Brown
[00:15:28] S0: Could probably give it slash goal and just, like, yeah, tell it to go nuts. But I I think at this point, it could 100% do the reverse solver if I just give it slash goal. I don't think it's at the level yet where it could do, like, the full poker solver if I gave it just slash goal and told it, yeah, go go run for a month. But we're going to pretty soon be at that point where I I probably could just tell it, like, yeah, go work on this for a month, and then come back to me with a a full, complete PokerSolver that's state of the art. And the problem is if you want to evaluate the capabilities of a model, what it can do after running for a month, the only way to be fully sure is to actually run it for a month. And if you wanna know after six months, the only way to know fully is to run it for six months. Now there I'll I'll get to, like, things we could do to address that a little bit later, but, like, it's important to recognize that the model release cycle is look. We're releasing new models, like, every two or three months at this point. And so a model comes out, it takes two or three months to push it to its limits, and then you have another model come out. And so nobody actually knows what the ceiling of capabilities are for these models because nobody's actually run them for long enough to really tell. When slash goal came out, for example, I mean, people started running things that it took over a week for it to finish, and so people actually didn't realize that this was a big deal until after a week, until a week after it was released.
00:16:45 · Sarah Guo, Noam Brown
[00:16:45] S1: Mhmm. [00:16:46] S0: I think that's gonna be more and more true. You know, the implications of that are, I think, pretty interesting because what do the labs do to, like, fully evaluate their models before they're released? It's actually very difficult because, yeah, you would have to the only way to to really do the evaluations is then delay the model release cycle. And, you know, there's a lot of competitive pressure right now to not do that. [00:17:06] S1: Do you think there's, like, exciting latent capability in the models that are already released that people have not fully explored given timeline?
00:17:14 · Noam Brown
[00:17:14] S0: I think absolutely. I think actually a really great example is the Erdos unit distance problem. So for the viewers that don't know, like, we used an internal model at OpenAI a few weeks ago to disprove the unit Erdos unit distance conjecture. Now I'm not a mathematician, but this seems like it was a a pretty big deal. The in the math community, it was, like, the first first problem that a lot of mathematicians had really spent a lot of time on, and the model was able to do something that they weren't able to do and do it in a way that was actually interesting and useful for mathematicians. Honestly, it did it at a budget that was dirt cheap. I mean, we didn't put a lot of effort into this. We just we trained a new model, and we were just curious what it could do, and we ran it through some problems. And this one, at a pretty low budget, it was like, oh, yeah. I think I have a disproof, and then we were able to verify that, yeah, that disproof is correct. After we announced the results, a bunch of people found that you could get the answer out of 5.5 as well. If now it's not as simple as just asking 5.5, hey. Here's the neural unit distance conjecture. What's the disprove? You had to scaffold it a bit. You had to, like, steer it a bit. And so somebody found, okay. You asked 5.5. List a bunch of ways that you could tackle this problem. And then for it it lists one of the paths that are actually promising to get to the disprove. And then it you tell it, like, okay. Explore this some more. And then if you do this enough times, it actually ends up arriving at the disprove. Now what this means is you could, in principle, ask 5.5 to you know, as as a general purpose scaffold, list a bunch of different strategies, and then for each strategy, tell it to investigate that strategy. And then it would probably be able to arrive at the disprove with a general purpose scaffold. Now that scaffold would be very expensive. I mean, it would probably cost I I just ballpark, like, a thousand to 10 to a $100,000. But it would be possible, and it would have been possible for somebody to disprove the Erdosia to disc distance conjecture before we did using a general purpose model. And nobody had explored sufficiently, well, happens if I put a $100,000 worth of compute into 5.5? What could it do?