No Priors: Artificial Intelligence | Technology | Startups
00:11:17 · Noam Brown, Sarah Guo
And I wouldn't be surprised if, six months or a year from now, the model is able to do zero shot an entire poker solver, basically my entire PhD thesis, in one go. [00:11:26] S1: Let's talk about the larger implications of needing to evaluate these models relative to, let's say, speed of their reasoning or efficiency versus token volume, right, or dollar budget or whatever your scaler is. Can you describe some of the larger implications in your essay, including around safety evaluations? [00:11:49] S0: Yeah. The safety evaluations thing, it's it's a bit of an inconvenient truth thing where okay. So I guess for background, a lot of the all of the labs have these things called either responsible scaling policies, preparedness frameworks. They go by various names. But the idea is that whenever a model's released, they go through a series of evaluations to measure, are there dangerous capabilities? Could these models do things that we're we we wouldn't want, a bad actor to do? And if the model isn't very capable, then it's no big deal. But if it is very capable, if it could be used, for example, to make bioweapons, then you want to put in mitigations against that. [00:12:23] S1: Mhmm.
00:12:23 · Sarah Guo, Noam Brown
[00:12:23] S1: Mhmm. [00:12:24] S0: But the question is, okay. Well, how do you evaluate whether the model is capable of that? And they have, like, various protocols about, like, how they do these evaluations. But a lot of these frameworks were developed around the era of ChatGPT, either before or after, when test time compute scaling was not really as much of a thing. Mhmm. And it made sense. Like, with GPT three, you couldn't scale test time compute. Like, if you gave it a budget of $10,000,000 and said, okay. Well, let's see what g p t three can do, it really can't do that much more than what you could do with, like, $10 or $1. The preparedness frameworks and responsible scaling policies, they don't really account for the amount of test time compute. They just say, okay. Well, what's the capability of the model? The problem is we're in a world now where the capability of the model is a function of how much money you put into it, basically. [00:13:08] S1: Mhmm. [00:13:08] S0: If you give it a budget of $10,000, it can do a lot more than what it can do with a budget of $10. If you give it a budget of $10,000,000, it could do even more. And so at what budget should you evaluate these models? The policies that exist today don't really address that question. [00:13:25] S1: Mhmm.
00:13:25 · Sarah Guo, Noam Brown
[00:13:25] S1: Mhmm. [00:13:25] S0: Some do some do better than others, but for the most part, this is not really a factor that's being heavily considered. Now whether it should be released anyway, I don't I don't wanna wade into this question. I think there's, you know, there's arguments on both sides. But I think the important thing to recognize is that this is a question that is not being we're we're just kind of, like, you know, pretending that this issue doesn't exist, and I think it's important to just, you know, one way or the other account for it. [00:13:50] S1: Yeah. It was the mirror image of the capability question of if the models can continue to do more and more without asymptoting on some tasks at very large budgets, then they should also be able to do so for tasks we don't want them to do as a society. Right? And so testing for that and what budget is allocated, it also seems out of sync from the model release cycle itself. Right? There's been this acceleration of, you know, you get a new model every sometimes few days and weeks at this point versus six months. And you have a line in the essay where you say, like, the the only way to truly evaluate an agent on some very long running task might be to run it for a year, and that's gonna be true of both, like, useful and negative tasks. Right? And so how do you think about that versus the model release cycle? [00:14:44] S0: Yeah. This this is also an interesting dynamic where, basically, as the models have become stronger, they've they're more they're better able to operate over longer horizons.
00:14:44 · Noam Brown, Sarah Guo
[00:14:44] S0: Yeah. This this is also an interesting dynamic where, basically, as the models have become stronger, they've they're more they're better able to operate over longer horizons. [00:14:53] S1: Mhmm. [00:14:54] S0: So, again, with g p d three, if you wanted to run it for, you know, a week, there's really not much you could do to scaffold it into something useful that could actually run for a week. But we're seeing now with the most recent models that you can actually scaffold, for example, 5.5 into doing a series of experiments that can run for weeks, for months. [00:15:12] S1: Have you given your poker solver task, like, infinite budget yet? [00:15:19] S0: I haven't really scaffolded something together where I just tell it, like, okay, just run this for for weeks. I think I could it I I Until it has [00:15:27] S1: some types.
00:15:28 · Noam Brown
[00:15:28] S0: Could probably give it slash goal and just, like, yeah, tell it to go nuts. But I I think at this point, it could 100% do the reverse solver if I just give it slash goal. I don't think it's at the level yet where it could do, like, the full poker solver if I gave it just slash goal and told it, yeah, go go run for a month. But we're going to pretty soon be at that point where I I probably could just tell it, like, yeah, go work on this for a month, and then come back to me with a a full, complete PokerSolver that's state of the art. And the problem is if you want to evaluate the capabilities of a model, what it can do after running for a month, the only way to be fully sure is to actually run it for a month. And if you wanna know after six months, the only way to know fully is to run it for six months. Now there I'll I'll get to, like, things we could do to address that a little bit later, but, like, it's important to recognize that the model release cycle is look. We're releasing new models, like, every two or three months at this point. And so a model comes out, it takes two or three months to push it to its limits, and then you have another model come out. And so nobody actually knows what the ceiling of capabilities are for these models because nobody's actually run them for long enough to really tell. When slash goal came out, for example, I mean, people started running things that it took over a week for it to finish, and so people actually didn't realize that this was a big deal until after a week, until a week after it was released.
00:16:45 · Sarah Guo, Noam Brown
[00:16:45] S1: Mhmm. [00:16:46] S0: I think that's gonna be more and more true. You know, the implications of that are, I think, pretty interesting because what do the labs do to, like, fully evaluate their models before they're released? It's actually very difficult because, yeah, you would have to the only way to to really do the evaluations is then delay the model release cycle. And, you know, there's a lot of competitive pressure right now to not do that. [00:17:06] S1: Do you think there's, like, exciting latent capability in the models that are already released that people have not fully explored given timeline?