No Priors: Artificial Intelligence | Technology | Startups
00:05:42 · Noam Brown, Sarah Guo
[00:05:42] S0: What do you mean by not thinking long enough? [00:05:44] S1: If you can build an agent or control the amount of test time compute being used, like, there's what is done by the model itself, and there's what the user can do. Do you think that, you know, the industry is using test time compute an optimal amount, way undershooting it, or it's, you know, it's a problem in the models where they just need to be able to do that thinking faster? [00:06:08] S0: I I think it depends on the problem. I think this idea that the models, you just let them think for a week or whatever, then they respond. It's it sounds nice, and, yes, the benchmarks look great, but it's not very practical when working because, like, okay, you ask the model a question, then you sit there for a week waiting for it to come back to you. Mhmm. I think what people have have found most effective is to kind of, like, iterate quickly with the models. And so the thinking time, I think, needs to be flexible. When it makes sense to respond quickly to the user, it should respond quickly. And then when it makes sense to think for a long time and the user wants it to think for a long time, then it makes sense to think for a long time. I think people have been striking the right balance given what they have to deal with right now.
00:06:08 · Noam Brown, Sarah Guo
[00:06:08] S0: I I think it depends on the problem. I think this idea that the models, you just let them think for a week or whatever, then they respond. It's it sounds nice, and, yes, the benchmarks look great, but it's not very practical when working because, like, okay, you ask the model a question, then you sit there for a week waiting for it to come back to you. Mhmm. I think what people have have found most effective is to kind of, like, iterate quickly with the models. And so the thinking time, I think, needs to be flexible. When it makes sense to respond quickly to the user, it should respond quickly. And then when it makes sense to think for a long time and the user wants it to think for a long time, then it makes sense to think for a long time. I think people have been striking the right balance given what they have to deal with right now. [00:06:47] S1: How would you characterize know, there's a lot of talk about benchmark maxing and the ability to gain different benchmarks. What would you characterize the, like, landscape of benchmarks as today? And then do you have, like, favorites that you think are more indicative of capability
00:07:02 · Noam Brown
[00:07:02] S0: than others? So So the benchmark maxing thing is also motivation for for writing an essay that I think it's really easy to show you can do much better than previous benchmarks or or previous previous models on benchmarks by just, for example, scaffolding a bunch of models together. So if you say, okay. Well, we're going to instead of just running this model once, we're gonna run it five times and take the best of the five responses or, like, ask a judge which one it it thinks is best, then you can get much higher scores than that model. And so it's really easy to make something that looks a lot better on paper but is actually not better once you control for the amount of test time compute. That is one thing that I'm worried about when it comes to benchmark maxing. I mean, it, like it's it's a little misleading is the only concern that I have. And then as far as, like, the benchmarks themselves, I think there is always a risk of, like, just optimizing for the benchmark. And I've I've certainly encouraged my team, and I think at OpenAI, we're pretty good about not trying to optimize for for specific benchmarks. But once you put out a benchmark, it's there's it's always at risk of just being optimized for. And I think one way to one way to address that is to keep held out private sets that isn't publicly available.
00:08:10 · Sarah Guo, Noam Brown
[00:08:10] S1: The most popular fallback advice for, you know, figure out if a model is significantly better or not is to just play with it for a while. Do you have anything more sophisticated than that that you suggest people do? Like, do you create your own set of new evaluations each time besides private holdback at OpenAI? [00:08:30] S0: I think everybody has their own set of questions that they like to ask the model whenever it comes out. [00:08:34] S1: Mhmm. [00:08:34] S0: For me lately, it's been I I use them to make poker bots and see how good they can make a poker bot. I think it's a nice eval because there is very little open source code for making poker bots. And there's a lot of published essay there's a lot of published papers on it, but you really have to reason through everything. And it's, like it requires a lot of just reasoning and iteration and, like, a lot of small gotchas that I can kind of I've already worked through myself, so I can see where the models fail along the way. They've gotten really good at it now. [00:09:04] S1: Can you describe perhaps, like, with your poker bot creation, like, how reasoning might have progressed in model releases for you guys over a few releases?
00:09:15 · Noam Brown
[00:09:15] S0: Yeah. When the early models were really bad at it. Like, they could not basically do anything. And then 5.2, I was able to work with it to make a reverse solver. So that's, like, the final stage Mhmm. Of poker. And that itself was, I thought, really impressive. I I had to work with it a little bit, but I was actually really impressed because I was able to make the reverse solver probably about five times faster than I would have alone. There were a couple things that I got tripped up on. Blockers was always a big, big issue. But overall, like, you know, with a with a bit of gentle steering, it just kind of like I I kinda felt like a grad student where, okay, they would run into issues, but at least, like, I would know what those issues were and know how to fix it. And I could just make suggestions, and it would go off and then do it. And then pretty quickly, would actually come back with something really good. Mhmm. And then especially the optimization, thought was very impressive. It was able to make it, like, 10 times faster than what I was able to do because it was just able to optimize the code so well. The downsides with 5.2 is I felt like it was gaslighting me a lot, and I always had to be very careful checking it and making sure, like, okay. Is it actually doing what it said it did? Are there any things that are, like, glaring issues that it's not recognizing or it's just pretending aren't issues? I remember there was, like, one point where for one of the models I was playing around with it, not 5.2, I kind of, like, as a as a unit test, I told it, okay. Well, let's say I have a $100 in the pot, and I fold. How much am I losing? And the model said $92. And I was like, that's crazy. I I have a $100 in the pot, I just fold it. How do I not lose a $100? And it said, oh, you know, it's 92. It's close to a 100. It's fine. It's no big deal. And I was like, clearly, this is a problem. Right? So the models did have this problem where they would gaslight you a lot. Mhmm. But once we got to 5.5, I actually thought it was way better. It was able to basically do a zero shot. And in fact, I've been working on just doing a full scale poker solver, and it it's basically able to do the whole thing with some gentle steering from me.
00:11:17 · Noam Brown, Sarah Guo
And I wouldn't be surprised if, six months or a year from now, the model is able to do zero shot an entire poker solver, basically my entire PhD thesis, in one go. [00:11:26] S1: Let's talk about the larger implications of needing to evaluate these models relative to, let's say, speed of their reasoning or efficiency versus token volume, right, or dollar budget or whatever your scaler is. Can you describe some of the larger implications in your essay, including around safety evaluations? [00:11:49] S0: Yeah. The safety evaluations thing, it's it's a bit of an inconvenient truth thing where okay. So I guess for background, a lot of the all of the labs have these things called either responsible scaling policies, preparedness frameworks. They go by various names. But the idea is that whenever a model's released, they go through a series of evaluations to measure, are there dangerous capabilities? Could these models do things that we're we we wouldn't want, a bad actor to do? And if the model isn't very capable, then it's no big deal. But if it is very capable, if it could be used, for example, to make bioweapons, then you want to put in mitigations against that. [00:12:23] S1: Mhmm.