No Priors: Artificial Intelligence | Technology | Startups
00:04:20 · Sarah Guo, Noam Brown
[00:04:20] S1: Given the model evaluation cycle and the fact that performance does not asymptote for many tasks over quite a long period of time, what do you do about that issue, the fact that some of the evals that you would want to run are both beyond the scope of budget or time that's reasonable given the current model release cycle? [00:04:40] S0: I mean, think for things like cyber, we've seen, and actually the AISI in their evaluations has shown that the models continue to improve at a 100,000,000 tokens. You know, if you run them for a 100,000,000 tokens, they're still improving at beyond that point. Mhmm. And that can take a very long time to run. But you also do see that, like, the performance is is it's not just, like, a discontinuous jump. It's actually, like, you can see the slope of improvement over those 100,000,000 tokens. And so you you could probably do some kind of evaluation up to a certain budget and then just say, okay. Well, this is what we project the performance to look like. And I think this this there hasn't been a lot of research on this yet. I actually think this would be a great paper to publish if there's any academics out there looking for something to research. Can you predict what the performance looks like at an inference budget of, let's say, $10,000, only using inference budgets up to 10 or 100 or 10 or $100? [00:05:34] S1: So maybe an orthogonal question for you. Do you think users are systematically, like, not thinking long enough with their models about problems? [00:05:42] S0: What do you mean by not thinking long enough?
00:05:42 · Noam Brown, Sarah Guo
[00:05:42] S0: What do you mean by not thinking long enough? [00:05:44] S1: If you can build an agent or control the amount of test time compute being used, like, there's what is done by the model itself, and there's what the user can do. Do you think that, you know, the industry is using test time compute an optimal amount, way undershooting it, or it's, you know, it's a problem in the models where they just need to be able to do that thinking faster? [00:06:08] S0: I I think it depends on the problem. I think this idea that the models, you just let them think for a week or whatever, then they respond. It's it sounds nice, and, yes, the benchmarks look great, but it's not very practical when working because, like, okay, you ask the model a question, then you sit there for a week waiting for it to come back to you. Mhmm. I think what people have have found most effective is to kind of, like, iterate quickly with the models. And so the thinking time, I think, needs to be flexible. When it makes sense to respond quickly to the user, it should respond quickly. And then when it makes sense to think for a long time and the user wants it to think for a long time, then it makes sense to think for a long time. I think people have been striking the right balance given what they have to deal with right now.
00:06:08 · Noam Brown, Sarah Guo
[00:06:08] S0: I I think it depends on the problem. I think this idea that the models, you just let them think for a week or whatever, then they respond. It's it sounds nice, and, yes, the benchmarks look great, but it's not very practical when working because, like, okay, you ask the model a question, then you sit there for a week waiting for it to come back to you. Mhmm. I think what people have have found most effective is to kind of, like, iterate quickly with the models. And so the thinking time, I think, needs to be flexible. When it makes sense to respond quickly to the user, it should respond quickly. And then when it makes sense to think for a long time and the user wants it to think for a long time, then it makes sense to think for a long time. I think people have been striking the right balance given what they have to deal with right now. [00:06:47] S1: How would you characterize know, there's a lot of talk about benchmark maxing and the ability to gain different benchmarks. What would you characterize the, like, landscape of benchmarks as today? And then do you have, like, favorites that you think are more indicative of capability
00:07:02 · Noam Brown
[00:07:02] S0: than others? So So the benchmark maxing thing is also motivation for for writing an essay that I think it's really easy to show you can do much better than previous benchmarks or or previous previous models on benchmarks by just, for example, scaffolding a bunch of models together. So if you say, okay. Well, we're going to instead of just running this model once, we're gonna run it five times and take the best of the five responses or, like, ask a judge which one it it thinks is best, then you can get much higher scores than that model. And so it's really easy to make something that looks a lot better on paper but is actually not better once you control for the amount of test time compute. That is one thing that I'm worried about when it comes to benchmark maxing. I mean, it, like it's it's a little misleading is the only concern that I have. And then as far as, like, the benchmarks themselves, I think there is always a risk of, like, just optimizing for the benchmark. And I've I've certainly encouraged my team, and I think at OpenAI, we're pretty good about not trying to optimize for for specific benchmarks. But once you put out a benchmark, it's there's it's always at risk of just being optimized for. And I think one way to one way to address that is to keep held out private sets that isn't publicly available.
00:08:10 · Sarah Guo, Noam Brown
[00:08:10] S1: The most popular fallback advice for, you know, figure out if a model is significantly better or not is to just play with it for a while. Do you have anything more sophisticated than that that you suggest people do? Like, do you create your own set of new evaluations each time besides private holdback at OpenAI? [00:08:30] S0: I think everybody has their own set of questions that they like to ask the model whenever it comes out. [00:08:34] S1: Mhmm. [00:08:34] S0: For me lately, it's been I I use them to make poker bots and see how good they can make a poker bot. I think it's a nice eval because there is very little open source code for making poker bots. And there's a lot of published essay there's a lot of published papers on it, but you really have to reason through everything. And it's, like it requires a lot of just reasoning and iteration and, like, a lot of small gotchas that I can kind of I've already worked through myself, so I can see where the models fail along the way. They've gotten really good at it now. [00:09:04] S1: Can you describe perhaps, like, with your poker bot creation, like, how reasoning might have progressed in model releases for you guys over a few releases?
00:09:15 · Noam Brown
[00:09:15] S0: Yeah. When the early models were really bad at it. Like, they could not basically do anything. And then 5.2, I was able to work with it to make a reverse solver. So that's, like, the final stage Mhmm. Of poker. And that itself was, I thought, really impressive. I I had to work with it a little bit, but I was actually really impressed because I was able to make the reverse solver probably about five times faster than I would have alone. There were a couple things that I got tripped up on. Blockers was always a big, big issue. But overall, like, you know, with a with a bit of gentle steering, it just kind of like I I kinda felt like a grad student where, okay, they would run into issues, but at least, like, I would know what those issues were and know how to fix it. And I could just make suggestions, and it would go off and then do it. And then pretty quickly, would actually come back with something really good. Mhmm. And then especially the optimization, thought was very impressive. It was able to make it, like, 10 times faster than what I was able to do because it was just able to optimize the code so well. The downsides with 5.2 is I felt like it was gaslighting me a lot, and I always had to be very careful checking it and making sure, like, okay. Is it actually doing what it said it did? Are there any things that are, like, glaring issues that it's not recognizing or it's just pretending aren't issues? I remember there was, like, one point where for one of the models I was playing around with it, not 5.2, I kind of, like, as a as a unit test, I told it, okay. Well, let's say I have a $100 in the pot, and I fold. How much am I losing? And the model said $92. And I was like, that's crazy. I I have a $100 in the pot, I just fold it. How do I not lose a $100? And it said, oh, you know, it's 92. It's close to a 100. It's fine. It's no big deal. And I was like, clearly, this is a problem. Right? So the models did have this problem where they would gaslight you a lot. Mhmm. But once we got to 5.5, I actually thought it was way better. It was able to basically do a zero shot. And in fact, I've been working on just doing a full scale poker solver, and it it's basically able to do the whole thing with some gentle steering from me.