00:08:47 · Ruby Justice Thelot, Parv Mahajan
[00:08:47] S1: I agree. [00:08:47] S0: These kind of broad open ended tasks. And, also, whether... You know, how important are these, like, broad unspecified tasks to AR and D? That's also very unclear. So... Yeah. Yeah. It's pretty uncertain. I think the thing I wanna make clear about support is in an ideal world, we would achieve transparency about evidence, which means we'd be able to make a lot of raw evidence or at least some level of evidence public. And then everyone in the public could come to their own conclusions using their own modeling and update their timelines accordingly. Fact The that we weren't able to do that means that the public has to trust METR more than is ideal. And, yeah, I think that's, like, one of the most important takeaways from this. What [00:09:26] S1: trying to frame this question.: [00:09:28] S0: Yeah.: [00:09:30] S1: Basically, in Anthropic's risk report in August, they said that they had not yet crossed their acceleration warning threshold.: Yeah.: METR said that they considered that Anthropic maybe actually did cross this threshold then. What are the sort of cruxes between whether Anthropic did or did not cross this, And how would they be how would they be behaving differently under their own frameworks, if they had?
00:10:00 · Parv Mahajan, Ruby Justice Thelot
[00:10:00] S0: Yeah, yeah. So we also, along this with this report, we published a review of the automated AR and D risk section of the of their August 2026 risk report. And what we found is that, basically, I think based on reading the report, I think a reasonable reader, based on reading their public version of the report, think like, oh, they're substantially below 2x. Or like, they're below 2x, but they have some uncertainty. Where, in fact, if you use, their numbers, you find that it actually comes out to about 1.97 x, depending on the exact measure you use, which is very close to 2x, right, which is the threshold. So, yeah, overall, I I think what we found is that probably the report, like, overstates the amount of confidence that Anthropic should have. And, if METR was writing the report, and especially if METR staff were examining all the evidence that Anthropic had access to, we would likely think that the threshold, is very probable or even maybe likely that the threshold of two x had been crossed. Again, how does this affect Anthropic governance, etcetera? METR's job is not to make those decisions. METR's is to, like, comment on the state of the evidence and try to get this evidence up to the public.: [00:11:07] S1: Right. So Sam McCandlish, who's Anthropic's responsible scaling officer and a co founder, just posted on X for the first time yesterday on this report, and said that Anthropic will publish its own analysis of research progress in the coming weeks, whatever that means. So what might METR look for in such a report?:
00:11:07 · Ruby Justice Thelot, Parv Mahajan
[00:11:07] S1: Right. So Sam McCandlish, who's Anthropic's responsible scaling officer and a co founder, just posted on X for the first time yesterday on this report, and said that Anthropic will publish its own analysis of research progress in the coming weeks, whatever that means. So what might METR look for in such a report?: [00:11:29] S0: Yeah. I think one thing that would be exciting from this report, and in general reports that companies are producing about progress towards RSI, would be making as much raw data available, especially about improvements in different parts of the training pipeline that were done by humans versus AI and, like, some of the things that they redacted from our report and making that available so the public can do their own analysis. This is, like, trying to get closer to the transparency about evidence that we talk about in our risk assessment proposal. Our goal, again, is that, like, no individual company... Yeah. No individual organization, including METR, should be kind of the arbiter of risk. Instead, individuals, policymakers, the public should be able to do their own analyses and come to their own conclusions. [00:12:08] S1: Right. And then one other thing is that Anthropic's self reported uplift in their Slack polls ranged, like, massively, like, by a factor of, like, 10 x. I think it was like 1.2 x to 12 x or something. Like, why? Why do you think these subjective estimations of uplift varied so much?
00:12:08 · Ruby Justice Thelot, Parv Mahajan
[00:12:08] S1: Right. And then one other thing is that Anthropic's self reported uplift in their Slack polls ranged, like, massively, like, by a factor of, like, 10 x. I think it was like 1.2 x to 12 x or something. Like, why? Why do you think these subjective estimations of uplift varied so much? [00:12:28] S0: Yeah. Yeah. So so we found, like, a somewhat similar result in in 2025 where we saw that, you know, people thought they were being uplifted by, like, around 20% when in fact, they would be, like, down lifted. And and later, our methodology was, like, a lot more uncertain. We were, like, unable to to really come to firm conclusions here. But in general, I think, self reported uplift studies have, like, various issues. For example, if you use, like, the format that you're presenting them in or, like, the type of question, there's, like, these are problems that, for example, social scientists or other people that work on surveys are very familiar with. And these get even harder when you're thinking about kind of these, like, long, fuzzy tasks that you're not necessarily, like, doing very intense time tracking if you're a machine learning researcher at Anthropic. So, yeah, I think, overall, these self reported upload studies are extremely hard to interpret as as we say in the report. And I wouldn't, I think a rapid increase in reported uplift is interesting, but I would not put too much stock in the exact number. And like you said, the error bars on this are so big that it's just one line of evidence among many that you have to consider.:
00:13:27 · Ruby Justice Thelot, Parv Mahajan
[00:13:27] S1: Right. So my last question before we go. A lot of this kind of... Basically, almost all of what we've talked about in the last stretch of time is totally inscrutable to the public. So like, if you were talking to a audience of basically normal people, like, what are the most important things for them to understand from this report? [00:13:45] S0: Yeah. Yeah. Here... Here's here's what I would say. So so the audience I have in mind right now is, like, my parents or, like, my my sister or things like this. The thing I would say is, like, it looks like Anthropic. AI is progressing a lot faster. And in fact, they're, like, kind of building the tools to accelerate themselves. And it seems like this has been happening since at least late twenty twenty five, but the state of the evidence is such that we're very uncertain about what this means for the future. And, also, like, we're very uncertain about the causes of this or, like, exactly what happened. And what makes this worse is that, Anthropic has redacted large parts of the evidence, so we can't make it... You know, we we can't, give the public and scientists a raw conclusion so that they can come to you with, like, firmer, you know, kind of firmer conclusions about about how the future will go. I think it's important that we continue to do these embedded exercises and labs get information out to the public. But, also, labs should feel free to, when possible, unilaterally just give this information so that we can understand where this future of this powerful technology is going.
00:13:45 · Parv Mahajan, Ruby Justice Thelot, S3
[00:13:45] S0: Yeah. Yeah. Here... Here's here's what I would say. So so the audience I have in mind right now is, like, my parents or, like, my my sister or things like this. The thing I would say is, like, it looks like Anthropic. AI is progressing a lot faster. And in fact, they're, like, kind of building the tools to accelerate themselves. And it seems like this has been happening since at least late twenty twenty five, but the state of the evidence is such that we're very uncertain about what this means for the future. And, also, like, we're very uncertain about the causes of this or, like, exactly what happened. And what makes this worse is that, Anthropic has redacted large parts of the evidence, so we can't make it... You know, we we can't, give the public and scientists a raw conclusion so that they can come to you with, like, firmer, you know, kind of firmer conclusions about about how the future will go. I think it's important that we continue to do these embedded exercises and labs get information out to the public. But, also, labs should feel free to, when possible, unilaterally just give this information so that we can understand where this future of this powerful technology is going. [00:14:48] S1: Right. Well, this is very interesting and informative. Thank you so much, Parv, for coming to MTS. [00:14:54] S0: Yeah. Thanks, Thelot. Bye. Ad[00:14:57] S3: A huge thanks to MTS sponsors.