00:04:13 · Ruby Justice Thelot, Parv Mahajan
[00:04:13] S1: What do you mean your methods would not be able to find it? [00:04:16] S0: Yeah. Yeah. So so, basically, some several of the methods that we're using at the time, like, we we... There's not that much data, right? There's only so many internal models, and we only have about a year of data. Also, the methods that we... It... It's hard to talk about this because much of the evidence... Important evidence here was redacted. But, like, basically, the lines of evidence that we used, which included, like, interviews with researchers, self reported uplift, investigations of different parts of the training pipeline, and contributions of both AI and humans to different parts of the training pipeline, and Anthropic ECI, we're not confident that any of them would be able to show this kind of hyperbolic growth you're describing even if it did exist. So I think this is, again, like a method development question and, like, a open science question. [00:04:57] S1: Like, we just have no idea how to measure hyperbolic growth if it were occurring? [00:05:01] S0: Yeah. So there's, like, a couple things that would be, like, very obviously high... Like, if we saw hyperbolic growth, we would expect very obvious things. Like, for example, if, like, a 100% of the architectural innovations at Anthropic or another company were happening because of, you know, because of AI, then we could be like, oh, yeah. Hyperbolic growth is probably taking place. Currently, we don't have enough evidence, and there's especially not enough public evidence to rule this in or out. Another
00:05:01 · Parv Mahajan, Ruby Justice Thelot
[00:05:01] S0: Yeah. So there's, like, a couple things that would be, like, very obviously high... Like, if we saw hyperbolic growth, we would expect very obvious things. Like, for example, if, like, a 100% of the architectural innovations at Anthropic or another company were happening because of, you know, because of AI, then we could be like, oh, yeah. Hyperbolic growth is probably taking place. Currently, we don't have enough evidence, and there's especially not enough public evidence to rule this in or out. Another [00:05:25] S1: thing that I found kind of strange was like, a lot of people reacted very strongly to the graph that METR published of ECI accelerating after October 2025 or so. Except it looked to me trend line of the public models was basically the same as the trend line of the internal models, just shifted around a month later. So why do you think people reacted so strongly to this in particular?
00:05:25 · Ruby Justice Thelot, Parv Mahajan
[00:05:25] S1: thing that I found kind of strange was like, a lot of people reacted very strongly to the graph that METR published of ECI accelerating after October 2025 or so. Except it looked to me trend line of the public models was basically the same as the trend line of the internal models, just shifted around a month later. So why do you think people reacted so strongly to this in particular? [00:05:49] S0: Yeah. Yeah. I I I don't wanna comment on people's reactions to it, but I can talk a little bit about the modeling behind it. So I think one of the important things about this report is we did a lot of robustness checks on this kind of, like, kink that we see around, like, October 2025. And we find that it's pretty robust. And especially that, the main reason for this kink that you've... Like like you said, like, don't see under, like, some, assumptions about public benchmarks is largely because we're... We were able to benchmark these internal models. And, also, we, like, did some, like, methodological changes that, that that that made this more robust. I'm like, if you include, for example, public models from other companies, and we talk about this in the report, or if you, like, some changes to... For example, you don't have internal models, then the kink becomes a lot less obvious. And in fact, you saw this, for example, in, like, the, METR's preview system card and system cards after that, which is why we're, like... We're interested in figuring out whether this applies to other, like, other companies in the industry as well and whether this will continue.
00:05:49 · Parv Mahajan, Ruby Justice Thelot
[00:05:49] S0: Yeah. Yeah. I I I don't wanna comment on people's reactions to it, but I can talk a little bit about the modeling behind it. So I think one of the important things about this report is we did a lot of robustness checks on this kind of, like, kink that we see around, like, October 2025. And we find that it's pretty robust. And especially that, the main reason for this kink that you've... Like like you said, like, don't see under, like, some, assumptions about public benchmarks is largely because we're... We were able to benchmark these internal models. And, also, we, like, did some, like, methodological changes that, that that that made this more robust. I'm like, if you include, for example, public models from other companies, and we talk about this in the report, or if you, like, some changes to... For example, you don't have internal models, then the kink becomes a lot less obvious. And in fact, you saw this, for example, in, like, the, METR's preview system card and system cards after that, which is why we're, like... We're interested in figuring out whether this applies to other, like, other companies in the industry as well and whether this will continue. [00:06:50] S1: Mhmm. So also in the report, METR said they found no convincing bottlenecks in algorithms, training compute, experiment compute, or data over the next twelve months, but has low confidence in this. So what, like, what would make you have confidence that there are or are not bottlenecks?
00:06:50 · Ruby Justice Thelot, Parv Mahajan, S2
[00:06:50] S1: Mhmm. So also in the report, METR said they found no convincing bottlenecks in algorithms, training compute, experiment compute, or data over the next twelve months, but has low confidence in this. So what, like, what would make you have confidence that there are or are not bottlenecks? [00:07:08] S0: Yeah. So so, again, I wasn't one of the report authors, and I don't wanna... I don't have access to any confidential information here. But we we we published on kind of the kinds of evidence we might need to see to be confident that there are, like, no bottlenecks to anything or that there might be bottlenecks. And based on based on the evidence that Anthropic gave us, which was later redacted as well as our analysis of this evidence, the the report authors are confident that at least training compute is is not a bottleneck. I Ad[00:07:41] S2: wanna shout out our sponsor, Lovable. You know the app you've been meaning to build, the internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable code base you can inspect and change plus two way GitHub sync. Lovable also handles the back end and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. Lovable. Dev. Now, onto the episode.
00:07:41 · S2, Ruby Justice Thelot, Parv Mahajan
Ad[00:07:41] S2: wanna shout out our sponsor, Lovable. You know the app you've been meaning to build, the internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable code base you can inspect and change plus two way GitHub sync. Lovable also handles the back end and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. Lovable. Dev. Now, onto the episode. [00:08:19] S1: I don't know if you can comment on this, but, you know, if you can, based on your estimates, vibes of the redacted version of this report, how have your timelines shifted? [00:08:32] S0: Yeah. That's a good question. Honestly, my my timelines haven't shifted that much. Although, again, I think the modeling here is extremely unclear because I think a large part of your modeling should hinge on whether you think these models are getting better at these, like, not very well specified tasks [00:08:47] S1: I agree.