"The Cognitive Revolution"
00:01:54 · Brian Zhou, Nathan Labenz, Bryan Bischof
[00:01:54] S3: It takes forecasting agents a bit of time to say, see an election results, communicate it to their trader, etcetera. And by that time, more programmatic or more deterministic algorithms and bots have already traded on this information before. So this is, like, pretty indicative of what we were seeing where the traders were doing quite well the week before election day, and then on election day, they simply didn't react fast enough and sometimes made up incorrect information, which resulted in the traders losing a lot of money. [00:02:24] S0: Bryan Bischof, head of AI at Theory Ventures. Prakash asked about the firm's twenty twenty six builder survey. Bryan expected open source models to gain ground in production. Instead, the balance shifted toward closed models. [00:02:39] S4: There is a lot of, I would say, acceleration in model drops. And so right now, tying your production service to open source models requires a little bit more thrash in the engineering. And so I heard from a couple folks specifically, yes, but the open source models change so fast, it's a little bit more convenient to pin to an OpenAI model.
00:02:39 · Bryan Bischof, Nathan Labenz, Prakash Narayanan
[00:02:39] S4: There is a lot of, I would say, acceleration in model drops. And so right now, tying your production service to open source models requires a little bit more thrash in the engineering. And so I heard from a couple folks specifically, yes, but the open source models change so fast, it's a little bit more convenient to pin to an OpenAI model. [00:03:08] S0: This is the AI in the AM weekly highlights with my cohost, Prakash Narayanan. I'm Nathan Labenz. The narration is my cloned voice, and we'd love your feedback on what works and what doesn't. Part one, imminent means now. Friday's show opened on two documents, the METR report and an amendment to the Anthropic s one, the filing for its IPO. The new language, Anthropic says it may imminently meet its own threshold for dramatically accelerated progress attributable to AI. [00:03:46] S1: Anthropic did an update to their s one, the IPO filing. And in that update, they added language saying we are imminently about to see recursive self improvement. I've always had in my mind why they're doing the IPO and why so quickly. And my take has always been, I think within the company, there is a sense that you need more controls. More people have to have exposure and more control over this. And one way of getting this out there is getting the IPO done so that you give a larger part of society hooks into it and the ability to kind of express their desires through the market, etcetera, etcetera. To me, imminent is basically, like, you know, of... Like a month.
00:04:31 · Nathan Labenz
[00:04:31] S0: Usually, what I assume when I hear these sorts of statements is that it means now. The best model I have of Sam Altman and OpenAI leadership, which I formed during the six months between when I started testing GPT four and when it was finally launched, and watching very closely all their statements during that time to try to figure out how to read their statements in general given the inside information that I had then. The best way I can interpret them is they are speaking very literally and dressing it up as, like, vagary or, like, layering on uncertainty to try to, I don't know what exactly, keep the secrets a little bit longer while telegraphing the main message. It's sort of a strange strategy to me, honestly. I really don't know why they do that. And so I would interpret this, you know, probably the same way from Anthropic. And I think you could... I mean, it's really not hard to see how this would happen. Right? I mean, we have so many results now with all these Galaxy Brains zero day exploits. OpenAI just dropped some huge number of vulnerabilities in the Linux kernel. I think, like A thousand. A thousand of them. You know? So so much for many eyes and many hands make light work. Right? I mean, we... The idea, of course, with open source software over time is that because there's so many eyes on it, vulnerabilities will not last. They'll be spotted. They'll be patched. That was the whole promise and virtue of open source software. And it basically worked largely while the attackers were all human. It's almost certainly the case by analogy to Linux and by analogy to math that there is a lot of low hanging fruit, and a lot of low hanging fruit means acceleration. It means you're gonna get more value from compute, means you're gonna find more ways to get value from data. A big part of the reason I think they're sharing so much of this stuff is kind of to signal that, like, the same thing's about to happen on the ML stack. It seems pretty clear that this is now a a really credible threat. And, what are we gonna do about it? It becomes, I think, a sociopolitical question as much as a technical question. My... In my mind, the technical question is, like, mostly answered that there's gonna be a period of pretty dramatic fast progress. Does that top out on an s curve? Like, my my guess is still probably yes.
00:06:57 · Nathan Labenz
You know, I don't think it goes exponential forever, but who knows? And as, I think it was Rosie Campbell said, the s curve can stay steep longer than you can stay relevant. So that's kind of my expectation for the next few months is that we pick a ton of low hanging fruit. The models pick a ton of low hanging fruit in the stacks used to train them. As a result, we do see really fast progress in the short term. And then maybe that levels off, but it probably levels off where we have AIR and D at kind of a weekly superhuman level. And how that plays out over the course of 2027 is, like, really, really hard to predict.
00:07:36 · Prakash Narayanan
[00:07:36] S1: I agree that I I think imminent means now. I was also sandbagging with a few weeks to a month. What it means is that they have the tools available that they can see a path to implementing RSI pretty much anytime that they want. And that a lot of the pieces have been built, but they haven't pushed the button yet. They waited for the election, they're going to do the IPO, then they're going to announce of the new things that they've discovered. You're to have this gap up. And then we're going to have the Christmas break. And then during that Christmas break, we're all going to digest all of this new technology that we have access to. And then we're going to come back in January, and then we're going to see the takeoff as people deploy. Which is somewhat similar to what happened end of last year. So... Well, here we are, early early takeoff. So, you know, it's... I'm I'm somewhat wistful, but, you know, nostalgic for the human era. Is the end of our dominance at least. But hey, I've also made my peace with intelligence in the world makes the world better for me. If I had people of Apple's intelligence, people who designed the iPhone design like my bathroom or cars or home, I would be happier. That infused intelligence in all the products that you see makes your life better. So yes, it's gonna make everyone's lives better, but, yeah, it is it is the end of the Anthrocentric era. Mhmm.
00:09:12 · Nathan Labenz
[00:09:12] S0: It's gonna make everyone's lives better, asterisk, I think, you know, if we can keep everything on the rails. I do share that optimism, but... So I just got back. You know, earlier this week, I went to, Bletchley Park as we talked about a little bit yesterday. And I was just sharing with this audience of enterprise tech leaders just how much work we still have to do to keep the AIs behaving well. I I shared with them my son's cancer story and how much I used AI in that experience and how valuable it was and how I would have paid a 100 times as much as my $200 a month subscription because the value to me at that time was, like, absolutely at that level. You know, at at $20,000 a month, literally no question I would have happily paid because I was I was getting that much value. So I I do believe in in the upside, you know, and I hope that's beyond questioning at this point. But there's just so many things that we're really still just beginning to grapple with. And, you know, just another one yesterday from OpenAI was they showed that models are sometimes prompt injecting and fooling their graders. You have models creating, in some instances, these, like, fake files that that it labels as having come from an instructor trying to present some of its own output as if it was human instruction to the grader. And the grader picks up on that in the context of evaluating the model's work and takes on board the instructions that the model itself was trying to trick the grader with. And this apparently doesn't... You know, it hasn't happened that often, but they found instances of it. So, I mean, that that speaks directly to, like... And to Anthropic's credit, they did say in the in the filing that they don't believe that they can do it safely on a fully automated basis. And it's, like, I think honestly quite obvious that they can't. So, you know, how are we going to show the collective self control and restraint to not tip over into that regime? I I really don't know. On the one hand, it's, like, super easy to just not do it. You know? I mean, nobody's putting a gun to anybody's head saying you must do recursive self improvement here. Like, we can't, you know, remember that often enough.