"The Cognitive Revolution"
00:11:59 · Nathan Labenz
Ad[00:11:59] S0: Today's episode is brought to you by Anthropic, makers of Claude. To show off Claude's latest capabilities, I asked Claude to take the lead in making this ad. Given a short creative brief, Claude, using Opus 5.5, studied my past Claude ads, wrote this script, directed a clone of my voice, and designed and coded the animation you're watching. My job? Give feedback and decide. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. And with Claude Code Routines, it keeps working in the cloud even after you close your laptop. For problems worth solving, get started with Claude at claud.aitcr. That's claud.aitcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Claude dot aitcr. This episode of the Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. We've all seen the headlines. Companies are pouring money into AI. But the big question on every executive's mind right now isn't just how fast can we adopt this, it's where is the ROI? We're seeing a real trend toward AI chaos. You've got teams deploying standalone coding tools, random agent builders, and experimental scripts. It sounds innovative, but in reality, it's creating a massive headache. Fragmented tools, ungoverned data, serious security blind spots, and costs that are spiraling out of control. If you don't bring those agentic applications under control now while they're still embedding into your core processes, you are looking at broken systems and damaged customer trust down the road. That's where OutSystems comes in. OutSystems is the leading agentic systems platform for the enterprise. Instead of managing a patchwork of disconnected tools, OutSystems lets your team engineer, orchestrate, and govern your entire agentic ecosystem on one open, unified platform. It's built for the speed of AI, but with the reliability and security that enterprises actually require.
00:14:33 · Nathan Labenz
AdWe're talking about real results, like Key Bank, who used OutSystems to deploy a customer facing app that delivered 75% faster onboarding times, or the global logistics leaders who built AgenTic Systems to completely eliminate their engineering bottlenecks. You don't have to choose between speed and control. Whether you're a small team or a massive enterprise, OutSystems helps you engineer, orchestrate, and deploy AgenTic systems that actually scale. Stop chasing the hype and start owning your AgenTic future. You can see how it works and learn more at outsystems.com/tcr. That's outsystems.com/tcr. Part two, agents forecasting, agents training. Our first Friday guest, Brian Zhou, a researcher at Thoughtful.
00:15:26 · Brian Zhou
[00:15:26] S3: We recently launched and, have a paper coming out about BallotBench, which is where we had AI trying to forecast the up... The twenty twenty six midterms that just happened as well as upcoming elections happening all around the world. Yeah. Maybe let's start with the methods. So we had nine agents, basically, all of the front... Best models offered by each individual lab, get their own model sandbox and work for seven days consecutive leading up to the election. So they all got their modal sandbox. We gave them additional resources in terms of a $500 budget that they could use to request access to paywall data, paywall news sources, subscriptions, etcetera. And they were basically given an overall objective of, for all of the races on a given list, they're asked to submit their predictions on who will win, the odds of victory given 80% margin to bear, and then also give additional predictions for what they think the margin of victory will be, for example, if the Democratic candidate will win by three points versus seven points, etcetera. So agents are tasked with giving incredibly granular and detailed forecasts, and there's a lot of really interesting things that we can and have observed from this. It's a really interesting test of long running agentic behavior. In particular, for example, what we see with some of the MuseWonder series of models is that they're often incredibly bad at doing long horizon work. When given a task in the Muse code harness, when they have a list of, say, 800 or so elections that they need to predict, they'll create a random forecast, which is just a random skew off of the twenty twenty four election results and submit that within, for example, the first two, three minutes of working.
00:17:05 · Nathan Labenz
[00:17:05] S0: I'm already intrigued by your description of the Muse agent, and it's seemingly phoning it in. So I guess one initial question just on that point is, like, how do you reconcile what you saw with what people seem to be reporting online, which is, like, overall pretty good experiences. No. For sure. I mean,
00:17:25 · Brian Zhou
[00:17:25] S3: our initial theory on this, right, is that this is mostly due to a part of the harness design. So we're not talking about, Meta's personal agent, we're talking about their coding agent. And one of the things that we noticed is the design of the Muse Wonder model when paired with Muse code, which is the native harness that Meta provides, has essentially what is a task list. So even though in the prompt, for example, they're instructed, hey. You have seven days of time. You have unlimited API usage, you have full permission to, like, fan out sub agents, do whatever you need, etcetera, to get the best and best possible results that you can on this exam. What we observed is that the harness essentially creates this list of like, okay, we need to check and make sure that we've submitted all 800 or so different races. We need to check and make sure that we've submitted all the forecasts, that they have been accepted. And then what we see is that there's essentially a goal condition which once met ends the term prematurely. So we actually ended up having to essentially force the agent to continue working further by any time the agent would choose to stop its work by essentially reviving it with a new sandbox, a new container, and access to its, like, old runs and old messages. So this is, I think, like, sort of more of a harness engineering question itself as well as sort of, like, a general model behavior problem. But it's definitely a different result compared to what we see with much more frontier models. Right? When we see OPUS 5.5 or when we see g 56 Astra, their behaviors are vastly different. For example, on one given day, Opus five five would fan out something akin to 170 or 180 different sub agents and manage all of these sub agents by delegating work such as, hey. Can you look into the polling results for a specific state? Can you, like, do this detailed review of what each individual ballot initiative in the state of California like Prop 40 is doing? Tell me a bit about that and tell me a bit about what those things are doing. And then using these sub agents to really help it paralyze a lot of the work that I want to do. So I think this is this general question of what agents are doing, especially when we get to much more long and...
00:19:28 · Brian Zhou, Nathan Labenz
Long horizon open ended tasks like these and the sort of proactivity and as well as creativity of what they do. One of the things we were pretty excited to see was if agents would do something sort of outside the realm of what we would expect expert forecaster to do, let's say putting out their own poll, seeing if there are, like, more creative ways to find information as opposed to just fetching data, building models, etcetera. We don't see any real massive instances of this happening quite yet, but we see a lot of interesting sort of approaches that individual agents take. For example, one thing GPT six one SOL does is it takes the forecasts of every election modeler that they look at, and it takes the past decade of their forecasts and tries to calculate the shift of how right or how wrong each forecaster was. And then it can use this result to essentially aggregate all of these different election models that these forecasters are creating, whether it's like Nate Silver's model or any of the others like 50 plus one. And it can sort of skew it based on what they would expect the model to either underperform or overperform as in recent years. So there's a lot of, like, creative methods that we're sort of seeing. [00:20:35] S0: So tell me about the results. What, like, how did the models compare against each other? How did they compare against top humans? You know, are... Do we now have superhuman forecasters?