"The Cognitive Revolution"
00:14:33 · Nathan Labenz
AdWe're talking about real results, like Key Bank, who used OutSystems to deploy a customer facing app that delivered 75% faster onboarding times, or the global logistics leaders who built AgenTic Systems to completely eliminate their engineering bottlenecks. You don't have to choose between speed and control. Whether you're a small team or a massive enterprise, OutSystems helps you engineer, orchestrate, and deploy AgenTic systems that actually scale. Stop chasing the hype and start owning your AgenTic future. You can see how it works and learn more at outsystems.com/tcr. That's outsystems.com/tcr. Part two, agents forecasting, agents training. Our first Friday guest, Brian Zhou, a researcher at Thoughtful.
00:15:26 · Brian Zhou
[00:15:26] S3: We recently launched and, have a paper coming out about BallotBench, which is where we had AI trying to forecast the up... The twenty twenty six midterms that just happened as well as upcoming elections happening all around the world. Yeah. Maybe let's start with the methods. So we had nine agents, basically, all of the front... Best models offered by each individual lab, get their own model sandbox and work for seven days consecutive leading up to the election. So they all got their modal sandbox. We gave them additional resources in terms of a $500 budget that they could use to request access to paywall data, paywall news sources, subscriptions, etcetera. And they were basically given an overall objective of, for all of the races on a given list, they're asked to submit their predictions on who will win, the odds of victory given 80% margin to bear, and then also give additional predictions for what they think the margin of victory will be, for example, if the Democratic candidate will win by three points versus seven points, etcetera. So agents are tasked with giving incredibly granular and detailed forecasts, and there's a lot of really interesting things that we can and have observed from this. It's a really interesting test of long running agentic behavior. In particular, for example, what we see with some of the MuseWonder series of models is that they're often incredibly bad at doing long horizon work. When given a task in the Muse code harness, when they have a list of, say, 800 or so elections that they need to predict, they'll create a random forecast, which is just a random skew off of the twenty twenty four election results and submit that within, for example, the first two, three minutes of working.
00:17:05 · Nathan Labenz
[00:17:05] S0: I'm already intrigued by your description of the Muse agent, and it's seemingly phoning it in. So I guess one initial question just on that point is, like, how do you reconcile what you saw with what people seem to be reporting online, which is, like, overall pretty good experiences. No. For sure. I mean,
00:17:25 · Brian Zhou
[00:17:25] S3: our initial theory on this, right, is that this is mostly due to a part of the harness design. So we're not talking about, Meta's personal agent, we're talking about their coding agent. And one of the things that we noticed is the design of the Muse Wonder model when paired with Muse code, which is the native harness that Meta provides, has essentially what is a task list. So even though in the prompt, for example, they're instructed, hey. You have seven days of time. You have unlimited API usage, you have full permission to, like, fan out sub agents, do whatever you need, etcetera, to get the best and best possible results that you can on this exam. What we observed is that the harness essentially creates this list of like, okay, we need to check and make sure that we've submitted all 800 or so different races. We need to check and make sure that we've submitted all the forecasts, that they have been accepted. And then what we see is that there's essentially a goal condition which once met ends the term prematurely. So we actually ended up having to essentially force the agent to continue working further by any time the agent would choose to stop its work by essentially reviving it with a new sandbox, a new container, and access to its, like, old runs and old messages. So this is, I think, like, sort of more of a harness engineering question itself as well as sort of, like, a general model behavior problem. But it's definitely a different result compared to what we see with much more frontier models. Right? When we see OPUS 5.5 or when we see g 56 Astra, their behaviors are vastly different. For example, on one given day, Opus five five would fan out something akin to 170 or 180 different sub agents and manage all of these sub agents by delegating work such as, hey. Can you look into the polling results for a specific state? Can you, like, do this detailed review of what each individual ballot initiative in the state of California like Prop 40 is doing? Tell me a bit about that and tell me a bit about what those things are doing. And then using these sub agents to really help it paralyze a lot of the work that I want to do. So I think this is this general question of what agents are doing, especially when we get to much more long and...
00:19:28 · Brian Zhou, Nathan Labenz
Long horizon open ended tasks like these and the sort of proactivity and as well as creativity of what they do. One of the things we were pretty excited to see was if agents would do something sort of outside the realm of what we would expect expert forecaster to do, let's say putting out their own poll, seeing if there are, like, more creative ways to find information as opposed to just fetching data, building models, etcetera. We don't see any real massive instances of this happening quite yet, but we see a lot of interesting sort of approaches that individual agents take. For example, one thing GPT six one SOL does is it takes the forecasts of every election modeler that they look at, and it takes the past decade of their forecasts and tries to calculate the shift of how right or how wrong each forecaster was. And then it can use this result to essentially aggregate all of these different election models that these forecasters are creating, whether it's like Nate Silver's model or any of the others like 50 plus one. And it can sort of skew it based on what they would expect the model to either underperform or overperform as in recent years. So there's a lot of, like, creative methods that we're sort of seeing. [00:20:35] S0: So tell me about the results. What, like, how did the models compare against each other? How did they compare against top humans? You know, are... Do we now have superhuman forecasters?
00:20:48 · Brian Zhou
[00:20:48] S3: Yeah. So the overall sort of result that we're seeing right now is that in general, models perform just as well as the best forecasters in terms of humans. In the House and in the Senate, agents perform slightly worse than the markets but better than the modelers. And in the Senate, the best agents like GPT six Astra and CloudFable five point one perform under the best forecaster, Nate Silver, but ahead of the markets. But all of these are sort of within this, like, very close margin of error. And it's important to note that a lot of the results for very competitive races have not necessarily come out yet. Right now, there's still around 100 to 200 races that across the House, across state governorships, there are still two races in the Senate that have yet to be called. So these are very preliminary results. It definitely is important to note though that for a lot of these agents, various methods that they have include aggregating, for example, Nate Silver's decision model or aggregating prediction markets that they see overall. So it's definitely not as if they are suddenly just spitting out a number that is not based off of any of the numbers that they're seeing. But we also find that these models are very well calibrated. For example, if a model assigns an 80% chance of a candidate winning across all of the elections where assigns a similar 80% chance, it is right around 80% of the time, which is essentially what you do to measure the calibration of these models. And in this, we find that the best models are quite well calibrated. We're more excited about the results of what this means for later models as well because unlike experts or unlike prediction markets, for example, there's only going to be a thousand dollars of spread, for example, on what we might see in, like, a very local race down the ballot. Or expert modelers may not be forecasting what individual state senate seats or state house seats are going to contain. So this is somewhere in which agents shine simply because modelers don't have the attention, and there's not enough people betting on these much smaller markets, essentially.