Decoder with Nilay Patel
00:38:09 · Nilay Patel, S3
[00:38:09] S2: There are there are several costs associated with this. And one, I think, is just like the most common in business, which is, well, if we find out there's a problem, we'll have to pay to fix it. So what if we just wait? Right? Like, you can see that coming. What if we find out that we are catastrophically in trouble? And then there's how much will it cost to fix it? And then there's what will we do to fix it? And the argument that I've heard about what will we do to fix it is, well, we'll just unleash a swarm of defensive AI tools that are watching the networks and watching the code. And they'll make a bunch of autonomous decisions, and they'll fight the autonomous script kiddies who are armed with RTX on their Mac studios. And something about that argument in my head. I've heard, for example, Palo Alto Networks CEO, Nikesh has made this argument. Right? You need you need to give the defensive agent some autonomous capabilities. And I'm like, well, hold on. Like, we... OpenAI can't control the swarm that went and attacked Hugging Face. How would I control the defensive agent from doing something equally damaging to prevent an attack? And I I don't think that there's a technical solution there that makes it clear what to do once you found all of the vulnerabilities. [00:39:16] S3: Couple things. One [00:39:18] S2: I told you I was gonna only ask dumb questions. I just wanna No.
00:39:18 · Nilay Patel, S3
[00:39:18] S2: I told you I was gonna only ask dumb questions. I just wanna No. [00:39:21] S3: Okay. That is a fantastic question. You have not asked a dumb question yet. I'm I'm honest. I... You have not asked a dumb question yet. That's a fantastic question. Okay. I'm gonna give you a shorthand that you can use going forward. If somebody actually uses the term swarm, then [00:39:35] S2: I don't know if... I mean, to be clear, I don't know if Nikesh used the word swarm.
00:39:38 · S3
[00:39:38] S3: Okay. I'm not saying he did. Yeah. And I have great respect for Nikesh. He is one of the best businessmen in in all of security. So I... This is not on him, be... But Swarm is used all the time. Yeah. But when I hear somebody use Swarm, I immediately discount all of their technical opinions by about 80%. The... It is going to become... It is already, and it will become the most overused term in all of security. Okay. The actual, quote, unquote, swarm from OpenAI was something between a different time, 700 to 1,400 agents. That had to run on... It it... It's hard to tell because we don't have the actual, like, size of the models, but, you know, 300,000,000 to almost $1,000,000,000 in hardware. That doesn't include the actual data center cost. It... The actual hugging face attack was... I I did some back of the envelope map... Math. It may be a million dollars in electricity. Right? No attacker is going to do that. And and in the end, that was stupid because these things were actually trapped. They were not prompted to attack Hugging Face. Most of them were doing something totally different and were kind of enlisted by the one thing that actually wanted to get to Hugging Face. They used a tiny fraction of their token budget to help their little friend, and then they had to tunnel their way out. So it was the most inefficient way to possibly attack Hugging Face. Almost nobody is going to be attacked by a swarm. So if somebody says swarm, they do not actually understand at a low... At the bottom technical level what's going on. The RTX attacks are almost certainly one single agent, maybe two.
00:41:17 · Nilay Patel, S3
[00:41:17] S2: By the way, I was the person who said swarm, just to be clear in this one. [00:41:20] S3: I Keep going. Right. And so now you won't. [00:41:22] S2: That I am. [00:41:22] S3: You can use that as a good filter for your future guest. It is driving me insane because it's everywhere. Right? It is absolutely everywhere. A couple of things are correct. One, we are going to have to turn over defensive actions at least the first couple of levels to AI. It is... That has already been true just because, yo, any reasonable sized organization generates terabytes of logs, if not petabytes of logs. So you already have AI, if not LLMs, much more traditional AI, like, classifiers looking at it. But then actual decisions need to be turned over to [00:41:59] S2: AI Yeah.
00:42:00 · S3
[00:42:00] S3: With human oversight. That is true. There is no way to just release defensive agents on your network, but you can build defensive agents that are plugged in to lots of things. They are plugged into your logs. They are plugged into your EDR. They are plugged into your cloud, and they have the instructions on what they're allowed to do. My preference is there are deterministic limits to what they're allowed to do. Those are hard to necessarily create ahead of time, but they are important. So, for example, we are building all this on top of Devin, our own SOC. Mhmm. Right? But we are putting in place deterministic limits. Like, it has the ability to turn off an account in Okta, but only so many accounts, so it can't turn off the entire company. Like, if it goes nuts and it says, everybody's a North Korean, it can't turn off everybody in the company. If it hits its limit, you know, no matter what we get paged, but if it hits its limit, we... It pages us a lot and says, hit my limit. Right? And then a human being has to then approve anybody above that limit. Limit. Right? Yeah. Obviously, if it turns off five people, then something has gone really, really wrong. Right? And so, yes, you need defense, but there's no way to just, like, let a bunch of bots loose to do all the defense. The only way we're gonna make it through this is you have to autonomously fix software bugs. So I am currently helping a company where we are onboarding all of their repositories. We are scanning with Mythos and a bunch of other models, and we are finding and fixing thousands of bugs, verifying with tests that we are writing, merging those PRs automatically. If you are renovating old code, the only way you're gonna survive this era is to do that on an automatic basis. You you can't... That would take years with humans, and you can do it in days. And, certainly, that is the place I would start, and then the next step is the operational side for sure.
00:43:56 · Nilay Patel, S3
[00:43:56] S2: It feels like I just keep swirling back to the the economics of all of this, But it it really feels like the other way you could solve it is by saying to the big model companies, if your models are used to go attack a company, you'll be liable for it. Right? We're gonna make it really costly, and then they should figure out whatever alignment that they need to figure out whether or not that's even possible. And the pressure on the open weight models will go up because we'll all be paying the frontier companies for defense, and they'll get better and better and better because we've created some economic condition for them to get better. But it doesn't seem like you think that's likely. Yeah. I just saw your face. [00:44:34] S3: I I think I think it's really dangerous. So so there's two different kinds of the models are attacking. So the the models have escaped and directly attacked people. I think there should be liability there. Right? But that is agent harnesses running at companies using their computers, their IP addresses, attacking people. That is, I wouldn't say totally solved, but it is much, much better. Like, the stories we're hearing now are still stories from the Hugging Face era. They're just coming out lately. [00:45:03] S2: I like that the era was, like, three months ago.