Decoder with Nilay Patel
00:41:17 · Nilay Patel, S3
[00:41:17] S2: By the way, I was the person who said swarm, just to be clear in this one. [00:41:20] S3: I Keep going. Right. And so now you won't. [00:41:22] S2: That I am. [00:41:22] S3: You can use that as a good filter for your future guest. It is driving me insane because it's everywhere. Right? It is absolutely everywhere. A couple of things are correct. One, we are going to have to turn over defensive actions at least the first couple of levels to AI. It is... That has already been true just because, yo, any reasonable sized organization generates terabytes of logs, if not petabytes of logs. So you already have AI, if not LLMs, much more traditional AI, like, classifiers looking at it. But then actual decisions need to be turned over to [00:41:59] S2: AI Yeah.
00:42:00 · S3
[00:42:00] S3: With human oversight. That is true. There is no way to just release defensive agents on your network, but you can build defensive agents that are plugged in to lots of things. They are plugged into your logs. They are plugged into your EDR. They are plugged into your cloud, and they have the instructions on what they're allowed to do. My preference is there are deterministic limits to what they're allowed to do. Those are hard to necessarily create ahead of time, but they are important. So, for example, we are building all this on top of Devin, our own SOC. Mhmm. Right? But we are putting in place deterministic limits. Like, it has the ability to turn off an account in Okta, but only so many accounts, so it can't turn off the entire company. Like, if it goes nuts and it says, everybody's a North Korean, it can't turn off everybody in the company. If it hits its limit, you know, no matter what we get paged, but if it hits its limit, we... It pages us a lot and says, hit my limit. Right? And then a human being has to then approve anybody above that limit. Limit. Right? Yeah. Obviously, if it turns off five people, then something has gone really, really wrong. Right? And so, yes, you need defense, but there's no way to just, like, let a bunch of bots loose to do all the defense. The only way we're gonna make it through this is you have to autonomously fix software bugs. So I am currently helping a company where we are onboarding all of their repositories. We are scanning with Mythos and a bunch of other models, and we are finding and fixing thousands of bugs, verifying with tests that we are writing, merging those PRs automatically. If you are renovating old code, the only way you're gonna survive this era is to do that on an automatic basis. You you can't... That would take years with humans, and you can do it in days. And, certainly, that is the place I would start, and then the next step is the operational side for sure.
00:43:56 · Nilay Patel, S3
[00:43:56] S2: It feels like I just keep swirling back to the the economics of all of this, But it it really feels like the other way you could solve it is by saying to the big model companies, if your models are used to go attack a company, you'll be liable for it. Right? We're gonna make it really costly, and then they should figure out whatever alignment that they need to figure out whether or not that's even possible. And the pressure on the open weight models will go up because we'll all be paying the frontier companies for defense, and they'll get better and better and better because we've created some economic condition for them to get better. But it doesn't seem like you think that's likely. Yeah. I just saw your face. [00:44:34] S3: I I think I think it's really dangerous. So so there's two different kinds of the models are attacking. So the the models have escaped and directly attacked people. I think there should be liability there. Right? But that is agent harnesses running at companies using their computers, their IP addresses, attacking people. That is, I wouldn't say totally solved, but it is much, much better. Like, the stories we're hearing now are still stories from the Hugging Face era. They're just coming out lately. [00:45:03] S2: I like that the era was, like, three months ago.
00:45:03 · Nilay Patel, S3
[00:45:03] S2: I like that the era was, like, three months ago. [00:45:06] S3: Right. But it's like Yeah. Basically, there's been a massive change. This was effectively mostly an organizational problem. It was not a technology problem. It is that the PhD researchers at these companies are gods on top of Olympus, and the security teams are plumbers and were not allowed to touch the evaluation networks. And all of a sudden, this problem happened, and the security teams are like, hey. How do you like us now? [00:45:33] S2: Maybe you
00:45:33 · Nilay Patel, S3
[00:45:33] S2: Maybe you [00:45:34] S3: should have talked to us. Right? Yes. And so that was the problem with, like, OpenAI. And then, like, the Anthropic and Meta stuff is they had this. They all shared this outsourced provider who did a really bad job, right, and has gone slapped hard. And so that is solvable. Right? That is not the problem here. The problem here is not OpenAI Anthropics models escaping from OpenAI Anthropic causing problems. The problem is is bad guys using models from a variety of sources. There's already a problem of defenders getting refusals from OpenAI Anthropic, and it is a real, real challenge that if I am trying to do defense of anthropic being like, oh, it looks like you're a bad guy. So I think we have to be extremely careful, create a situation where if I'm trying to do defense and I'm the middle of the worst day of my career, that all of a sudden I get cut off. I think we have to be extremely careful about that because it is actually... As somebody who now has this challenge, it's actually quite challenging to disambiguate between people doing defense and offense in some cases. We are getting better collectively at this, but it is a challenge. It's also something that's really pissing the rest of the world off because, you know, Americans are, like, 5% of the planet. There are lots of people who need to do cyber defense around the world, and they are they are treated much more poorly. Europe, Asia, legitimate companies are treated much more poorly from a cyber defense perspective.
00:46:54 · Nilay Patel, S3
[00:46:54] S2: This is, if you add it all up, this is a good argument to slow down. Right? It... To say, we need to catch up to the capabilities, the models. We need to build some of these systems. We need to initiate everyone into a way of thinking where the cost of prevention is is worth it and not just the cost of the cure. Do you think that it's likely that we we will slow down? Do think that we should? [00:47:16] S3: If I had my druthers, I would be fine slowing down on the frontier and then catching up on the cheap... Cheaper models for defense. I'm a defense guy. I have to, like, live in the world that other people who work six blocks for me Yeah. And, you know, who are way, way, way, way richer. Make... I live in their world, man. Yeah. So, like, I get asked this question all the time, and it's like, I just do the best I can within the world other people make. [00:47:44] S2: Yeah. Do you think that the the big labs have woken up? You you mentioned that their cybersecurity teams, they've gotten more access. They've they've stopped what you might call the hugging face error, or at least they've they've changed the structures of those companies. Do you think that will help?