Tech Talks Daily
00:06:49 · Neil C. Hughes
[00:06:49] S0: Yeah. And there's been so many other incidents like this as well. I think there was one. I think it was at Meta where they were trying to shut it down, and it checked... Started speaking in its own language and prevented it it from actually being shut down. There's another way. I think there's a manager being blackmailed when he tried to shut it down. He was having an affair, so he threatened to spill the beans if he didn't play ball. But, I mean, when we look through our news feeds, the most dramatic framing of all these instances that AI has gone rogue. I mean, mentioned Hugging Face. There are 700 agents working together. But what we don't talk about enough, I think, is they were working together to solve a legitimate problem. It wasn't anything sinister as such. It was, I've been given this mission. I'm gonna complete this mission no matter what gets in the way. So why do you think the the more realistic danger is often an agent pursuing a legitimate, objective in in a way that nobody anticipated.
00:07:40 · Jonathan Trayers
[00:07:40] S1: Yeah. Look, I think before we jump into it, think... If I give a a tiny bit about my background. Right? Yeah. Like, my background is in security architecture and solution architecture. So everything that we do, we do it within guardrails. And and I think that's the interesting part when you talk about rogue. Like, rogue is a great headline. Right? But it implies that the agent is intending to be malicious. Right? Or what actually, in this case, was that the agent was doing exactly what it was asked to do. It was asked to pass the test. The goal was legitimate, but nobody thought to rule out that it could get out to do so. Right? Actually, I I have a really... I think I think it's a funny example personally. Right? So if you actually... I saw some news there back in August in Melbourne. I'm not sure if you saw it. It was about a guy who... In Melbourne and he had this gym membership. Right? And it was a Pilates class that was always full. And he was tired of trying to going online every day to try to book his Pilates class. Right? So what did he do? He... Obviously, a tech guy. He utilized OpenClaw with one of the Anthropic models, and he gave the the bot the task to go and book his class. Right? Yeah. But actually what happened was the the agent that he had found a vulnerability flow within the the software of the Jib class. And it actually found out that it could book well in advance and also cancel other people's request. Right? So I think it's really fine. You're you're talking about, like, the rogue and we talk about the malicious one, but it can happen to anyone. Right? Yeah. The elements here, if you look at lessons learned or rogue, the bot did what it was asked to do. Right? Yeah. Okay? The the lessons learned, you have to look back at. It's on two sides here. One, the person who utilized the agent essentially didn't give a guardrails to say, only book it through official means. Right? That's just an example of a guardrail. And then the platys class, they obviously didn't have sufficient security and due diligence on their website when they developed it. So, you know, it found a very simple flaw, an authorization flaw to be able to meet its needs. Right? Like, outside that, you go, well, would would I have done it? Would I be allowed to do it? Would I be, you know, would I be prosecuted for hacking?
00:10:05 · Jonathan Trayers, Neil C. Hughes
But, actually, you know, it didn't do anything outside what it was told to do. [00:10:10] S0: Wow. So as any AI tool moves from advising a human to taking actions with credentials, system access, and the ability to trigger workflows, who's more exposed here? Is it SMEs? Is it large enterprises? Does it matter if they're deploying fewer agents and less complex systems? What are you seeing here? Who's most at risk from this? Mean, you mentioned a gym there. It's not a Fortune 500 company. Right? [00:10:35] S1: I think everyone's at risk [00:10:37] S0: Yeah.
00:10:37 · Jonathan Trayers
[00:10:37] S1: In one way or another. There's two aspects in this case, and of a colleague I work with and we all talk around this in the industry, kind of to be able to reduce staff risk is all around being brilliant at the basics. Right? We're not trying to invent anything new to stop, to reduce the risk of your organization having an issue. But we look at being brilliant at the basics. Essentially, if you take it from a cyber security perspective, your risk is linked to the blast radius that's available. Right? So as an adviser, the worst case that an agent would do would be... Is to give a human the wrong answer and the human ignores that. Right? Yeah. In terms of an agent, the worst thing it can do is what access it has to it. So we're giving something with machine learning speed at 03:00 in the morning to be able to do an automate action and it can only... It can do whatever it has access to. Right? So, you know, it can only act based on the relevant permissions. So therefore, the damage it causes is linked to identity and the permissions. If I take my career and I started in cyber security fifteen plus years ago, I'd always seen cyber in terms of identity and access management as a really kind of administrative task back then. So it was just a task that had to be done. The word cyber security in the frameworks that we utilize today wasn't industrialized in in the manner that we utilize it day on day out right now. So therefore, like, if you look at that kind of kind of aspect right now, if we reduce the risk, we're essentially removing the amount of permissions. In my view, that's probably higher for a small medium enterprise. Why? In a small medium enterprise, they don't have the same level of regulation and due diligence. So they probably give people in their environment with more access than you would have in a regulated environment where you'd be looking down at the granular permissions that they need to do a specific task. Right? Yeah. But actually, when you look at an agent in that exact environment, they probably will give it more access than it should have. And because of that, the damage that it could cause if it goes wrong is higher. Right? So I think that's the really interesting one. If if you just today or this week look at the mayhem of PHP, right, as a as a really good example. There's two lenses to look at that PHP.
00:13:12 · Jonathan Trayers
We haven't found that vulnerability for ten plus years. Right? An AI frontier model found that vulnerability. If we take it from the advisory perspective, that's great. Everyone in the world will now know that we have to fix something. But actually, the time that from the disclosure of the vulnerability and having to patch is so short right now that there's so much, you know, publicity of the amount of websites being hacked because people aren't keeping up to the speed of the the patch management. And... Right? So kinda to close that point, right, that you're asking, I said be brilliant at the basics. Said Yep. If you have a rigorous process in your environment no matter what size you are, If you know that the vulnerability is raised, you have a process to immediately patch your devices and everyone knows your vulnerability management and you know your environment, you're probably in a good place. Right? So look, I think that the round way about it to... In that question is to say that, you know, the agentic piece and the taking actions does bring a lot more risk. But from an organization perspective, as long as you're pulling to the basics, probably are in in good hands.
00:14:30 · Neil C. Hughes
[00:14:30] S0: Yeah. So many great points there, especially around identity access management. I've seen so many new start ups appearing now to try and manage with this influx of agents that we're seeing. And I appreciate this next question. It is going to be almost a podcast episode on its own. But if we've got somebody listening and they're... Before that organization grants an agent access to a critical system, We're talking about guardrails, permissions, and monitoring controls. What should they be putting in place as standard? What what should be nonnegotiable right from the beginning?