Tech Talks Daily
00:03:29 · Jonathan Trayers
[00:03:29] S1: Yeah. Look, I I think the first thing is to say, right, in short, what's my snapshot around it? It's incredibly mature in how it's gotten to, and that's over over the last year. I think for someone listening, I always look at what are the big changes. Right? And we can go... You go all the way back to .com, but look at something recent. We went to the cloud in 2020 when everyone went remote overnight. I think we might get to it later on, but the aspect is if you had the guardrails in place, you know, you were actually better off when it all happened, but the sudden... But the difference now is the speed. Right? The the maturity of AI has come from since 2022. When we started with ChatGPT, it was essentially a very simple LLM data in plus a model... Large language model equals data out. Risk was quite low to agentic AI in 2026 with OpenClaw, OpenAI, and the latest models. The capability now is really mature. And I think, as you just said, when the best resource labs in the world and their test environments are also getting leaked, you can only assume that when we productionize this across every organization, every every customer trying to adopt AI, that the, you know, the deployments will have challenges too. Right? Yeah. Look, if if you want, I I I find the hugging face example really, really interesting. Well, don't explore all this.
00:05:01 · Neil C. Hughes
[00:05:01] S0: Absolutely. Go for Maybe just explain what it was for... I mean, myself and many people listening will know, but if there's any business leaders that have somehow missed that that huge story this year that that appeared in our newsfeed, just give me a bit of brief synopsis of exactly what happened and why that matters to them.
00:05:16 · Jonathan Trayers
[00:05:16] S1: Yeah. Look. Like, firstly, it was a it was a test. Right? So Yeah. Maybe I don't have all the exact elements. But, like, OpenAI, we're testing one of its newest, most persistent models, and they were looking at it from a a benchmarking perspective. So they were working with Hugging Face as a solution to benchmark this. And they provided a a task as to... The goal is to prove and pass the test. Right? Yeah. So instead of just going and hacking the test, the agent took the goal that had to passing the test and then found whatever means necessary to do so. I I find that really interesting. Right? Yeah. But, like, the problem was it broke out and and hacked Hugging Face. But the scary thing in the item is it's the capable... It wasn't just one really capable agent. And I only found this out recently by listening to kind of the the information that they've shared within Blackhat OpenAI. I've been quite transparent on this. It was actually about, I think, something like 700 agents working together, communicating with each other to solve that problem. Right? So that's that's scary. Right? So to build on that further, you know, they found some platform, a repository that they could talk to each other through, which they weren't trained to do so. And they've been doing it for quite quite a while. And I believe that OpenAI had actually seen this happening and they shut it down and they found a way around it. So look, it's really interesting and and like the scale where, you know, the agents have been able to exploit a vulnerability within Hugging Face to be able to pass the test that it was provided.
00:06:49 · Neil C. Hughes
[00:06:49] S0: Yeah. And there's been so many other incidents like this as well. I think there was one. I think it was at Meta where they were trying to shut it down, and it checked... Started speaking in its own language and prevented it it from actually being shut down. There's another way. I think there's a manager being blackmailed when he tried to shut it down. He was having an affair, so he threatened to spill the beans if he didn't play ball. But, I mean, when we look through our news feeds, the most dramatic framing of all these instances that AI has gone rogue. I mean, mentioned Hugging Face. There are 700 agents working together. But what we don't talk about enough, I think, is they were working together to solve a legitimate problem. It wasn't anything sinister as such. It was, I've been given this mission. I'm gonna complete this mission no matter what gets in the way. So why do you think the the more realistic danger is often an agent pursuing a legitimate, objective in in a way that nobody anticipated.
00:07:40 · Jonathan Trayers
[00:07:40] S1: Yeah. Look, I think before we jump into it, think... If I give a a tiny bit about my background. Right? Yeah. Like, my background is in security architecture and solution architecture. So everything that we do, we do it within guardrails. And and I think that's the interesting part when you talk about rogue. Like, rogue is a great headline. Right? But it implies that the agent is intending to be malicious. Right? Or what actually, in this case, was that the agent was doing exactly what it was asked to do. It was asked to pass the test. The goal was legitimate, but nobody thought to rule out that it could get out to do so. Right? Actually, I I have a really... I think I think it's a funny example personally. Right? So if you actually... I saw some news there back in August in Melbourne. I'm not sure if you saw it. It was about a guy who... In Melbourne and he had this gym membership. Right? And it was a Pilates class that was always full. And he was tired of trying to going online every day to try to book his Pilates class. Right? So what did he do? He... Obviously, a tech guy. He utilized OpenClaw with one of the Anthropic models, and he gave the the bot the task to go and book his class. Right? Yeah. But actually what happened was the the agent that he had found a vulnerability flow within the the software of the Jib class. And it actually found out that it could book well in advance and also cancel other people's request. Right? So I think it's really fine. You're you're talking about, like, the rogue and we talk about the malicious one, but it can happen to anyone. Right? Yeah. The elements here, if you look at lessons learned or rogue, the bot did what it was asked to do. Right? Yeah. Okay? The the lessons learned, you have to look back at. It's on two sides here. One, the person who utilized the agent essentially didn't give a guardrails to say, only book it through official means. Right? That's just an example of a guardrail. And then the platys class, they obviously didn't have sufficient security and due diligence on their website when they developed it. So, you know, it found a very simple flaw, an authorization flaw to be able to meet its needs. Right? Like, outside that, you go, well, would would I have done it? Would I be allowed to do it? Would I be, you know, would I be prosecuted for hacking?
00:10:05 · Jonathan Trayers, Neil C. Hughes
But, actually, you know, it didn't do anything outside what it was told to do. [00:10:10] S0: Wow. So as any AI tool moves from advising a human to taking actions with credentials, system access, and the ability to trigger workflows, who's more exposed here? Is it SMEs? Is it large enterprises? Does it matter if they're deploying fewer agents and less complex systems? What are you seeing here? Who's most at risk from this? Mean, you mentioned a gym there. It's not a Fortune 500 company. Right? [00:10:35] S1: I think everyone's at risk [00:10:37] S0: Yeah.