Tech Talks Daily
00:10:37 · Jonathan Trayers
[00:10:37] S1: In one way or another. There's two aspects in this case, and of a colleague I work with and we all talk around this in the industry, kind of to be able to reduce staff risk is all around being brilliant at the basics. Right? We're not trying to invent anything new to stop, to reduce the risk of your organization having an issue. But we look at being brilliant at the basics. Essentially, if you take it from a cyber security perspective, your risk is linked to the blast radius that's available. Right? So as an adviser, the worst case that an agent would do would be... Is to give a human the wrong answer and the human ignores that. Right? Yeah. In terms of an agent, the worst thing it can do is what access it has to it. So we're giving something with machine learning speed at 03:00 in the morning to be able to do an automate action and it can only... It can do whatever it has access to. Right? So, you know, it can only act based on the relevant permissions. So therefore, the damage it causes is linked to identity and the permissions. If I take my career and I started in cyber security fifteen plus years ago, I'd always seen cyber in terms of identity and access management as a really kind of administrative task back then. So it was just a task that had to be done. The word cyber security in the frameworks that we utilize today wasn't industrialized in in the manner that we utilize it day on day out right now. So therefore, like, if you look at that kind of kind of aspect right now, if we reduce the risk, we're essentially removing the amount of permissions. In my view, that's probably higher for a small medium enterprise. Why? In a small medium enterprise, they don't have the same level of regulation and due diligence. So they probably give people in their environment with more access than you would have in a regulated environment where you'd be looking down at the granular permissions that they need to do a specific task. Right? Yeah. But actually, when you look at an agent in that exact environment, they probably will give it more access than it should have. And because of that, the damage that it could cause if it goes wrong is higher. Right? So I think that's the really interesting one. If if you just today or this week look at the mayhem of PHP, right, as a as a really good example. There's two lenses to look at that PHP.
00:13:12 · Jonathan Trayers
We haven't found that vulnerability for ten plus years. Right? An AI frontier model found that vulnerability. If we take it from the advisory perspective, that's great. Everyone in the world will now know that we have to fix something. But actually, the time that from the disclosure of the vulnerability and having to patch is so short right now that there's so much, you know, publicity of the amount of websites being hacked because people aren't keeping up to the speed of the the patch management. And... Right? So kinda to close that point, right, that you're asking, I said be brilliant at the basics. Said Yep. If you have a rigorous process in your environment no matter what size you are, If you know that the vulnerability is raised, you have a process to immediately patch your devices and everyone knows your vulnerability management and you know your environment, you're probably in a good place. Right? So look, I think that the round way about it to... In that question is to say that, you know, the agentic piece and the taking actions does bring a lot more risk. But from an organization perspective, as long as you're pulling to the basics, probably are in in good hands.
00:14:30 · Neil C. Hughes
[00:14:30] S0: Yeah. So many great points there, especially around identity access management. I've seen so many new start ups appearing now to try and manage with this influx of agents that we're seeing. And I appreciate this next question. It is going to be almost a podcast episode on its own. But if we've got somebody listening and they're... Before that organization grants an agent access to a critical system, We're talking about guardrails, permissions, and monitoring controls. What should they be putting in place as standard? What what should be nonnegotiable right from the beginning?
00:15:03 · Jonathan Trayers
[00:15:03] S1: I think the the first thing you look at it is that if I look at the guardrails, you have... We... I feel something in in my day to day as kind of an enterprise AI security model or blueprint. Right? And it it essentially... It it comes in layers. It's a defense in-depth. Right? So you're talking around different layers around governance, you're talking around aspects around the protection mechanism, and then you're looking at what the AI can do, so around the decision making. Right? And then finally is the control after and the visibility. Firstly, as an organization, start by putting in a framework and a governance model. And the non negotiable firstly is what use case are you bringing it in? Have you permission to bring that into your organization? And have you considered the risk around it? Right? Too many organizations are just adopting AI on the fly. If you are in an organization today and you said that I'm making a big change the way we work, that starts from typically the board down, giving investments to senior management, accepting the risk of the potential new projects bringing brought through. And with that risk, you have a financial commitment and a team stood up to do that. However, with AI right now, we completely seem to have missed that. Right? We just think that anyone can be their own developer and they can add permissions. And especially potentially in the small and medium enterprises where, you know, you don't have the same level of governance. Right? In in place. So look, to start, if you're utilizing agents specifically, because I think agents is where the risks are in this case, you know, we're talking about in that protection domain. What permissions does the system have access to and limit us, firstly. Second one, be very tight in terms of the prompt. So, like, if you think about that AI decision making, it it happens at a kind of three stages. That's how I look at it. Right? It's your inputs. Right? So what systems have you access to? What data has that access to? You have the model itself on the prompt. So what model has it been trained on and what data? And then the prompt equals the outputs. So the only thing that you can limit here is the inputs and the decision making element. So you need to bring tighter control around what's allowed and the guardrails in place.
00:17:26 · Jonathan Trayers, Neil C. Hughes
And then after it's monitored, monitor it and have a kill switch. And a kill switch should be that every agent that talks to a system specifically, so the agent should have its own identity, And also, every single build connection that's allowed to should have its own specific identity that you can cut off all of them at any point and have that fully controlled. [00:17:49] S0: So much great advice there. And as we've said a few times here, behind the headlines of AI going rogue or agents going rogue is that they're just trying to complete a task or over time will slowly drift away from their initial objective. So how should businesses better test for unexpected behavior when the the possible failure modes are so difficult to predict in advance?
00:18:13 · Jonathan Trayers
[00:18:13] S1: That's a difficult one, to be honest. You can't predict predict every failure mode. That's the point, like, you've just said. So stop trying to essentially predict them all and reduce the blast radius. I think it's back to my last point. Yeah. Assume that'll surprise you. Look at cyber today. Right? I think anyone who's been successful in cyber is assuming breach and have the the necessary kind of protections and layer defense inside it, such as micro segmentation, really strong identity, so that if their perimeter is breached during lockdown to a specific area, right, and that's controlled, and then monitor and continuously detect what's happening. So if you're thinking about bringing in an agent, if you take that same philosophy and one, you verify it within a controlled environment, a sandbox for example, watch what access you're giving it to it and making sure that its outcome is exactly what you expect. Next, move it towards a shadow mode. As I kind of mentioned to you in terms of my day to day job, it's around deterministic autonomous flows. So therefore you have human in the loop. So test it for a few weeks with human in the loop. And secondly, you shouldn't give a finance something that processes your invoices and something that pays your invoices the same. You can have a human in the loop. Two potential agents doing two jobs, but someone guard with the guardrails in between. So make sure that when you're testing the behavior, you know what happens. And then have a runbook and kinda monitor as it grows. See what is happening. See the logic behind it. A lot of people just turn on agents and leave them be. Right? That's where, you know, we would have problems down the line. And from a industry perspective, my experience is this whole centralized platform for managing agents, Example, agent three six five as a as a Microsoft solution. You have times three b as well in solutions that we we work with as well. And they're all new. Right? So you can't assume that every organization knows how to adopt them or has adopted them because most investments of this nature typically come with a five year plan. But five years ago, this didn't exist. So how put this beyond a five year plan? So I think the basics is start small. Look at that that blueprint of keeping isolated, only give it the same permissions, and monitor it, and I think you'll be okay.