Tech Talks Daily
00:15:03 · Jonathan Trayers
[00:15:03] S1: I think the the first thing you look at it is that if I look at the guardrails, you have... We... I feel something in in my day to day as kind of an enterprise AI security model or blueprint. Right? And it it essentially... It it comes in layers. It's a defense in-depth. Right? So you're talking around different layers around governance, you're talking around aspects around the protection mechanism, and then you're looking at what the AI can do, so around the decision making. Right? And then finally is the control after and the visibility. Firstly, as an organization, start by putting in a framework and a governance model. And the non negotiable firstly is what use case are you bringing it in? Have you permission to bring that into your organization? And have you considered the risk around it? Right? Too many organizations are just adopting AI on the fly. If you are in an organization today and you said that I'm making a big change the way we work, that starts from typically the board down, giving investments to senior management, accepting the risk of the potential new projects bringing brought through. And with that risk, you have a financial commitment and a team stood up to do that. However, with AI right now, we completely seem to have missed that. Right? We just think that anyone can be their own developer and they can add permissions. And especially potentially in the small and medium enterprises where, you know, you don't have the same level of governance. Right? In in place. So look, to start, if you're utilizing agents specifically, because I think agents is where the risks are in this case, you know, we're talking about in that protection domain. What permissions does the system have access to and limit us, firstly. Second one, be very tight in terms of the prompt. So, like, if you think about that AI decision making, it it happens at a kind of three stages. That's how I look at it. Right? It's your inputs. Right? So what systems have you access to? What data has that access to? You have the model itself on the prompt. So what model has it been trained on and what data? And then the prompt equals the outputs. So the only thing that you can limit here is the inputs and the decision making element. So you need to bring tighter control around what's allowed and the guardrails in place.
00:17:26 · Jonathan Trayers, Neil C. Hughes
And then after it's monitored, monitor it and have a kill switch. And a kill switch should be that every agent that talks to a system specifically, so the agent should have its own identity, And also, every single build connection that's allowed to should have its own specific identity that you can cut off all of them at any point and have that fully controlled. [00:17:49] S0: So much great advice there. And as we've said a few times here, behind the headlines of AI going rogue or agents going rogue is that they're just trying to complete a task or over time will slowly drift away from their initial objective. So how should businesses better test for unexpected behavior when the the possible failure modes are so difficult to predict in advance?
00:18:13 · Jonathan Trayers
[00:18:13] S1: That's a difficult one, to be honest. You can't predict predict every failure mode. That's the point, like, you've just said. So stop trying to essentially predict them all and reduce the blast radius. I think it's back to my last point. Yeah. Assume that'll surprise you. Look at cyber today. Right? I think anyone who's been successful in cyber is assuming breach and have the the necessary kind of protections and layer defense inside it, such as micro segmentation, really strong identity, so that if their perimeter is breached during lockdown to a specific area, right, and that's controlled, and then monitor and continuously detect what's happening. So if you're thinking about bringing in an agent, if you take that same philosophy and one, you verify it within a controlled environment, a sandbox for example, watch what access you're giving it to it and making sure that its outcome is exactly what you expect. Next, move it towards a shadow mode. As I kind of mentioned to you in terms of my day to day job, it's around deterministic autonomous flows. So therefore you have human in the loop. So test it for a few weeks with human in the loop. And secondly, you shouldn't give a finance something that processes your invoices and something that pays your invoices the same. You can have a human in the loop. Two potential agents doing two jobs, but someone guard with the guardrails in between. So make sure that when you're testing the behavior, you know what happens. And then have a runbook and kinda monitor as it grows. See what is happening. See the logic behind it. A lot of people just turn on agents and leave them be. Right? That's where, you know, we would have problems down the line. And from a industry perspective, my experience is this whole centralized platform for managing agents, Example, agent three six five as a as a Microsoft solution. You have times three b as well in solutions that we we work with as well. And they're all new. Right? So you can't assume that every organization knows how to adopt them or has adopted them because most investments of this nature typically come with a five year plan. But five years ago, this didn't exist. So how put this beyond a five year plan? So I think the basics is start small. Look at that that blueprint of keeping isolated, only give it the same permissions, and monitor it, and I think you'll be okay.
00:20:50 · Neil C. Hughes
[00:20:50] S0: And the other big question we've gotta bring in here is accountability. Of course, in the old days, human tech teams would all point to each other like the Spider Man memes, and it's not my network. It's it's your application. It's not my application. It's the the third party that manages it. So right now, though, how should accountability be divided between the vendor, the deploying organization, and the the people who approved its access when an autonomous agent could cause harm or something unexpected? Where where does accountability sit?
00:21:21 · Jonathan Trayers
[00:21:21] S1: If we just take AI outside this for a second. Right? Just think about where accountability and any kind of specific action is. You can outsource the task. You can't outsource accountability. You know, essentially, if you look at any of the directives across Europe, miss two, DORA, EU AI Act, as an example, the risk is at the management's desk. A regulator won't accept that an agent did it no more than they'll accept the intern did it. So they count the miller that he sits at the business. Right? So that's the first kind of threat. Right? Then you look at the agent itself. Right? The agent itself, you purchased an agent with clear documented guidelines. Right? So I don't know how the accountability would would be in in a in a in a... Answer that you are deploying it with your environment. You are responsible there. You have purchased a product to do something and you you own the accountability and responsibility. However, I do find when I was looking at the articles over the last couple of days, and I'm not sure if you saw anything around the new AI safety bill that they've... Yeah. And and Anthropic pitches to say that the bill should block a model if something like 50 people should be killed. I... Up to 50 people. And I I can't, like, comprehend that. Like, that is absolute madness. Right? The fact that we are saying that they would block a model if a large number of people. I think I think you killed was the the lines that I've read across the the news articles, but I think that comes back to accountability. Firstly, I I think the... In my opinion, the model owners are responsible for for one level of guardrails in place, and then the day to day use is the organization's management and board.
00:23:23 · Neil C. Hughes
[00:23:23] S0: Lovely. And finally, for anyone listening, let's give them a a few valuable takeaways here. If we have someone listening in a... Maybe an SME, they wanna move quickly enough to benefit from AgenTek AI without, leaning too much on governance or slowing things down or creating problems further on down the line. Any any advice, tips on on what they should be doing right from the outset?