Masters of Scale
00:42:15 · John Donvan, Meredith Whittaker
[00:42:15] S2: front And are malicious. [00:42:17] S0: Could be malicious. You know, we have a prompt injection that researchers showed was, you know, kind [00:42:23] S2: of able [00:42:24] S0: to, through a calendar setting, putting, you know, malicious text in a calendar setting, sort of a calendar invite, and then asking the agent sort of summarize my calendar for me, it summarizes everything including this text that prompted the agent to turn on a boiler, open the shutters and otherwise use someone's smart home devices through a calendar summary. Now, wasn't, you know, that was an example that was used, but this is the type of collateral consequences that are clearly possible through the, you know, allowing systems like this to... [00:43:04] S2: So it sounds like a good one to bring to you, Neil, in terms again of your relative optimism that the challenges are addressable. Let's talk about prompt injection. Do you have thoughts on that?
00:43:16 · Neil Chilson
[00:43:16] S1: So, you know, there's there's prompt injection, you can try to deal with prompt injection in lots of different ways. You might have filters on the front end, you certainly have controls on the back end about what the agent can do and permissions, and then in the middle you might try to train the agent to recognize these things. And I don't know that... All of these are defense in-depth. You want to do all of these things, and we can make we can make these systems safer from these types of attacks. I mean, is not that different from the types of attacks that we had with viruses where, you know, you used to play Russian roulette putting a floppy disk. I don't know if I had to explain the floppy disk to your audience. Yeah. Maybe. But, you know, you used to play Russian roulette putting a floppy disk in your hard drive or in... Into your floppy drive because it might have a virus on it. This was rampant in the nineties, and we we didn't solve it. It's not a solved problem, but it is a problem that is much... It is managed, and I think that this is a similar technology issue. To the extent there are signals in documents, you know, white text that we don't notice. I think the great example of that was the person who put that in their resume and it like bumped their resume to the top of the the screening pile because they, you know, that's what they had put in the prompt injection. That's a signal that computers can recognize just like any other signal, and I sort of like fundamentally think that if a human can recognize a signal, we can train a system to recognize that signal as well. And so the question is, how far, how many layers do we wanna put on? What kind of controls do we put on a system to make it safe enough for the uses in which people are using it? There is no such thing as a 100% safety. There's no such thing as a 100% privacy. And these are engineering problems and we can deal with them.
00:45:13 · Meredith Whittaker
[00:45:13] S0: I guess where I wanna index on this is like, who is we? Because we talk about we can train a system, we can have controls at the front end and the back end, and then, you know, my mom, my dad goes to the app store and downloads Muse because Muse is advertised as a real chipper little helper that can do everything, gives it permission to everything, has no idea that Muse is rummaging through their contact list, rummaging through their email, you know, able to send data off-site, you know, that clicking that terms of service gives it that type of pervasive access and suddenly it's spending a bunch of money on, you know, through their credit card on some website they never saw. You know, who is responsible here? Because I I don't feel comfortable at all with the idea that, like, in some subject... In some subjective future, we kind of will maybe do this so we should sort of, you know, move along into this and not, you know, recognize that there are real dangers now. Right? 11% of anthropics, you know, Claude, agentic tests, 11% of the time with these guardrails, it failed. So, why would we trust it now? And why wouldn't we put the burden of care, the duty of care on companies that are right now sort of staring down the barrel of trillion dollar valuations instead of having us be the sort of collateral crumple zones for, you know, the test guinea pigs for technology that, like, we know is constitutively insecure and not fit for purpose when that purpose needs to be responsibly execute these tasks without spilling my banking data online, which is a, you know, a cost I'm not willing to pay.
00:47:00 · Neil Chilson
[00:47:00] S1: So I I I mean, I think these companies under the law are liable for the vast majority of the types of harms that you're talking about. And if those types of harms happen, they have... They will face quite strict scrutiny both from the law... The legal community as well as from their investors. And so I do think that we have incentives there to solve these problems. That's what I mean by the we. I mean that we have a structure, and maybe we could tweak it, maybe we can make it better, but we have a structure that creates an incentive to deliver products that don't harm consumers. And that that that structure that... Or that structure should benefit consumers. And I think we could tweak those incentives, maybe we could tweak the the liability, but I think that structure is ultimately the path towards the we, which is society, figuring out how to use these tools in a way that is beneficial and that outweighs the harms that they could cause. And I I think that has been the path of every single technology that we've developed in the past. It's certainly been the path of many of the technologies that we all use every day. Liability is a real thing for companies that cause harm to consumers. We need to we need to impose that. I'm not trying to let anybody off the hook, but but the but the solution is not to stop the technology and say, hey, we're not gonna improve this. It's gonna say... It's to say, hey, here are some of the challenges. Let's try to tackle these challenges. Meredith,
00:48:30 · John Donvan
[00:48:30] S2: in this best foot forward part of Neil's argument, he's making the case that the agents actually deliver a whole lot of good to individuals and empowerment and convenience, etcetera. Do you refute that piece of it? Again, setting aside the trade offs that you're talking about, do you think he's right that they have a lot of benefits to deliver?
00:48:52 · Meredith Whittaker
[00:48:52] S0: I think, you know, look, where they work as intended, which is not all the time, they are convenience tools. You know, I don't think anyone's arguing against that. And again, agents is a very woolly term. You know, for coding, agents are doing things that are very very useful. Right? Coding is a field where we have a huge amount of data and where it's sort of objectively verifiable, your code runs or it crashes your compiler and you can kind of get a sense of what's there and sort of move on. I think when we're talking about these kind of consumer agents, which is where my concern is, because, you know, they are hollowing out Signal's ability to provide privacy by demanding or coercing data access in ways that are, you know, I think architecturally irresponsible at a technical level. You know, irrespective of the convenience of, you know, not having to do your own Christmas shopping, right, you're looking at a hollowing out of our fundamental cyber security architectures in a way that is again, you know, it is reconstituting more and more power and data access in the hands of the existing winners of the AI ecosystem. So, you have agents who are sort of grabbing data from the application layer, you have agentic integrations that are becoming kind of the front end of the internet, the interface of the internet, making it harder and harder for independent website developers, independent application developers, you know, to have a place in that ecosystem. So, you know, sure they are... They can be convenient, but the argument isn't about convenience. The argument here is about should we trust them? And I think all of the evidence points to they are not trustworthy, so we should not