Masters of Scale
00:04:41 · Meredith Whittaker, Jeff Berman
You have Perplexity's agentic browser hacked by a Reddit comment via prompt injection. You have 11% of Anthropics Claude Copilot experiment, or Claude Chrome experiment, 11% of the time, even with guardrails, it was not able to resist prompt injection. We have researchers showing a prompt injection attack on Google Calendar that led an agent to turn on the boiler in someone's smart home. So when we're talking about trust, I think the answer is fairly obvious. No, of course you shouldn't trust these fallible systems, But we should be pushing to build trustworthy systems. Because in security, trust comes with verification. I'm not asking you to signal. I'm not asking you to trust me. I'm asking you to look at our code and recognize that it's trustworthy, and if it isn't, come back to me, and we will fix it. [00:05:34] S3: Thank you. Neil, three minutes and twelve seconds. Great. So
00:05:42 · Neil Chilson
[00:05:42] S1: for twenty years, the Internet has been built on a bargain. Data that... About you that you couldn't use, companies pooled, they sold ads against it, and they used that money to fund very useful services. That bargain, meant that companies learned a lot about us, but we didn't learn that much about ourselves. AgenTic AI flips that bargain. We all now have, each of us has access to powerful software tools that can comb through the mountain of our data for us. So, I use a lot of the applications, a lot of the agent use cases that Meredith referred to. I give it access to my calendar, it manages that, it monitors my family's finances, I use it to troubleshoot all the time. Thousands of people do lots of other things that are like that, you know, comparison shopping online, and I I take the point that a lot of that's driven by convenience. But these are tasks that only the wealthy could delegate in the past. If you don't have an executive agent, these are the types of things that you have to do yourself. So, there is a huge amount of convenience here. Agents give us agency, the theme of this event. And yes, that does mean that we have to share our data, but that data is being used for our purposes, not the old bargain, of the Internet. And here's the surprise. These agents can... One of the things that they can make more convenient are some of the practices that we might use to increase privacy. So today, we trade privacy for convenience all of the time. But agents don't care about privacy. Or... Sorry. They may not care about privacy. We'll find out. But they they don't care about inconvenience. They don't care about inconvenience. An agent doesn't get bored reading privacy policies or debating between them. It doesn't get bored... It doesn't get confused by using Tor or NoScript in order to, you know, get through some of these complicated things. These are tools that humans find very burdensome and cumbersome. Agents don't care about that. They can do those types of practices for us. And so agents have the time and the patience to make privacy more convenient for us. Earlier today in the, in my my hotel room, I used Muse to unregister me to block, block me on five of the major, data brokers in The US, and and it took a six word prompt. That was helpful.
00:08:16 · Neil Chilson, John Donvan, Jeff Berman
Now, has spent her career protecting the privacy of communications, and so obviously, she's reasonably worried that agents will open a new window into our private lives. But look at the status quo. There are many windows into our private lives and I hope to convince you that agents can stand on our side of the glass. They can help us track the trackers and they can help us close the windows that we never even intended to open. So I totally agree with Meredith, we need to build more secure systems, but agents can improve the old internet bargain. They can do this by making privacy easier, and more convenient, and they can do it by having you use your data rather than somebody else. [00:09:03] S2: Thank [00:09:03] S3: you, Neil. Meredith, I hear Neil saying that the agents give us an opportunity not only for a better privacy bargain, but we can actually enhance our data privacy by using the agents. Why is he wrong?
00:09:15 · Meredith Whittaker
[00:09:15] S0: Well, I think the agents aren't ours in most cases. When we're talking about Meta's Muse, we had a person show up to someone else's house because Muse had given that person the address to that house, because that person said, hey, Muse, can you manage my Facebook marketplace for me? And Muse took that to mean we're giving out your address, and then impersonating you when this guy stands outside your door confused, wondering why you didn't come out to get the product he was gonna buy. You have Muse now publishing private information online, company revenue, loan application agreements, other proprietary data, that it's unclear how that leaked. But because of Muse's architecture, because of the way these convenient consumer agents are being built, again, they have access to everything. We're breaking down the sandboxes, which is a paradigm in security engineering. We are giving them root permissions, the key to your house, the key to your digital ecosystem to do what they can to combine that data, to make determinations about that data, and we recognize that there is a fatal flaw. This is prompt injection in LLM driven agents, where they don't... They aren't alive, they aren't human. Sorry, guys who think you are. But they they respond to this data as if it were an order, as if it were a fact. And so we have seen countless breaches. We have seen agents make perilous mistakes, and I think, yes, I could ask an agent to drive tour for me, but that's not, again, if we're talking about Meta's Muse, that's not my agent. Finally, we learned through reporting recently that Muse is logging people's contact lists, logging the people they talk to. So you install Muse, and I am in your contact list. Muse now has my information. They are logging when we talk to each other. They are creating dossiers about me and your communications with me, Cambridge Analytica style. And so you're making a choice for your convenience that also implicates me. Now, that might not be serious. We're privileged people. We're in San Francisco. Our threat model may be fairly moderate, but that could be extraordinarily serious for somebody working on human rights in an authoritarian country who is trying to communicate data with somebody outside of that country.
00:11:43 · Meredith Whittaker, Jeff Berman, Neil Chilson
And so this, again, I don't care about winning this debate, I care about looking this in the face and recognizing we are eating the seed corn of our cyber security when we install these things in the name of convenience without thinking through the perilous architectures that we have used to create them. [00:12:03] S3: So Neil, given the check... Given the given the challenge of segmenting and locking down data, why is the convenience value worth more than the risk we're creating here? [00:12:17] S1: Well, I I think it's easy for us to dismiss these types of challenges, but again, the the goal here is not that agents are perfect or that they're perfect protectors of security. The question is, how do they compare to the status quo? And the status quo is not great. Most people don't use Signal, unfortunately. And the status quo is full of people who are gathering information about us in part because we've made that convenience trade off over and over and over. So Meredith is totally right that we need secure agents. We need better agents, but the challenge is not to not use. Remember what the proposition is. The proposition is, should we trust agents with our information And the... And... Or should we not trust agents? And it's a conditional. It depends. It really depends on the agent. It really depends on the use and it really depends on the access that it's asking for. Most of the agents right now, I don't let them write online, I let them read online for the most part.
00:12:17 · Neil Chilson, Jeff Berman
[00:12:17] S1: Well, I I think it's easy for us to dismiss these types of challenges, but again, the the goal here is not that agents are perfect or that they're perfect protectors of security. The question is, how do they compare to the status quo? And the status quo is not great. Most people don't use Signal, unfortunately. And the status quo is full of people who are gathering information about us in part because we've made that convenience trade off over and over and over. So Meredith is totally right that we need secure agents. We need better agents, but the challenge is not to not use. Remember what the proposition is. The proposition is, should we trust agents with our information And the... And... Or should we not trust agents? And it's a conditional. It depends. It really depends on the agent. It really depends on the use and it really depends on the access that it's asking for. Most of the agents right now, I don't let them write online, I let them read online for the most part. [00:13:14] S3: And And I so think Meredith's it's all about those saying, trade then you've got risk of prompt injection. Right? You've you've got concerns on top of concerns. So if we can't segment and lock down data and we have concerns about prompt prompt injection, why does the convenience outweigh the risk?