Masters of Scale
00:01:46 · Jeff Berman
[00:01:46] S3: With us today, Meredith Whittaker, the president of Signal, and Neil Chilson, the head of AI policy at the Abundance Institute. We have a tight clock. I'll remind you that the resolution on the table is, should we stop trusting AI agents with our information? Three minute opening statements for each of you. Meredith, we'll start with you.
00:02:05 · Meredith Whittaker
[00:02:05] S0: Alright. I don't know. Did we start trusting agents with our information or are we just kinda using them because we need to? Right? Look, I run Signal, and my job is pretty serious. I need to protect people's data because they use Signal for privacy in contexts where it's often a life or death situation. And so this is not a theoretical debate. I don't really care about winning. What I care about is facing these issues clearly and coming up with solutions so we can build trustworthy technology that actually deserves our trust. Now, look, you want to use an agent, it is convenient, you want to book your birthday party and invite your friends. That's a great agentic marketing case, and it's the kind of thing we hear from agent companies all the time. But, of course, in order for an agent that is running on your consumer device to do that, it needs access to your contacts, it needs access to know who your friends are, it needs your credit card information, it needs to be able to drive your browser and effectively impersonate you to search for a restaurant, to put your credit card down, to make that reservation. It needs, in this case, maybe access to your calendar to figure out what dates are available for you and your friends. And then if you're using Signal, like you probably should be, it needs access to your Signal messages to communicate with your group text and to figure out with your friends what actually works to book that party. So, on the one hand, you have the promise of a magic genie that's gonna, you know, do something that's really convenient. Right? This stuff takes time. It can be inefficient. And on the other hand, you have the paradigm of what a consumer agent is doing right now. Which is, one, it needs deep permissions, what we'd call root permissions in, you know, my technical world. It needs to be able to act without explicit consent, doing a number of steps to achieve a goal, or at least it thinks achieve a goal. And it needs to be able to respond to natural language, to text, in context, as if that text were a command, and follow that text command. And this is where we get to prompt injection, which is an extraordinarily serious issue, and this is how you hijack an agent. There's already a huge number of examples of prompt injection leading to problems. A zero click exploit pulled corporate data off of Microsoft's Copilot.
00:04:41 · Meredith Whittaker, Jeff Berman
You have Perplexity's agentic browser hacked by a Reddit comment via prompt injection. You have 11% of Anthropics Claude Copilot experiment, or Claude Chrome experiment, 11% of the time, even with guardrails, it was not able to resist prompt injection. We have researchers showing a prompt injection attack on Google Calendar that led an agent to turn on the boiler in someone's smart home. So when we're talking about trust, I think the answer is fairly obvious. No, of course you shouldn't trust these fallible systems, But we should be pushing to build trustworthy systems. Because in security, trust comes with verification. I'm not asking you to signal. I'm not asking you to trust me. I'm asking you to look at our code and recognize that it's trustworthy, and if it isn't, come back to me, and we will fix it. [00:05:34] S3: Thank you. Neil, three minutes and twelve seconds. Great. So
00:05:42 · Neil Chilson
[00:05:42] S1: for twenty years, the Internet has been built on a bargain. Data that... About you that you couldn't use, companies pooled, they sold ads against it, and they used that money to fund very useful services. That bargain, meant that companies learned a lot about us, but we didn't learn that much about ourselves. AgenTic AI flips that bargain. We all now have, each of us has access to powerful software tools that can comb through the mountain of our data for us. So, I use a lot of the applications, a lot of the agent use cases that Meredith referred to. I give it access to my calendar, it manages that, it monitors my family's finances, I use it to troubleshoot all the time. Thousands of people do lots of other things that are like that, you know, comparison shopping online, and I I take the point that a lot of that's driven by convenience. But these are tasks that only the wealthy could delegate in the past. If you don't have an executive agent, these are the types of things that you have to do yourself. So, there is a huge amount of convenience here. Agents give us agency, the theme of this event. And yes, that does mean that we have to share our data, but that data is being used for our purposes, not the old bargain, of the Internet. And here's the surprise. These agents can... One of the things that they can make more convenient are some of the practices that we might use to increase privacy. So today, we trade privacy for convenience all of the time. But agents don't care about privacy. Or... Sorry. They may not care about privacy. We'll find out. But they they don't care about inconvenience. They don't care about inconvenience. An agent doesn't get bored reading privacy policies or debating between them. It doesn't get bored... It doesn't get confused by using Tor or NoScript in order to, you know, get through some of these complicated things. These are tools that humans find very burdensome and cumbersome. Agents don't care about that. They can do those types of practices for us. And so agents have the time and the patience to make privacy more convenient for us. Earlier today in the, in my my hotel room, I used Muse to unregister me to block, block me on five of the major, data brokers in The US, and and it took a six word prompt. That was helpful.
00:08:16 · Neil Chilson, John Donvan, Jeff Berman
Now, has spent her career protecting the privacy of communications, and so obviously, she's reasonably worried that agents will open a new window into our private lives. But look at the status quo. There are many windows into our private lives and I hope to convince you that agents can stand on our side of the glass. They can help us track the trackers and they can help us close the windows that we never even intended to open. So I totally agree with Meredith, we need to build more secure systems, but agents can improve the old internet bargain. They can do this by making privacy easier, and more convenient, and they can do it by having you use your data rather than somebody else. [00:09:03] S2: Thank [00:09:03] S3: you, Neil. Meredith, I hear Neil saying that the agents give us an opportunity not only for a better privacy bargain, but we can actually enhance our data privacy by using the agents. Why is he wrong?
00:09:15 · Meredith Whittaker
[00:09:15] S0: Well, I think the agents aren't ours in most cases. When we're talking about Meta's Muse, we had a person show up to someone else's house because Muse had given that person the address to that house, because that person said, hey, Muse, can you manage my Facebook marketplace for me? And Muse took that to mean we're giving out your address, and then impersonating you when this guy stands outside your door confused, wondering why you didn't come out to get the product he was gonna buy. You have Muse now publishing private information online, company revenue, loan application agreements, other proprietary data, that it's unclear how that leaked. But because of Muse's architecture, because of the way these convenient consumer agents are being built, again, they have access to everything. We're breaking down the sandboxes, which is a paradigm in security engineering. We are giving them root permissions, the key to your house, the key to your digital ecosystem to do what they can to combine that data, to make determinations about that data, and we recognize that there is a fatal flaw. This is prompt injection in LLM driven agents, where they don't... They aren't alive, they aren't human. Sorry, guys who think you are. But they they respond to this data as if it were an order, as if it were a fact. And so we have seen countless breaches. We have seen agents make perilous mistakes, and I think, yes, I could ask an agent to drive tour for me, but that's not, again, if we're talking about Meta's Muse, that's not my agent. Finally, we learned through reporting recently that Muse is logging people's contact lists, logging the people they talk to. So you install Muse, and I am in your contact list. Muse now has my information. They are logging when we talk to each other. They are creating dossiers about me and your communications with me, Cambridge Analytica style. And so you're making a choice for your convenience that also implicates me. Now, that might not be serious. We're privileged people. We're in San Francisco. Our threat model may be fairly moderate, but that could be extraordinarily serious for somebody working on human rights in an authoritarian country who is trying to communicate data with somebody outside of that country.