Masters of Scale
00:09:15 · Meredith Whittaker
[00:09:15] S0: Well, I think the agents aren't ours in most cases. When we're talking about Meta's Muse, we had a person show up to someone else's house because Muse had given that person the address to that house, because that person said, hey, Muse, can you manage my Facebook marketplace for me? And Muse took that to mean we're giving out your address, and then impersonating you when this guy stands outside your door confused, wondering why you didn't come out to get the product he was gonna buy. You have Muse now publishing private information online, company revenue, loan application agreements, other proprietary data, that it's unclear how that leaked. But because of Muse's architecture, because of the way these convenient consumer agents are being built, again, they have access to everything. We're breaking down the sandboxes, which is a paradigm in security engineering. We are giving them root permissions, the key to your house, the key to your digital ecosystem to do what they can to combine that data, to make determinations about that data, and we recognize that there is a fatal flaw. This is prompt injection in LLM driven agents, where they don't... They aren't alive, they aren't human. Sorry, guys who think you are. But they they respond to this data as if it were an order, as if it were a fact. And so we have seen countless breaches. We have seen agents make perilous mistakes, and I think, yes, I could ask an agent to drive tour for me, but that's not, again, if we're talking about Meta's Muse, that's not my agent. Finally, we learned through reporting recently that Muse is logging people's contact lists, logging the people they talk to. So you install Muse, and I am in your contact list. Muse now has my information. They are logging when we talk to each other. They are creating dossiers about me and your communications with me, Cambridge Analytica style. And so you're making a choice for your convenience that also implicates me. Now, that might not be serious. We're privileged people. We're in San Francisco. Our threat model may be fairly moderate, but that could be extraordinarily serious for somebody working on human rights in an authoritarian country who is trying to communicate data with somebody outside of that country.
00:11:43 · Meredith Whittaker, Jeff Berman, Neil Chilson
And so this, again, I don't care about winning this debate, I care about looking this in the face and recognizing we are eating the seed corn of our cyber security when we install these things in the name of convenience without thinking through the perilous architectures that we have used to create them. [00:12:03] S3: So Neil, given the check... Given the given the challenge of segmenting and locking down data, why is the convenience value worth more than the risk we're creating here? [00:12:17] S1: Well, I I think it's easy for us to dismiss these types of challenges, but again, the the goal here is not that agents are perfect or that they're perfect protectors of security. The question is, how do they compare to the status quo? And the status quo is not great. Most people don't use Signal, unfortunately. And the status quo is full of people who are gathering information about us in part because we've made that convenience trade off over and over and over. So Meredith is totally right that we need secure agents. We need better agents, but the challenge is not to not use. Remember what the proposition is. The proposition is, should we trust agents with our information And the... And... Or should we not trust agents? And it's a conditional. It depends. It really depends on the agent. It really depends on the use and it really depends on the access that it's asking for. Most of the agents right now, I don't let them write online, I let them read online for the most part.
00:12:17 · Neil Chilson, Jeff Berman
[00:12:17] S1: Well, I I think it's easy for us to dismiss these types of challenges, but again, the the goal here is not that agents are perfect or that they're perfect protectors of security. The question is, how do they compare to the status quo? And the status quo is not great. Most people don't use Signal, unfortunately. And the status quo is full of people who are gathering information about us in part because we've made that convenience trade off over and over and over. So Meredith is totally right that we need secure agents. We need better agents, but the challenge is not to not use. Remember what the proposition is. The proposition is, should we trust agents with our information And the... And... Or should we not trust agents? And it's a conditional. It depends. It really depends on the agent. It really depends on the use and it really depends on the access that it's asking for. Most of the agents right now, I don't let them write online, I let them read online for the most part. [00:13:14] S3: And And I so think Meredith's it's all about those saying, trade then you've got risk of prompt injection. Right? You've you've got concerns on top of concerns. So if we can't segment and lock down data and we have concerns about prompt prompt injection, why does the convenience outweigh the risk?
00:13:14 · Jeff Berman, Neil Chilson, John Donvan
[00:13:14] S3: And And I so think Meredith's it's all about those saying, trade then you've got risk of prompt injection. Right? You've you've got concerns on top of concerns. So if we can't segment and lock down data and we have concerns about prompt prompt injection, why does the convenience outweigh the risk? [00:13:27] S1: So, I think the only way we get to solve these problems is the same way that we solve problems in messaging, right? We didn't give up on messaging. We didn't stop messaging because there were risks and there were there were platforms that were built poorly. We built better platforms. Meredith was a key part of that, and so I think that that's what we need to do here. We're going to learn trust and we're going to learn how to improve these products by using them, not by standing back and letting people who don't care about privacy build them. [00:13:57] S3: Meredith, one of the points I heard Neil make in his opening is, [00:14:01] S2: to your [00:14:01] S3: point about us being privileged people, I'm guessing you have an executive administrative assistant, many of them in the room do. They probably have access to an enormous amount of your life, your contact information, etcetera. Most people can't afford to have someone do that. And this is a massive time saver, a massive efficiency creator for people who don't have the means otherwise. What's really the difference between you're having a human admin and you're having a digital admin?
00:14:01 · Jeff Berman, Meredith Whittaker, Neil Chilson
[00:14:01] S3: point about us being privileged people, I'm guessing you have an executive administrative assistant, many of them in the room do. They probably have access to an enormous amount of your life, your contact information, etcetera. Most people can't afford to have someone do that. And this is a massive time saver, a massive efficiency creator for people who don't have the means otherwise. What's really the difference between you're having a human admin and you're having a digital admin? [00:14:27] S0: Well, Emma, my assistant, is incredible, but if Emma messed up like this, I would fire her. There is labor contract that protects my relationship with Emma. She understands, as a human being, how sacrosanct the information that she has access to is, and she's accountable for it. She's not a bot that is installed at a pervasive level, who's sort of fumbling through text, you know, that that could lead to a serious catastrophe, which, you know, when you have a duty of care like Signal, again, this is not a game. This is not sort of a casual, like, oopsie, learning as we go, a bunch of data source spilled, there's a number of human rights workers now not able to get out of a country. Right? [00:15:14] S1: Right.
00:15:14 · Neil Chilson, Meredith Whittaker, Jeff Berman
[00:15:14] S1: Right. [00:15:14] S0: And I think this is this is not something where we can sort of hand wave around this. There are paradigmatic issues with the way agents are being built. Again, they are breaking down the walls we have put up to protect cybersecurity. The architecture of a lot of these agents, if you compare it to targeted malware, looks very, very similar. Get pervasive permissions on your device, root permissions, access to all of the data at the application level that comes through the device, hooking into screen buffers, using your accessibility tool kit, doing screenshots, running that through an optical character recognition system to create a database of everything that goes through your screen, then processing that, sending it off-site to process on third party serve... Off -site servers to, you know, get your inference, to send it back, and then driving your browser or any app as if it were you. Chase Bank just blocked Muse, because they cannot authenticate that it's an actual user using the bank with that extremely personal information, or whether it's an agent that's been prop injected to, like, take that information, and they see the liability concerns as too high. [00:16:26] S3: But they've blocked all agents. Right? They've not blocked all agents. [00:16:29] S0: Well, they've blocked the agent from the company that has a history of doing things at the, you know, the the most excessive level.