Masters of Scale
00:24:17 · Meredith Whittaker
[00:24:17] S0: Well, look, again, I think it's really important that we come down to technical brass tacks here, and recognize that there is a real issue with these systems. Agency is about choice, but if you choose for me, if you choose to install Muse, and Muse is sucking up my communication data and our data, that's not really a choice that I made. And I think it is a mistake to look at this as sort of an issue where individuals are just choosing to use this, given the vast distinction between the marketing that people are hearing about super intelligent, conscious AI that can do everything magically on the one hand, and the technical reality of what it takes to make these real life systems that depend on the hardware and the software and the libraries and the supply chains and the device access of agents actually function to do these things. And I would, you know, again say that what we are looking at as Signal is extremely stark. We're looking at the introduction of agents into the operating system in ways that are nullifying or threatening to nullify our ability as independent application developers to protect your privacy and security at the application level. Because when you are using the accessibility tool kit, when you are poisoning the water, the operating system that Signal and all other applications swim in, in order to grab data, in order to execute an agentic command, our encryption, which has been verified, which is formally verified, which is sort of mathematically proven to be robust, that protects your privacy, that means you don't have to trust me, you don't have to trust Signal Based on Vibes, You can look at the code, and you can verify that trust is earned. That's no longer the shield. The shield is whatever that agent that now has access to your messages may or may not do with it. That is the vector of attack, and that is hollowing out the idea of privacy and security at the application layer, at the website layer. It is actually reversing the promise of an open Internet where everyone can participate and grabbing data and authority back to the massive companies already owning AI.
00:26:30 · Jeff Berman
[00:26:30] S3: Thank you, Meredith. Neil, two minutes for your closing.
00:26:36 · Neil Chilson
[00:26:36] S1: So thanks Jeff, thanks to the audience, and thanks especially to Meredith. I think Meredith is absolutely right that there are... Alright. I could stop there, I suppose. But Meredith is right that when I share information, it also shares other people's information. This is not a problem that's unique to agents. It is the nature of the world, and this is a challenge for Signal in part because they have, they own one part of that communication, and so things that happen outside of that communication make it very difficult for them to make promises and keep those promises to their customers, and part of that is an education, task for them on the, on... For their customers, and so I totally buy that, but the promise here is not that agents won't make mistakes, they will make mistakes, the companies that build them will make mistakes, the people that use them, you and I, we will make mistakes as well, but but do we disengage totally from this technology? No. We build it better. That's what we did in the communication space. That's what Meredith has done so such a great job on and we need to keep doing that. That's that's our effort here and agents can not only help do that, they can help protect our privacy in some of the ways that are simple, maybe they're not as technically sophisticated, but for the average person, being able to file and register and opt out of all the data brokers with a six word prompt is pretty powerful privacy protection, and so I want to say that these tools can be very useful to protect privacy. We need to make sure that we can do that. They can be on this side of the glass when the... For all the windows of people looking at us, they can help defend, and that is part of access, but there's benefits to access as well, and so I don't think this is as clear cut, and and dried as as Meredith has said, and I hope you'll vote no on the proposition because of that.
00:28:30 · Jeff Berman, John Donvan, Meredith Whittaker
[00:28:30] S3: Thank you. Please join me in thanking Neil and Meredith for being in our first Masters of Scale debate. Thank you. Thank you. [00:28:46] S2: Hi. I'm John Donvan, and we are back for the first time ever doing a kind of a post game, a post debate episode. You'd saw what just happened on this debate stage here at Masters of Scale in San Francisco, and I'm now sitting down with Neil and Meredith to talk about the debate that just happened and maybe get to parts of it that in the time allowed we didn't get to. I'm just... I I guess you still got the adrenaline going on. One thing I'd like to do though, this was an audience that really knew these issues. And I'd like to talk a little bit at a level, a little bit more at a level for general understanding on parts of this. But for beginners, we didn't really have a chance in that debate itself to learn that much about each of you. So I wanna ask you each a question. I'll start with you, Meredith. In your work, your profession, your thinking in a sort of biographical sense led you to be on that stage taking that side of the issue? [00:29:44] S0: Well, look, I started working at Google in 2006. And so I watched a business model that was reliant on grabbing as much data as possible about as many people as possible using that to create models that were then, you know, sold to advertisers to enhance their business. I watched that metastasize from the inside and I became concerned [00:30:11] S2: about It's a pejorative word, but metastasize.
00:30:11 · John Donvan, Meredith Whittaker
[00:30:11] S2: about It's a pejorative word, but metastasize. [00:30:14] S0: It's a it's a metaphor, but I guess it's pejorative because I, you know, I do think that is simply structurally a dangerous business model to have a handful of large companies, largely jurisdiction in, you know, one one country with extraordinarily intimate amounts of data about global populations. You know, you can have the best philosopher kings running those companies now, but that executive team shifts, that board shifts, the regime under which the country is governed shifts, and that is a turnkey recipe for extraordinary social control, for extraordinary access to our intimate lives, for the ability for those in power to control and oppress us based on access to our most intimate thoughts, to our communications, to our thinking, to our creative practices, to our intimacies. [00:31:11] S2: So, really, talking about I twenty years ago, you had these alarms. [00:31:16] S0: Yes, suddenly it has been twenty years. [00:31:19] S2: And twenty years ago, were you relatively isolated in your community with these concerns?
00:31:26 · Meredith Whittaker
[00:31:26] S0: It was not was not as clear to everyone. I was working through these questions myself. You know, there was a dashboard that anyone at Google could access at the time that I started where you could see people's Gmail accounts and other extremely sensitive information, and that was for debugging, for, you know, kinda digging into these things to figure out a problem. Of course, very, very quickly after I started, there was an engineer fired for using that dashboard for stalking. Right? And so it was, you know, these little things sort of you're like, wait, how are we justifying this? How does this work? I started reading the the history of the Stasi. I started reading the history of IBM's participation with the Nazi regime during the holocaust, where the efficiencies that allowed you to classify everyone to determine, you know, who was Jewish, who was Romani, who was queer, to figure out where they lived, who their family members, to round them up, were facilitated by punch card census that were designed in collaboration with that country company. And then I said, you know what? That is really horrifying to think about. Okay, so how would you instrument the kind of data that I see flowing through these infrastructures every day for that? Okay. Well, how would we guard against them? Okay. There's technologies like encryption, there are, you know, security technologies we could use. Okay. Well, let me invest my time and my care into trying to build those, and I was part of a community at that point that was sort of a, you know, hacker, open source folks, people that, you know, were referred to as sort of tinfoil hat before Snowden, who were trying to build independent technology that would help remedy or prevent some of those harms. Now, I think my theory of change and my understanding of it was a little naive at that point, because my sense was if the technology exists, then it will be adopted. I didn't understand the political economic foundation that, you know, actually, large companies who are based on this data gathering, this surveillance business model, where they need all this data, they need this scale in order to fuel this particular peculiar economic form, are never gonna apply those technologies, whether they exist or not,