Masters of Scale
00:28:30 · Jeff Berman, John Donvan, Meredith Whittaker
[00:28:30] S3: Thank you. Please join me in thanking Neil and Meredith for being in our first Masters of Scale debate. Thank you. Thank you. [00:28:46] S2: Hi. I'm John Donvan, and we are back for the first time ever doing a kind of a post game, a post debate episode. You'd saw what just happened on this debate stage here at Masters of Scale in San Francisco, and I'm now sitting down with Neil and Meredith to talk about the debate that just happened and maybe get to parts of it that in the time allowed we didn't get to. I'm just... I I guess you still got the adrenaline going on. One thing I'd like to do though, this was an audience that really knew these issues. And I'd like to talk a little bit at a level, a little bit more at a level for general understanding on parts of this. But for beginners, we didn't really have a chance in that debate itself to learn that much about each of you. So I wanna ask you each a question. I'll start with you, Meredith. In your work, your profession, your thinking in a sort of biographical sense led you to be on that stage taking that side of the issue? [00:29:44] S0: Well, look, I started working at Google in 2006. And so I watched a business model that was reliant on grabbing as much data as possible about as many people as possible using that to create models that were then, you know, sold to advertisers to enhance their business. I watched that metastasize from the inside and I became concerned [00:30:11] S2: about It's a pejorative word, but metastasize.
00:30:11 · John Donvan, Meredith Whittaker
[00:30:11] S2: about It's a pejorative word, but metastasize. [00:30:14] S0: It's a it's a metaphor, but I guess it's pejorative because I, you know, I do think that is simply structurally a dangerous business model to have a handful of large companies, largely jurisdiction in, you know, one one country with extraordinarily intimate amounts of data about global populations. You know, you can have the best philosopher kings running those companies now, but that executive team shifts, that board shifts, the regime under which the country is governed shifts, and that is a turnkey recipe for extraordinary social control, for extraordinary access to our intimate lives, for the ability for those in power to control and oppress us based on access to our most intimate thoughts, to our communications, to our thinking, to our creative practices, to our intimacies. [00:31:11] S2: So, really, talking about I twenty years ago, you had these alarms. [00:31:16] S0: Yes, suddenly it has been twenty years. [00:31:19] S2: And twenty years ago, were you relatively isolated in your community with these concerns?
00:31:26 · Meredith Whittaker
[00:31:26] S0: It was not was not as clear to everyone. I was working through these questions myself. You know, there was a dashboard that anyone at Google could access at the time that I started where you could see people's Gmail accounts and other extremely sensitive information, and that was for debugging, for, you know, kinda digging into these things to figure out a problem. Of course, very, very quickly after I started, there was an engineer fired for using that dashboard for stalking. Right? And so it was, you know, these little things sort of you're like, wait, how are we justifying this? How does this work? I started reading the the history of the Stasi. I started reading the history of IBM's participation with the Nazi regime during the holocaust, where the efficiencies that allowed you to classify everyone to determine, you know, who was Jewish, who was Romani, who was queer, to figure out where they lived, who their family members, to round them up, were facilitated by punch card census that were designed in collaboration with that country company. And then I said, you know what? That is really horrifying to think about. Okay, so how would you instrument the kind of data that I see flowing through these infrastructures every day for that? Okay. Well, how would we guard against them? Okay. There's technologies like encryption, there are, you know, security technologies we could use. Okay. Well, let me invest my time and my care into trying to build those, and I was part of a community at that point that was sort of a, you know, hacker, open source folks, people that, you know, were referred to as sort of tinfoil hat before Snowden, who were trying to build independent technology that would help remedy or prevent some of those harms. Now, I think my theory of change and my understanding of it was a little naive at that point, because my sense was if the technology exists, then it will be adopted. I didn't understand the political economic foundation that, you know, actually, large companies who are based on this data gathering, this surveillance business model, where they need all this data, they need this scale in order to fuel this particular peculiar economic form, are never gonna apply those technologies, whether they exist or not,
00:33:38 · John Donvan, Meredith Whittaker, Neil Chilson
[00:33:38] S2: No. To not take our [00:33:40] S0: I mean, you know, Facebook with none of our data doesn't have a business model. Right? Signal is a nonprofit because there is no business model for collecting no data. [00:33:50] S2: Yeah. [00:33:50] S0: That's the economic engine of the particular tech industry we have right now. [00:33:55] S2: Neil, throughout the debate you were in the difficult position of actually having some nuance in the sense that you were saying, yeah a lot of what Meredith is saying is concerning but we can find ways to deal with it. Seem confident that we'll find ways to deal with it. Same question that I put to Meredith. Going back to your time in this field, what gives you that, I guess I [00:34:21] S1: would say relative optimism? So you know, I was the first kid in my family to go to college, and I was very interested in computers. And so I went and I studied computers. But what got me interested in computers was complexity theory, actually. And so this is the idea. I mean, people have heard about the butterfly effect and lots of other versions of it. But one thing that comes out of complexity theory or complex systems is that you can't control them, but they can do very powerful things. And you can influence them. You can poke your finger in here and there. You can influence them, but you can't control them. And so this idea of control has like sort of... Control versus dynamism has been the major way that has driven like sort of how I think about the world. [00:35:09] S2: What do mean by that?
00:35:11 · Neil Chilson
[00:35:11] S1: The idea that idea that we can design a structure and that it will... And that we can predict into the future how it will work, that just has really extreme limits in a in a complex world where we deal with all sorts of systems of individual actors and even physical phenomena that can't be simulated in any way that is more cost effective than just running the system itself. And so what I mean there is basically that insight or that lesson that I learned even at a young age has sort of bled all through my my graduate school training into law school and into the policy world where now when I see complex problems, I often see people grasping for control over those systems, and I worry about that as having unintended consequences. And in the flip side, I don't see people recognizing the power and the dynamism that can happen from empowering individuals and and operating under a, you know, a system of even laws across them and and what they can do and what they can accomplish in that space. And so when I see hard problems, and, you know, Meredith laid out a bunch of them with the current system, in fact I tried to sort of frame up my part of the debate as pointing to many of these current problems in the system, I also see a lot of benefits from that system. Know, I as a kid, connecting to the Internet was a major way that broadened my horizons that I would have never had. And I was connecting to the Internet in day and age where, believe me, it was not safe for kids. It was was all like college kids on there, on old BBSs and stuff. So I see
00:36:52 · John Donvan, Neil Chilson
[00:36:52] S2: that tell people what a BBS is who aren't gonna know because they weren't around. [00:36:56] S1: Yeah. So a BBS is like an old message board. It actually stands for bullet bulletin board service that somebody would run on their computer and other people could call in with their computers and leave messages for each other on these bulletin board systems. And and it was the first way that I found a community of people who were interested in the same things that I was, and that process has gotten so much easier over time in part because of the powerful systems that large companies and small companies have built in this space. And I was trying to lay out that dynamic in in the debate about the trade offs that we make in this space between privacy and convenience, and that it's not a clear cut line. There are benefits to not having privacy all the time and it's... And and and in fact, the primary way that we communicate with other people is sharing information and so... And and get things done in this world. And so it's very important to think about the trade offs there and many people have very different needs in all of those spaces and so we need solutions that fit lots of different people's demands. We don't need one solution that tries to tries to stuff everybody in the same box.