Masters of Scale
00:54:23 · Jeff Berman, Meredith Whittaker
[00:54:23] S3: But me [00:54:23] S0: and this much safer than one. Is architecturally how these systems work at That's a technical not true. And so you can put harnesses in there and Muse itself is, you know, kind of... We already have the example that I shared on stage of Muse giving out a user's home address to someone else than impersonating that user in the context of a Facebook marketplace sale. Now, I'm someone who's had a stalker or two in my life. That is terrifying to me. Right? So I might trust it. Do you know, like I trust you to sell my old champagne flutes on Facebook marketplace. But suddenly that trust is breached. And there's someone at my home and Muse is saying like, I'll be right outside. I'll be right outside. I'm at the door. And somebody is very confused. Right? Who else could it give that that address to? And so I don't... I I just... I think we need to recognize that again, at the level of the fundamental architectures and the the way these agents are being integrated at the consumer level, it's very difficult to just sort of cabin off. I trust you just to do this task. And in fact, a thing that did one task and checks in every time is not an agent in the current collusional It's
00:55:36 · Neil Chilson, Meredith Whittaker
[00:55:36] S1: it's just maybe... It's it's an agent that's learning how to be trustworthy. And I think the Muse example is is particularly interesting because there are plenty of things that I would not advise people to install, say, OpenClaw on their home computer because it's their home computer and their computer has lots of things. But with Muse, we have an isolated computer that is in Meta's cloud, and that means that if something goes wrong on that computer, it doesn't mess all the things up that are on my computer. That is safer for consumers in many ways than installing an agent on their own device. And so in that way, there's a benefit. And then the example about the the Facebook marketplace, I am not sure how people deliver champagne flutes to somebody who bought them without arranging a place for them to meet, and so some of this is the the user learning what it means to charge an agent with a task. This agent did not hand out the address to random people, it handed out the address to the person who purchased something. [00:56:42] S0: Who wanted to purchase it. [00:56:44] S1: Who wanted to purchase it. And that's... [00:56:45] S0: That the user who handed it... Muse the task had no idea this is how it was working. [00:56:51] S1: I I I... Like I said, things will go wrong and people will learn. I think...
00:56:51 · Neil Chilson, John Donvan, Meredith Whittaker
[00:56:51] S1: I I I... Like I said, things will go wrong and people will learn. I think... [00:56:55] S2: Meredith, let let let's just look a little bit towards potential solutions. What is your... How would you want to address the issue? I mean Neil is saying that his great concern is that a move against agents would shut down a development that could help a lot of people. That's a strong part of his argument. I'm not sure if that's what you wanna do. Do you wanna stop the development of agents or do you wanna improve their use, which is what his argument is, that they can be improved. [00:57:20] S0: Well, the proposition that we are debating is should we trust them as they stand [00:57:25] S2: or I agree, but I'm moving
00:57:26 · Meredith Whittaker
[00:57:26] S0: beyond it. I get it. Get we're sort of water skiing through many many oceans right now. No, absolutely not. I think, you know, technologies can have, you know, cases that are you know, useful to be tautological about it. But I think it's, you know, we need to recognize that right now we have a lot of companies that are... Have spent billions and billions of dollars on developing AI. The CapEx versus revenue for the AI industry is a kind of like open closed secret. It's a little nerve racking to look at the market and the debt structuring of, you know, large scale AI right now. And there is a deep desire to be an AI leader and to find a consumer market fit for technologies that have, you know, cost a lot and on which so many lofty promises are riding. And so, you are seeing a rush to reckless deployments, to integrating agents into operating systems, to Meta Muse which, you know, a lot is banking on that. Their compute spend is high watering right now. And we are seeing the fundamentals of cyber security being cast aside as if we, you know, all just sort of, you know, got struck by lightning and forgot that these run on software and hardware, that our supply chains are buggy, that they require sort of, you know, infrastructures that at the network layer, at the software layer, at other layers of the stack can be hacked and perturbed. And that to actually use these systems, we're not talking about a new godhead strike, you know, that has come down from the clouds as a sentient machine learning system. We're talking about software and hardware infrastructures that are being integrated in ways that are deeply unsafe. And now if I'd said that three months ago, I would have to do some explaining. But now we have the hugging face incident. We have Microsoft's Copilot exfiltrating data. We have prompt injection attacks all over the place and we have a week where the PHP, you know, supply chain issues, the the bug in PHP means that 75% of websites are vulnerable in ways
00:59:41 · Neil Chilson, John Donvan, Meredith Whittaker
[00:59:41] S1: that Bug identified by them. [00:59:42] S2: So what I what I hear, Neil, from Which [00:59:44] S1: is not... [00:59:44] S0: You know, like, I'm not arguing that AI is bad. [00:59:46] S2: Right. I I I [00:59:47] S0: am making a high stakes argument that the way this is being rolled out is extremely perilous to the infrastructure, the, you know, the computational infrastructure, the network infrastructure, the Internet, that is at this point the nervous system of core institutions, our governments, our, you know, critical infrastructure Right energy [01:00:05] S2: now, you're saying don't trust because of what you called a reckless rush. So Neil... [01:00:09] S0: I'm saying it's not trustworthy. [01:00:11] S2: Not trustworthy now. You're not saying it can't ever be, but you're saying right now, if you look at the language of the proposition, it's not trustworthy because of what Meredith Neil is describing as a reckless rush. That this this is a bad moment in terms of what's going on. What what's your response to that? If you're not arguing for the future, but you're arguing for this moment. [01:00:29] S1: Well, I think the the interesting examples, you know, Meredith said earlier that coding was one of the areas where these are extremely useful. And all the examples that she gave of the the biggest threats are also in that space. And so this is the first frontier, this is the space where we are trying to figure out how do we balance this properly. How do we deploy this technology so that defenders have the advantage over the offense? And...
01:00:29 · Neil Chilson, John Donvan
[01:00:29] S1: Well, I think the the interesting examples, you know, Meredith said earlier that coding was one of the areas where these are extremely useful. And all the examples that she gave of the the biggest threats are also in that space. And so this is the first frontier, this is the space where we are trying to figure out how do we balance this properly. How do we deploy this technology so that defenders have the advantage over the offense? And... [01:00:57] S2: But are are you not con... Do you not share some of the deep concerns he's talking about this moment that we're in? [01:01:01] S1: Of course. Of course. [01:01:02] S2: You do. [01:01:03] S1: Of course. We need to do this right. And so the question is, do we have the right structures and incentives in place to make that happen? [01:01:11] S2: Do you think so? [01:01:11] S1: And I think there are many things that we can improve on. I think there should be a lot more coordination between the companies. I think we need a lot more research into this space. I think we also need to make sure that companies aren't, you know, using this language of agents to to get off the hook. That, oh, it was our agent who did this and therefore, it's not... It's kind of out of our control. That's that's not fair, and I don't think under the law that that stands up. So we we do need responsibility for these companies, for the damage that their agents cause, and I I think I think the current law is there, but there are some gaps to fill for sure.