00:22:19 · Dmitri Dolgov
[00:22:19] S0: That's a great question. Look, architecture's come and go. I just mentioned a few really big waves of innovation that we've seen. The foundation is kind of all of the machinery that allows you to grab the fruits of the latest state of the art technology at any moment in time. And that's your evaluation, that's your data, that's infrastructure. If you don't have that, if you don't have those strong fundamentals, then you kind of evolve into a tech stack that kind of works and has certain implicit properties that have organically emerged over time. But it's... It can be very, very difficult, if not impossible, to make big changes to it. Right? And then, you know, there's plenty of companies, I think, in the history of technology development that kind of got stuck in those local minima. So what you want if you have... Well, conversely, if you have those strong fundamentals, that allows you to take really big technical swings. Again, whether it's transformers or convnets or what what what have you. And land kind of big disruptive change in technology without the disruption of momentum of your product and without seeing regressions in the field, which, of course, is so important for us because we're operating in a safety critical environment. So, if anything, I would say that's probably the most difficult thing that we've built at Waymo, and probably the most important muscle that we, as a company, develop over time. It's not, again, the technology of the day. It's this ability to bet on the latest and greatest innovation and then take big disruptive steps in technology development without breaking momentum of product development and deployment.
00:24:14 · Ruchi Sanghvi, Dmitri Dolgov, Aditya Agarwal
[00:24:14] S1: So to repeat, the key would be infrastructure, data, and then evaluation. [00:24:20] S0: That's right. [00:24:20] S2: Given the model that you have trained and then you have distilled to kind of run on these cars, if that was the only IP that Waymo had today, right? Like, Okay, you have these models that have been trained, but you didn't actually have kind of like the historical trove of essentially evals and data and so on, would that be an asset in like, would that be a sufficient asset to start kind of like a new AV company?
00:24:43 · Dmitri Dolgov
[00:24:43] S0: Yeah. I can give you the, you know, the model that we have right now. I don't think you would know what to do with it. Like, you wouldn't, you know, be able to validate it if you wanted to go, you know, apply it to a new deployment. I I don't think you would have the machinery to validate it and deploy with confidence, let let alone iterate on it. Right? So it really comes down to the fundamentals of infrastructure, metrics, eval, training recipes. Right? It's not it's not static. Right? It's all about just how, you know, quickly you can iterate and your level of confidence and understanding the properties of their thing they're building. So it's much more about kind of the machine for building the product as opposed to a snapshot of the product you have at any moment in time, Right? And this is an industry where it's always been very easy to get started. Right? In 2009, it was a dozen of us, in a course of about, what is it, twelve to eighteen months, build a system that could drive those 10 routes that were very difficult and with no human invention done each. But it's been seventeen years to get to where we are today. And it's this kind of the natural property of many of these technical curves. Right? They start out steep and then they, you know, flatten out. And with all of every breakthrough that we've seen over the years in technology, whether it's, you know, hardware, computers, off board, onboard, ML, massive, it really has a drastic impact on, you know, the early part of the curve and getting started has, you know, I think been getting easier. Right? So nowadays, can take your, you know, favorite, model, state of the art, some VLM or VLA, yeah, there you go. Pick your favorite one and go from kind of do open loop end to end imitation learning on it. And you will very quickly, with, you know, tiny budgets and a tiny team, you know, in a matter of, you know, weeks, if not days, get something that works. That You
00:26:43 · Ruchi Sanghvi, Dmitri Dolgov
[00:26:43] S1: need something that works and drives. [00:26:45] S0: It's not gonna be good enough for a product, but it will be very impressive. Right? It has a drastic impact on that early part of the curve, but it doesn't make... If you're talking about superhuman performance, if you're talking about deployment at scale, it has a much muted impact on kind of the hard part of the problem. [00:27:02] S1: And is the hard part reliability? [00:27:05] S0: Performance and reliability. It's getting all those nines in the safety of your system and reliability is one of the key important parts of it. [00:27:17] S1: So I've heard but not experienced that every nine takes 10x the effort? [00:27:23] S0: That tends to be roughly the rule, yeah. [00:27:25] S1: So which nine was the hardest? I mean, for those who don't know what a nine is, can you?
00:27:25 · Ruchi Sanghvi, Dmitri Dolgov
[00:27:25] S1: So which nine was the hardest? I mean, for those who don't know what a nine is, can you? [00:27:31] S0: If you look at whether it's the performance of the system, pick anything around safety or quality. If you look at the reliability of the system, you have some metrics, and you want to get to a certain number of nines for it to be deployable with confidence in the physical world. Right? And then, again, this is another way to describe that property that I mentioned, where you kind of start out steep, it's very easy to get to the first 90%, and then it's, you know, that much more difficult to get to the, you know, next, you know, ninety nine and ninety nine point nine nine nine nine. Right? And the way you end up solving those subsequent nines oftentimes requires, you know, very different approaches than the previous ones. So your question was which one is the hardest? Well, okay, the one you're working on right now is always the hardest but also the most interesting, the most exciting one. Which maybe that's [00:28:26] S1: nine let you deploy Waymo's at such a large scale?
00:28:26 · Ruchi Sanghvi, Dmitri Dolgov
[00:28:26] S1: nine let you deploy Waymo's at such a large scale? [00:28:31] S0: Well, I guess the first big step was to... We kind of had our usual zero to one moment, and then one to 10 out of 100 and so forth. The zero to one, that was a really big step. That was in 2015 when we, for the first time, operate a vehicle without a human driver on public roads. So that was a huge step. This was... And this is kind of this this massive transition from a demo or driver assist product to a full autonomous system. So that was definitely a step function. That step forced us to think very deeply about the performance and reliability of the system across the whole stack, from the platform itself, the car, you know, how reliable is the car? Do you have redundant actuation? What if the brakes fail? What if the steering fails? What if you lose power? The hardware that we're putting on the car, the sensors on the compute, do we have redundancy there? Do we have the right number of nines in that part of the stack? As well as the software that we are putting on the vehicle. And very importantly, at that time, we had to deeply think about this question of what does it mean for an autonomous vehicle to be ready? You know, what's good enough? So this was the first version of what we refer to as our safety and readiness framework, which is this robust, multifaceted machine that we've built over the years that is data driven, that allows us to repeatedly ask this question of, is the system ready and can we deploy? So that was 2015.