Y Combinator Startup Podcast
00:09:25 · Chelsea Finn
The first is that they spend a lot of time on dead end trajectories. And maybe this is okay if you're just spending compute on it, but this would cost a lot in the physical world. So we can look at a concrete example, say that we want a robot to construct cardboard boxes and stack them, on the right. And in this trajectory right here, the robot accidentally grabbed two boxes that are flush against each other. And if we let it continue, it would just continue to try to fold that box rather than separate out the two boxes. And trying to fold two boxes together isn't useful data that will teach the model how to get better at the task. And so that would be kind of wasting a lot of time on the robot attempting to go down the wrong path for solving the problem. And so instead of spending a lot of time trying to do that task, what we'll do is we'll actually have a human intervene and show the robot what to do and how to recover from that situation. And so what you can see here is a human is tele operating and intervening with the robot and showing it that to recover from this situation, it needs to essentially try to separate out the two boxes. It then kind of puts its gripper in, sees it if the... Then if the robot could autonomously recover, it doesn't autonomously recover. And so the person then intervenes again, to help, help it get back on the right track so that we're efficiently using the data on the robot. Okay. So this is the first thing that we can do, which is we can show the robot how to recover early, or how to recover so that we're not spending time on dead end trajectories, or at the very least, just terminate the episode early. Now, the second thing that we can do is, PPO and GRPO and these kinds of algorithms, they make many attempts at a single prompt. And depending on the algorithm, they're they're essentially trying to estimate for these different responses, what is a good response and what's a bad response. And so even for an individual prompt, they're actually going to kind of roll out like 10 or 50 times, for that individual prompt. And they're doing this because they're trying to estimate the value of these different, attempts to then kind of up weight or kind of increase the likelihood of good things and decrease the likelihood of bad things.
00:11:42 · Chelsea Finn
But we can actually amortize this cost, rather than trying to collect a lot of attempts for a single prompt, we can amortize this across different prompts and learn a much more general value estimate of what's good and what's bad and use this to improve with our autonomous experience. And so what this looks like is, we can train a general purpose value function on lots of kind of videos of the robot experience. And this can learn things like if it accidentally unfolds a shirt when it's trying to fold, that that's bad and that's making negative progress, it's shown in red. Or if it's making kind of forward progress, it recognizes that as well. And the same value function can also estimate what's good and bad for a completely different scenario, in this case, for retrieving an item from a fridge. And so this sort of kind of general purpose value model that's kind of predicting basically the time to success can basically significantly reduce the amount of attempts needed to learn how to improve from experience. So with these two kind of improvements to a reinforcement learning system, we kind of have a general improvement algorithm that trains a foundation model on diverse data, then collects experience from that with a human intervening as necessary to help prevent dead end trajectories, and then train a general purpose estimate of what's good and bad, the value function, and then use that to improve the model. And with this sort of improvement, we're able to, kind of fine tune a foundation model to higher degrees of performance. So, in the task of making, a latte, in this case, we'll be making a latte in collaboration with a person, where the robot is in charge of making the espresso, and the person is in charge of steaming the milk. This is what the, what the kind of model looks like. The model is directly controlling the joints of the robot using, the images from the robot's cameras as input. And we can see that the model, is able to do the pretty challenging task of inserting the portafilter, waiting the appropriate amount of time for the espresso to dispense, pouring the steamed milk into the cup. And then the last part of this task is actually the most challenging, where it needs to take a very full latte, cup of latte and transfer that over to the coaster. And so here's actually the observation that the robot sees directly.
00:14:04 · Chelsea Finn
And you can see that the policy is super delicate and able to balance the cup appropriately and smoothly so that the latte doesn't spill. So this is the kind of gives you a sense of the difficulty of this kind of task. And kind of going back to this reliability question, we took this policy and we ran it not just once, but we ran it for thirteen hours straight. And we basically wanted to evaluate, is this policy not only good at making a latte once, but can it do so reliably to the extent that it would be needed to be useful in the real world? And so here's a time lapse of that process. And indeed, we found that the robot was reliable enough to be useful for long stretches of time without making mistakes frequently. Now, the same algorithm isn't specific for making lattes, of course. And so we also applied this to other applications as well. Dandelion Chocolate Factory is a few blocks from our office. And so we took a workflow that they typically have a person do, which is to construct these cardboard boxes, label them, and stack them. And we trained our robot to basically do exactly their real workflow, and trained it with the reinforcement learning algorithm that I talked about to get a policy that is far more reliable at constructing, labeling, and stacking these boxes. And then we also applied this algorithm to fold and close as well. And we wanted, in this case, to not just test how well the model could do one task in one environment, but to do it in many environments. And so these are clothing items that the robot has never seen before in a home that's never seen before, it's able to do so and act autonomously for an extended period of time. Now, videos aren't, don't always kind of show everything. And so we also quantitatively measured the reliability of these models. And we care both about the reliability as well as the speed, like how many boxes can it build per hour. And so we're gonna measure throughput, which kind of couples both success rate and speed. And we find that over the phases of training from pre training to like an SFT like stage to an RL post training like stage, we see a drastic increase in success rate and seed in throughput. And specifically around a 2x throughput just from the RL stage itself, showing how we can get much greater reliability from reinforcement learning.
00:16:24 · Chelsea Finn
And for the Espresso task, if we look specifically at the success rate, we achieved a 90% or over 90% success rate on making Espresso. So the takeaways for this part is that we we can develop a scalable recipe for high reliability of complex robotic manipulation tasks. And we saw, in this case, a two x higher throughput from using experience and interventions. But most importantly, we saw how we can achieve long term autonomy in real workflows that people actually care about in the real world. And this is what it's going to take, I think, for robots to be useful in the real world. Now there's also a lot more work and a lot more opportunities. The... We actually only ran a few iterations of improvement of this algorithm. And with more iterations, we should be able to see even greater improvement or even greater reliability. And even with this improvement, the robot still makes mistakes. It's also still slower than people. And so, there's many... Like a ton of room for improvement for developing even more powerful recipes. So we've seen long term autonomy, for these different workflows. But there's actually one more ingredient that I'd like to talk about for enabling robots to be autonomous and useful for long periods of time. And that ingredient is memory. So you might be surprised to hear that most state of the art foundation models for robotics have no memory or no context. They're just operating on the current sensor observations, the current camera readings, and predicting actions based off of that. And you actually can, do short motor skills. You can do repetitive tasks without memory. The videos that I showed before didn't have any context either. But if you wanna do a long task that involves multiple different steps in sequence, then memory is critical for tracking progress of the steps that you've completed so far. So if it's critical for doing these kinds of long horizon tasks, then why don't these models have any context or memory?
00:18:22 · Chelsea Finn
There's a couple of reasons for this that are technical and I'll talk through one of them, which is that if you naively approach memory, and try to feed in context, like past video to a robot foundation model, say that you would just pass in ten seconds of video, maybe this ten seconds of video is sampled at 50 Hertz, which is a common control frequency in robotics, and you feed in all four camera streams on the robot, and you use around kind of two fifty six tokens per image, this corresponds to passing in half a million tokens into your model, which is a lot of tokens. And trying to do that in real time, into your model right now is quite challenging. Even if you subsample to one frame per second, you're still gonna be passing in 10,000 tokens into your model, which at least right now is prohibitively expensive for these models, and that's still only ten seconds of memory. So, I don't have time to go into the technical details of exactly what we did here, but, we also developed a solution for this context problem. And specifically, we developed a system that has memory at multiple timescales. The first is a short term video memory that has about ten seconds of video memory, but is done so and computed much more efficiently than naively passing it into the model. And then for longer memory, for memory that spans multiple minutes or multiple hours, we don't necessarily need video of exactly what happened in that past history. And so instead, we represent memory for those parts in text, where we summarize what happened, in text based and then incorporate that much more compressed textual summary of what happened over the past ten, fifteen minutes into the model as well. And with this sort of kind of memory at multiple different timescales, we're able to enable robots to do tasks that can operate for ten or fifteen minutes at a time completely autonomously. And what's different from the previous slide, or the kind of what I showed previously is that this task isn't repetitive. So this is gonna be a ten to fifteen minute task that's involved cleaning a kitchen. And the robot isn't just repeatedly making espresso over and over again. What it involves is wiping the counter with a sponge, kind of then drying the counter with a paper towel, throwing away the paper towel.
00:20:39 · Chelsea Finn
Next, it's going to put away the mustard into the fridge, then it will put the dishes away into the cabinet, wash some of the dirty dishes in the sink, and so on. And so, by incorporating memory, it's able to do a task that requires keeping track of all of these different steps, that is done to, clean the kitchen and successfully operate for, ten to fifteen minutes completely autonomously. Great. So those were a couple ingredients for long term autonomy. Now I'd like to build on that and actually take those ingredients and put it into a general purpose model, that can do everything that I showed before, but also can do that in a single model and can do some other things as well. And to think about developing such a general purpose model, I think it's really helpful to contextualize like where robotics is at, within the timeline of other developments in generalist AI. And if we think about how kind of generalist AI systems have evolved over the past, like fifteen years, I think the first major milestone was in 2012, when we saw that a deep learning system trained from scratch. This was the first time that it topped an external benchmark. And all of the previous methods for that benchmark were specifically designed for that application. So all of the previous methods, specifically, this was the ImageNet benchmark, were designed specifically for image classification. Whereas this was the first time that a deep learning based system actually outperformed those more specialist systems. And this is kind of a much more general algorithm that wasn't specifically designed for image recognition. And then just a couple of years later, we found that we weren't just training algorithms from scratch, but we were able to get models like pre trained models that are useful for fine tuning to downstream tasks. And so it became the norm to take a model that was pre trained on ImageNet and then fine tune it on a downstream task. And we actually saw better performance from using that pre trained model like BERT or like an ImageNet pre trained model. And then from there, I think the next big phase and the next big transition in generalist AI models wasn't using pretrained models, but kind of moving from a pretraining fine tuning regime to a regime where we're just using generalist models out of the box. This is with models like the start of GPT-two.