The Frontier with Brendan Foody
00:41:29 · Jonathan Frankle
[00:41:29] S0: God, I wish I knew. Um, and I can't wait to find out. I can tell you my best guess right now, knowing that we'll laugh at this in three years, just as much as we can laugh that I said you'd never need more than eight GPUs back in 2022. But I think, you know, there will be a little bit of like all humans will become more like managers a little bit. We're all gonna have to learn the skills of like how to declaratively describe a task and then check whether the task was done well selectively, know which questions to ask, kind of almost context-free to figure out whether something was done well. We're all gonna be writing a lot of process. We're all gonna become kind of program managers in a sense. We're gonna write process and figure out how to build pipelines of agents. So we're all gonna kind of like become our own architects of teams. But I think we're all really gonna have to become critical thinkers and get pushed to the boundary of what we do. Nothing is gonna be wrote anymore because if it's wrote, we can have an agent do it, which on the one hand is a little bit scary because some of us are very comfortable with our rote task. I know how to do performance reviews really well at this point. I know how to be a good research manager. Um, and that's scary for me because I'm gonna have to be more creative every day. But it's also gonna force each of us to like really get to the hard and interesting stuff that makes our work worthwhile. In the same way that, like, if you don't have to sit down with a typewriter and typeset, or you don't have to figure out the punch card and walk to the mainframe and plug it in. You can think about building software that's really creative. I was meeting with, you know, someone who used to be high up at a bank who used to work on mergers in the 80s, and he was telling me what they were like, because I'd mentioned the punch card example. He was like, Yeah, a merger used to be a stack of punch cards. And I asked him, so how many mergers did you do a year? Three. And I asked him how sophisticated were the mergers, and he was like, they were all the same structure. What about today? And he was like, Yeah, an associate can do three mergers, and every merger is creative and interesting.
00:43:12 · Jonathan Frankle
And it's led to more dynamism in the economy because we can now conceive of transactions that wouldn't have made sense because we can handle the scale and complexity of it because we have computers. And I think that's what we're going to see is just the sophistication of the tasks. An example I love is like in the insurance world, the idea that, like, okay, in California, there are wildfires, so we will cancel all insurance policies in California for homeowners is such a 2026 thing to do. In 2030, I would hope that with AI, every single home is being like evaluated by a drone that's taking footage and looking at the environment and analyzing it multiple times a year. And every policy is custom written. And we would never cancel all the policies in California because there was a wildfire in LA. We would look at the risk specific to each home. An insurance adjuster is going to be thinking about not like rubber stamp this policy. Oh, this policy doesn't meet this one criteria. Let me follow the process. Cool, AI can do that. They'll be thinking about how do I ensure. Person's home that I could never have ensured before. Yeah. We'll push to the boundaries of work. And I don't know, to me, that's I hate using the word abundance in like I don't mean it in the political sense, and I don't mean it in the um, you know, AGI sense. I mean it in the sense of like a lot of things will be easier to create and demand will go up for them. We'll need more people to work on them. The work will be more interesting, and more people will have access to things like an estate plan at a really low cost. We'll all have access to better tax preparation, like these very boring things that we actually spend a lot of time on, but the life of an accountant will also get a lot more interesting. And I'm really excited about that world.
00:44:36 · Brendan Foody
[00:44:36] S1: Right now, creating Evals or training agents is a small fraction of a percent of the economy. How many multiples larger do you expect that to be in five years?
00:44:45 · Jonathan Frankle
[00:44:45] S0: I mean, you know, not to flatter you, but I think it's gonna get pretty big. Um, it's hard for me to say off the bat. I think these things can grow beyond, way beyond our imagination. As with everything, the way it's going to grow is that we have to reduce the cost. We have to make it easier, more accessible for everyone. The market size will grow. The cost per eval, I hope, will go down by several orders of magnitude. And the market size will go up by several more orders of magnitude, just as with any other good. The fact that digital computers have come down so much in cost has meant we have them in our pockets and our watches and now our smoke detectors. I hope that evals feel like that. Where in five years, you don't need to go to Mercore and say, here's tens of millions of dollars, please give me a really good eval for this task. You go to Mercore and you say, here's $1,000 for a license to some amazing product that I'm giving to this one person, but I've actually given it to every single employee at my company. And now I'm spending hundreds of millions or billions of dollars on Mercore and the market has grown, the technology has changed and become democratized, just as any digital technology does. I hope we get there the way that software has. It used to be that you needed to be at a university or a lab and know the right person so you could get access to the mainframe, learn how to use punch cards, and get your time on the computer, potentially in the middle of the night, because these things ran 24-7, so that you could get your program to run, find out what happened, and come back the next day to try again. That's what evals feel like today. I hope we get to the point where it feels like clawed code for building an eval. Yeah. The demand for software has gone way up because it's become really accessible. And I hope we get there with evals where the demand for evals looks like the demand for software, the market size gets big, and anyone can do this.
00:46:21 · Brendan Foody
[00:46:21] S1: If you had one final parting piece of advice for enterprises that are looking to adopt AI, what would it be?
00:46:29 · Jonathan Frankle
[00:46:29] S0: I think I would have two. One is to like try to like live in almost the dialectic of containing both all the possibilities of AI and the perils and risks as we roll out this new technology that we still don't really understand fully, and contain both, like be able to keep this middle perspective that there are both incredible opportunities and some risks you need to be cognizant of. It's so easy to fall into one bucket or the other. Like this is dangerous, no AI, or this is incredible. We're gonna use AI for everything. No more workers, no more anything, just tokens to the moon. We have to live in this in-between where the internet changed the world, but also, you know, didn't change a lot of things. The other piece I always say to enterprises and to the world at large is be willing to experiment, like controlled sandboxes where you can experiment a little bit and try things. Let people bottom up try to use this technology. I think, in some sense, if we look back at the internet, like the internet was kind of built from first principles, or the web at least was built from first principles in a democratized way. Anyone can put a website online. That meant a lot of people did some really stupid things, some really bad things, and some really silly things. Like we ended up with like all the like Adobe Flash videos from the late 90s and early 2000s that you weren't even around when they existed, but were hilarious at the time. And you know, we got the beginning of memes. Open source software, software that anyone can see the code to and edit. Nobody's gonna make money off of that, nobody's gonna bother on that, changed the world. So the idea that within your business or within the world, like this technology is just available to you and you can try to create something, most things you try to create are gonna be either useless, kind of silly, or maybe even a little dumb and risky, but a handful are going to change your business or change the world. And creating the space for folks to do that, I think is really important. I mean, the best piece of advice I've gotten about computing is you never know who's going to change the world. And so you should try to give as many opportunities to as many people as possible. It's like, what would the world look like without Wikipedia? It's kind of hard to imagine.