00:11:15 · Brett Caughran, Matt Stockton
[00:11:15] S0: That makes sense. And you've talked a lot about the bitter lesson of AI engineering on Twitter. Just walk through sort of your experience with the bitter lesson. Like to what degree have many of these fixes been obsoleted by the next iteration of the model? And where do you think we are on that obsolescence curve? [00:11:36] S1: Yeah, yeah, it's interesting. So it's kind of like, you know, the speed of iteration in this space is so fast that basically you kind of have to be positioned to kind of like throw away a lot of your work or repurpose it or redo it. And I guess like a couple of good examples of this in my experience is, you know, way back when we're getting these early models like GPT-four-zero, you'd have to do a lot of things to instruct the model on how to get work done. You'd have to be very specific. You'd have to overly prescribe what the model model does, right? And now the newer models, GPT-five 0.5, you don't actually want to constrain them as much, right? You actually want to let them breathe a little bit when you provide instructions. And, you know, ultimately, you keep having to reinvent things when you're building with these tools, which is hard because it's the kind of the life cycle of building is shrinking in that you kind of have to reevaluate how you're using these tools. And I think that's just going to continue to happen. You kind of have to have your, you have to be aware of the techniques that are useful for the time period in which you're building, right? And those change relatively quickly.
00:13:05 · Khe Hy
[00:13:05] S2: How do you advise folks, especially ones that don't have the in house expertise, you're mentioning things like Rag and vector databases. I know enough to be dangerous to ask one or two follow-up questions on that topic. But a full time CIO, you know, small team of PMs, they can't invest that time. And so how do you, like, how does someone from the outside know kind of the unknown unknowns to be dangerous from an infrastructure, AI infrastructure perspective, while not getting lost in the weeds, you know, besides hiring three of us?
00:13:51 · Matt Stockton
[00:13:51] S1: Yeah, no, that's a really good question. Like, how do you, if this is not the thing, has a job, and it's not to learn how to use AI, right? They have an actual job. And so how you stay in tune to what's happening? My view, and this is just, it's kind of silly to say it, is like, you just kind of have to be in it. Like, you have to use the tools. You have to spend time experimenting. There's no replacement for just going and doing the thing, right? Like going to try something. Even for me who's in the weeds, these things change, right, over time, and so you just have to be in it. So I think it's more, it's almost more like a process or a team operations concern. It's like you kind of have to both have people at the ground level that are curious and willing to experiment and have the time to experiment. And I think also you kind of have to have the view as a firm of like, Hey, we're going to spend time using this, and we're going to try to integrate it into our processes. In terms of where to look to learn, I mean, Anthropic, OpenAI, all of their resources, all of their training is really good. Their documentation is great. So you can just kind of go there and do their tutorials, read their docs, and get up to speed relatively quickly. But I think the most important thing is actually spending the time doing it, which it's hard, right? Doing something else and it's not learning AI.
00:15:31 · Khe Hy
[00:15:31] S2: Brett, I'm curious in your world, you see folks that, CIOs and PMs that if they're talking to you, they have the vision that this stuff is important, but they probably are super time crunched that they're not firing up a Cloud Code instance on a Saturday night to test the Rag implementation. How are you finding that? And how are you seeing that kind of in the folks that you talk to kind of balance the, you know, I've got to, like, I need to be the CIO that uses the tool. At the same time, I'm the CIO. I've got 9,000 responsibilities and playing with Cloud Code is probably not one of them that takes priority.
00:16:11 · Brett Caughran
[00:16:11] S0: I think we've probably all seen the same thing of usually at a firm of 20, you have two or three who had some sort of moment and they get a little bit of AI psychosis. I say that kindly because I think we probably all three have some degree of AI psychosis. And I think what stimulates that, at least for me, is like trying something that you think is impossible for AI to do and it actually does it. And you're like, Holy moly. Like, I can't believe that these tools can do these things. That takes to your point, Matt, like a bunch of experimentation, but that is such a seductive, addictive feeling. And all of a sudden you're doing, you're sort of trying other things late nights and weekends. So I'd say pretty much every firm has like one or two of those people that are trying to bang the drum for the other individuals to sort of bring them along. I think the single player versus multi player game like really, really resonates. I think that's the tricky challenge of how do you diffuse that information to the broader firm. Because most of those people, like you ask them how many hours they've spent experimenting to get to that level of fluency with the tools, and it's usually like hundreds of hours. So that's a trick. And then even me, almost every week, the interface on Codex changes or Cloud Cowork changes. You're like, Wait, what's this button? What is Codex projects? What's perplexity computer workflows? Just staying up with the architectural changes of the UI is hard enough to do. But...
00:17:58 · Matt Stockton
[00:17:58] S1: Yeah, I really love your example of trying something that you don't think is possible for these models to do. I agree that when you kind of frame it for people to do that and then they go do it, that's kind of generally when they hit their moment because, you know, lots of people are still using these tools more like a Google search for what it's worth. They don't really truly understand how capable these tools are. I mean, recent example for me is I've been really into trying to take written content that I've used the models to build using upstream data sources, instead of just having that written content, basically turn that written content into an interactive website. So you think about you you built a company deep dive, or you're doing some research on some topic, right? And, you know, you get this 10 page document. It's like, okay, do you really wanna share that? Is someone gonna read it? Turn it into a website, have... Send them a self contained HTML, JavaScript CSS, where they can just click around. And, like, that just kinda works now. Like, if you if you if you tell the model to turn this stuff into a website, you can get an HTML, send it to someone. Like, I don't know. If you would've asked me if that was possible, like, six months ago, I'd be like, no way. Can't. It can't just one shot a file that you can share that someone can just click around in.