The Information's TITV
00:48:08 · Dan Shipper, Akash Pasricha
[00:48:08] S2: this is counterintuitive, but I think it's I think it's really interesting, and it says something about where things are going. Because everybody at this event... If you're here, there's 400 people here. If you're here, you love AI. And I've talked to so many people at this event who are working in big companies who are getting so much pressure to use AI as much as possible, which is... They love. It's good for them. It's it's like... It's what it's what they want. And I think the expectations from management or executives that are not familiar with AI for how fast you can go before you set up a good system to do it well, there... It's like... It's kind of out of whack. And so there's something really interesting where for AI pill people inside of big companies, the way that they're being managed, I think, is is out of whack with, like, what is realistic and and the and the the best way to start a high quality AI project, which is to get the basics right first and build, like, build slowly, and then you can go fast. Go slow to go [00:49:11] S0: fast. And this is kind of interesting though because you're saying management's expectations are too high, and yet we've got headlines every week saying x y z company is putting a cap on how much you you you spent on AI.
00:49:11 · Akash Pasricha, Dan Shipper, Paul Kedrosky
[00:49:11] S0: fast. And this is kind of interesting though because you're saying management's expectations are too high, and yet we've got headlines every week saying x y z company is putting a cap on how much you you you spent on AI. [00:49:25] S2: And there's, you know, and there's there's different managers in different parts of the economy, and and and I think there's this flip flop from, like, token maxing to, like, token token minning. And and and to me, what what that says is if you're an executive, you're... If you're a CEO and you are not... You don't have your hands in this technology all the time, the only thing that you can do is follow the trend and be like, oh, token maxing or oh, no. Token maxing is over and now we're gonna put put budget caps. That's the only thing you can do and both don't work. Because the reality is that, yeah, forcing people to spend as many tokens as possible when your organization is not set up to do that well, it's not gonna work. Obviously, you're just gonna get people who are gonna spend tokens on things that don't don't matter. But but if your organization is set up to use AI well and you and you unleash the people who are really good at doing it, you actually probably should be token maxing because the more tokens you spend... If your if your setup is good, the more token you... The more tokens you spend, the more high quality output you [00:50:22] S1: get. Right.
00:50:22 · Paul Kedrosky, Dan Shipper, Akash Pasricha
[00:50:22] S1: get. Right. [00:50:23] S2: Right. And so I don't think there's I don't think there's one right answer, but the the the... In terms of, like, a headline token maxing or token budgets, but the the right answer is always you have to have your hands in it in order to make good decisions. [00:50:35] S0: Right. Okay. So let's let's talk about the tokens and the model specifically. So you've been doing these vibe checks for for how long been [00:50:43] S2: Three years [00:50:44] S0: or four years? Yeah. Three years of vibe checks. You've probably done dozens by now. [00:50:47] S2: I mean, [00:50:47] S0: more time. Maybe. So I wanna talk about where the models are going. And I asked you a question a couple months ago when you came on the show about what the models are not yet good at. [00:51:00] S2: And What did I say? Do you remember? [00:51:03] S0: I I don't. The the reason I'm asking you again is because we've had a couple models come out since then. [00:51:11] S1: Yeah. [00:51:12] S0: So I'm gonna re ask you the question, what are they not good at right now?
00:51:16 · Dan Shipper
[00:51:16] S2: That's a that's a really good one. Okay. So, you know, obviously, the models, they're out there, like, hacking into, like, government, like, infrastructure and solving Navier Stokes. So they're, like, they're very good. And there are some models that are coming out that are not out yet that are very, very good. And I I think will change how we think about things like building software, for example. And they are very bad at a lot of things. A really a really simple example is models often leak their context in in the way that they do their output. So an example might be, you've ever had the experience of working with a model and you're like... You're you're you're you're making a landing page together and you're like, yeah, I don't like this headline because it it mentions x and and... But it should mention y. And then the model's like, great. I'm gonna rewrite the landing page, and it says, we're not gonna talk about x. We're we're gonna we're gonna write y, and that puts it in the copy of the landing page. Like, is a... That's a leak in the way that it that it that it thinks into the artifact that it's producing. And this is actually quite common even with the smartest models. And more generally, the category I would put this in is coherence, is when a model gives you an answer, does it have an accurate understanding of what it knows, what you know, and what should be in the thing that you're making together? And very often, models break coherence. And even though they're out there swarming and and solving all these amazing problems, something that's that that basic, you and I would not have a particular problem knowing, is still something that they need to improve a lot on.
00:52:58 · Akash Pasricha, Dan Shipper
[00:52:58] S0: Okay. So which lab is doing the best job of solving that? [00:53:00] S2: Oh, you know, there's there's actually, like, a really good dark horse here, which is Google. Right. [00:53:05] S0: You just talk on about Gemini being [00:53:07] S2: I've seen slept on a bit. Gemini is just really good. It's not a good coder, but it is really good... A really good writer. And there... It knows things about how to write that you probably know that are about, like, how to tell a story where each part of your argument, you have to... Like, the the the one paragraph has to lead into the next. It has to set it up so it causes the next thing to happen. Like, it knows how to do that in a way that even, like, Opus or Fable or Astra, they just sort of, like, jump and then you get that, like, this is, like, this is sort of slap. Like, it all kind of makes sense if you don't if you don't look at it too carefully. But once you really look at it, it doesn't make any sense. Gemini doesn't do that. And I have no idea why. My own... My my one theory is that there is just an advantage to being Google. Like, you you have actually organized the world's information for the last twenty years, and that allows you to, like, get a data mix that's different. [00:54:02] S0: Right.
00:54:02 · Akash Pasricha, Dan Shipper
[00:54:02] S0: Right. [00:54:02] S2: And it it it reminds me using Gemini how strange it is that almost all the other models sound a lot alike. Like, their data mixes are very similar. And Gemini is the first one that I've used, like a a frontier model, the first frontier model I've used in a while that just feels like it's trained on something else. And that's really cool. [00:54:21] S0: So so this is kind of interesting. I mean, so so Google, if I hear you correctly, you're saying that if you want to basically, what, be a creative writer, you know, Google is the model suite you can use. Where I'm going with this is actually, I I wanna get your take on, if we go through each of the labs, I wanna know from you what type of work you think they're prioritizing. Sounds like Google and the Gemini family, it's creative work that you think they're prioritizing [00:54:51] S2: I wouldn't say creative writing. I'm thinking more about, like, non fiction writing, like the writing you might do do at work. Or, you know, for us at Every, like, we we write, like, long form pieces. I wrote this, like, 8,000 word piece on the history of Codex, like, [00:55:06] S0: Codex It's is having that still personal personal.