The Information's TITV
00:46:28 · Akash Pasricha, Andrew Ambrosino, Dan Shipper
[00:46:28] S0: But let me just ask you the flip side to that, which is that you talk about personifying these agents. Yes. I mean, there is kind of a whole school of thought that we shouldn't be personifying them too much and treating them like humans because of the fact that, you know, as these capabilities get better, that could be a risk. So, you know, where where do you sort of sit on that balance? [00:46:49] S5: Yeah. I I understand this take. I think with dots, you're seeing us embody but not personify, which is, you know, kind of a little bit nuanced. And, you know, I think that we've done a very good job with that product, we'll iterate, and we'll see how, you know, these things, these things will surely evolve over time. [00:47:08] S0: Right. [00:47:09] S5: But I think that there's there's a a, you know, threading that has to happen here. [00:47:13] S0: Yeah. Great. Well, Andrew, I wanna thank you for Thank you. Joining us. That is Andrew Ambrosino from OpenAI here on TITV. I'm here with Dan Shipper, the CEO and founder of Every. Dan, welcome to the show. It's great to have [00:47:40] S5: you here. [00:47:41] S2: Thanks for having me, and thanks for coming to Thesis. [00:47:43] S0: Wait. Is this like your Super Bowl? [00:47:44] S2: Like Yes. In fact... Well, you know, GPT 8 probably would be, like, my my true Super Bowl, but barring that, this this is a really special gathering, and it's it's really great to have
00:47:44 · Dan Shipper, Akash Pasricha
[00:47:44] S2: Like Yes. In fact... Well, you know, GPT 8 probably would be, like, my my true Super Bowl, but barring that, this this is a really special gathering, and it's it's really great to have [00:47:57] S0: you here. Well, it's a great event. I'm excited to talk about all that's news right now in AI. Tell me, what's the most counterintuitive thing you've heard on the ground here today at the event about AI? See, I don't know if [00:48:08] S2: this is counterintuitive, but I think it's I think it's really interesting, and it says something about where things are going. Because everybody at this event... If you're here, there's 400 people here. If you're here, you love AI. And I've talked to so many people at this event who are working in big companies who are getting so much pressure to use AI as much as possible, which is... They love. It's good for them. It's it's like... It's what it's what they want. And I think the expectations from management or executives that are not familiar with AI for how fast you can go before you set up a good system to do it well, there... It's like... It's kind of out of whack. And so there's something really interesting where for AI pill people inside of big companies, the way that they're being managed, I think, is is out of whack with, like, what is realistic and and the and the the best way to start a high quality AI project, which is to get the basics right first and build, like, build slowly, and then you can go fast. Go slow to go
00:48:08 · Dan Shipper, Akash Pasricha
[00:48:08] S2: this is counterintuitive, but I think it's I think it's really interesting, and it says something about where things are going. Because everybody at this event... If you're here, there's 400 people here. If you're here, you love AI. And I've talked to so many people at this event who are working in big companies who are getting so much pressure to use AI as much as possible, which is... They love. It's good for them. It's it's like... It's what it's what they want. And I think the expectations from management or executives that are not familiar with AI for how fast you can go before you set up a good system to do it well, there... It's like... It's kind of out of whack. And so there's something really interesting where for AI pill people inside of big companies, the way that they're being managed, I think, is is out of whack with, like, what is realistic and and the and the the best way to start a high quality AI project, which is to get the basics right first and build, like, build slowly, and then you can go fast. Go slow to go [00:49:11] S0: fast. And this is kind of interesting though because you're saying management's expectations are too high, and yet we've got headlines every week saying x y z company is putting a cap on how much you you you spent on AI.
00:49:11 · Akash Pasricha, Dan Shipper, Paul Kedrosky
[00:49:11] S0: fast. And this is kind of interesting though because you're saying management's expectations are too high, and yet we've got headlines every week saying x y z company is putting a cap on how much you you you spent on AI. [00:49:25] S2: And there's, you know, and there's there's different managers in different parts of the economy, and and and I think there's this flip flop from, like, token maxing to, like, token token minning. And and and to me, what what that says is if you're an executive, you're... If you're a CEO and you are not... You don't have your hands in this technology all the time, the only thing that you can do is follow the trend and be like, oh, token maxing or oh, no. Token maxing is over and now we're gonna put put budget caps. That's the only thing you can do and both don't work. Because the reality is that, yeah, forcing people to spend as many tokens as possible when your organization is not set up to do that well, it's not gonna work. Obviously, you're just gonna get people who are gonna spend tokens on things that don't don't matter. But but if your organization is set up to use AI well and you and you unleash the people who are really good at doing it, you actually probably should be token maxing because the more tokens you spend... If your if your setup is good, the more token you... The more tokens you spend, the more high quality output you [00:50:22] S1: get. Right.
00:50:22 · Paul Kedrosky, Dan Shipper, Akash Pasricha
[00:50:22] S1: get. Right. [00:50:23] S2: Right. And so I don't think there's I don't think there's one right answer, but the the the... In terms of, like, a headline token maxing or token budgets, but the the right answer is always you have to have your hands in it in order to make good decisions. [00:50:35] S0: Right. Okay. So let's let's talk about the tokens and the model specifically. So you've been doing these vibe checks for for how long been [00:50:43] S2: Three years [00:50:44] S0: or four years? Yeah. Three years of vibe checks. You've probably done dozens by now. [00:50:47] S2: I mean, [00:50:47] S0: more time. Maybe. So I wanna talk about where the models are going. And I asked you a question a couple months ago when you came on the show about what the models are not yet good at. [00:51:00] S2: And What did I say? Do you remember? [00:51:03] S0: I I don't. The the reason I'm asking you again is because we've had a couple models come out since then. [00:51:11] S1: Yeah. [00:51:12] S0: So I'm gonna re ask you the question, what are they not good at right now?
00:51:16 · Dan Shipper
[00:51:16] S2: That's a that's a really good one. Okay. So, you know, obviously, the models, they're out there, like, hacking into, like, government, like, infrastructure and solving Navier Stokes. So they're, like, they're very good. And there are some models that are coming out that are not out yet that are very, very good. And I I think will change how we think about things like building software, for example. And they are very bad at a lot of things. A really a really simple example is models often leak their context in in the way that they do their output. So an example might be, you've ever had the experience of working with a model and you're like... You're you're you're you're making a landing page together and you're like, yeah, I don't like this headline because it it mentions x and and... But it should mention y. And then the model's like, great. I'm gonna rewrite the landing page, and it says, we're not gonna talk about x. We're we're gonna we're gonna write y, and that puts it in the copy of the landing page. Like, is a... That's a leak in the way that it that it that it thinks into the artifact that it's producing. And this is actually quite common even with the smartest models. And more generally, the category I would put this in is coherence, is when a model gives you an answer, does it have an accurate understanding of what it knows, what you know, and what should be in the thing that you're making together? And very often, models break coherence. And even though they're out there swarming and and solving all these amazing problems, something that's that that basic, you and I would not have a particular problem knowing, is still something that they need to improve a lot on.