Latent Space: The AI Engineer Podcast
01:11:53 · Shawn Wang, Mihail Eric
[01:11:53] S0: Broadly, because that is our audience. [01:11:56] S1: Mhmm. Take a look at that. Yeah. I think that, again, it's function of how we just scale. Right? And I think that for the enterprise, there are a number of issues that aren't always I almost feel like the answer is not even always just more agents, and and since we're still actually on the week five agent ready code bases, like, I talk to companies of, like, a certain scale, the the things that are holding them back are really more just, like, low level DevOps, CICD stuff, right? Their merge queues are getting backed up. It takes too long to stand up an app. There's just actually more foundational things that have almost nothing to do with, that throwing in one more SRE bot, or a la a bot wouldn't be the answer to the problem, because there's more systemic things that they need to figure out. And so... So that brings us to [01:12:38] S0: the week six.
01:12:39 · Mihail Eric
[01:12:39] S1: Yeah. Exactly right. So that's where we get into... Yeah. Exactly right. Yeah. There we go. So the MergeQ and beyond. Yeah. This is a love letter of sorts to modern, gentic code review. Yeah, I think that biggest thing that maybe happened between last year and this year was just it suddenly became very clear that code review is a bottleneck, especially once you people got really, really, appealed and the velocity on generation became so high that, you know, you could, in theory, like, put up 30 plus PRs a day, and then what? Like, what's... At what point do you decide that you either have to institute yourself and put yourself in the middle as some way gate and review things in the traditional way, but then you're literally the constraint, you're the choke point of the velocity, and that's not good. Right? And then what was the point of generating at scale? Or you say, well, how do we maybe get the best of both worlds? Where we're instituting ourselves enough that we have trust in what's getting merged, but we're also able to kind of merge at scale, but we're also not kind of freewheeling it here, and anything goes. So a lot of, I think, what I wanna touch on here is just that middle ground, right? Which is like, there are... There is a class of problems that if you've put in thoughtful design into your agent readiness, you've put in thoughtful design around your verifiability, there's a class of problems that you can feel comfortable, you know, some percentage of the time, essentially, like auto approving. And what has to be true from a kind of a design perspective, from like an architecture perspective, to just auto approve? And when do you have to insert yourself and say, actually, here, I still wanna review this carefully? And even if you do review it carefully, are you still doing line by line review, are you doing something else? Right? So yeah, some of this.
01:14:13 · Shawn Wang, Mihail Eric
[01:14:13] S0: One of our hottest topics of this year, we have a guest writer program for Latent Space. This is from Ankit from Aviator. [01:14:20] S1: Nice. [01:14:20] S0: And he has a whole thing on killing the code review, because I... You know, that is, like, maybe the final boss Yeah. Of of the... That's that's a bottleneck. He has a few, gates by which he does this. I don't know if you... You've talked to Funky or... [01:14:34] S1: Yeah. No. I've I've read this post. Yeah. I haven't spoken to him personally, but I've I've definitely... [01:14:38] S0: Mostly right. Any criticisms? [01:14:40] S1: No. I think this is right. [01:14:40] S0: I mean, [01:14:41] S1: like, broadly right. I guess you mentioned one thing, like, last boss. Maybe not... I I think there's two choke points. There's the review, and there's probably, like, the intent or, like, kind of the thing at the top. Right? Like, [01:14:49] S0: is the And there's some there's some stats as well that that I like to pull up. Everyone's stats is different. [01:14:54] S1: For sure, for sure. [01:14:54] S0: But clearly, more people are merging. Clearly, the number of reviews are not increasing in line with merge Yeah. That's
01:14:54 · Shawn Wang, Mihail Eric
[01:14:54] S0: But clearly, more people are merging. Clearly, the number of reviews are not increasing in line with merge Yeah. That's [01:15:03] S1: right. That's And if you look at any modern sort of company that's really invested in this and thought about this carefully, and some people have already done posts about this, and I'm sure we'll do posts about this as well. But I think recently, like, the Amplitudes and, I think Cloudflare has has done a good one. I think Augment has also done a good one, where they talk about the design of their AI code review system. These are a bunch of really good, if you wanna share some of those links. Think they're fantastically written posts. Yeah. So shout out to... I actually, shout out to Cloudflare, because I think, like, very, like, kind of, like, know, in a sense, like, legacy company, or, like, you know, bigger company. But I think they've done a fantastic job in terms of just their their content around the AI, the AIS DLC. So I think they have a good post there. But TLDR, yes, I think this is true. I think that there's... We're still finding that sweet spot. What is auto approvable? What is not really auto approvable? And if it is not auto approvable, is even in that world of something that has to be somewhat human reviewed, what do we review? Are we still doing the line by line? Are we doing some combination of the spec? And, you know, I think this... Yeah. Again, it's a whole separate question there as well. But...
01:15:03 · Mihail Eric, Shawn Wang
[01:15:03] S1: right. That's And if you look at any modern sort of company that's really invested in this and thought about this carefully, and some people have already done posts about this, and I'm sure we'll do posts about this as well. But I think recently, like, the Amplitudes and, I think Cloudflare has has done a good one. I think Augment has also done a good one, where they talk about the design of their AI code review system. These are a bunch of really good, if you wanna share some of those links. Think they're fantastically written posts. Yeah. So shout out to... I actually, shout out to Cloudflare, because I think, like, very, like, kind of, like, know, in a sense, like, legacy company, or, like, you know, bigger company. But I think they've done a fantastic job in terms of just their their content around the AI, the AIS DLC. So I think they have a good post there. But TLDR, yes, I think this is true. I think that there's... We're still finding that sweet spot. What is auto approvable? What is not really auto approvable? And if it is not auto approvable, is even in that world of something that has to be somewhat human reviewed, what do we review? Are we still doing the line by line? Are we doing some combination of the spec? And, you know, I think this... Yeah. Again, it's a whole separate question there as well. But... [01:16:09] S0: Are are people keeping, are you guys keeping the spec up to date at all times? The spec is no way. Right?
01:16:09 · Shawn Wang, Mihail Eric
[01:16:09] S0: Are are people keeping, are you guys keeping the spec up to date at all times? The spec is no way. Right? [01:16:16] S1: Yeah. No. Well, I mean, okay. In a few different places, right? So, like, there's a spec that as it exists in, like, a linear issue, and there's a spec as it exists in, like, the context of a PR description, right? So you can, like, you can have your PR description be the place where you, like, describe more or less the entirety of the implementation in a more condensed format. And as long as that's the most up to date thing representing this point in time of the code base, I think that's sufficient, actually. So we're thoughtful about our peer descriptions. We make them be not just, like, you know, a one paragraph. It's more like, it is effectively like a spec of what has happened in the PR, and there's even, like, kind of line file level pointers to things where things have changed, so you not only get the high level of what's happened, but even the kind of the construction of the files and sort of the service level boundaries that you're touching. So yeah, I think that's where we've invested quite a bit.