Latent Space: The AI Engineer Podcast
01:01:51 · Mihail Eric
[01:01:51] S1: we... I think there's this, like, a thing in general principle that I that I generally... Which is, like, you build to learn and you buy to keep. Or sorry. You buy to learn and you build to keep. Yeah. The other way around. So so you you buy to learn and you build to keep. I And think that's generally true here, where in the early days when we were still trying to figure out, like, you know, get our feet under us a little bit, we were, you know, code reviews, as example, one of them where we spent a lot of time looking at different code review platforms. It turns out today, still use Devin because we've actually still find a lot of value in how the system's been built. So, you know, shout out to the Cognition team. But then along the way, we did try out some, like, SRE tools, and, you know, we tried some sort of, like, off the shelf, maybe more, like, background agent platforms. Decided that it was just given our infrastructure, given how we had set up our own cloud platform, it just made more sense to kinda build this out internally. Of course, we're not dogmatic. We we are open to revisiting them if if something comes out. But here, I felt... I think usually we ended up building in house when we felt that there was no, like, mature offering out there. So CodeRabbit is maybe an interesting example, because we did do a whole survey of CodeRabbit tools maybe over a year ago. At the time I'll show it out, CodeRabbit, right? Yeah, CodeRabbit and them. There's also CodeRabbit, Graphite, eGraphite for a while before they got acquired. Like, Ellipsis and... Which, like, a YC company before the fact. So... There's a bunch. Right? I mean, so so there were a lot of folks that we did try. And they they were okay, but I I don't think they had, like, crossed the threshold of, like, their immediately useful yet. And so then we kind of, like... We sort of, like, put it aside a little bit. This was, you know, I'm saying, like, well over a year ago. Then we kind of went back to this and sort of organically. I mean, again, again, shout out to the to the dev team and and Cognition folks where we tried it again. I think, I had sort of like gotten a... That somebody had told me, like, hey, you should try this and and, you know, it seems like it's gotten a lot better. And I've been actually... During Silas' talk, during the class last year, and it was just good.
01:03:38 · Mihail Eric, Shawn Wang, S2
And it it was picked organically. Like that's... [01:03:40] S0: So purely vibes. Right? Like, this is... There's no way to even... [01:03:42] S1: There was... Yeah. This this was like... I I mean, okay. Okay. Here... Here's what I'll say. When we did that first evaluation the first time around, where we didn't really pick any vendor, and we were like, there's no vendor that we're super happy with, I did do, like, an analysis where I looked at our PRs, basically, it's sort like a, you know, set of pie diagrams, like, here's what this is solving. Are we we happy with the of the trade off of verbosity and how often it's posting with utility? And so there was... It was more statistically driven than just vibes. The second time around, I wasn't gonna push hard on it. The first time, I was kinda like, hey, we should adopt an AI code tool. Second time, I was like, no, I'm not gonna do that again. But then organically, people were just like, Devin is, like, good. Like, can can you, like, add me to this? And so today, people are sort just, like, getting organically added. So this is a vendor where, you know, one of the cases where we're still leaning to the vendor. You know, we'll see in the long term what that looks like, but everything outside of that, we just didn't feel like maturity was there. [01:04:30] S0: Yeah. And it's not painful enough that you would, like, rip out, rip out Devin to write you on. [01:04:34] S1: That's right. [01:04:35] S0: That's interesting. [01:04:35] S1: Yeah. It's... [01:04:36] S2: It's a good good talk for Devin.
01:04:36 · S2, Shawn Wang, Mihail Eric
[01:04:36] S2: It's a good good talk for Devin. [01:04:41] S0: Even know, you know, for the record, I'm only an advisor. I'm not that involved anymore, but, like, it's interesting to to hear about it externally. Mhmm. Because internally, there is a certain amount of, let's call it, the bubble, where on Slack, you only see your positive things, because people only want to circulate positive things. But you have no dog in the fight. You just choose the [01:05:03] S1: best tool. I mean, if you want to get us free credit, totally... That'd helpful too. Say saying [01:05:08] S0: no no I to think Silas will hook you up. Okay. So there's that, right? And that informs all your teaching work. [01:05:17] S1: Yes. That's right. [01:05:17] S0: That's right. I think that's just really good. And we can bring in more of this as you like, because I think this lived experience is very good. Yeah. I'm happy to go back to agent ready code bases. So week four and five kind of [01:05:32] S1: matching Yeah, they blend a little bit. Yeah, that's right. [01:05:36] S0: Whatever you want to, whatever strong beliefs you have introduced in Monaco [01:05:40] S1: Yeah. [01:05:40] S0: And recommend people to start with.
01:05:42 · Mihail Eric
[01:05:42] S1: Yeah. No. For sure. Yeah. Week four and five, I would say they they are a little bit... Four is maybe a little bit more, like, conceptual foundations and ways to do this, because there are a lot of ways to customize agents. And then I would say week five is where I really kinda, like, lean into, alright, let's think about this and apply this in practice on, like, a true code base. And so here, I will definitely, you know, give a shout out to to the factory team and and, and wanna just highlight the value of something like a readiness report. I I think that the main piece that I want to reinforce, something like agent ready code code bases is to me like a precursor and a bit of a building block toward everything to come afterwards. When I say everything to come afterwards, I mean, like, people always got, you know, for a while, and I think people still are, like, super hyped on, like, multi agents, like, paralyzing agents, like, tens of agents doing all kinds of things. It's like you actually need to set a foundation in order to do that well, and the foundation is how verifiable is your code base, how, like, well structured is your code base, how understandable is it by an agent. And so I think, like, agent readiness to me is, like, almost almost like piece one of the next four to five weeks where... Maybe until, like, week eight or nine where where I spend more time talking about, like, broadly, you know, AI software teams and, like, kind of org level structure. But definitely building up the building blocks for background agents of the software factory. I think this is where it all kind of gets interesting from that perspective.
01:06:59 · Shawn Wang, Mihail Eric
[01:06:59] S0: Yeah. One thing that I observed you talking about from Monaco, but it's not here, is the SRE side. When you close the loop, you actually want to have the agent read your logs for builds, and then ideally read your analytics for live production. [01:07:15] S1: That's right, that's right, yeah. Not explicitly, but it's a really good shout out. So I think that I'm gonna weave that in in like two places. One is in the background agents, week, and then also in the software factory week. So you're absolutely right. I do... It's a really good call out. This idea of, like, trigger slash event based closing of the loop, which, you know, now is one of the one of the two or three entry points when people talk about agents kicking off tasks in the software factory, that's absolutely right. That'll come in there. [01:07:40] S0: Yeah. It's interesting that people call it trigger or event based, which I guess that is the actual technical implementation of it. Yeah. It just seems too low level almost. Mhmm. But I... So so the trigger is usually a cron job. Right? Like, let's let's be real. [01:07:55] S1: Sure. Sure. [01:07:55] S0: Sure. Event based is usually a cron job that, like, surfaces an event, and then it's event based. But it's all crons.
01:07:55 · Shawn Wang, Mihail Eric
[01:07:55] S0: Sure. Event based is usually a cron job that, like, surfaces an event, and then it's event based. But it's all crons. [01:08:02] S1: Yeah. Yeah. That's right. That's right. Yeah. Yeah. Or, yeah, like, webhooks or something like that. Yeah, that's absolutely right. Yes. And, you know, the technical term, of course, because at the end of the day, yeah, being sort of like a a person by by by trade and and, yeah, we we we call those things, and so I guess that's... I fall back on that terminology. [01:08:19] S0: Yeah. And have any AI SRE companies made an impression on you?