Latent Space: The AI Engineer Podcast
00:58:56 · Shawn Wang
[00:58:56] S0: Yeah. Interesting to have... You talk very differently than a normal head of AI that I might also talk to, because you're also a former founder. And you're also used to thinking about these things in this way. And it's interesting to... It's important to have that Right. In your role. Yeah. Cool. I I I think you've you've introduced Monaco enough there. And then and obviously, I think as Monaco, the story develops, we can revisit. Sure. Sure. Maybe the bridge back is also what is coding or agent native coding or modern software development like inside of Monaco? You know, I think you're... When you're teaching, you're maybe trying to be here's a lay of the land, but maybe inside of Monaco, you guys are more opinionated on one thing versus the other?
00:59:40 · Mihail Eric, Shawn Wang
[00:59:40] S1: Yeah. Yeah. No. I mean, have... So we've made investments in coding agents from basically day one, and we've learned a lot. We have our own scars that we picked up in learnings. And I think we've now been getting much deeper into sharing some of our learnings. So in our blog, which might be worth sharing here, we've been increasingly posting some of what, maybe not that. So that one is more of a product, the one at the very top. This stuff, I would say, yeah, when AI gets desperate and before, like scaling contributions, build or buy, SRE. So we spend a lot of time looking at different providers, looking at vendors, end up building a lot of things in house. So for us, something like a background agent platform, we have our own version of like a Ramp Inspect that we call MonacoDR, which is... It's built on the Coder platform, but it is what we use to essentially scale up our engineering product. [01:00:26] S0: Coder like coder.com?
01:00:26 · Shawn Wang, Mihail Eric, S2
[01:00:26] S0: Coder like coder.com? [01:00:27] S1: Coder.com. Yeah. Oh. Yeah. Okay. We actually... Yeah. But have sort of adapted it to some of our use cases. And, yeah, it's where a lot of our agent development happens. It's just this really well designed piece of software for essentially isolating of of sort of, development environments, scaling out to multi agents, you know, being able to run these, like, long long horizon agentic coding agent tasks. So Monaco, shout out to John Tracy and and co for for building that out. And and then, yeah, like, from there, we've also built our own versions of, like, you know, like a like a Slack bot, which is called HAL. But, you know, effectively HAL is, like, this part AI SRE, part like customer success, part like kind of QA Engineer, because it it it lives in our Slack, it serves all the purposes of helping us debug things and put up PRs when things are broken, but also used by our own salespeople to find out how the product works, find out, like, how to get to the bottom of something. So there's, like, interesting questions we've had to answer around the different personalities that Hal has, depending on whether he's talking, like, a salesperson versus a versus, like, a develop... Developer. And so systems like this that we built out over time a lot in the last, like, year. So shout out to Yunja for for Hal and and John Tracy as well. [01:01:40] S0: So you're building a lot in house. Yeah. You also use other tools? [01:01:45] S1: We do use other tools. [01:01:46] S0: So what is the line? [01:01:48] S2: Builders versus buy.
01:01:48 · S2, Mihail Eric, Shawn Wang
[01:01:48] S2: Builders versus buy. [01:01:49] S1: What was that? Yeah. The Builders buy. It's a good question. [01:01:51] S0: And
01:01:51 · Mihail Eric
[01:01:51] S1: we... I think there's this, like, a thing in general principle that I that I generally... Which is, like, you build to learn and you buy to keep. Or sorry. You buy to learn and you build to keep. Yeah. The other way around. So so you you buy to learn and you build to keep. I And think that's generally true here, where in the early days when we were still trying to figure out, like, you know, get our feet under us a little bit, we were, you know, code reviews, as example, one of them where we spent a lot of time looking at different code review platforms. It turns out today, still use Devin because we've actually still find a lot of value in how the system's been built. So, you know, shout out to the Cognition team. But then along the way, we did try out some, like, SRE tools, and, you know, we tried some sort of, like, off the shelf, maybe more, like, background agent platforms. Decided that it was just given our infrastructure, given how we had set up our own cloud platform, it just made more sense to kinda build this out internally. Of course, we're not dogmatic. We we are open to revisiting them if if something comes out. But here, I felt... I think usually we ended up building in house when we felt that there was no, like, mature offering out there. So CodeRabbit is maybe an interesting example, because we did do a whole survey of CodeRabbit tools maybe over a year ago. At the time I'll show it out, CodeRabbit, right? Yeah, CodeRabbit and them. There's also CodeRabbit, Graphite, eGraphite for a while before they got acquired. Like, Ellipsis and... Which, like, a YC company before the fact. So... There's a bunch. Right? I mean, so so there were a lot of folks that we did try. And they they were okay, but I I don't think they had, like, crossed the threshold of, like, their immediately useful yet. And so then we kind of, like... We sort of, like, put it aside a little bit. This was, you know, I'm saying, like, well over a year ago. Then we kind of went back to this and sort of organically. I mean, again, again, shout out to the to the dev team and and Cognition folks where we tried it again. I think, I had sort of like gotten a... That somebody had told me, like, hey, you should try this and and, you know, it seems like it's gotten a lot better. And I've been actually... During Silas' talk, during the class last year, and it was just good.
01:03:38 · Mihail Eric, Shawn Wang, S2
And it it was picked organically. Like that's... [01:03:40] S0: So purely vibes. Right? Like, this is... There's no way to even... [01:03:42] S1: There was... Yeah. This this was like... I I mean, okay. Okay. Here... Here's what I'll say. When we did that first evaluation the first time around, where we didn't really pick any vendor, and we were like, there's no vendor that we're super happy with, I did do, like, an analysis where I looked at our PRs, basically, it's sort like a, you know, set of pie diagrams, like, here's what this is solving. Are we we happy with the of the trade off of verbosity and how often it's posting with utility? And so there was... It was more statistically driven than just vibes. The second time around, I wasn't gonna push hard on it. The first time, I was kinda like, hey, we should adopt an AI code tool. Second time, I was like, no, I'm not gonna do that again. But then organically, people were just like, Devin is, like, good. Like, can can you, like, add me to this? And so today, people are sort just, like, getting organically added. So this is a vendor where, you know, one of the cases where we're still leaning to the vendor. You know, we'll see in the long term what that looks like, but everything outside of that, we just didn't feel like maturity was there. [01:04:30] S0: Yeah. And it's not painful enough that you would, like, rip out, rip out Devin to write you on. [01:04:34] S1: That's right. [01:04:35] S0: That's interesting. [01:04:35] S1: Yeah. It's... [01:04:36] S2: It's a good good talk for Devin.