Latent Space: The AI Engineer Podcast
00:31:39 · S2, Mihail Eric, Shawn Wang
[00:31:39] S2: I think you don't want to fault them for growing a system prompt and then cutting it. At some level, it's also how the model was, right? Weaker older models, you need to give them examples of this is good, this is bad. And then obviously you cut that away at some point, right? And that's just part of learning what has been done, right? You can apply the same towards, if you're using a local coding agent with not the strongest model, you might need to add these examples, right? [00:32:05] S1: That's right. Is that correct? Yeah. Yeah. And then, of course, there's interesting AI learnings to fall out of that, which is why, as the models get better, when are you over constraining it by adding those examples, right? [00:32:16] S0: And then how you [00:32:16] S1: can pull away the over constraining. That's, I think, part of the learning as well. [00:32:20] S0: Okay. Week two, you're going to talk about context engineering. Repet, I haven't seen, but there's many versions of spectrum development. What would you highlight to a broader audience about what you're putting in here that people should know? [00:32:35] S2: Yeah. This is like the one hour version of the course. Hey, exactly. [00:32:40] S0: Please condense everything. The answer [00:32:42] S1: is going just... That's fair, yeah. Like... [00:32:46] S0: Well, yeah. It's it's also a trailer. Right? It's like, well, if you want more... [00:32:48] S2: If you're signing up next next year or next fall for courses, you know, this is your one hour.
00:32:48 · S2, Mihail Eric, Shawn Wang
[00:32:48] S2: If you're signing up next next year or next fall for courses, you know, this is your one hour. [00:32:53] S1: This is like this is like... Almost like the quibi of the class. You know I mean? Where whether it's it's too long to be a trailer. It's too short to be the class. Is this like... What I guess people will double best part, [00:33:02] S2: if it's a trailer, by the next time you teach it, it'll change again. [00:33:05] S1: That's true, [00:33:06] S0: that's true,
00:33:07 · Mihail Eric
[00:33:07] S1: that's true. Yeah. Well, so it's okay. So a good call out on engineers, it's funny. I think that the main thing that I just wanna reinforce here is, and I see this even when I'm, like, interviewing folks. Very few people in industry even progress beyond, like, just vanilla prompting. Right? There's, like, a form of, like, hey, do me the thing. Right? Like, do do the thing. And and that, certainly as the models get good, that can carry you a certain distance. But inevitably, you know, any folks that I speak to that are... Come out of, like, a Google or some place where there's a massive, like, code base, the reality is you have to be lot more thoughtful about how you're, like, ingesting context. Right? And so thinking through what are some of those principles and those techniques around how to, like, point, you know... And I sort of, you know, the rapid framework that I'm sort of describing here is is functionally similar to, like, DEX's RPI, right, or something of that nature, where you have this, like, re research kind of proposal, and sort of, like, a spec, and then and then you do, like, your implementation. And then you kind of tie together with some testing and whatnot. So what I just wanted to reinforce here is that, like, you should have this step, especially when you're entering unfamiliar code bases, when you're trying to, sort of infuse into the context something that an agent needs to understand about a very large code base. There's value in something that is a research phase. And then in the case of this particular sort of framework that I'm putting forward, it's also then suggests some proposal where there's a few alternative ways of solving a problem once you've done the research. And so this is where I also invite the human in the loop to decision be maker on one approach versus the other. And then you enter the plan, which is kind of the spec. All to say, the TLDR is like, there is a way to do this that actually does scale out to code bases that are millions of lines long. And it's not just the vanilla prompt. And I think that's what I wanna highlight. If you're gonna go work in these big code bases, you should use something a little bit more involved.
00:34:51 · S2, Shawn Wang
[00:34:51] S2: Anything to add on context engineering? [00:34:54] S0: I think that probably the weirdest thing about how I think about context right now is when you split out to a sub agent, [00:35:02] S2: because [00:35:04] S0: subagents can benefit from inheriting your entire context of the main thread or not. Sometimes there's a regular disagreement on on that. And then sometimes you want to condense what the subagent does and researchers put it back onto the main thread. And it's very hacky. I don't like anything I'm doing. So I don't have any answers. All I know is, like, when you want to fan out work Mhmm. It is not just about you have a big spec and you you, you know, you do RPI, you do RREPIT, whatever. It is not one agent anymore. It is about fanning out work and making sure they all have the right context and they all have the right guidance. If you have a review agent reviewing all the other works, you have... Make that guy have have everything. And you always go through this phase of expanding contracts, like, oh my god. These guys don't have enough context. I'm gonna give them more context. Okay. Great. And then they're gonna overture you into some random one off comment that you made a while ago and then make some really stupid mistakes because you you didn't mean it that way, but, like, like, took it as the law. Right. Something like that. And so I don't know what what the word is I'm saying. Right? This is, like, multi agent management for context.
00:36:12 · Mihail Eric, Shawn Wang, S2
[00:36:12] S1: Yeah. Yeah. Yeah. Yeah. I think that that's absolutely right. There... There's, like, some aspect of, this is maybe the part that's more, maybe, among the most art than science. Right? It's like some combination, I think, to your point. Yes, the fan out for the exploration, the fan out for the subtasks, then the consolidation, compaction, whatever you might want to call it. I think being thoughtful about refreshing of the context, understanding when things are getting a little bit too bloated. I think Dex had this concept of the dumb zone and sort of being a little bit more thoughtful about the dumb zone. [00:36:39] S0: He's good at these equating things. [00:36:40] S1: Oh, Yeah. [00:36:40] S2: Think The little diagrams. [00:36:41] S1: The little diagrams. Oh, mean How did you [00:36:43] S0: have him on? [00:36:44] S1: Yeah. Well, think he's... I mean, he definitely informed a lot of the curriculum here. And so he's... Which is fantastic, I think, to think around all this. No, I think the diagrams are huge. That's useful.
00:36:53 · S2, Mihail Eric
[00:36:53] S2: I would add, breadcrumbs for people that are trying to look into this. It's good to see best context engineering preferences, like on your sub agent inheriting cloud managed agents has a version where sessions are separate and it's not inherited. This used to be a big topic before reasoning models when agents didn't run as long. We had shorter context windows. Everyone knows very, very managing that precious context window, right? LingQing used to write a lot of blogs about this in good context engineering. All this stuff is good for people to know. I think in the student context as well, you can expect the average student to take your class because it's a great class and they'll know all these things. You should know them, know them better as well. This is a fun space for research as well. One of the papers, research papers presented at AIE, they looked at the Delta in the Claude code system prompt, what got changed. And they plotted a whole bunch of changes in performance. They tried to reverse engineer it. It's just good research that you can do. There's another one that also looked at context window management and how it affects performance downstream. They weren't the best papers, but like it's quite accessible and it puts out good blog posts. So people that are trying to work on it, I just recommend it. It's a good area you can easily poke into. [00:38:15] S1: Yeah. Take a shout out. [00:38:16] S2: Agent skills, CLI?