The Pragmatic Engineer
01:07:08 · Gergely Orosz, Maude Lemaire
[01:07:08] S2: Let's be opinionated. I like it. [01:07:11] S1: Naively, naively what it was doing was, say you have a bunch of things, you're finding a bunch of tasks that don't depend on each other, right? But they're latency sensitive. What's the classic solution? [01:07:24] S2: You paralyze them. [01:07:25] S1: You paralyze them. However, again, product hat, right? What if basically no one is ever doing all four of those things? [01:07:35] S2: And why are you doing all of them? At the same time, you're wasting resources, latency, you're increasing load for no reason. [01:07:43] S1: And so I remember, I think it was Opus that was very excited that it found this opportunity to paralyze a bunch of these tasks. And I was like, oh, on first glance, sure, that seems... Like a good thing to do, except that the problem, even if you paralyze them, the longest one is always going to be the bottleneck regardless. And the longest one was always so much longer than the other three. And the other three tasks were being run such a small 1% of users were doing any of the other three. And so it ended up being a complete waste of time. Like this was not the optimization that it was like selling to me as being like the panacea to improving time to first token. And a lot of the things that were actually quite helpful were really silly things like, I mean, silly, like increasing the TTL on what we stored in Redis where we could within reason.
01:07:43 · Maude Lemaire, Gergely Orosz
[01:07:43] S1: And so I remember, I think it was Opus that was very excited that it found this opportunity to paralyze a bunch of these tasks. And I was like, oh, on first glance, sure, that seems... Like a good thing to do, except that the problem, even if you paralyze them, the longest one is always going to be the bottleneck regardless. And the longest one was always so much longer than the other three. And the other three tasks were being run such a small 1% of users were doing any of the other three. And so it ended up being a complete waste of time. Like this was not the optimization that it was like selling to me as being like the panacea to improving time to first token. And a lot of the things that were actually quite helpful were really silly things like, I mean, silly, like increasing the TTL on what we stored in Redis where we could within reason. [01:08:42] S2: How do you use ELLs? Because so far my sense is... Talking through your career, you look at a problem, you understand it, you figure out how to do it, you fail sometimes or make mistakes, you learn from it, you learn from others, including right now in the last few years where AI is increasingly everywhere, but as you're Making systems more reliable, more performant. It sounds like you're still doing so much thinking. I sense that you're using the LLM on the side to generate some ideas to say, no, this is wrong. No, that was a good idea. Okay, let me use it. But unlike a lot of software engineers I talk to, even on this [01:09:19] S1: podcast,
01:09:19 · Maude Lemaire, Gergely Orosz
[01:09:19] S1: podcast, [01:09:21] S2: most of them are saying like, oh, so much of my work that I used to do, the LLM does it. I now can do more things. I can focus on higher. I'm not getting that sense from you. [01:09:29] S1: Yeah, I mean, I'm spending all day like chatting with an LLM in some way, shape or form. I think like I still do a lot of the normal engineering stuff, which is like I'm on call. Sometimes I just have to fix a silly bug. But I do tend to I do a lot of sort of preliminary exploration, but then verification manually. [01:09:54] S2: And I sense that a lot. Like, it sounds like even when you get something from an LM or whatever, you will verify, you will make sure that you understand that this is right. It sounds like you don't have a judgment if it's going to be right or wrong, but you need to verify it, right?
01:10:06 · Maude Lemaire
[01:10:06] S1: Yeah, yeah. Because I think a lot of the problems that I work on are more of the, like, product platform-y problems. And the whole point is to solve a problem holistically And to create that foundation that you can build on top of, right? And if you only built the foundation to cover like 80% of the house, like what's the point? Because that other 20% that was not built on the foundation is going to be really shaky and impact the rest of the house that you did build. And if you already did that over here, well, nothing's really stopping you from, say, tacking on another 10% off past the foundation and then that spiraling, right? And so I think especially now that I think AI is so good at just duplicating things in places because that seems fine and it doesn't have a sense of it's not great at going, but why? But why? 20 levels removed. It leans more towards, I will just tack on, I'll just build an extra bathroom past the foundation. That's fine, right? Like you asked me to build an extra bathroom, that's where I'm doing it. Rather than questioning the underlying fundamental assumptions, which are, should one build a bathroom not on top of a foundation? And those are the questions that I'm asking myself all the time. And I think part of it is because I'm coming from a product platform perspective.
01:11:40 · Gergely Orosz
[01:11:40] S2: I have a sense that the product platform work, the platform work reliably performance, it will only be more important because, again, we'll have systems that have so much load, so much test. What advice would you have for a software engineer who What are paths that you've seen people make it onto platforms that you worked on? How did they transition?
01:12:12 · Maude Lemaire
[01:12:12] S1: I'm such a strong advocate for more product folks, more interchange actually between product folks and product platform teams. I've seen it so many times where product platform teams, are spun up out of product engineers who care a lot about the fundamentals. They care about reliability. They care about performance already. They're trying to improve that and inject that into the way that they're building product all the time. And I think the best product platform folks that I've worked with get to a point where they've seen a pattern enough times in the things that they are building as product engineers. And they're like, I want to dig in and solve this problem across the board. Like this specific problem, I understand really deeply from having implemented it two, three, four times. I want to go solve it once and for all and be done and like have this thing that other people can also use. Right. They get annoyed with the toil and then they go and maybe embed with the product platform team or they like spin one up with some other folks and then they do that. And then a lot of times those folks end up staying on product platform teams and continuing to really like the product platform work. But then they get further and further detached from what product development actually feels like. And I think that's where you see sort of the chasm that ends up being created between product platform teams and product teams is just there's not enough cross pollination between the teams. And so one of the ways that I've seen that chasm merge back together is having like rotation cycles between those teams, sort of always having someone on the product platform team, having a core set of folks who are like maintaining the main systems and platforms that they've built, but sort of always having someone on rotation from a product team. On the product platform team bringing like, oh, well, I don't know what you all are doing here, but like on product, we're just like YOLO shipping all this AI slop all the time. Like what you're reading your code, what?