The Pragmatic Engineer
01:05:43 · Gergely Orosz, Maude Lemaire
[01:05:43] S2: Please don't do that. [01:05:44] S1: Always do it in the same place. The problem with duplication is just like so much worse when you're dealing with And so one of the first things that I ended up having to do was like, honestly, manually instrument a lot of these flows and actually read through the code myself and understand, okay, this chunk of processing is synchronous. This chunk of processing is asynchronous and put all of those together. And one of the ways that I started doing that was I had it generate a Like big sequence diagram of everything that goes into time to first token. So like from the time you hit submit to when you receive that first visible token and it was huge. And I went to Staples and I had them printed out on like big blueprint paper because I'm still a really analog person. And then from that, I went through and I was reading the code and I was annotating in on my big piece of paper. Like, okay, this is correct or this is incorrect. This piece we could probably do something about. This other piece, no, actually definitely needs to happen before this other piece. One of the other things that I was finding that it was doing really painfully wrong is that it was finding sort of the classic things that could... [01:07:00] S2: What was doing what the LM was doing wrong? [01:07:02] S1: Oh, yeah. So what it was doing wrong was finding... Well, wrong is kind of the wrong word. [01:07:08] S2: Let's be opinionated. I like it.
01:07:08 · Gergely Orosz, Maude Lemaire
[01:07:08] S2: Let's be opinionated. I like it. [01:07:11] S1: Naively, naively what it was doing was, say you have a bunch of things, you're finding a bunch of tasks that don't depend on each other, right? But they're latency sensitive. What's the classic solution? [01:07:24] S2: You paralyze them. [01:07:25] S1: You paralyze them. However, again, product hat, right? What if basically no one is ever doing all four of those things? [01:07:35] S2: And why are you doing all of them? At the same time, you're wasting resources, latency, you're increasing load for no reason. [01:07:43] S1: And so I remember, I think it was Opus that was very excited that it found this opportunity to paralyze a bunch of these tasks. And I was like, oh, on first glance, sure, that seems... Like a good thing to do, except that the problem, even if you paralyze them, the longest one is always going to be the bottleneck regardless. And the longest one was always so much longer than the other three. And the other three tasks were being run such a small 1% of users were doing any of the other three. And so it ended up being a complete waste of time. Like this was not the optimization that it was like selling to me as being like the panacea to improving time to first token. And a lot of the things that were actually quite helpful were really silly things like, I mean, silly, like increasing the TTL on what we stored in Redis where we could within reason.
01:07:43 · Maude Lemaire, Gergely Orosz
[01:07:43] S1: And so I remember, I think it was Opus that was very excited that it found this opportunity to paralyze a bunch of these tasks. And I was like, oh, on first glance, sure, that seems... Like a good thing to do, except that the problem, even if you paralyze them, the longest one is always going to be the bottleneck regardless. And the longest one was always so much longer than the other three. And the other three tasks were being run such a small 1% of users were doing any of the other three. And so it ended up being a complete waste of time. Like this was not the optimization that it was like selling to me as being like the panacea to improving time to first token. And a lot of the things that were actually quite helpful were really silly things like, I mean, silly, like increasing the TTL on what we stored in Redis where we could within reason. [01:08:42] S2: How do you use ELLs? Because so far my sense is... Talking through your career, you look at a problem, you understand it, you figure out how to do it, you fail sometimes or make mistakes, you learn from it, you learn from others, including right now in the last few years where AI is increasingly everywhere, but as you're Making systems more reliable, more performant. It sounds like you're still doing so much thinking. I sense that you're using the LLM on the side to generate some ideas to say, no, this is wrong. No, that was a good idea. Okay, let me use it. But unlike a lot of software engineers I talk to, even on this [01:09:19] S1: podcast,
01:09:19 · Maude Lemaire, Gergely Orosz
[01:09:19] S1: podcast, [01:09:21] S2: most of them are saying like, oh, so much of my work that I used to do, the LLM does it. I now can do more things. I can focus on higher. I'm not getting that sense from you. [01:09:29] S1: Yeah, I mean, I'm spending all day like chatting with an LLM in some way, shape or form. I think like I still do a lot of the normal engineering stuff, which is like I'm on call. Sometimes I just have to fix a silly bug. But I do tend to I do a lot of sort of preliminary exploration, but then verification manually. [01:09:54] S2: And I sense that a lot. Like, it sounds like even when you get something from an LM or whatever, you will verify, you will make sure that you understand that this is right. It sounds like you don't have a judgment if it's going to be right or wrong, but you need to verify it, right?
01:10:06 · Maude Lemaire
[01:10:06] S1: Yeah, yeah. Because I think a lot of the problems that I work on are more of the, like, product platform-y problems. And the whole point is to solve a problem holistically And to create that foundation that you can build on top of, right? And if you only built the foundation to cover like 80% of the house, like what's the point? Because that other 20% that was not built on the foundation is going to be really shaky and impact the rest of the house that you did build. And if you already did that over here, well, nothing's really stopping you from, say, tacking on another 10% off past the foundation and then that spiraling, right? And so I think especially now that I think AI is so good at just duplicating things in places because that seems fine and it doesn't have a sense of it's not great at going, but why? But why? 20 levels removed. It leans more towards, I will just tack on, I'll just build an extra bathroom past the foundation. That's fine, right? Like you asked me to build an extra bathroom, that's where I'm doing it. Rather than questioning the underlying fundamental assumptions, which are, should one build a bathroom not on top of a foundation? And those are the questions that I'm asking myself all the time. And I think part of it is because I'm coming from a product platform perspective.
01:11:40 · Gergely Orosz
[01:11:40] S2: I have a sense that the product platform work, the platform work reliably performance, it will only be more important because, again, we'll have systems that have so much load, so much test. What advice would you have for a software engineer who What are paths that you've seen people make it onto platforms that you worked on? How did they transition?