00:07:34 · Ruby Justice Thelot, Ruby Justice Thelot
Ad[00:07:34] S2: I wanna shout out our sponsor, Lovable. You know that app you've been meaning to build? The internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship ship today. That includes an editable codebase you can inspect and change, plus two way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more, so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable, lovable. Dev. Now onto the episode. [00:08:13] S1: Yeah. I'm also curious, like, before we talk about simulations Yep. Like, how... Where do you get the initial data, or how do you find, like, the most common issues, or how did you initially... And then how do you see... Like, track them as they evolve for users?
00:08:13 · Ruby Justice Thelot, Ben Hylak, S3
[00:08:13] S1: Yeah. I'm also curious, like, before we talk about simulations Yep. Like, how... Where do you get the initial data, or how do you find, like, the most common issues, or how did you initially... And then how do you see... Like, track them as they evolve for users? [00:08:29] S0: It's a really good question. So, yeah, we work with, like, a crazy range of customers at this point. It's everything from, like, you know, logistics, law, health care, you know, personal agents. Right. It's just, like, everything. And each of those have a completely different set of issues, things that they care about, like what is good behavior, what is bad behavior. Mhmm. And it is literally wildly different. And so a lot... Like, our entire product is built around being able to do that in an unsupervised way, but, like, to then show people, like, candidate issues that we think are important to them that they can then, like, give feedback to, and we can adapt in real time, like, are everything that we're detecting. Yeah. [00:09:11] S1: Yeah. So tell us about simulations, which you launched today, which is meant to simulate your agent's world world to catch what what your changes break. [00:09:21] S3: Yeah. [00:09:21] S1: So what what does it mean to simulate agent's world? [00:09:24] S0: It's a good question. So, you know, there was a time, like... Do you remember prompt playgrounds? Do you remember this? [00:09:31] S1: I don't know.
00:09:31 · Ben Hylak
[00:09:31] S0: I don't know. Okay. So, like, people that were, like, making chatbots, like, three years, four years ago, whatever. It's like OpenAI and Anthropic. They have these, like, prompt playgrounds where it's like you... Because, like, a chatbot was essentially just a prompt. Mhmm. Right? It was just this, like, text, you you know. And then you'd have this little playground, and you'd edit the prompt, and you'd rerun it, and you would see the new Mhmm. The thing is that, like, agents have gotten so, you know, complicated and so complex that, like, there's not some simple prompt that defines their behavior anymore. Yeah. It's like, yeah, there is a prompt, but there's also skills, the tools that they have access to. Those tools have definitions that explain what the tools are for. And, like, then there's a... Like, you you you kind of can even statically analyze an agent. You kind of have to put it in motion because there's all of these... Whether it's feature flags or conditionals, like what plan the user's on, all these sorts of things that go into assembling the context that's available. And then, of course, there's memory and all these other things. And so what that means is that, like, most teams right now are not able to iterate on their agent in a meaningful way without shipping to production. Mhmm. And so... And I think that production is a very accurate kind of source of truth as far as the problems that really matter and all these sorts of things. But also, like, we wanna prevent problems, you know, I think especially as the stakes get higher, it's not enough to just catch problems that are happening in production. You have to have to prevent them in a meaningful way.
00:10:51 · Ruby Justice Thelot, Ben Hylak
[00:10:51] S1: Yes. Yeah. Yeah. [00:10:53] S0: I think to date, the way it's been done is like just, you know, normal kind of standard evals, which are like, you define these test cases for every, you know, capability your agent has essentially, and then you run them. You know, teams will collect, like, thousands of them. And nowadays, it's just like you ask an agent to make the evals. Yeah. But it's it's very backwards if you think about it. It's like, you you know, we want agents to do things we have never thought about before. And what... Like, so defining a test case for every capability is, like, a little silly. And so we we took a different approach, which is, like, we wanna show you what your change is going to change. So the easiest way to do that is if we can make a way, which is what we did, to re simulate past things that happened in production. That means for, you know, a specific user was using a personal agent, we can simulate all the services that agent was connected to at that point in time, their email, their calendar, you know, like, they were ordering something on DoorDash using the DoorDash MCP and their personal agent, like, we'll simulate that whole service. Fill it with a mix of real, when it's available, and synthetic data. What it means is, like, we can run those simulations and show you, like, well, here's what that... If you change the model, here's what's going to happen as a result.
00:12:08 · Ruby Justice Thelot, Ben Hylak
[00:12:08] S1: Oh, I see. Okay. Okay. Is there any way that that helps beyond, like, a model change? Or can you Yes. Sort of, like, rank, you know, what would have been the best outcome, like, for the agent or a better decision that the agent would have made. [00:12:21] S0: Yeah. Totally. So it's, like, models, prompts, tools, like, any harness change, essentially, is something that we can we can simulate. [00:12:28] S1: Yeah. I'm curious your thoughts on, like, that problem of agents. Like, you want them to do something new, you want them to think out of the box. Like, the amount of times that I literally, in the prompt, contain and think about this, like, in a novel way or, you know, or I'm like, search all existing, like, libraries for such and such, like, niche or something. [00:12:49] S0: Yes. [00:12:49] S1: And then I... That's my way of, like, prompting some sort of out of the box, like, creative instinct in an agent. But, like, what are your broader thoughts about how this gets solved that it's not just the user prompting?
00:12:49 · Ruby Justice Thelot, Ben Hylak
[00:12:49] S1: And then I... That's my way of, like, prompting some sort of out of the box, like, creative instinct in an agent. But, like, what are your broader thoughts about how this gets solved that it's not just the user prompting? [00:13:03] S0: Well, I think that... I I do think a lot of the future will be about building better ways to sort of align agents to what humans want. Like, I don't know if that's going to be, like, through BCI, or if that's gonna be just better. I think I think there's a lot that the products themselves can do Mhmm. To align to people and understand you better. And I and... Yeah. I I I think that... I I really think that that that's that's the gap right now is, like, you know, you tell an agent what you want, and it just does something that's, like, completely different from what you wanted, and you realize that it's often a communication gap as well. [00:13:41] S1: Yeah. Yeah. Yeah. Yeah. You think that the... Like, right now, the ways that we have of agents, like, learning over time, adapting to your preferences is just memory? [00:13:50] S0: Yep. [00:13:50] S1: Is is there something in between or added that could, like, make that adaptation happen faster? [00:13:56] S0: Yeah. It feels like there will be. I I do think that, like, it's very obvious that that on things like tone, for example, it just really struggles to adapt to people. I do think that frontier models are actually worse at these things. [00:14:13] S1: Than earlier models?