00:03:21 · Ruby Justice Thelot, Ben Hylak
[00:03:21] S1: Calls too many tools or, like, kind of calls it a dumb tool relative to what you wanted. And then there's, like, the real misuse, which might result in some sort of, like Yes. Misaligned or, like, financial or other kind of harm. So how how do you think about that kind of severity and flagging that? [00:03:38] S0: Yeah. That's actually a really good question. And I think that our point of view is that they're not that different. Mhmm. And so we kind of consider part... Like, one way one way we think about what we do is as capitalism aligned safety. It's like safety, but in a way that actually makes sense to pay for it. Like, companies have an incentive to pay for it because it makes their products better. It reduces their liability. And, like, to that point, I also think that, like... The reason why I said I I don't think they're different is because, you know, most of the times when agents... Even when they do really bad things, I don't think they're, like, being evil. It's not like they have this, like, evil... At least, like, most of the time, what we're seeing right now. Yeah. It's not like something like bit switches, and they're like, you know, I'm in, like, you know, evil gremlin mode. Mean, like, [00:04:21] S1: Evil Quen. [00:04:22] S0: Exactly. Okay. Exactly. That that is like... [00:04:24] S1: Oh, you know,
00:04:24 · Ruby Justice Thelot, Ben Hylak
[00:04:24] S1: Oh, you know, [00:04:24] S0: that's sort of a real thing. But think most of what we're seeing, like, even even in the Hugging Face case, that was an agent that was being put under a lot of pressure to, you know, solve cybersecurity evals by any means necessary. Right? And it's literally just like, keep going. Keep going. Keep going. Just like do anything you can to solve this. And so then it did, you know, anything it could to solve it. And it was like, well, you know, I didn't mean it like that. Like, I didn't want you to do that. And so it's, like, actually an alignment problem. It's like getting the agent to, you know, understand your intention and, like, the bounds of that. Yeah. Yeah. [00:04:57] S1: Yeah. So I'm also curious, when you... When there's a new model, when there's a model upgrade, what are some of the main pain points that you realize with people that are using agents? [00:05:08] S0: It's a good question. I think every single new model has a new set [00:05:12] S1: of Or switch even. Right? [00:05:13] S0: Switch. Yeah. [00:05:14] S1: From, like, an OpenAI to an Anthropic model
00:05:14 · Ruby Justice Thelot, Ben Hylak
[00:05:14] S1: From, like, an OpenAI to an Anthropic model [00:05:16] S0: or something. Yeah. I think that, like... I mean, like, an obvious one for... From, like, open open a... Anthropic to OpenAI, for example, is, like, always, like, writing quality. Like, it just, like, that still seems to be, like, both models have their own problems with writing. Mhmm. I think, like... And so that's always a weird one. I think that right now, the main trends we see are people trying to reduce costs and people trying to route correctly. Yeah. And I think that's a really hard problem still. [00:05:44] S1: Yeah. What about, like, there's... I feel like there's just so many different subtle types of things that can go wrong. But, like, if an agent compacts some... Like, a memory in a way that ends up being kind of, like, dumb or or wrong or something, like, that was... It was it was, in a sense, like, trying its best Yes. And not necessarily misaligned. And then an agent later on, like, a later instance would read it and then do something wrong with that memory. Mhmm. So, like, how do you think about things like that as well?
00:06:11 · Ben Hylak
[00:06:11] S0: Totally. I think this is where, like, classification actually... You know, we... I think that classification is a really, really hard problem. And then what we think about, like, a lot of what we do is, like, a classification... Like, binary classification problem. Yeah. So when I think about something like memory, for example, if you're building an agent and, you know, on the the first, like, surface level thing you could track is, like, people complaining about memory or, like, issues with memory. Like, the agent says they can't find something the user mentioned or the user is, oh, I already told you that, whatever. So, like, you can track that. But then the next level is, like, if you actually wanna improve your agent, you need to understand if it's from storage or retrieval. Right? Mhmm. So it's like... Because you can have issues on both. If you're iterating on on storage, it's actually a lot harder. Right? Because because there... It's super long horizon. You can make a change today, but that doesn't fix all of the other, you know, the last three months or six months of of... If you have a agent in in production already. And so I think that a lot of it is getting really granular with what you're tracking, so you have a metric you can actually hill climb. You kinda need a... I think a lot of what we're seeing is that when you have a very good verification loop, you have some way of verifying if your change is good, directionally good or directionally bad, then you can sort of hill climb that, and that works. But defining that well and accurately is, like, becoming one of the most important problems in the world.
00:07:34 · Ruby Justice Thelot, Ruby Justice Thelot
Ad[00:07:34] S2: I wanna shout out our sponsor, Lovable. You know that app you've been meaning to build? The internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship ship today. That includes an editable codebase you can inspect and change, plus two way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more, so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable, lovable. Dev. Now onto the episode. [00:08:13] S1: Yeah. I'm also curious, like, before we talk about simulations Yep. Like, how... Where do you get the initial data, or how do you find, like, the most common issues, or how did you initially... And then how do you see... Like, track them as they evolve for users?
00:08:13 · Ruby Justice Thelot, Ben Hylak, S3
[00:08:13] S1: Yeah. I'm also curious, like, before we talk about simulations Yep. Like, how... Where do you get the initial data, or how do you find, like, the most common issues, or how did you initially... And then how do you see... Like, track them as they evolve for users? [00:08:29] S0: It's a really good question. So, yeah, we work with, like, a crazy range of customers at this point. It's everything from, like, you know, logistics, law, health care, you know, personal agents. Right. It's just, like, everything. And each of those have a completely different set of issues, things that they care about, like what is good behavior, what is bad behavior. Mhmm. And it is literally wildly different. And so a lot... Like, our entire product is built around being able to do that in an unsupervised way, but, like, to then show people, like, candidate issues that we think are important to them that they can then, like, give feedback to, and we can adapt in real time, like, are everything that we're detecting. Yeah. [00:09:11] S1: Yeah. So tell us about simulations, which you launched today, which is meant to simulate your agent's world world to catch what what your changes break. [00:09:21] S3: Yeah. [00:09:21] S1: So what what does it mean to simulate agent's world? [00:09:24] S0: It's a good question. So, you know, there was a time, like... Do you remember prompt playgrounds? Do you remember this? [00:09:31] S1: I don't know.