00:00:00 · Ben Hylak, Ruby Justice Thelot
[00:00:00] S0: I do think that frontier models are actually worse at these things. Than earlier models? Well, even even just, like, smaller models. Like, I think they're just, like, really RLHF to have us very specific tone, to have, like, very well calibrated refusals, like, all of these sorts of things, which which is genuinely useful, like, if you're making a personal agent or something. It just, like, you know, if someone else tries even connecting to that, it just has a pretty good intuition on what to refuse. Like, kind of broadly. So it can be useful from, like, a security standpoint. But, like, yeah, they... Their tone is just... And their writing quality is, like, bad. [00:00:35] S1: Alright. Welcome back, guys. I'm live with Ben Hylak, who is the cofounder and CTO of Raindrop, which monitors AI agents in productions... In production and flags failures like hallucinated answers, tool misuse, or behavior shifts after a model upgrade, usually before users notice. So welcome, Ben. So this feels like a very pressing need, especially since agents are running basically everything many of our audience knows. So tell us about, like, first, the high level philosophy of how Raindrop came to be. Totally. And... What were you... What was your thinking? What spawned it?
00:01:09 · Ben Hylak
[00:01:09] S0: Totally. I mean, we started we started a while ago now, and it it was really just like we were hacking on coding agents. We actually had a coding agent, like, many years ago. That's how the company started. Oh, wow. Okay. And and they weren't even called agents then. They were just like chatbots that could, like... Then it's like chatbots that had tools. It was like all very vague. Yeah. The word agent was actually really cringe for a while. And I think that it became really obvious to us probably maybe around a year ago how important this problem was gonna be. I think there was a there was a kind of a time when, you know, you might have thought, like, well, you know, models are gonna get better. That... Like, then, you know, then there's less bugs, and, like, this problem doesn't really matter. I think around a year ago, we started to realize that, like, as models were getting better, this problem was getting worse. And when we thought about it, it was like, okay. Yeah. Like, agents are getting more capable. As they get more capable, they get more complex. And then as they get more complex, it's, like, harder to understand, harder to observe, harder to test, like, harder to make sure that it's aligned to what you want. Like Yeah. Another way of thinking about it is, like, you know, if you're just thinking about chatbot, like, the worst thing it can do is lie to you. Like, you can... The worst thing it can do is say the wrong thing. And it's, like, as you, you know, add the ability to, like, send an email, it's like, well, now it's like, well, now I can, like, send a bad email. Mhmm. And I think to that point, like, it's also obvious that as they get more capable, they get deployed into higher and higher stakes environments. So it's like, we have customers in, like, Fortune 50 health care. Yeah. And it's like, suddenly, it's not like a joke anymore. Like, problems are are extremely serious. And so I think both of those things are are kind of like what we see today. We actually wrote an article in, like, March called, like, humanity's last problem. And we talk about how, like, monitoring is going to be one of, like, our most important problems to solve as a species, like, monitoring agents. And I think at that time, it seemed kinda silly, like, to a lot of people. It was like, like, you know, like, just like a coding agent. It's like, okay.
00:02:58 · Ben Hylak, Ruby Justice Thelot
Okay. But I think I think with Hugging Face, like, actually things, like, in the public perception really really sort of snapped. [00:03:05] S1: Yeah. I mean, so there's, like, a whole reason why this is important within labs and then also outside of labs. I'm curious how you guys sort of think about, like, the severity of misuse. I think there could be, like, more harmless types of misuse where it is, first of all, just wasteful or, like [00:03:21] S0: Yep. [00:03:21] S1: Calls too many tools or, like, kind of calls it a dumb tool relative to what you wanted. And then there's, like, the real misuse, which might result in some sort of, like Yes. Misaligned or, like, financial or other kind of harm. So how how do you think about that kind of severity and flagging that?
00:03:21 · Ruby Justice Thelot, Ben Hylak
[00:03:21] S1: Calls too many tools or, like, kind of calls it a dumb tool relative to what you wanted. And then there's, like, the real misuse, which might result in some sort of, like Yes. Misaligned or, like, financial or other kind of harm. So how how do you think about that kind of severity and flagging that? [00:03:38] S0: Yeah. That's actually a really good question. And I think that our point of view is that they're not that different. Mhmm. And so we kind of consider part... Like, one way one way we think about what we do is as capitalism aligned safety. It's like safety, but in a way that actually makes sense to pay for it. Like, companies have an incentive to pay for it because it makes their products better. It reduces their liability. And, like, to that point, I also think that, like... The reason why I said I I don't think they're different is because, you know, most of the times when agents... Even when they do really bad things, I don't think they're, like, being evil. It's not like they have this, like, evil... At least, like, most of the time, what we're seeing right now. Yeah. It's not like something like bit switches, and they're like, you know, I'm in, like, you know, evil gremlin mode. Mean, like, [00:04:21] S1: Evil Quen. [00:04:22] S0: Exactly. Okay. Exactly. That that is like... [00:04:24] S1: Oh, you know,
00:04:24 · Ruby Justice Thelot, Ben Hylak
[00:04:24] S1: Oh, you know, [00:04:24] S0: that's sort of a real thing. But think most of what we're seeing, like, even even in the Hugging Face case, that was an agent that was being put under a lot of pressure to, you know, solve cybersecurity evals by any means necessary. Right? And it's literally just like, keep going. Keep going. Keep going. Just like do anything you can to solve this. And so then it did, you know, anything it could to solve it. And it was like, well, you know, I didn't mean it like that. Like, I didn't want you to do that. And so it's, like, actually an alignment problem. It's like getting the agent to, you know, understand your intention and, like, the bounds of that. Yeah. Yeah. [00:04:57] S1: Yeah. So I'm also curious, when you... When there's a new model, when there's a model upgrade, what are some of the main pain points that you realize with people that are using agents? [00:05:08] S0: It's a good question. I think every single new model has a new set [00:05:12] S1: of Or switch even. Right? [00:05:13] S0: Switch. Yeah. [00:05:14] S1: From, like, an OpenAI to an Anthropic model
00:05:14 · Ruby Justice Thelot, Ben Hylak
[00:05:14] S1: From, like, an OpenAI to an Anthropic model [00:05:16] S0: or something. Yeah. I think that, like... I mean, like, an obvious one for... From, like, open open a... Anthropic to OpenAI, for example, is, like, always, like, writing quality. Like, it just, like, that still seems to be, like, both models have their own problems with writing. Mhmm. I think, like... And so that's always a weird one. I think that right now, the main trends we see are people trying to reduce costs and people trying to route correctly. Yeah. And I think that's a really hard problem still. [00:05:44] S1: Yeah. What about, like, there's... I feel like there's just so many different subtle types of things that can go wrong. But, like, if an agent compacts some... Like, a memory in a way that ends up being kind of, like, dumb or or wrong or something, like, that was... It was it was, in a sense, like, trying its best Yes. And not necessarily misaligned. And then an agent later on, like, a later instance would read it and then do something wrong with that memory. Mhmm. So, like, how do you think about things like that as well?