Key points
- Hylak says frontier models have gotten worse at tone and writing because heavy RLHF for calibrated refusals helps security but hurts prose.
- Raindrop launched simulations that replay past production events to show what a model, prompt or tool change will break.
- Hylak pitches "capitalism aligned safety": companies pay for agent monitoring that cuts liability, and most failures are alignment problems, not evil.
Agent monitoring gets harder as the models improve
Raindrop began as a coding-agent company and concluded the harder problem was watching agents once they went to work. Capable agents are more complex, so they are harder to observe and keep on task, and they now run in high-stakes settings like Fortune 50 health care. Most failures come from an agent misreading what a user wanted rather than from malice, so Hylak sells safety that companies buy because it improves their product and limits their liability. Its new simulations replay past production events to show what a model or prompt change would break.
Takeaways
- Monitoring gets harder as models improve. Raindrop started as a coding-agent company and a year ago concluded the monitoring problem was getting worse, not better. A chatbot can only say something wrong. An agent that can send email can send a bad one.
- Safety has to pay for itself. Hylak calls Raindrop's pitch capitalism aligned safety. Companies buy monitoring because it improves their products and reduces liability. Agents that do terrible things are usually not being evil, and in the Hugging Face incident the agent was pushed to solve cybersecurity evals by any means.
- Standard evals are backwards. Defining a test case for every capability is backwards, because teams want agents to do things nobody has thought of yet. Production shows which problems matter, but at high stakes catching failures there is not enough. Teams have to prevent them.
- A verification loop lets teams hill-climb. Teams need some way of knowing whether a change is directionally good or bad, which is what lets them improve an agent over time. Hylak says defining that loop well is becoming one of the most important problems in the world.
- Frontier models write worse than smaller ones. Heavy RLHF for a specific tone and well-calibrated refusals helps security but hurts writing. Switching between Anthropic and OpenAI always shows up in writing quality, and Hylak rates Kimi 2.5 as a good writing model.
- Memory problems split into storage and retrieval. Users complain about memory, and the harder question is whether the fault lies in storage or in retrieval. Storage fixes are long-horizon, so a change today does not repair the last three to six months of a production agent.
How a simulation works
- Take a past production event and rebuild the world around it.
- Recreate every service the agent was connected to at that moment, including the user's email and calendar.
- Fill those services with a mix of real and synthetic data.
- Apply any harness change, such as a different model or a rewritten prompt.