Latent Space: The AI Engineer Podcast
01:44:40 · Colin
[01:44:40] S2: Yeah. Yeah. So NeurIPS. And this one, I think we're using Opus 4.8 just because I think that was what was available when we started building this. And, you know, what we found in the the... Again, one of the challenges with these rubrics is you you start creating these, like, super complex tasks, and then someone's gotta verify them and check them, right? And that can take a lot. I can tell you from personal experience that trying to go through one of these models and tie out, I know, 146 rubric criteria is extremely challenging, right? Time consuming, exhausting. So if you use, you know, verifiers, agentic verifiers to grade, what we found with, you know, pre Gandalf was that they they were unreliable. You know, you you have this you have this Excel spreadsheet with, again, you know, 2,600 formulas, and you're asking it to check a binary. Right? Like, net income equals a 100,000,000 in, you know, 2026. And it would it would sometimes, you know, falsely, you know, meet a criteria because it found that number, but it didn't necessarily, you know, understand, you know, where it was looking in the in the model. So, you know, our research team spent a lot of time building this grader to give it, you know, more context from what I understand with with the environment and basically the inputs that were, you know, feeding these tasks so the grader had better context to understand, no. I'm looking at an LBO model when I say, you know, net income. It should be here, and this is the year I'm looking at, which means this is the column I'm looking at. And we were able to get a lot of improvement. So the, you know, the verifier, I think, was, you know, over 90% in agreement with humans and able to cache things that sometime... You know, we... We'll run this. We'll we'll have two humans grade the same rubrics. Right? And it's not it's not one to one. You'd hope it is, but people make mistakes. Right? So having that verifier there to also check their work and check each other's work, you know, helped us, you know, better understand how this stuff can be deployed in the real world.
01:46:40 · Sahil, Shawn Wang
[01:46:40] S0: It's worth actually talking about that because so much of when you build that grader informs you about, you know, where in the rubric that you've constructed do you need to be more precise or more clear? You know, I think one example that we had in building one of these tasks was, you know, the model or one of the rubric items was, you know, net income is a 100 or something. Right? And what the agent did was it basically just took a tab in Excel, you know, made a new tab in Excel and just literally had the number like, you know, or just like listed every number from, like, one to, like, a million. Right? And so the number a 100 was present, and the greater was just looking for the number a 100 somewhere and found it. Right? And so that taught us a lot. You know, there's a lot of problems, right, in that, but it taught us a lot about the task construction and the task design, taught us about, like, you know, where... How do we need to be more precise with the rubric or, you know, such... You know, in such a way where the greater could, in the spirit of the task, make sure it's aligned with what we want. [01:47:45] S5: There's a lot of little So bit you told me about the reward hack where it just spammed out a [01:47:48] S0: Exactly. Numbers. [01:47:49] S5: What what was that?
01:47:49 · Shawn Wang, Sahil
[01:47:49] S5: What what was that? [01:47:50] S0: Yes. It was it this. Right? It was, you know, a lot of... You know, the the rubric we'd constructed was, you know, look for originally. Right? So it's like early days, but, you know, look for a net income number of a 100. Right? And that was like the word or the, like, specific rubric criteria. So what the model did is it made a tab in the Excel sheet that just listed every number from like one to a million. And so then every number that was in the rubric was there. It found it. Right? In the in the cheap that obviously isn't the spirit of the task to to to full reward hack. And so what that taught us is, okay, we have to be more precise with the Rubrik criteria, or maybe we have to redesign the task a little bit where there's a little bit more of a template to work from in terms of the financial model, like the Excel model that the the agent is gonna build from to make it in such a way where if you have a greater and that rubric criteria and the net income is a 100, it's actually only looking for something that is where the net income is listed as a 100 and not just the number a 100 somewhere in the financial [01:48:53] S5: Yeah. Yeah. Fair enough. We're gonna wrap up soon because we do have a a full slate, but one thing we wanted to... We prepped before this interview was talking about where the future of finance and AI is going. You were asking me what I thought was incomplete. I I don't know. Do you have thoughts before I before I interject with mine?
01:48:53 · Shawn Wang, Colin
[01:48:53] S5: Yeah. Yeah. Fair enough. We're gonna wrap up soon because we do have a a full slate, but one thing we wanted to... We prepped before this interview was talking about where the future of finance and AI is going. You were asking me what I thought was incomplete. I I don't know. Do you have thoughts before I before I interject with mine? [01:49:15] S2: I think that we're just gonna have to keep building more realistic environments. Right? I think that it's it's unlikely that you're gonna see these companies based on conversations I've had change how they work on a deal. I mean, some of these companies have been around for three hundred years. Right? You're gonna change the culture of a company that's been dominating finance, you know, across six continents to conform to your new technology. Like, I'm just not buying it. Right? So the... Our industry really has to meet them where we can make an interface that is transparent and reliable and trustworthy so that there is zero friction in using it as much as possible. Because if it's if it's just limited to, you know, note taking or even just assembling key pieces, like, sure, that's helpful, but it's not like a transformative, you know, sea change or step change in in productivity. And so I think we're just gonna keep we're gonna keep pushing the frontier with Atlas, you know, in in other, you know, data companies on how we can make things more more realistic, more challenging so that the product that, you know, that this industry is producing is is more palatable, and so we get more deployment and integration.
01:50:23 · Sahil, Shawn Wang
[01:50:23] S0: Yeah. It's like kinda echo a lot of that. Right? Where the the bar for using this in the real world for real banks is very, very high in finance. And so one, how do you build that trust? One way or one sort of component to building that trust, hopefully, can be benchmarks like this where, you know, people, you know, can look into the tasks and be like, wow. Like, it's really cool that that models can do that or that agents can do that. And we're gonna have to make those more and more and more complex and reflective of what the actual production environment looks like for for these agents in a bank. And, again, kinda like what mentioned earlier, we think Atlas is is a great great step in the right direction. There's still a long way to go before we're actually simulating, like, a full bank the way that it looks like in the real world. Right? So I think that's gonna be just a huge continuous, you know, area of development for data. The the second place where I I think this is gonna have to play or is gonna be a lot of attention on in the next, you know, number of years is this interpretability component. Because finance is just such a high consequence industry that if you can't build... You know, if you can't inspect live what is actually happening, doesn't matter how well you perform on benchmarks. Like, I need to feel as a banker that there is like interpretability for me. [01:51:46] S5: Yeah. And then the models do output grounding in sources as we saw.
01:51:46 · Shawn Wang, Sahil
[01:51:46] S5: Yeah. And then the models do output grounding in sources as we saw. [01:51:50] S0: Yeah. But like, you know, in that tax example, right, just like drawing attention to that and knowing like when to draw attention to what I think is gonna be pretty important. And I think that's gonna, you know, call it alignment, call it interpretability, whatever it is. But I think that's gonna play a real, real role in finance. But, yeah, curious.