Latent Space: The AI Engineer Podcast
01:38:29 · Alessio Fanelli
[01:38:29] S4: I think I think there's a nuance there. So to push back as much as you're giving a harsh, there's no value in it, it's still eval signal for how to build the system around it. Right? So maybe you have a human in the loop. Maybe there's a guardrail. Maybe it solves the first half and this is where you define, okay, the system we have today only works up until this point and the humans take over the rest. Right? So there's some value and it's good to know, but yeah, you you don't wanna full send this system autonomously. Right?
01:39:00 · Sahil
[01:39:00] S0: I think it depends on the perspective, right? I think it's totally correct. But for this benchmark, for example, we interviewed hundreds of bankers. And one question we asked that thought was really interesting is, would you rather start from this output Yeah. Or not? And the vast majority said that they would have rather started from scratch. And so this... I said, like, sit with that for a second. Right? Like, the vast majority said they would have rather started from scratch. I think you're totally right from the lab perspective. Right? Having the eval there to be like, hey. You know, we gotta get better at surfacing these tax treatments or whatever whatever ways those manifest. There's tons of those examples. Right? It can certainly help you direct where do I need to improve steerability of the model, where do I need to improve... You know, where might I need more training data, where might I need... You know, you can figure out the solutions and solve for those. But if you're looking at it from the perspective of, like, how do I get this used in a bank? And you're talking a lot more about interpretability and, like, this human agent interaction that starts to matter a lot more because in a in a profession like banking where things are such high consequence and small mistakes like this can have, like, incredible, you know, downstream effects, yeah, you just... It can't be 79%. You know? Fair. And if it is, it has to be able to surface and interact with the human into... In in that way. And so if I answer things, it's really interesting example where it feels like so many things are verifiable, but the bar for, you know, what you have to hit is so high that the the paradigm between how humans and agents interact has to has to be really thought through.
01:40:42 · Shawn Wang, Alessio Fanelli
[01:40:42] S5: Yeah. Well, you know, I think the sort of number representation to get the message across the researchers is something you can play with. Right? Like, you can just define, like, well, this is the critical thing. Right? The the rest is, like, sort of the normal checks that, like, generalize across all the cases. But this... For this example task, this was the central emphasis anyway. So maybe maybe just like the weighting of the numbers needs to shift over time. Would... Should we should we broaden out or it was... Was there more in this... Was there less? [01:41:12] S4: I actually have a question to take this to make it more broad. You know, you've shown us the environment, the task, how complex it is. How do you come up with 93 checkpoints, 1,400 points across these categories? Penalty traps, I think that's pretty intuitive. These are hard blockers, dependency gate as well. But what goes into that rubric, that nuance? How do you kind of split across this? Is there a higher level learning that we can take away? So outside of this example, making good rubrics for environments.
01:41:46 · Colin, Shawn Wang
[01:41:46] S2: Yeah. So again, a big emphasis is making that as deterministic as possible. So our key kind of anchor is how do we architect the task and the environment so that it is is gradable on a rubric? It's a valid criteria. The the sections are created by our experts with multiple layers of review to kind of hit the key the key pieces, as well as, you know, there's that there's that tail that is, you know, delivery and formatting, but the meat of it is the actual analysis. So that's... Every one of these... We did, again, 13 different environments with probably 20 different task types, and they could be, you know, CLO portfolio modeling, data center development, you know, kind of basic m and a tasks, LBOs, like this stuff. So the the different sections are are different... Or sorry. The the the breakout of the rubric... Rubrics are different depending on the the specific prompt and inputs and the environment, but, you know, that's basically how it was determined is literally just, you know, our experts, what pieces are are critical that that move the the analysis. [01:43:03] S5: Yeah. Great. Is there... Is there a section... There's a page 12. I don't know if we covered everything. [01:43:08] S2: Oh, this is... This kind of goes into the, like, the actual scoring. [01:43:12] S5: Oh, there's other things that it got wrong. Okay.
01:43:12 · Shawn Wang, Colin, Alessio Fanelli, Sahil
[01:43:12] S5: Oh, there's other things that it got wrong. Okay. [01:43:14] S2: Yep. So it also got wrong. It it wired up its revenue build a little incorrectly. So that's a that's a big piece of the valuation. But we we saw... You know, again, like, we we still see errors where it will just... It will reference the wrong cell. You know? It... It'll reference the tax rate instead of the growth rate or something. [01:43:35] S4: There's nuances too. Right? Like, it can get steps incorrect. And even towards the end when it rechecks at work... Its work, it may find these errors, which is pretty common in enchanted loops. Right? Like, before you output, check your work. Oh, I found out I messed up step 12, and then continue, and this this checks for all this. [01:43:52] S2: Definitely. [01:43:52] S0: Yeah. I like that you [01:43:53] S5: have a 1% tolerance. That's pretty good. That's like... Yeah. No. There's gonna be rounding errors. You know? [01:43:57] S2: Yeah. Yeah. I mean, we... Again, we try to make this as, you know, deterministic as so there's only really one right answer Yeah. With, you know, everything that it was given. [01:44:05] S4: Yeah. And it's not only you have a 1% tolerance for those that can't see the screen, it's 1.27% off with a 1% tolerance, which gives it a zero out of 30 on her word. So strict. Strict. [01:44:16] S2: Mhmm.
01:44:16 · Colin, Shawn Wang
[01:44:16] S2: Mhmm. [01:44:18] S5: Okay. I I don't know if we covered semi verifiable rewards enough. So the... Basically, this is what I would call, like, a model grader type thing. What model do you run, or what do the... What model do the bank... Do the labs run to grade the other models? [01:44:34] S2: So we we built a grader called Gandalf. I think in this Yeah. [01:44:38] S5: You submitted a paper.