Latent Space: The AI Engineer Podcast
01:30:53 · Alessio Fanelli
[01:30:53] S4: I think it's always cool to see these trajectories and actual example and environment walk through it. How do you translate this to the RL side? So one is the rubric of what you pull. Two is there's a lot of work here that's done for thirty minutes that is correct. In this case, there's one error, sometimes there's multiple. How do you transition that out into useful reward signal because you did a lot of inference and a lot of it's correct? Maybe start off with the rubric of how this is graded and answer as you
01:31:23 · Colin
[01:31:23] S2: Yeah. So creating the rubric is by far the most challenging part of, you know, these semi verifiable domains. Generally, it's very easy or or or straightforward to say, like, this output is a 0%. You know, they return absolutely nothing like a blank page or fail... Okay. Under... Understood. Or this output is a 100%. Like, oh, they they nailed it perfectly, and I, you know, I would... I wouldn't change anything. This is perfect. What's really hard is every step in between, particularly when there's there's multiple variables at play. Right? Maybe maybe model a is better at, you know, the the cash flow build down, but model b is better at the balance sheet. Well, how do you how do you how do you balance those two things? Right? And so we spend a lot of time building these rubrics with experts, and the goal is to make them as deterministic as possible. Right? So we we intentionally create the environment, the inputs, to be... You know, there's only one right answer. Right? And, you know, there may be conventions in the economy or the, you know, the industry where there there could be two defensible or reasonable, you know, approaches, but we just... We... You have to make absolutely sure in the environment that, okay. There are two reasonable approaches, but actually this note from Sahil says, you know, use this approach. Right? So then we can we can grade it on the rubric. So that's how we that's how we approach the rubrics. Generally, the rubrics are about 50% just hard checks. So this is like a simplification, but, you know, half of the rubric is is the numbers. Like, is, you know, is is the year one EBITDA right? Is the year two cash balance right? And we think that's reflective, and that is reflective of the industry standards. Like, you could argue it's even higher.
01:33:09 · Alessio Fanelli
[01:33:09] S4: So at a high level, it's like the key steps of solving a task. There's some binary answer. You should have step one to five. You should have this. That's this binary. Yes or no?
01:33:18 · Colin
[01:33:18] S2: Yeah. So... Yeah. So we we grade the actual numerical outputs for, you know, half half of the the points or criteria. And and in general, in in our benchmark, there's, I think, on on average, a 146 different criteria. And then the remainder of the the points are allocated across construction, so not necessarily, you know, the numbers, but but really drilling down into the process. And so this this is the gate that that we've been talking about. And this is, you know, I would also consider a construction output because, you know, the step up is built as purchase price plus estimated working capital liabilities, less tax basis and current assets, less tax basis, appreciable assets, and recovered straight line over fifteen years. This is... This was discussed and shown in the environment multiple ways. You know, we knew that the agent looked at the source material, and this is, you know, fair to grade it on. And it got it wrong, again, because, you know, something in, you know, it's it's weights to to to looked at what it was calculating and forgot, you know, that this is a special exception and and kind of maybe reverted to its treatment of goodwill for other scenarios. But that's how we think about the rubric, and we can use the rubric. And so sorry, this gate shuts off everything else in this section. It's just like a VP would. Right? If I... You know, if I'm showing... If I'm going through a model, you know, with my managing banker or whatever and, you know, Row 9 is wrong, like, they, you know, they check everything else, but once they get to Row 9, that's wrong. You know, there's a bus in your model or there's a gremlin or something, You know, they circle it and send it back. Right? They don't just keep... They don't keep checking because number one, they've lost trust in you because they, you know, they see this error, and it's clearly... You you didn't get this right, so why are you gonna get everything else? Mhmm. Right? But also because these errors compound. Again, this is, you know, this is over 2,000 interconnected formulas. And so if there's one error, that error pollutes every other number in it. So... And because it's realistic, we shut off the gate for every... You know, all 180 points in the section dealing with the tax mechanics.
01:35:34 · Colin
And so we can use this rubric, which again has a lot of specificity on on the numbers, and you can you can actually just logically go through. Well, it actually got, you know, years one, two, and three correct for these numbers. It got years four wrong. So now you can you can direct your attention to what happened between year three and year year four that there's there's an error there. Right? And so we can use we can use the rubric, and we use the trajectory to come up with kind of the the failure modes and understand which models are strong and weak in in various types of tasks. The three biggest issues that we saw across models for this benchmark were financial logic where it was unable to unable to, you know, exercise this kind of expertise, in context with the environment as well as scope and completeness. So it might it might have gotten through, you know, steps one through nine of... But it left step 10 unfinished. Right? So it's so it's an unfinished model. So, yeah, financial logic still... You know, we... You have to make the tasks more complicated, but it's still failing these, you know, pretty consistently across every every model family.
01:36:43 · Sahil
[01:36:43] S0: And I think for... On rubrics specifically. Right? Like you... The shape of this task can be used for evaluation. It could also be used for RL. And so how you think about the score probably differs. Right? To your point, if the model's getting a lot of the things right, I don't wanna give it no... As you know, reward at all during an RL run. But for the sake of an eval Yep. Where you're like, can I actually use this thing in the real world, like, at a bank where a banker would trust it, that has a different sort of threshold, right, on what you need? And so one thing the team did, which I which I think is really cool, is just separating, you know, what's truly critical for real world usage where if you're you gonna evaluate it, like what would just make this totally untenable, like that tax treatment example, like that's a critical component that would just lose you trust immediately in a meeting with your your client or with your VP. But, you know, broking it out in a way where when you're doing an RL run, you can sort of still have, you know, enough granularity in the reward to say, hey. This still scores like... You know, if you look at the thing, it's 30 points. Right? And maybe this whole section's, what, a 180 points out of... How many... What's the total? 1,000 something? Fourteen sixty. Right? So, like, you can still give assignment accordingly for RL, And obviously that that helps. But I think on the evaluation side, what's what's really important is... I think I was kinda kinda find this funny. Labs will often be like, oh, like our model scores 79% on this Not bad. On this benchmark. Yeah. But not interesting anymore. It's like, you know, imagine you just had an employee who just kept getting everything 79% over the line and it was like, cool, I'm done. Never gonna look at it again. It's like this unusable.