00:06:46 · Brett Caughran
[00:06:46] S1: Yeah, yeah, which is, it's wild. Like I guess we would need to bring some experts in to explain exactly how they cook that in. But I am launching a new eval, which is a real life eval. I'm taking 100 companies and I'm running 100 earnings previews as earnings season starts next week. And I'm going to basically give a fable two tasks: forecast revenue, EBITDA and EPS accurately and accurately forecast whether the stock is going to trade up or down on the earnings print. So that's a very tight sandbox of judgment. And there's hiding from that. I'm going to keep maybe we'll talk about that, the results once earnings season is over. I'll put some of that on Twitter, obviously. That's great. But I thought that'd be a fun I was inspired, I think the talk in my network for last week was this Thinking Machines Bridgewater report where the researchers took a large language model, had native judgment and financial tasks in the high 40s, overlaid financial sort of a judgment layer, financial expertise, and judgment tasks raised up to 75, like the mid-70s. That wasn't the most academically rigorous report, but I think directionally was inspiring to a lot of people thinking about this combination of human judgment with large language model judgment, that the combination could drive better results. And so I've taken a lot of time to codify into my earnings preview process the frameworks of judgment that lead to my estimation of whether a stop will trade up or down, things like setups and expectations and buy side whisper and business momentum and guidance trajectory, etcetera. And so I'm excited to unleash that with the Fable model in this exercise. I'd be shocked if it does really well, but I want to just at least kick this off because then the iteration into version 2.0 and feeding that back, I think, is a fun dataset to start building.
00:09:09 · Khe Hy, Brett Caughran
[00:09:09] S0: Can I ask, so I hadn't seen that paper? When you say they're adding the judgment, are they fine tuning the model with that? Or is it in a skill or in a prompt? How is that judgment being incorporated into [00:09:25] S1: the process? I don't know the specifics. I read it sort of relatively quickly, and maybe we need to try and get one of the authors in. That would be a fun conversation. But I think the conclusion was they took this sort of codified analytical judgment of the Bridgewater team and overlaid that on the large language model approach. And the combination was valuable. And sort of in a very few way, I've started to see that, right? When I started to see, okay, take this approach, almost train it how to think, codify in some of the setup patterns that I've traded in the past. It's a quick study. Like the models have been a quick study on those patterns and following on the rails of those that pattern recognition power, there's been a few instances where I'm like, that was like nailed it pretty accurately. Not in a scientific way, hence I want to start sort of doing this in a broader subset to see if did I just get lucky and the heads flip, you know, head flipped on the coin twice in a row. So I'm inferring skill, which is why my plan to do this in a more systematic way in the earnings prints coming up starting next week.
00:10:48 · Khe Hy, Brett Caughran
[00:10:48] S0: That's cool. One thing I'll flag, I know we advertise this podcast as not having the answers, but us just talking through what's on our radars. And someone who knows a lot more about LLMs than I do had mentioned something called, I think it's called braintrust. Dev, and it's a platform for running evals. And he had actually suggested that I do an exercise kind of like the one you're describing. I'm not close enough to fundamental equity and investing to do that, but he had suggested that I do that using this platform. So just putting it on your radar to maybe check it out that there's actually like a structured environment to run these evals, and we can run that. [00:11:34] S1: Oh, that's interesting. That's interesting. Yeah, that's one of the areas in our work where we're really, like, I've evals a lot on vibes, right? Like, you know, you try, you shoot off a few workflows, you're like, ah, this looks good. But the problem is, even when you shot off five things in Opus 4.8, like you kind of, like sometimes it works, sometimes it doesn't. So it's very hard to scientifically measure vibes. One of the things we're trying to do with clients is create a more scientific eval set of the various MCP inputs, right? So who are the modeling MCP vendors? Let's create an eval set so we can more scientifically score some of those vendors on accuracy and capability. So I'd say that's like a work in progress on our side. But yeah, good call out on the brain. Trust our Devil, take a look at that.
00:12:29 · Khe Hy
[00:12:29] S0: Cool. On my end, I'm a little late to the game, but it's something that I'm having a lot of fun with. So we're both Karpathy, Andre Karpathy fanboys. And probably, you know, like he drops a banger of a tweet every six weeks, and the AI world just, like, realigns around this new idea. So maybe two or three tweet cycles ago, so that might be, you know, eighteen weeks ago, he had something, he called it kind of this, like, self updating wiki, like a knowledge wiki. And I kind of didn't have time to focus on it when it came out. And I'll circle back to the wiki with a very common problem that many of my clients are having right now. And that problem is they've gotten really good at co work. Some are using code, you know. And so now they've got good prompts. They've got good skills, all that stuff. They've connected their MCPs. But let's say they've got an investment, and in case, like private equity fund, they have an investment, and there's 500 documents in that folder. Let's say that's, you know, the investment is, I don't know, Sweetgreen, right? So they have this Sweetgreen, assuming it was private. They point it to the Sweetgreen folder. They've got 500 documents in that folder, and it's not even organized and so on. And so if they wanna ask questions about it, they could be very tactical questions like, you know, how has the store same sales growth changed over quarter over quarter? Or they could be more qualitative questions like, how should this thesis, how is this thesis being informed by inflation, right? Tariffs, right? So you point at CORT can't go through 500 files, right? And so what it does is it kind of cherry picks it. And what it really does is it does, like, basically, like, lot of, like command F. So like, let's say you're doing tariffs. It's gonna do like command F on your 500 files and be like, tariff, command F, inflation, command F. And then it's gonna grab 17 files, pull them into context, and then try to answer your question. Question. And it starts to break down more and more as you have more and more documents, more and more MCP. MCP breaks down, too, because the context windows are too small when you're dealing with, like, huge volumes of data.
00:14:50 · Khe Hy
So that's the problem that people are starting to have, And it's even more complicated by the fact that there's PDFs, there's Excel files, there's, you know, 200 page, you know, credit agreements and so on. So here's where the Carpathi wiki comes in, is that the way this works is it says we're not gonna we're going to create a map of your information, an ontology of your information. You might hear that word, people throwing that word around. And so what it does is it basically, let's say, it takes the 500 files, and one at a time, well, first it will convert it into markdown, and then it will basically read it. And as it reads it, it kind of creates a summary at the top with tags and metadata. And then it creates links of ideas inside the text that it just read. So if you have something like, I don't know, tariffs, right, it will just, like, find all the references to tariffs. Then you ingest the next file, and it does the same thing, ex... With one exception. As it finds tariffs, it then looks in all the other files and says, are there any references to tariffs in the other files? And then they start to link. And you kind of have this kind of like, you could think of it as kind of like a private worldwide web, right? Like a little wiki, right? And so what's cool about this is that the LLM is doing the ingestion, so like taking the file, creating the links. And then as you get new information, it just keeps updating it. So there's this like self improving process. Now, I'm gonna be honest with you, like, again, it's another one of these things, like, hard to know if it's working versus the naive approach, which is, you know, just point it at 500 with Fable. But that being said, I I know that some clients have done this, like, you know, maybe, like, a concentrated... Like, a PE fund that's, like, 15 investments. They're like, we're gonna go through this process for our 15 investments. We're not gonna look at the 200 we passed on, but we'll do the 15. We understand it well. Every analyst can kinda own their name in the wiki. It's not like your style of investing where the turn... You know, you're talking six year investment period, so you're not flipping names, and there's not data you're responding to every second. It's a little bit different.
00:17:16 · Khe Hy, Brett Caughran
But what you start to have is this knowledge graph that is much easier for an LLM to traverse without blowing a ton of tokens and by increasing the depth of accuracy. So early days, early days in my own exploration on it. I do have a few clients that, like, I don't wanna say, like, I brought it to them. Like, they kind of came up with it. We, like, they came up with it themselves and we compared notes. You know, a lot of this is around this idea of, like, if you can get all your files neatly organized in Markdown with good naming, good metadata inside, like, a nice, clean folder structure that the LLM can go in and read it much better. And so you're starting to see... And again, I encourage everyone or I discourage anyone from doing this, oh, we're gonna do twenty years of data across this. Like, no, no. Pick, like, five names and start. Pick one name and start. And so that's starting to happen, and and I'll be able to report back on the progress. But I I have seen some green shoots that, again, this is more in private equity, but it works. [00:18:24] S1: That's interesting. And I've heard more about the same concept of just making your file system more legible to the agents. How are clients doing that? Is that a skill that coworker or codex can go in and organize? Like that even gives me hives when I start to think about letting codex rip on my internal files to make them legible, or people just pressing the button and hoping for the best?