00:00:00 · Khe Hy, Brett Caughran
[00:00:00] S0: So I gotta admit that I was excited when the government had banned Fable out of the gates. I think they were all things that that, like, Opus probably could have done with a lot of steering, but Fable pretty much one shot at them. [00:00:13] S1: Converting a 120 page operating manual on building a hedge fund style financial model is getting, like, quite a bit easier. Welcome to the next episode of Invest with AI, where we explore the intersection of fundamental investing and artificial intelligence. We have a short duo episode today where we'll do just a little bit around the horn vibe check on AI and investing. And one of the key topics people are discussing these days is the new launch of Fable, Kay. So what are the vibes on Fable by your assessment? [00:00:56] S0: Yeah, great to be here. Great to see you. So I gotta admit that I was excited when the government had banned Fable out of the gates, not because I believe in any paternalistic intervention of any sorts, but I was just like, Shoot, I'm not ready. And I think that, you know, with all the hype on these, like, big, expensive models, I gotta be honest with you, and I wonder how you approach this, is I don't have a great set of evals. And by evals, it is like, do I [00:01:27] S1: have
00:01:27 · Khe Hy
[00:01:27] S0: a complex task or series of tasks with a lot of good context that is hard and that prior models have not fully succeeded at? And so I don't have like a predetermined set of evals, but I do have one thing that I run pretty regularly with our, whenever a new model comes out. And that's basically I take, I have a giant folder of like everything my business has ever done. So you're talking like 200 granola calls, 50 decks, MCP into my email, different like surveys that people have filled out the recordings of all my trainings. And I dump that in and I say, build me a website, create a strategy document for me, create a presentation from scratch just based on like all of these materials. And I gotta say the result was pretty good. Like, the website was beautiful. The deck was pretty good. But most importantly, the strategy doc, because that's where, you know, I'm intimately connected with my business. Like, know it better than anyone else. So, like, if someone's gonna give me a unique insight, I can... I'm like, that's a unique insight. And I'll point here, and again, this is not the most robust eval, but what I'll say is I had a call, a prospecting call, with someone that was not even really a prospect, just kind of like half networking, half biz dev. But they said something to me that I was like, oh, that's very that's like, I feel seen. Like, you saw something that I didn't see in my own business. And it's one of those things that's, like, a little uncomfortable where I know I should pursue it, but I'm not pursuing it. And there's, like, reasons why I'm not pursuing it. And it was just one, like, one minute inside probably ten thousand minutes of phone calls. And it jumped right on that one minute, and it's like, you should be pushing so much harder on this. And that was... Again, I don't wanna overanalyze this, but I had forgotten that that was a very salient point when they made it, and it was once. It didn't happen five times. It happened once. And the fact that it read through ten thousand minutes of calls and was like, this is the thing you need to focus on, was pretty shocking to me. So that's what I'll say on the like formal eval side of things or the closest thing I have at formal. I had to build a lot of things from scratch. So I had to build like a tool that any client of mine can describe themselves in a multistep process.
00:04:17 · Khe Hy
Like, I'm a fundamental investor. This is what I look at. This is my investment horizon. And then it generates, like, 20 skills that they can then copy and paste into Claude. And I had it use all the skills that I had ever created for any of my clients, but also comb, like, YouTube and GitHub and, like, any public repository. So that was pretty cool. I think they were all things that that, like, Opus probably could have done with a lot of steering, but Fable pretty much one shotted them. Now I can't tell you about the price because I'm on the subsidy still, so, like, it's hard to know, like, was that, like, $500 worth of credits or 30? I I don't know. Probably closer to 500 than 30 if I had to guess. Was it worth $500? Probably not. Was it worth 30? Abs... So freaking lutely.
00:05:13 · Brett Caughran, Khe Hy
[00:05:13] S1: Very, very interesting. I agree so far that like the eval is hard. Like eval is hard. I have a few that I still in the pipeline to run, like simple things like the level two financial modeling tasks that are just complex and heretofore the models haven't been able to accomplish. So I'll sort of get to those. I've been traveling a little bit. But I was running one where I was trying to turn the process to identify signals around good times to short an individual stock, and Fable did an amazing job of sort of going back into the history of good times to short that stock and created a really coherent pattern recognition engine that historically I would have had to steer quite a bit. It would have taken me two or three iterations to say, Hey, we're not worried about a goodwill impairment or GAAP earnings. These are sort of the things that ultimately are triggers to stocks, and I may have to feed that in transcripts, etcetera, or my own research. And it one shotted it, was a little bit scary, sort of catching that same fundamental dynamic that you just mentioned, like materiality or taste, like what's the right word for it? [00:06:40] S0: Yeah, it's like incisive judgment. Yeah. Like people in haystack stuff.
00:06:46 · Brett Caughran
[00:06:46] S1: Yeah, yeah, which is, it's wild. Like I guess we would need to bring some experts in to explain exactly how they cook that in. But I am launching a new eval, which is a real life eval. I'm taking 100 companies and I'm running 100 earnings previews as earnings season starts next week. And I'm going to basically give a fable two tasks: forecast revenue, EBITDA and EPS accurately and accurately forecast whether the stock is going to trade up or down on the earnings print. So that's a very tight sandbox of judgment. And there's hiding from that. I'm going to keep maybe we'll talk about that, the results once earnings season is over. I'll put some of that on Twitter, obviously. That's great. But I thought that'd be a fun I was inspired, I think the talk in my network for last week was this Thinking Machines Bridgewater report where the researchers took a large language model, had native judgment and financial tasks in the high 40s, overlaid financial sort of a judgment layer, financial expertise, and judgment tasks raised up to 75, like the mid-70s. That wasn't the most academically rigorous report, but I think directionally was inspiring to a lot of people thinking about this combination of human judgment with large language model judgment, that the combination could drive better results. And so I've taken a lot of time to codify into my earnings preview process the frameworks of judgment that lead to my estimation of whether a stop will trade up or down, things like setups and expectations and buy side whisper and business momentum and guidance trajectory, etcetera. And so I'm excited to unleash that with the Fable model in this exercise. I'd be shocked if it does really well, but I want to just at least kick this off because then the iteration into version 2.0 and feeding that back, I think, is a fun dataset to start building.
00:09:09 · Khe Hy, Brett Caughran
[00:09:09] S0: Can I ask, so I hadn't seen that paper? When you say they're adding the judgment, are they fine tuning the model with that? Or is it in a skill or in a prompt? How is that judgment being incorporated into [00:09:25] S1: the process? I don't know the specifics. I read it sort of relatively quickly, and maybe we need to try and get one of the authors in. That would be a fun conversation. But I think the conclusion was they took this sort of codified analytical judgment of the Bridgewater team and overlaid that on the large language model approach. And the combination was valuable. And sort of in a very few way, I've started to see that, right? When I started to see, okay, take this approach, almost train it how to think, codify in some of the setup patterns that I've traded in the past. It's a quick study. Like the models have been a quick study on those patterns and following on the rails of those that pattern recognition power, there's been a few instances where I'm like, that was like nailed it pretty accurately. Not in a scientific way, hence I want to start sort of doing this in a broader subset to see if did I just get lucky and the heads flip, you know, head flipped on the coin twice in a row. So I'm inferring skill, which is why my plan to do this in a more systematic way in the earnings prints coming up starting next week.