00:38:47 · Brett Caughran, Kris Bennatti
[00:38:47] S1: Interesting. Long has that model been live for, and how do you track, like, efficacy? Is there any way to sort of distill that down into a batting average or hit rate or alpha, etcetera? [00:38:58] S0: Yeah. So it's been a while since we've done a backtest versus share price. But I believe there's about like a 14 differential in performance. So, underperforming by about 15% for high risk companies back But when we did we've been doing this since 2019. Very popular among actuarial teams at D and O insurers. So, we do a lot of tracking based on SCA securities class actions, which is the most important fraud indicator for them. So, high risk companies are three times more likely to be subject to an SCA of any kind and every company that has a score of 70 or higher has about a one in three chance of SEC enforcement. So, it's very high precision. Yeah. And, of course, the other two out of three also likely frauds just subject to SEC enforcement. [00:40:04] S1: Yeah. Yeah. Yeah. Conversation we had one time is just how sort of natively good LLMs have been at identifying risk. And I think the sort of conclusion was there's a lot of evidence in the training corpus to sort of pattern recognize. Am remembering that conversation correctly? Like, why why why have LLMs been sort of... Why why has this been sort of a a strong native skill of of AI, the sort of risk dimension dimension? [00:40:34] S0: We've had mixed results actually with out of the box LLMs. [00:40:41] S1: Okay.
00:40:41 · Brett Caughran, Kris Bennatti
[00:40:41] S1: Okay. [00:40:41] S0: So some things that have worked well are find headwinds, those types of requests. Where we've seen issues is with more complex elements. So I've actually been somewhat disappointed with out of the box LLMs and being able to understand, say, bank's risk or off balance sheet risk in a way that's a bit more nuanced. So, yeah, I think there's still a couple gaps in understanding that personally I would have expected to be fixed by now. Yeah. I think the reason for that, though, is that there is just not a lot of good forensic information on the web to learn from. [00:41:44] S1: Okay. [00:41:44] S0: Where there is a lot of good information about general risks, like operational risks, etcetera, but there's not a ton of good forensic accounting stuff out there. Up until two years ago, until we wrote a blog post on it, if you Googled what is a critical audit matter, it would tell you the wrong thing. [00:42:03] S1: Yeah. There's sort of a weird circ too. Like when I'll search for things about buy side and AI, it'll bring my own blog post back to me. You probably... Yeah. [00:42:13] S0: Yeah. No. [00:42:14] S1: Wait, no, I want someone smarter than me telling me about it. I don't want to read my own. Yeah. Read my own blogs, but a lot of Hudson Labs stuff shows up there too. It's funny sort of getting your own documents fed back to you.
00:42:14 · Brett Caughran, Kris Bennatti
[00:42:14] S1: Wait, no, I want someone smarter than me telling me about it. I don't want to read my own. Yeah. Read my own blogs, but a lot of Hudson Labs stuff shows up there too. It's funny sort of getting your own documents fed back to you. [00:42:27] S0: Honestly, the reason we created a blog post about critical audit matters is because it was bothering me because I kept reading short reports that would reference the Google result for critical audit matter, which had clearly been written by AI. There's all of these short reports going out being like, oh, you know, this company has a critical audit matter related to revenue, ergo fraud. And I'm going, you know, single software company has a critical audit matter related to revenue. This is... We need to we need to clear this up. So Oh, okay. I I I do think there is still still a few gaps in understanding much, much better than it was a year ago. But in some of the more nuanced disclosure, there's... I'm still finding some reasoning error.
00:43:23 · Brett Caughran
[00:43:23] S1: Okay. Makes sense. Yeah. What... Super, super, super interesting, I think sort of like a very low calorie way for investors to just make better decisions, like having some sort of like systematic... If I sort of think about like the agent path and the agent stack, some sort of like forensic accounting risk element in that agent stack is critical. Like if I can do this in a seamless way before I make a trading decision, having sort of a risk checklist, forensic risk score, like, there's a one in three chance of this thing being a fraud. Like, I'd like to know that before I make the trading decision instead of after I make the trading decision. Odds of restatement risk, etcetera, which those are sort of very painful eight Ks to sort of wake up to. Having the systematizing your decision dashboard just seems like an absolute no brainer to me.
00:44:17 · Kris Bennatti, Brett Caughran
[00:44:17] S0: Agreed. And one of the things that's cool, we talked a little bit about cost and efficiency and if you try to run Hudson Labs risk scores out of the box, even for us, it costs hundreds of thousands of dollars to run the Hudson Labs forensic risk score over a ten year period, which is a sort of somewhat interesting element of AI. And one of the reasons that we're now finding it harder to support quant funds, which I think is a pretty, just because, you know, whenever you're updating something in that stack, if you have to rerun ten years of data, becomes harder to offer the state of the art functionality to purely quant funds who need that back testing, which I think is a pretty interesting development in AI where we've all been thinking about like, aren't LLM's amazing for quant funds? But we end up running into these cost constraints where we're really only able to offer state of the art functionality to people who don't need ten years of data. [00:45:35] S1: Yeah. How work? Does I mean, so I haven't tried this yet, this conversation has inspired me to try this. So I'm a healthcare investor. Can I create like a risk skill where it's like a three page risk checklist on a name, pulling in all of my underlying research on those companies, but also pulling in the Hudson Labs forensic risk score, pulling from the 52,000,000 data point vector database via MCP? Like, is that something where I could sort of press that button and get sort of a custom made risk report on any of my companies? [00:46:19] S0: We should chat. Let's chat about, try to get you set up.
00:46:19 · Kris Bennatti, Brett Caughran, Khe Hy
[00:46:19] S0: We should chat. Let's chat about, try to get you set up. [00:46:23] S1: Cool. Alright. Sounds great. Can I [00:46:27] S2: ask about, on this cost question, the, there's so much happening, right? So you have Fable that is 10x more expensive than You have all these, a lot of my clients have become heavy co work users. And guess what? That's not a cheap harness, turns out. Then you have Chinese models, open weight models. How are you seeing this kind of cost? And also, people care about cost. Like, people stopped caring about people didn't care about costs three months ago. And now a lot of people, maybe some of the mega funds excluded, don't care. What's your take on the shifting tokenomics landscape as it relates to investment decision making?