00:08:10 · Brett Caughran
[00:08:10] S0: Yeah, that's a great question. I think we probably have the same personal Sherpa in the learning journey of Andre Karpathy. And I think he's done such a service to both of us in terms of what he shared publicly on YouTube with many of his videos. And I just find him to sort of be the rare Venn diagram of technical fluency on these tools and also not having the right incentives, not trying to hype his company to raise valuation. And the way he explains things is very good. I think Kay and I both recommend his videos on YouTube as a good zero to one in learning what LLMs are. And I had a similar insight to you on 'one, 'three with the deep research functionality. That was the first time where I had something in my research stack that I found to be institutional grade, particularly if you pointed it towards open web information, things where there was a lot of government data or something where the raw data was in the open arena. Deep research reports were wonderful at distillation. And for really at the same time, the spring of twenty five, I found myself starting research processes with deep research reports, which I found to be often better than sell side reports and customize the specific thing. There were certain use cases, I come from the quant world where natural language processing is sort of a key fundamental alpha. And from a hedge fund world where we're constantly trying to understand the body language of management, that word selection really matters. There's information laden in word selection. So I think that was an initial use case that I found quite helpful out of chatbots. Is the company using more bullish or more bearish language? But these were small pieces, right? With my Sherpa Andre, I think it was in October, he went on Dwarcache and he called agent Slop and said, It's going to be a ten year journey. And so I'm like, All right, well, Andre says it, it's good enough for me. Everyone talking about agents, I never really understood the agentic thing in '25 either. OpenAI came out with their agent functionality and it's just what is this thing? It just didn't work at all. So I wrote it off as hype and followed Andre over Christmas as he sort of went through the coding agent thing. And he came out in January. He's like, listen, like what I said in October is completely different. It's completely changed.
00:10:40 · Brett Caughran, Khe Hy
So I started going down sort of the understanding of agents. And I have a full set of like evals and different things I want to try to do with these tools and chatbots failed the majority of them. I think when did Claude sort of come out with Claude Code and Claude CoWork? But that was really like the big agentic shift of the Claude. Exactly. Of the Claude CoWork. [00:11:03] S1: Yeah, Claude Code was I think Q2 of twenty five and co work was January '20 [00:11:10] S0: Yeah, so the Claude code, like the December, January moment was like a big inflection in the agentic revolution. And even people talk about Claude Code and an IDE, but even the things you can do in Cloud Chat are now agentic. I can go build Excel, you know, build Excel things just directly in Cloud Chat. So Cloud really, I think, took this big step ahead from OpenAI, and you've seen more agentic tools. It's really just the last ninety days, there was like, call it 5% of my workflows, five to 10% of my workflows as an investor was the TAM for AI in '25. Now I feel like it's pushing up to 30, 40, 50, 60%. And the imagination you can apply with these tools now is well beyond what you could do with chatbots. Now there's still some issues that we'll explore this chatbots. Issues, many jagged edges, context window limitations haven't gone away. Multi document retrieval accuracy hasn't completely been solved. But the pace of evolution just in six months is, I think, well ahead of the bull case I envisioned, you know, summer twenty five.
00:11:10 · Brett Caughran, Khe Hy
[00:11:10] S0: Yeah, so the Claude code, like the December, January moment was like a big inflection in the agentic revolution. And even people talk about Claude Code and an IDE, but even the things you can do in Cloud Chat are now agentic. I can go build Excel, you know, build Excel things just directly in Cloud Chat. So Cloud really, I think, took this big step ahead from OpenAI, and you've seen more agentic tools. It's really just the last ninety days, there was like, call it 5% of my workflows, five to 10% of my workflows as an investor was the TAM for AI in '25. Now I feel like it's pushing up to 30, 40, 50, 60%. And the imagination you can apply with these tools now is well beyond what you could do with chatbots. Now there's still some issues that we'll explore this chatbots. Issues, many jagged edges, context window limitations haven't gone away. Multi document retrieval accuracy hasn't completely been solved. But the pace of evolution just in six months is, I think, well ahead of the bull case I envisioned, you know, summer twenty five. [00:12:34] S1: Can I ask you because I feel like the the word agentic gets used a lot? And what is how do you personally define the word? And then I'm happy to share my definition.
00:12:45 · Brett Caughran
[00:12:45] S0: Yeah. No, I'm curious yours. I, I think about the investor toolbox as like workflows, kind of end to end workflows. Hey, I want to go do an earnings preview process or this company has a new CEO, I want to go and explore the background and the potential of this CEO. And within that, there can be 20 steps along the process. I want to go understand or her stock grant, past performance, do a background analysis on if this person has nefarious historical affiliations, etc. In the past, I had to create each of those as individual prompts because as you know, a prompt past five or six pages started to lead to a garbled output with chatbots. So I couldn't build a workflow to do all of these things. And many of the tool providers tried to do that, chaining together prompts and building all of this complex engineering around it. Now I can one shot that an agent, So with a skills it really just makes the tools much more user friendly. It gives me an end to end workflow that still isn't perfect, but is shocking in terms of the quality and caliber. So my three zero six prompts have gone down to 30, really like 18 orchestrated pipelines and then individual skills that I can pull off the shelf to dig in more deeply. So you're seeing this abstraction in the complexity of working with these tools. And there's been innovations around MCP and MCP and data as well, too, that sort of the one example you'd see that if you went into ChatGPT last year and asked a question, often the quantitative metric will be pulled from a blog somewhere, right? So if I went in and I said, what is the ten year history of Nvidia's earnings per share? These were exercises that a chatbot couldn't answer accurately. So it's like, if I can't answer a basic question like that, like get out of my face with this AGI thing. But now I can plug Claude Cowork into an MCP from Dilipa or Faxit or FinShat, and that data is piped in through the MCP process with, in my experimentation, much better accuracy. So it's not a solved problem, but there's been huge innovation. Do you think about agentic workflows and what agents unlock?
00:15:29 · Khe Hy
[00:15:29] S1: Think similar to what you said, so I hear kind of like end to end workflows being one versus kind of constantly steering the model. Okay, now tell me this. Now tell me this. The analogy I use is it's like the difference between Waymo and Google Maps, right? Waymo, you give it the destination and then you just kick back. You don't really care how it gets there as long as it gets you there within a standard deviation of your expectations for time and safety, right? And so with the AgenTic, it's like, hey, I would like to, you know, write this deck. But I don't really care if you do slide seven first or this, but I just have a general intent of what I'm looking for. And then the model kind of figures it out. Right. So that that would be kind of like the end to end. And obviously, you mentioned the connections as well. The other thing that I would add with agents is that they tend to, like, self correct along the way. So you might give it a PDF that's too big. And then without telling it, it will run like a PDF extraction tool, right? Like, I don't even know that the PDF extraction tool exists as a thing. I don't even know that PDF's too big. But the agent is making these decisions and what, you know, in AI language, they'll say tool calling, right? And not just tool calling like Slack or Dulupa, tool calling like these weird kind of utility help error apps to, like, chunk a PDF or to use what we would call Command F, you know, is called grep in, you know, in computer science land. And so the agent not only can it get you answers, but it can find and solve its own problems by accessing new tools, which again gets to that point that you mentioned. It makes the end to end workflow more accurate and longer running. So in many ways, that kind of full agreement with what you said, I would add kind of the tooling element that is kind of abstracted away that, like, I don't have to tell it to use the PDF extractor and then the ability for it to reason and act and self correct in the process.
00:17:42 · Brett Caughran
[00:17:42] S0: Yeah. And I'd add probably one of the it's not the flashiest AI pilled moment, but one of the big AI pill moments I had this year was the usability of agents for validation. And so we work in an industry where accuracy is critical. We have a fiduciary duty to our investors to conduct a rigorous investment process. And if you follow any of the finance benchmarks of LLM accuracy, they tend to sort of shake out in the 60 to 70% range. So I'm always like, all right, well, yes, for qualitative use cases where I'm putting in six transcripts that I can go and validate in the transcript, sure, Right? But for quantitative use cases, that was a very difficult mental hurdle for me to get over. Like I don't want to unleash LLMs on financial models where it's 65% accurate. Exercise of Like finding 35% error is like next word prediction, next token prediction, next word prediction, it's fine. Next number prediction is not fine. We don't want a stochastic exercise to sort of make up a plausible sounding number, right? And in a few of these use cases, like LLMs are very good at making something look plausible, which means that where I've seen quantitative errors, sometimes a quantitative error looks more plausible than the actual number, right? And so this was something where I was like pump the brakes on quantitative use cases in 25. So what's changed is MCP, the data's gotten more accurate. But the ability to build agentic, I was using is like, okay, let's check these numbers, use these four LM to check this LLM and go back and forth. And it was okay, but the ability to create agents to validate work now, it's exactly you pointed out to sort of go in tool call, check all these numbers, right? Even give me a checklist by hand to go in and spot check numbers that could really be problematic for my thesis. That to me is a big unlock with agents that didn't exist in chatbots. Curious, Kay, to understand a little bit more about on your consulting side, are there three or four common themes you see come up from clients in terms of the problem space that the firms you work with have in general that you help them walk through?