00:30:05 · Ying Hua
[00:30:05] S0: It's very easy now. So let's think about it this way. So let's say you have 50 data points to track, different data sources, right? One, you can auto run each one of them. And then you can have each one decide, is this an interesting thing? And say, of the 100 I run today, let's say I have a daily run of 100 data sources. And then out of the 20 of them is interesting. AI determines interesting could be a meaningful change can be versus consensus, whatever it is. And then you can just have an agent on top of it's like, hey, those things are interesting make a dashboard for me. I only want a few things that's interesting. That's fully customized right that that's definitely doable. But I think there's another layer of customizing you touched on that I think it's actually a harder problem. You're talking about different industry, domain knowledge. I go back to my domain knowledge point. I think how do you analyze a tech company is very different than how you analyze an insurance company. The metric you care about is different. So if you think about the terminals, if you think of some of the current vendors, when they generate AI, a lot of the standardized, right? Standardized, whether it's model KPIs, those are not very useful. Those are not useful. But that part of customization gets a little bit because the model layer or the platform layer needs to have domain knowledge in order to spit out things that matters. So one simple example I used to test this for every some of our competitors, every competitor I try, I used to ask them, hey give me like an earnings call summary for JP Morgan. If they ever spit it out margin revenue like ah this platform doesn't know anything. Like no one cares about revenue for JP Morgan or insurance company. Right. Like that's just like a very obvious thing. So that part of customization I think is a lot harder than what dashboard to service to you. I think that part is a part we've spent the last two and half years building a ticket level domain knowledge for that reason. I think that is the hardest piece of all of it. How do you connect different knowledges? How do you know which knowledge matter at what point? And how do you know when user asks a question you need to pull that specific piece of knowledge in order to answer that question properly. So I think part of it is easy, part of it is, I guess, TBD.
00:32:34 · Khe Hy, Ying Hua
[00:32:34] S2: Yeah. Interesting. One question, one topic that comes up often with some of my investing clients is this, like, separation of synthesis and judgment. [00:32:47] S0: Yes. [00:32:47] S2: Right? And so the, like, I'll often hear someone say, it's like, We're okay for AI to do the synthesis and the gathering. [00:32:55] S0: Yeah. [00:32:55] S2: But when it comes to the judgment, that resides with the human. That resides with the analyst. Yeah. First of all, I don't think that's a very bi- that line is not as binary as one would tend to believe. But in your examples, you've said that, you know, Okay, You know, you have these data, these discrepancies in data, and then you have an agent kind of flag the discrepancy. And so, you're basically giving the agent some form of judgment. Yeah. And you've come back oftentimes to the role of domain expertise in this. But I'd be curious on your kind of... What your view on that separation between AI does synthesis, humans do judgment. And then when AI starts to make more judgment, like, where is it coming? Where is that training coming from?
00:33:46 · Ying Hua
[00:33:46] S0: I think it's a great question. So I think in a dashboard example, I actually, in most of the cases that we build, I don't think we're actually relying on LLM to make judgments. We basically will give a rubric flag me when these criteria hits. So it's still a human filter on what matters and what not. Obviously they're disadvantaged of that. It could be a new environment I haven't thought of. I didn't really flag that because of that. So that's I think but most of that is going to be a human, rubric that LLM flags. Judgment is interesting. I would say at the current state, given the fact that a few things, number one is I don't believe models are truly reasoning or truly understanding. There is no true sense of understanding. It's like a mimic sense of understanding or mimic sense of reasoning. Given that there is no continuous learning, meaning, anything changes doesn't get included, and given that I don't think models are trained in the right way to solve financial puzzle. And by the way, think financial puzzle is also one of the hardest ones, right? I think like compared to AlphaGo, compared to poker, like this is much much harder. Given all those limitations, I actually agree with the fact that judgment should rely on with humans. So let me give you an example how we think about it. Number one is very early on we have instructed our platform to not give subjective things unless they're asked to do so. So everything should be relied on checkable verifiable objective things. That's number one. Number two is if you think about like the knowledge layer we built. The knowledge bay we built is purely facts, things that can be verified with facts. It can be synthesis to your point, right? It can be like how credible is management? Can give you a score, but the score is backable by two stats. Like every time, you know, 30% chance they actually missed the guidance they gave. That's not a very good track record. Even though it's giving opinion but it's backable by track. So all of the things we focus on are these trackable factual things. Subjective stuff we try to stay away with. I don't personally think LLM is at a point where it can do that yet. We have some ideas on how it can get better. We're playing with some stuff. It's kind of like how human analysts start from scratch, build intuitions, right? It's a very much I form a hypothesis.
00:36:16 · Ying Hua, Brett Caughran
I test it out. I look at the stock reaction. Did I get it right? Did I get it wrong? Then learn something new. Okay, I'm try it again. I think, we can definitely build that but I don't think the model is there yet. So I would generally agree to leave the judgment to human. And the other thing is I think human is just naturally better at forming patterns, especially new patterns. I have a three year old comparing AI with a three year old. You'll see like how much better a human is better at like just connecting the dots and actually like spit out a hypothesis or like oh actually that's pretty reasonable versus AI is very much like I think too much backward looking, too much like pattern finding. So I generally agree with that sentiment. [00:37:03] S1: Yeah. Yeah. One of the conversations we've had in the past has always been sort of thorny issue for quants and sort of persists in AI is sort of this issue of materiality. Even I was sort of going through SpaceX today and my AI agent was really convinced that Starlink ARPU was the core thing that's going to matter. I'm like, well, what about the perceived terminal value of space travel and AI? Why do these systems still have native materiality limitations in your mind?
00:37:40 · Ying Hua
[00:37:40] S0: That's the domain knowledge. Like to give you an example, if you ask that same question what is most important for SPAY X? But precede it by providing a lot of context. This is what's going on in the market right now. These are the valuation of AI stocks. These are all twists that Elon said. These are all the things. If you give it enough context, will know starting is probably not the most important thing. But that context rely on domain knowledge and that's the same reason why quants, historically struggle with that too. Once have talked to quants and they can mimic, they can make a automatic do the model line by line, But they still don't know which line item matters for a given quarter. So they cannot bet every single line. They have to pick three but they don't know which one. I think again this goes back to domain knowledge. And how do you form that domain knowledge? I think that's the hardest part. Think this is the part that human analyst kind of the advantage of human analysis based on all the domain that they based on all the history they had all the domain knowledge they have a mental map of what's important what's not important. The other thing is I think to be fair, like I think it's a hard thing. I remember towards the end of my time at BAM, sometimes I would be like, hey if you give me, the press release, earnings release the day before, even if you were to do that, I don't think my hit rate on how the stock is going react is going to be 100%. Even humans have a hard time doing that. What matters? What's factored in? So I think it's a hard part. So I think that's a domain knowledge issue And it really depends on when you ask that question, what other relevant contexts that LLMN need to have a sense of what's material, what's not.
00:39:27 · Brett Caughran
[00:39:27] S1: Yeah. One other thorny issue that I see... I I sort of run into in my build out is sort of the current primacy of Excel as a sort of the investors' workbench. And I'm sort of sort of learning that Excel is not very token efficient, and I sort of would historically push back on the AI people saying Excel isn't the great, isn't sort of the best quantitative dashboard, needs to move into Python, etcetera. How do we solve Excel sort of ingress, egress, like the sort of the brain of the investment process today? How do we how do we solve that? And what's the future look like from a spreadsheet perspective?