Business Breakdowns
· Matt Reustle
[00:00:00] S0: Yeah, I can tell you from doing the show, I get to experience all these different businesses and I'm trying to draw the patterns and match them up, and some of them come more simply than others. The mission-critical part of the supply chain that's a small percentage of the overall cost, but they're a dominant player, they have all the market share, pricing power should be there. That's one that's emerged and I think gained a lot of appreciation over time. And then there are the more subtle ones like what you described before, which really, articulating them can be more challenging. But that is an interesting approach, and actually something that I've seen a few investors who have a differentiated framework try to lean into – finding ways to take that proprietary knowledge or pattern recognition that they developed over time, and get this tool that can get the most out of it in terms of uncovering new opportunities. I think one of the things that you've made clear to me in other appearances that you've made and as we've talked, is there are skill sets to get the most out of AI, and I don't think they're always that obvious, or at least they weren't obvious to me. But I was hoping we could just talk through some of the things that seem to make a massive difference in terms of whether investors are getting the Google search better quality output, versus really building on what they're doing, becoming massively more efficient. And the first is just the prompt writing. So how would you articulate how important prompt writing is? Some of the basic things that you think investors benefit from when thinking about prompts, and just feel free to wax poetic on it.
· David Plon
[00:00:00] S1: It's an interesting question, that the answer changes every three months in terms of what makes an effective prompt, because the model capabilities change, as well as the tools available to the models are constantly changing. In the investment research context, and I think this is true for other contexts as well but particularly research-based workflows with large language models. The mental model that I think still works really well is, imagine you are writing an email to somebody maybe overseas who's going to work overnight and is going to be doing a task for you. And assume they're smart but maybe lack a lot of context on you. What information would you want them to have to be able to do a good job? Whatever you end up writing is probably a pretty good starting point for an effective prompt. And to dig a little bit deeper on some of the subcomponents, when I write prompts today, I usually outline a specific task and why the task is happening, in the same way where if you give an analyst, "Build this cost curve," they say, "Okay." If you say, "Hey, build this cost curve because I think it might be shifting and that could imply something about future changes in the pricing," that's helpful. So I usually provide some background context. I provide a task. Depending on the nature of what I'm looking for, I might specify an output. I might say, "Hey, I want your output to look like this: have an intro paragraph, then outline these data points, show me a table that has X, Y, and Z." That sometimes is useful; sometimes that can be hurtful in that you're constraining the model's output the same way you would a human analyst. You might want to give them some slack on the rope to be able to be creative in how they want to synthesize the information. And then I usually have a section of my prompts where I have specific task guidelines, and then also just general domain knowledge. So for instance, in terms of guidelines, these are things that I would just think about. Maybe I'm doing that guidance prompt. Make sure you capture soft guidance that doesn't have a specific figure on it, because otherwise if you say just capture all guidance, it's just going to capture a specific quantitative guidance.
· David Plon
And then in terms of transferring domain knowledge, I have like a set of bullet points that I try to encapsulate – hre's how I think about being an analyst and some of the things you might learn through your experience. One of those is I always remind the model, like, "Look, you're going to largely be reading commentary from management teams. They are always biased positively. It's important you take a skeptical eye to anything they're saying." And even something that simple really helps because the models have been trained to want to be helpful, and that usually influences them to be more positive than they otherwise should be. So just little things like that, where you could imagine taking an eager college student who's excited and believes in the good, and trying to impart a little wisdom – "Hey, actually management teams tend to inflate things a bit." Just to wrap that up, defining the task, the context behind the task, some specific outputs if I want it, some guidelines, and some kind of domain context, I think is a really good starting point. And if you can send that to a human and they read it and understand the task, you're probably going to do pretty well with the model as well.
· Matt Reustle
[00:00:00] S0: Yeah, the level of instruction, if you almost humanize it, that was one of the initial things that I think you mentioned months and months ago that made a material change for me. And then on the idea of context and where --- and we'll use an LLM, assume off-the-shelf LLM for this question – but when you're thinking about when to just give them massive freedom versus when to load in documents. How does that change the scale and the output quality in your experience?
· David Plon, Matt Reustle
[00:00:00] S1: So I think it's very task-specific. There's a few different dimensions which it depends upon. One is, is the nature of the task something defined and structured and maybe more quantitative? In which case yeah, it can be a pain if you're using an off-the-shelf solution, but upload the documents. Because otherwise, generally speaking, an off-the-shelf model is going to have to go crawl the web, and that's an imperfect, inefficient way to gather a bunch of structured information, and you'll most likely get hallucinations. Which is why, of course, companies like Portrait exist because all that stuff's preloaded. But to the extent that the goal is something more exploratory or creative, it can still be really useful to upload the relevant documents, but I would explicitly instruct the model, "Hey, I don't really know which way this answer is going to go, so pull on threads, follow it, chase down leads." And in that case, you can steer the model – especially modern ones that are quite responsive – to do a lot of web searching. Where this matters as well is how important the task is with respect to accuracy to usefulness. There are certain tasks where if you're 99% accurate, you're 0% useful, and that's like self-driving cars. Where the cost of an error is exceptionally low. So I think of model building like, if I presented a model that had an obvious flaw in it to my boss in any of my past roles, that would be pretty bad. I certainly did that on occasion. [00:00:00] S0: That makes two of us.
· Matt Reustle, David Plon
[00:00:00] S0: That makes two of us. [00:00:00] S1: Yeah. Maybe you're just doing an industry survey – "What's the history of this industry and how has it evolved to these three players?" If there's a factual error here or there and it's earlier in the research process and it's more of a triage exercise, that's okay, I'm fine with that. Having some calibration depending on where the task is in the process, the importance of the accuracy, and how structured you want the output to be, I would vary and experiment. [00:00:00] S0: Yeah, it's quite interesting to see where things matter based on precision versus being directionally right is where all the value comes from. On your point on experimentation, I'm curious how much this matters. And this is something that I mentally struggle with sometimes as well. You mentioned the technology is constantly changing, so by experimenting or getting better with certain skill sets that eventually the models are going to be able to do on their own – how much difference do you see in investors or yourself when you experiment with different things in creative ways, when you know that there's a standard, decent-quality way that you can solve it already? I know it's a broad question just about experimentation as a skill set, but I am curious if you've noticed any patterns with the clients that you work with, or you yourself and some of your teammates.