00:35:09 · Matt Stockton
[00:35:09] S1: Yeah. I mean, I think, you know, these models, it's funny, they have a skills creator skill, which you can go in and use to create skills. The point of that is like these models obviously know what they expect to see within these skills. So I think that's useful to try. Ultimately, I think it's more about... I like to... This concept of like hill climbing, right? Where you try a thing, you look at its outputs, and you use your brain to say, Hey, is this good or bad and why? Right? And so, like, ultimately, when I'm building these things out, you kind of have to have this iterative loop. So maybe you build something that does something, produces an output, and then you say, okay, like, let me record myself talking about what's good and bad about this specific thing. Let me take that feedback, the output of the skill, the skill itself, and do work with the model to iterate the skill itself, right? I think kind of refining it is a pattern that I like to use, just trying to continually improve it. And ideally, you know, for some of these things too, it's really important to have kind of systematic evaluations. Like if you're kind of producing something, you ideally have a way to test its outputs, which is tricky if it's text, right? Because it's sometimes very subjective. But how can you specify what correct is and have a way to kind of continually test that your skills are doing the right thing over time?
00:36:44 · Brett Caughran, Matt Stockton
[00:36:44] S0: Have you thought about building evals? Because that's sort of hard as well too. Even like I'll eval things across three different LLMs, and I'll get a little bit of a different scorecard on my on my evals. [00:36:56] S1: Yeah. Yeah. It's definitely... It's it's tricky with with written text. Right? It's a lot easier if you're evaluating. Oh, is this number correct? Did I extract this specific sentence from this specific, section of the the filing? With with with text, I mean, can obviously do something... The technique is LLM as judge where you're using a LLM itself to review the outputs, but that's a little tricky because then, okay, what is the judge doing? How are you evaluating that? I mean, I think oftentimes, like, when I'm building these workflows for the space and the output is like a written report, you get a lot of utility by just reviewing those with the person that's consuming it. Like you say, okay, well, let's step through this memo and you tell me what's good and what's bad about this. And we'll record that and transcribe that and use that as input to try to figure out, okay, how do I adjust this skill? So that's, I mean, that's not a automated evaluation, but it's kind of an evaluation loop, which I think is really valuable, right? Like, because you're kind of going back to the person who's using the outputs and getting their feedback and trying to integrate that back into the upstream instructions.
00:38:23 · Khe Hy
[00:38:23] S2: Matt, has your view I know early in the skills shift, you were a very big supporter of Skill Creator. I think I saw a tweet from you recently that is maybe more skeptical of Skill Creator. I know personally that I've pushed people to use Skill Creator. I use it all the time. And now it's these, like, overbuilt... Someone even asked me, Do you think they're overbuilding them to just, like, subconsciously have you use more tokens? I'm like, Dude, I just need you to print one number, and you wrote me a book.
00:38:58 · Matt Stockton
[00:38:58] S1: This kind of ties to a more general view I have here, is that these models are amazing, right? Like, you can type things in and get things back, and they're great, right? But there is this weird kind of like dopamine hit of like just getting things coming out of the machine to the point of like, sometimes, like, you know, you're not reading things that you should be reading. So clearly, you shouldn't say like, Go create me a skill that does a memo, and then just say, Okay, this skill's done, right? Like, there's a lot of value, I think, from like stepping away from these tools, like, and just opening a notepad on your system and, like, writing down what you want, like typing words or describing what you want. I think that is the failure mode is that, like, you don't actually spend the time to read what these things are producing, whether it be the skills themselves or some meta instructions that you're using. But it's a balance, right? I think, you know, I'm assuming that the companies have spent a bunch of time on these Skill Creator skills such that they produce the shape of output that does best in their system. But yeah, like you got to spend the time to make sure that what you're using is like what you actually intend to use.
00:40:17 · Brett Caughran
[00:40:17] S0: And the hill climber mode is the sort of mindset is so valuable. I've sort of thought about it as like the red pen phase, because I feel like the natural impulse of Wall Street users with these tools, as you run an earnings preview, you get it back and you're just like, Oh, this is awful. Right? Rather than saying, Okay, this isn't great, but how can I red pen this and give feedback back into the model to go back in and change the skill and do that two or three or four? I even find myself sort of having to nudge myself to hill climb with that red pen mode. Like you're just not going to get there right out of the box. Like the first shot is almost never going to be ideally how you want. Ideally how you want. One example I give is LLMs think that goodwill impairments are really, really scary things for companies. Like you can't buy the stock because there's going to be a goodwill impairment coming, which is just an accounting thing. So you get that back. You have to be like, no, don't. Never in the future cite goodwill impairment as a key risk to a company that's more of an accounting dynamic rather than a fundamental fundamental dynamics. There's a lot of these different debugging things that you have to go through to to get these things to speak your language.
00:41:38 · Matt Stockton, Brett Caughran
[00:41:38] S1: Yeah, yeah. Yeah, I think another thing that that's interesting too is, you know, these models are ultimately, they're good at, you know, distilling text, they're good at writing code to produce things. But they're ultimately nondeterministic, right? Like, they are nondeterministic machines, right? And so if you're trying to solve a problem that requires determinism, or is a calculation, like, sometimes it's better to just write the code or do it somewhere else. So like, you definitely... I mean, LLMs, this is a tool, right? And so like, you can't... For some of the things you're trying to do, maybe it's not the right tool for the job. And just like always having that in mind too is like, it's something that you can use in your broader workflow, but it's not always the thing you need to reach for everything. [00:42:31] S0: Matt, how do you see the tool stack evolving? I mean, there's sort of been this debate on using actual agents like OpenClaw and Ermes. I've seen more and more of my friends sort of shift from building in Claude Co. To actually using Claude Co. Work, for lean on the consistency of the harness. So it feels like there's a little bit of a trend of abstraction into the super apps. Here's your perspective on that and where you see things going over. Where do you think what do you see as like the current trend and where do you see things going in that regard?