00:44:47 · Priya, Maya, Anika
[00:44:47] S1: And so... And know your environment really well too. Because like every single enterprise is super different in how they use things and what softwares they use, like in general. And so it's basically like all the scaffolding around like the model itself. [00:45:01] S0: Yeah. And so the Claude Code is example of a harness, Cognition has a harness, Cursor is an example of a harness. Even things like open evidences like ask bar, like all these like chat applications, like ChatGPT is also an example of harness. Like, Harvey is an example of a harness. There's a lot of application layer companies that wrap a model in general. [00:45:25] S2: And I feel like one of the reasons people have been talking about harnesses a lot also is as people talk about not just having like a bunch of slop that can, like, affect your security or affect your entire, like, code base. Because one of the examples that gets given is, like, LLMs are not deterministic. And you can actually, like, see this. Like, sometimes if you, like, tell an LOM, like, hey, don't do this. It's not like an actual rule and it'll, like, ignore you, which I [00:45:53] S1: was gonna say. Honestly, he gets really upset about this. [00:45:57] S2: Well, it's just, like, insane. Like, I literally, like, written in my clod in, like, 20 different places, like, just stop using Updashers and, like, it'll keep doing it.
00:46:04 · Priya
[00:46:04] S1: Definitely, as we see more AI code that's generated... Because, like, cogeneration is so cheap now. Like, we all know that, and you can literally have anything do it. Things move to trusting the code that's written, like, verification. Like, it's the entire reason that you're, like, I'm still spending so much time before something can be shipped. Yeah. Because there is so much that goes into it on, like, the trust and verification layer. And I feel like there's been, like, Horizon incidents, like Horizon, like, so many things across, like, every company. Andrej Karpathy also said that vibe coding is passe. I don't know if you guys saw that quote. He was like, vibe coding is passe, and the real value now is in systems that can plan, execute, test, and iterate. And the really interesting thing... I don't know if you guys read it. I sent it in our chat. But MongoDB came out with this paper, and it was basically about, like, agent harnesses and the importance of them. And one of the examples was of Vercel. And so Vercel deleted 80% of their agent tools, and success went from 80 to a 100% in, like, outputting code. And the latency fell from seven hundred seconds to one hundred seconds, and they were way more efficient with the exact same model. And so that shows that, like, if you keep the model the same, like, the infrastructure that you build around it has, like, a significant difference in, like, what actually gets shipped to, like, production code bases, how fast it can be done, and also the cost that you're spending on it. Well, I
00:47:20 · Maya, Priya
[00:47:20] S0: think it all comes down to like literally prompting and context management, which is like prompting is like basically the question that you ask the model. And what the harness is doing is like attaching all of these prompts together. And then you attach too many prompts together or you don't like write the prompt in the most effective way, you get a worse output. Yeah. And so like, that's why there you see actually all these like harness benchmarks and Claude Code is not at the top of the harness benchmarks. Because if you actually spent time really thinking about like, like the minimal set of tools that you should give like a harness, you'll realize that it's not that many. And if you give more, it's like you get actually negative returns rather than like flat amount of returns. And often like in the harness, it's like the simpler that you can be, the better that you can be. But there's like this kind of like middle thing where it's like, okay, obviously having nothing is not great, but then having too much is too much. And so [00:48:09] S1: Figuring you could it out. [00:48:10] S0: Well, it's also like an experiment that you can run like... So there's this thing called like Gemma, I think is what it is. I don't know. Someone correct me probably. But you, can optimize the prompt you're giving. Like, you can actually like run an experiment on a ton of prompts and, like, optimize your prompt, like, deterministically to get the best output. So you can, like, like... Wait. What... [00:48:34] S1: What's the output that drama gives you?
00:48:34 · Priya, Maya, Anika
[00:48:34] S1: What's the output that drama gives you? [00:48:36] S0: Like... Oh, sorry. Gemma is, like, the technique. You can, like, give it, like, essentially, like, thousands of prompts and you can figure out, like, what is, like, the most optimal prompt. And so you could actually apply this to harnesses where you can give like a thousand different descriptions for tools and figure out which is like the best version of the description for the tool so that the tool is used most effectively by the model. [00:48:55] S1: Do you think that Anthropic will like optimize Claude Code to the point where, like, most people will just end up using that? [00:49:00] S0: I mean, I think that they are trying to. Yeah. [00:49:02] S1: Well, yeah. They're definitely trying to. [00:49:03] S2: It seems like a lot of Cognition's competitive advantage right now is how compliant it is with industry certifications Yeah. And like finance and health And I just don't know if that's a long term strategy because it seems like these health insurance companies are partnering... Like, all these really regulated industries are very pro AI and have pretty deep partnerships that they're developing with AI companies. I assume Anthropic included. Yeah. OpenAI included. [00:49:32] S1: I feel like... I actually... Well, I
00:49:32 · Priya, Maya, Anika
[00:49:32] S1: I feel like... I actually... Well, I [00:49:33] S0: was gonna I was gonna draw to some historical context. This one is not from the nineteen hundreds. It's just Google. But Google has always been like the number three player in cloud computing, like, or in in the cloud. Like g g... What is it? GCP. GCP. Has always been like number three. Yeah. But they've brought in their position to 14% of the market and they're they were profitable. And once they became profitable, they quickly shut off in terms of like their margins and stuff. And it took them a long time. Like they launched in 2008 and they only turned profitable in 2023. And now they're like, actually making a ton of money from it. And and the so, way they competed was just by like undercutting AWS and Azure as much as possible. [00:50:12] S1: Mhmm. [00:50:12] S0: And so it's like, yeah, you're gonna probably like, spend time doing things that are not sustainable. Like, maybe like this compliance thing is like the way that Devin is trying to be like, is trying to win the contracts. I'm sure they're also undercutting by some extent, and they're not profitable by some extent. But once you're in, you're in. And once you have like an enterprise... [00:50:31] S1: Once you're in in something like this, it's like so sticky. It's like the most sticky. [00:50:35] S2: Yeah. [00:50:36] S0: So I do think it's like... [00:50:38] S1: I I think that's what [00:50:39] S0: the VCs are betting on honestly. Like, this is the same thing with AMD. AMD has always been number three. But...
00:50:39 · Maya, Priya, Anika
[00:50:39] S0: the VCs are betting on honestly. Like, this is the same thing with AMD. AMD has always been number three. But... [00:50:44] S1: Yeah. But it's like fine. [00:50:45] S2: It's like a business for them. [00:50:46] S0: There's like... It's just like what we've realized is that it's a market where like winner doesn't take all and that like Yeah. It's fine. Yeah. Yeah. I think probably Cursor. [00:50:56] S2: Feel like AMD is a little bit different because it's more supply constrained. But yeah. [00:51:01] S0: Yeah. I could see what you're saying. [00:51:02] S1: But I [00:51:03] S0: don't think like AWS and like Azure were ever supply constrained. [00:51:06] S2: Yeah. No. That makes sense. They seem, like, really into the idea of staying independent. [00:51:12] S1: As they [00:51:13] S2: should, honestly. Oh, yeah. Yeah. [00:51:14] S1: It's super valuable. It's like them in factory right now. Yeah. Really? [00:51:18] S2: Why... Oh, factory is a coding agent? Yeah. [00:51:20] S1: Yeah. It's a coding harness. [00:51:21] S2: Well, dude, we use [00:51:22] S1: I think I've [00:51:23] S2: been saying it's a robotic [00:51:24] S1: You did. I I heard you say that on something, and I was like, it's not what it is. Okay. Well, Maya's camera died, and we have to go. This has been awesome. Exciting new update.