The MAD Podcast: How AI Gets Built — with Matt Turck
00:17:52 · Matt Turck
[00:17:52] S1: Sure. Since we're on the topic, I cannot resist asking for a double click on this. So what does that mean? How do you artificially extend a context window? Well, the the idea of a
00:18:02 · George Sivulka
[00:18:02] S0: context window is that you can think about all of the things in the context window. So as humans, we currently can remember whatever the last ten minutes of conversation, maybe not this entire conversation. But we also have bits and pieces over our life and our training and our early careers that we've collected that make us really good employees. AI, you've got a system prompt and then whatever, up to a million tokens to jam as much context as you can in there. When you think about what you actually want AI to do, you want it to reason over all of your data. Applications like Rag can jam the context window with as many search results as you want, But it's not gonna reason over that data. It'll just search for stuff that exists. By elongating the context window, ideally you'd be able to connect the dots, find stuff that's there, but also stuff that's not there, stuff that's missing. And what that means actually tangibly from a product perspective is how do we very elegantly use the current context window, scale that up, or jam as much of a current maximum context window runs into a single question to get the right answer. And so it's actually more of an infrastructure problem. It's more of a kind of patchwork or convolution problem of how you actually apply that with the current length of context windows. And then it's actually kind of an information theory thing where you have to think through how, you know, what I get out of each of these runs can be maximized every time, but also maximally compressive. So that I'm... You know, every single, you know, future iteration, it gets just the right information, nothing that's not relevant.
00:19:49 · Matt Turck, George Sivulka
[00:19:49] S1: As you mentioned a year or two ago, you released Matrix, which is the flagship product for Hebia. So let's get into it. What does that actually do? I think it was billed as the interface to AGI. What's almost my reality as a user of the product? What do I have in front of me? What does it do for me? [00:20:08] S0: If the most important job in the future will be how you actually manage AI agents or how you prompt these things at scale, You can think of Matrix as actually running a bunch of sub agents or an interface like even a Trello board where you can assign a lot of tasks, and then a bunch of agents will do these things. And so it could be a network or a multi agent platform where you can basically say, Hey, I want all of these things done. They all are related to each other in some specific way. And every cell in what looks like a grid output ends up becoming a task that's completed by an agent. [00:20:47] S1: And you mentioned you started in finance, and then you expanded to law and consulting. How do you customize something like this? And relatedly, how do you even decide which next vertical you need to go into?
00:21:01 · George Sivulka
[00:21:01] S0: One of the things that is a massive fallacy in AI applications today is verticalization as paramount. All of the VCs, a lot of entrepreneurs as well believe, because it's been true for the last twenty years, that building a very verticalized piece of software is the only answer. And so you've got plenty of startups that are like, okay, I'm only gonna be AI for blank. AI for compliance or AI for law or AI for X, Y, or Z. When you have a capabilities, like AI capabilities approach to the problem, you actually start to realize that generalization will beat specialization every single time. And what I mean by that is if you, let's say you want an investing agent, your investing agent will become a researching agent and the researching agent should become a learning agent. And then you get to something closer to AGI. If you want a legal agent, right? Your legal agent will become maybe like a diligence agent or something that's very hyper logical. And if it's really logical, it'll be basically the smartest possible agent and end up at AGI. Or if you're a banker, your banking agent will become a marketing agent, because banking is effectively dressing up a company to be sold. And then the marketing agent will become, okay, the persuasion agent. And the persuasion agent will become something closer to AGI. And what I'm trying to highlight here is that, it's a way to hack, like to prompt one or two things for a specific set of customers at this point in time today. But as AGI comes closer or is actually here, as these models improve, and as we, as people get better at using them, we're going to get way more interested in actually extending the context window or pulling these expert knobs or taking something that's general and writing our own prompts and building our own agents. Because the reason why you as an investor beat the market, isn't because you're all using the same chat GPT wrapper prompt. And the reason why you as a lawyer are getting paid $2,000 an hour, isn't because, Oh, I have the same shortcuts from AI for a law company. It's because you are an expert and you can customize the greatest and the best tools to the way that your firm works, to the way that you actually can find an edge. And that's not specific. The best investors aren't only looking at SEC filings.
00:23:31 · George Sivulka, Matt Turck
They're reading really crazy data sources that have nothing to do with finance. The best lawyers, whether they're persuasive or logical, they have backgrounds that are way outside of law. You don't want specialization, you want generalization. And a lot of our customers are realizing that. That's actually the superpower of Hebia. And that's why so many people are buying us. [00:23:55] S1: Don't seem to be a huge fan of chatbot as an interface. Nothing against chatbots. [00:24:00] S0: Talk [00:24:02] S1: about how different the Hebia interface is and what you're trying to achieve there?
00:24:02 · Matt Turck, George Sivulka
[00:24:02] S1: about how different the Hebia interface is and what you're trying to achieve there? [00:24:09] S0: I think chatbots are, in my eyes, like the TI 84 or like the HP 12 c, like a like a calculator. It's a one off, I'm gonna go and put in my equation and then get a response. Nobody does their taxes in a calculator. Nobody computes a DCF or like does serious knowledge work in a TI-eighty four. You like stop using them in high school. You might have a simulator on your laptop to like, because it's novel. The way that we work is we have these really broad, interfaces called spreadsheets. And the way that everyone interfaces with a computer or the fundamental paradigm of computing is through a spreadsheet. It's flexible, it's highly modular, you could call it an expert system. And that's actually the way that humans, billions of people use capable computing today. The spreadsheet in Excel is the most important software that ever been invented. Hebia takes an approach that's not dissimilar. Just like a single cell in Excel is a calculator. A single cell in Hebia could be a chat or it could be a variety of other AI operations. But the Matrix platform, how people use it is so much broader. It's so much more flexible. It's an expert platform. And we're building that expert platform for how humans will interface with AI agents. [00:25:27] S1: And so the way they interact with the product and realizing it's difficult to talk about an interface verbally, but it's it's more [00:25:34] S0: of invited me for a demo. I would have done a