The MAD Podcast: How AI Gets Built — with Matt Turck
00:25:34 · George Sivulka, Matt Turck
[00:25:34] S0: of invited me for a demo. I would have done a [00:25:36] S1: demo for you guys. Time. But it's it's a spreadsheet. Right? I mean, it's it's talk One about [00:25:43] S0: of the mods... So we think we're building effectively the new Microsoft Office suite. And just as Microsoft Office had Word and PowerPoint and Excel, we have Matrix and then actually a variety of other applications, some of which I can talk about, some of which I And we have a chat bot. And we think that there's going to be an entirely new productivity suite for how humans, how experts want to interface with AI without having to get in the weeds and code. And it's one of those applications. Yeah. [00:26:09] S1: Great. Alright. If that's okay with you, a little bit of a technical deep dive into how that whole works behind the scenes. So presumably, to start with, you need to connect with a bunch of sources of enterprise data and process the data. So how does that stage of ingestion work? Ingestion and indexing is one of the fundamental pieces of
00:26:33 · George Sivulka
[00:26:33] S0: having any knowledge work application. There's different steps to it. So first is just collecting the data sources and hooking into as many providers as possible. And that's not fun or not technical. The more interesting thing is actually once you have the data, how you index it. We believe that you can do a lot of pre processing. And I'm not talking about keyword search and building a BM25 index, or like having like some sort of semantic search index with embeddings, even your super long embeddings that are coming out. That's not actually that interesting to our users. Again, that's good at doing searches. It's good at finding things in the data, but you want to start to process information before the user even asks a question. You want these agents to be doing work ahead of time. What we do instead is we actually ingest documents and pre process, pre populate, depending on the doc type, depending on the context of the document, a really rich schema and understanding of each document. That ideally would be, Hey, we've pre indexed and pre done 90% of the work. At the same time, that's not enough for asking and running user questions on the fly. And for the 10% of the work, I. E, okay, this war just broke out here, or this crisis is happening here, or this event that you couldn't have predicted is happening, We use the same indexing engine and that same infrastructure to run things on the fly. And so you can consider any query that goes through Hebbia or that goes through Matrix as a combination of a lot of pre index work and then some stuff that's on the fly and custom. And the pipeline to do that from parsing documents, sometimes using multimodal documents to figure out formatting and structure and images, actually how we then go and run that index and what we're retrieving over and how do we save the work of the pre indexing agents. All of those are interesting search problems that I could talk about for a while. That's a bit of the thesis behind how we index.
00:28:27 · Matt Turck
[00:28:27] S1: And then perhaps that's part of what you just described, but I wrote down something about what you call the ISD architecture. So that's the Rag part after that. And you were saying that somewhere that Rag doesn't cut it for very hard problems. So what is your approach to RAG? And I assume a lot of people here know, but maybe use the opportunity to define what RAG is in
00:28:52 · George Sivulka
[00:28:52] S0: the first place. Sure. RAG is an architecture for using AI. That's retrieval augmented generation. It was first coined in a paper in March 2020 by a bunch of Facebook researchers. Hebia were actually the first to turn that into a product. So it's like a very close thing to my heart. So back in 2020, we were the first people to actually productionize it, roll it out. And the idea is that you could hook a search engine up to an LLM. It's basically what Perplexity uses. It's like a search engine to LLM, except over the web, always have an answer. Over offline and unstructured and private documents, you don't always have an answer. And what we say when we say rag doesn't cut it or that heavy, actually after making that as our baby, we had to kill the baby and we turned away from Rag to what we call ISD, is we said, well, we need an architecture that's closer to extending the context window versus just searching or calling an external tool. And ISD leverages that index. So a lot of work that has been done ahead of time. But then also it leverages a way of recursively kind of reading sub documents. So still leveraging tools, sometimes leveraging that infinite effective context window to kind of bubble up an answer. And it's really a decomposition agent at its core. So you ask a question instead of it just pinging a search tool, it can ping a variety of different tools, does rich decomposition, shows its work in the matrix, and then synthesizes a complete response towards the end.
00:30:22 · Matt Turck, George Sivulka
[00:30:22] S1: And do you have pieces still of, I guess, almost classical reg architecture, like re rankers and that type of stuff somewhere? Or you completely... [00:30:31] S0: Very funny thing. And my first five employees will get a laugh out of this and no one else. But we have to this day, most accurate re ranker that has ever been released. And we do not use it. We spent a lot, like the first year, year and a half, just training, embedding algorithms, training different versions of Colbert with multi embedding architectures for a single passage and then training re rankers. And we came up with a novel re ranker architecture, which four years later, academia and the industry have not beat, and we do not use it. And the reason we don't use it is because a lot of the time search isn't what's important. If we were building a public web API LLM, re rankers would be important, But we scrapped all of that for a really heavy infrastructure play that uses tons of large language model calls. It is really expensive from a latency and just latent dollar cost perspective, but achieves really accurate answers for deep research, for deep diligence tasks that people can only do in the heavy of platform. [00:31:36] S1: Great. Let's talk about the model part. I think you guys are very close to OpenAI. So do you use multiple OpenAI models? Do you use all this stuff
00:31:36 · Matt Turck, George Sivulka
[00:31:36] S1: Great. Let's talk about the model part. I think you guys are very close to OpenAI. So do you use multiple OpenAI models? Do you use all this stuff [00:31:45] S0: that you can talk about, open source, what have you? Yeah. We we... We've got partnerships with OpenAI, but also Anthropic and also, you know, Amazon. So we're we're playing the field. Don't don't tell Sam Altman. But the idea behind Hebia is we believe the model layer will become commoditized. It's no longer a hot take. So whatever models you wanna use, and hopefully, eventually, computing on the edge will be the way that you run these things. Just like you don't ping the cloud to run your Excel model, unless you're in Google Sheets. [00:32:15] S1: And then you've built some very smart stuff, what I can tell, around sort of scaling all of this. So you have this concept of Maximizer, which sounds like it's a router but smarter. What is it?