The MAD Podcast: How AI Gets Built — with Matt Turck
00:34:10 · Matt Turck
[00:34:10] S1: Inevitable question about hallucinations, you know, high stakes professional context where people have paid a lot of money to provide very reliable results to the customers. Is that something that, you know, in 2025 is as much of a problem as it was?
00:34:26 · George Sivulka
[00:34:26] S0: I think it's old news. And I think the only reason we still talk about it, it's like everyone talks about hallucinations, and, you know, no one knows if it's happening or what's going on. It's like, it was a problem back when the models were stupid, but they're obviously not stupid now. And I'd even say that they're way better than any human. So, when we start to think about like hallucinations, it's like, well, where does that come up? Like, have we actually seen a lot of that happen? Probably not recently. But the place where it does come up is actually a limitation of rec. It's when you've got the wrong documents or you're searching for something and it finds something that's broken. And what you're realizing is that the models are actually pretty good at doing reasoning. They're really good meta learners. Like the hallucination, whether it's a problem or not, no one really cares. What people care about is when these things fail because they don't have the right context or they're not reading the documents in the right way. And that's why we've done a lot of research on how to feed the right stuff at the right time. And if that's really, really expensive, we don't care because models are gonna become... Intelligence will become too cheap to meter. That's what our industry is predicated upon. So, we'll spend $10,000 in model costs per user a year. We wouldn't care. We don't, but indicative.
00:35:47 · Matt Turck
[00:35:47] S1: You just mentioned research. How does that work at at FBI? Do you guys have a concept of an AI lab within the company, or is it distributed across the team? And how important is research, especially in a context where, again, there seems to be something new every other day? How do you think about doing your own research versus just free writing whatever comes?
00:36:11 · George Sivulka
[00:36:11] S0: Yeah. I... It feels like there's something new every day. But also, like, if you've been tracking the field, it also doesn't. Like, it it kinda doesn't feel that different. I don't know if I'm allowed to say that, but it doesn't for me. There's new models, but they don't feel that different. Some of them are more sycophantic. That's the news. I think, and not a lot of people are saying it, but I think SF has a really big group think where it's like, okay, everyone's gonna go out and work on the same thing. There's this sort of memetic attraction rivalry between OpenAI and Anthropic. And what are they gonna do next? And like, oh, they have a new interface. So we're like, the chat bar got a little bit jumpier with the animation. And we kinda know, okay, they can make videos and it's too slow, and they can make images and it's better now. The issue is to come up with new interfaces and AI that can really do new things, I think you kind of have to avoid the group think. And so we currently don't have any AI researchers hired out in San Francisco. We don't even listen a lot to what people are requesting and really try to think about, okay, what is the tool that AGI would use? Like if you actually had AI that could choose any other AI tool to use, what would that look like? And we have that guide, what the interfaces look like. That's kind of how we came up with Matrix and a lot of the other interfaces that we're talking about. That's how we came up with ISD versus Rag and what the rest of the industry uses. It's kind of by avoiding what is going on on Twitter and just keeping our heads down. So we do have a research lab. It is not a standard research lab. It does spend a lot of time on information retrieval and context.
00:38:02 · Matt Turck, George Sivulka
[00:38:02] S1: So do you think that, to take the extreme of what you said, that research is actually slowing down? So there's test time compute that we're talking about that sort of felt like the major innovation, you know, over the last six months. But after we're done with Test Time Compute, are we sort of running out of tricks? I think there there will [00:38:21] S0: be more tricks. I do believe that AI will start to work on itself. I think that... I don't mean to be relatively pessimistic, but I wouldn't start a company in AI right now. I mean that from the perspective of the alpha is gone. When I was starting the company, everyone was working on crypto. And it was like, you know, the alpha was gone. I was like, okay, we're gonna work on AI. And so I don't know what it is. It's like the next thing. But I do think that the models will continue to get better. They'll get way cheaper. They'll get way faster, and the user experience will improve. I think longer term, just as I'm talking about how generalization beats specialization, you saw over the last twenty years of enterprise SaaS, Excel get unbundled into a million specialization things. It only made Excel more powerful. But there will still be specialization as a second wave to create a lot of value. But I don't see lots of very large changes in the near future. But maybe I've also gotten spoiled from how much stuff has changed. So I'll have to think about that one. [00:39:25] S1: Great. So everyone, pivot to crypto right now. Announcing the rebranding of the event. Crypto driven
00:39:25 · Matt Turck, George Sivulka
[00:39:25] S1: Great. So everyone, pivot to crypto right now. Announcing the rebranding of the event. Crypto driven [00:39:33] S0: Crypto Crypto New York City. Exactly. There we go. [00:39:37] S1: I wouldn't come. Switching tacks a little bit and talking about go to market. You've innovated there as well. I think among other things, you have a very different take on who salespeople should be for a company like Hebiah in terms of backgrounds. Do you wanna talk to that? [00:39:57] S0: Yeah. I'm happy to. I don't know if this audience is interested in like like sales playbook and lineages and all that stuff. [00:40:04] S1: But Yeah. I think there's a fair amount of people, you know, building companies.