Latent Space: The AI Engineer Podcast
01:34:55 · Mihail Eric, Shawn Wang
[01:34:55] S1: So, okay. So what I'm thinking here is if this is if this is the week design... More kind of geared toward scaling this out to, like, what happens when you take some principles... And a lot of this was informed by, like, the conversation I had with folks that were, you know, leading AI native teams AI native teams at, at respective companies. And so here, I think that something at MCP portal really becomes interesting once you're at a certain scale, where you effectively have to have a single point of entry to do things like governance, security, auditing, cost optimization. And so, you know, I... I'd be curious to hear what you see as the future or maybe lack thereof of something like this, but it to me is... And it's... If you believe in, like, an open router or, you know, some place... Like, a single point of entry for for many things to happen and connect to and be effectively, you know, governed, I feel like there's a world where this also can easily exist. [01:35:43] S0: Yeah. My my only tweak on this is that this folds into the OLM gateway. So they they are the exact same platform. [01:35:49] S1: That I will give you. That I will give you. I can I can accept that? That actually, you can can agree with.
01:35:49 · Mihail Eric, Shawn Wang
[01:35:49] S1: That I will give you. That I will give you. I can I can accept that? That actually, you can can agree with. [01:35:53] S0: Yeah. And and then then, by the way, it also folds into the other thing of, what did I say? Was off z, you know, permissioning and all these things and billing. So, like, to the limit, all infra converges to the same exact problems where they have the box in the middle and then you have the box in two sides. The, you know, godbox will solve everything. Right? And For the enterprise. True only if you take... If you understand the job of the godbox. And so separating MCP portals and gateways means you don't fully accept that they are, you know, the the the same layer of problem. In a very similar way, I also find that people don't understand database querying to have a lot of the similar things where they will separate filtering from ranking from... Or or just, like, general recommendation Mhmm. If they're the same problem. So then you just come... Because when you take all these things, you combine them in the same layer, then you can sort of cross depend on things in useful ways. If there are different layers, then they cannot.
01:35:53 · Shawn Wang, Mihail Eric
[01:35:53] S0: Yeah. And and then then, by the way, it also folds into the other thing of, what did I say? Was off z, you know, permissioning and all these things and billing. So, like, to the limit, all infra converges to the same exact problems where they have the box in the middle and then you have the box in two sides. The, you know, godbox will solve everything. Right? And For the enterprise. True only if you take... If you understand the job of the godbox. And so separating MCP portals and gateways means you don't fully accept that they are, you know, the the the same layer of problem. In a very similar way, I also find that people don't understand database querying to have a lot of the similar things where they will separate filtering from ranking from... Or or just, like, general recommendation Mhmm. If they're the same problem. So then you just come... Because when you take all these things, you combine them in the same layer, then you can sort of cross depend on things in useful ways. If there are different layers, then they cannot. [01:36:52] S1: Yeah. I actually will not disagree with this. I think maybe the... Yeah, so it's slightly pedagogical. I think that this breaking up of separate terms versus one, I think it is absolutely correct that the underlying architecture is the godbox, where one thing goes in and there's a bunch of other stuff that is some third party, some internal... Some first party that all kind of exists on the other side of the godbox. So, yes, I agree with this, actually.
01:36:52 · Mihail Eric, Shawn Wang
[01:36:52] S1: Yeah. I actually will not disagree with this. I think maybe the... Yeah, so it's slightly pedagogical. I think that this breaking up of separate terms versus one, I think it is absolutely correct that the underlying architecture is the godbox, where one thing goes in and there's a bunch of other stuff that is some third party, some internal... Some first party that all kind of exists on the other side of the godbox. So, yes, I agree with this, actually. [01:37:18] S0: I I have a I have a spicy spicy tweet. I... Oh, god. Where where is the godbox tweet? Oh, yeah. So this is the m m m x Sure. N Sure. M plus n Let me bring it up. Architecture. So anyone working in infra needs to understand. Okay. Congratulations. You now have a job in infrastructure or you want to be an infra engineer. You need to understand that your entire job is making god boxes and then taking them apart. So either... Your problem is either you don't have enough... You don't have a good god box, or your problem is someone else's god box you don't like, so now you have to take it apart and make your own. [01:37:51] S1: That's funny. That's very funny. I like that. That's very funny. Into the heat death of the universe. Wow.
01:37:51 · Mihail Eric, Shawn Wang, S2
[01:37:51] S1: That's funny. That's very funny. I like that. That's very funny. Into the heat death of the universe. Wow. [01:37:58] S0: That's funny. Which is a... This is a restatement of, like, there's only two ways to make software, which is you bundle them, you unbundle them. Right. Right. What are you bundling? Is It the things that go together, hopefully. And then maybe over time, people's opinions shift as what goes together, what should not go together. So there is that whole thing, maybe Monaco should be more of a point solution. Maybe Monaco should own everything. Right? And, like, there's the question of, like, in some ways, you're bundling for the the kind of startup segment that you serve. In some ways, you're unbundling from Salesforce because Salesforce... And then that's how you win market share from Salesforce. You unbundle from Salesforce. Mhmm. Mhmm. [01:38:39] S1: That's interesting. Yeah. That's that's an even broader statement. I mean, connecting it to broadly all of vertical AI or... Right now, like, bundling, unbundling. I mean, that's like the whole, like, Craigslist and Airbnb, you know, the the Tinder and all those are separate. Tinder. But, you know, like, a personals, basically. Yeah. Yeah. Yeah. Yeah. [01:38:57] S0: Yeah. Never thought it... Never thought thought about it that way. But, yes. And, you know, one one of the things that... So we just shipped the OpenRouter Pod today. Right? And OpenRouter, huge success. [01:39:06] S2: Mhmm.
01:39:06 · S2, Shawn Wang
[01:39:06] S2: Mhmm. [01:39:07] S0: Lots of people tried to build gateways. None of them had success with OpenRouter. OpenRouter only builds gateways. So there's an example of, like, this... They disagree with me that the the the router layer should also do ops because Arize is doing ops and observability and and what have you. And and they... And, like, the MCP gateway, they don't have that. And they they they just... They only handle model routing. And so, like, I I, like, I am... I love being wrong like that because they're like, we are the point solution, and we only do this one thing really, really well. And that's our that's our engineering focus, which is good, but it's also the marketing focus, which is nice. Mhmm.