00:01:29 · Steffen Hoellinger, Neil C. Hughes
[00:01:29] S1: and what... Steffen Hoellinger. I'm the field CTO with Confluent, particularly focused on Flink and AI related use cases. I'm pretty excited about today's announcement, particularly around Streamhouse and the additions to Confluent platform. [00:01:42] S0: There has been a lot of announcements. So we've spent the last few years hearing about AI pilots, proofs of concept, and ambitious investment plans. But from the conversations with enterprise leaders across Europe and everything you're seeing, how much progress are organizations making in turning those experiments into genuinely... Or or something generally that really improves business outcomes?
00:02:04 · Steffen Hoellinger, Neil C. Hughes
[00:02:04] S1: Yeah. I think there is a big change currently underway in regards to how organizations deal with data and particularly around how they process data. So historically, Kafka was always used as a system to distribute data but to kind of not transform it while it's traveling in the stream. I think the requirement of AI and of AI agents in particular induces the need to not only like move data from A to B but also like to increase the value while the data is traveling in the stream. And this is also why our strategy around Streamhouse and around Flink is particularly focused on transforming the data, joining the data, enriching the data, but also filtering the data which is incredibly important when you combine it with AI agents. So only the right data hits the agent at the right time in the right possible shape, size, and form as the ideal context, fresh context for these agents to run. And batch processing in this world of AI is just not good enough anymore. So this is why I think there is a need to introduce the category of Streamhouse as a new principle of data architecture of how you build these operational use cases, particularly focused with the needs and requirements of AI. [00:03:15] S0: And one of the reoccurring challenges from businesses that we keep hearing is that their data is spread across so many different systems, departments, and locations. So how big a barrier is that fragmentation, especially when companies want to introduce things like AI agents that need to understand exactly what's happening right across the organization?
00:03:15 · Neil C. Hughes, Steffen Hoellinger
[00:03:15] S0: And one of the reoccurring challenges from businesses that we keep hearing is that their data is spread across so many different systems, departments, and locations. So how big a barrier is that fragmentation, especially when companies want to introduce things like AI agents that need to understand exactly what's happening right across the organization? [00:03:34] S1: Yeah. I think it's a big problem, and it also shows why particularly around batch way of doing things in terms of moving data in a batch pipeline and a batch process takes long, is expensive, and is particularly not good enough for these AI, agentic AI use cases to work with that fresh data. So AI really needs fresh continuously aggregated data all the time to make good decisions and recommendations. And this is why batch fails in the in the world of AI. [00:04:04] S0: And of course, we are two European guys in The US at the moment. And Europe just has its own challenges around data sovereignty, privacy Yep. And indeed regulation. Do do you think these requirements could slow AI adoption down, or could they ultimately encourage European businesses to build more trustworthy and reliable systems? Which way do you see this?
00:04:23 · Steffen Hoellinger
[00:04:23] S1: Yeah. I think there is definitely a big requirement and big interest at the moment for sovereignty in that space which is also why we are building on open source and on open standards in that sense. So Kafka itself is the technology as you know that Confluent developed and the creators of of Kafka back in the days at LinkedIn are kind of the same people that started then in the company Confluent and then we adopted further open standards and open source technologies like Flink for stream processing as a leading technology to process data while it's traveling in the stream. In the world at the moment, a huge adoption for Flink particularly in China but also in Silicon Valley for moving data around but also processing it whilst moving in the stream. And then OpenTable formats like Iceberg and Delta. So all of this is kind of built on open source technology, open standards, but still presented as a fully integrated platform. So companies can actually rely on that they can run these systems in all kinds of ways in different parts and also in different scenarios in operational mission critical use cases and scenarios wherever you you require this this compute and this processing to to happen. So we have lots of scenarios where people run Kafka and Flink in hybrid cloud scenario where they run it on prem, where they run it in the battlefield at the edge. There are places where you wouldn't expect Kafka to run. And, of course, also, server end infrastructure to run those those systems.
00:05:55 · Neil C. Hughes, Steffen Hoellinger
[00:05:55] S0: And I'm curious you'll... You dangled a carrot there. You don't have to mention any names. Some of those scenarios where you said you you wouldn't expect them to run. Is there anything you can share around that? [00:06:04] S1: Yeah. Actually, there are there are few very interesting use cases also over the last few years. You have people ran Kafka, for example, in America's Cup cluster linking like two boats with each other. You had like the the race boat and you had like a a separate boat that was actually the boat where all the team members had like making the decisions and then you had cluster link... Linking between those two boats exchanging information at low latency. There was just one example of a scenario where you wouldn't expect people to run a Kafka cluster on. [00:06:31] S0: Really? Incredibly cool. I'm curious, are you seeing European organizations or some reconsidering where their data, where their AI workload should live, particularly when it comes to public cloud versus private infrastructure. Are you are you hearing a lot of those conversations?
00:06:46 · Steffen Hoellinger
[00:06:46] S1: Yes. Of course. I mean, it's it's definitely a big subject, particularly in regulated industries Yeah. And banking, insurance, but also some retailers that actually had to learn it the hard way over the last few years moving everything into the cloud, and then you had some disruptions because certain cloud region was, for example, offline. So we had a lot of these use cases where people rely on the resilience of Kafka, have used Kafka for mission critical use cases and they need to be able to, yeah, compute and to transact all the time. So they have local Kafka clusters running in the factory floor and running Kafka cluster in the retail store. And these are fully independent solutions where you even have compute where you have local models running locally and, yeah, making sure that retail store is always able to sell products to their customers and the factory is always able to function even if the link to the cloud is not working for a part of the time. And then, of course, what you can do is you can kind of catch up once the cloud comes back up or the the region becomes available again. You can kind of cluster link that with other regions and you can catch up on that. So never... You never lose data and you're always there. You're always able to process data and to keep keep doing your business.