Most enterprise AI never makes it to a single customer. New capabilities from Confluent target the two specific reasons why: security teams blocking sensitive data and developers drowning in tool-switching. Here is what the fix looks like in practice.

by Kasun Illankoon, Editor in Chief at Tech Revolt

[For more news, click here]

There is a particular kind of frustration that has become familiar in enterprise technology circles. A team has a working AI model, leadership has approved the budget, and the use case is genuine. Then the whole project stalls, not because the AI is wrong, but because the data feeding it is unsafe, fragmented, or simply not moving fast enough to be useful. The AI layer turns out to be the easy part. The data layer is where things break.

Confluent, now operating as an IBM company and the organisation that effectively commercialised Apache Kafka, has spent the better part of a year identifying this as the central obstacle to enterprise AI reaching production. This week, the company announced a set of new capabilities for Confluent Cloud and its Confluent Intelligence platform that are designed, with some precision, to remove the two most common reasons AI workloads stall before they reach a real user.

Those two reasons, according to Confluent's own analysis and data from the broader industry, are security risk and developer friction. The new features address both directly, and the approach is notable less for its ambition than for its specificity.

The bottleneck most AI roadmaps do not plan for

To understand why the data layer has become such a persistent obstacle, it helps to understand what production-ready AI actually requires. A model needs context, and that context needs to be current, governed, and trusted. In industries like financial services, healthcare, or insurance, that requirement collides almost immediately with data governance rules that were built for a different era.

Sensitive fields, personally identifiable information, records protected by regulation: all of it sits inside the data streams that would make AI applications genuinely useful, and security teams are not wrong to be cautious about letting it flow freely into AI pipelines.

The solution Confluent has shipped for this problem is a built-in machine learning function for PII detection and redaction that operates directly inside Flink SQL. The significance of the architectural choice here deserves some attention.

Rather than requiring teams to move data to a warehouse before scrubbing it, or to build custom detection code, or to route data through an external service, the redaction happens in the stream itself, at the point of processing. For teams in regulated industries, this distinction matters considerably. Data that never leaves the governed pipeline is data that security teams can more plausibly approve.

The second structural problem is the one developers encounter more directly. Managing real-time data pipelines requires jumping between tools: one environment to write the pipeline logic, another to inspect what is happening, another to debug, another to deploy. Every context switch adds latency to what is supposed to be a fast iteration cycle. In practice, many AI development teams spend more time navigating their toolchain than improving their applications.

"Most AI projects fail before they reach a single customer because the data layer breaks down. Teams have the models and the mandate, but security risks and fragmented data stop them from shipping. We're fixing that by making the streaming layer the foundation for secure, production-ready AI," said Sean Falconer, Head of AI, Confluent

What the MCP server actually changes for developers

The most structurally interesting announcement in this release is the fully managed Model Context Protocol server, now generally available for Confluent Cloud. MCP, the protocol developed by Anthropic that allows AI systems to interact with external tools through a standardised interface, has been gaining significant traction as a way to give AI agents real operational capabilities. Confluent's implementation allows developers to use AI as a control plane for streaming operations, meaning they can build, manage, and debug data pipelines through natural language rather than switching between dashboards and configuration files.

A second layer of this capability, called Agent Skills, encodes organisational best practices and approved workflows into the AI's operating parameters.

This is a meaningful addition because it addresses a genuine concern about agentic systems: that they will execute tasks in ways that are technically correct but organisationally problematic. By encoding standards into the agent's available actions, teams get the speed benefit of AI-driven operations without surrendering control over how those operations are actually performed.

For the data engineering community specifically, Confluent has also released a free open source dbt adapter that brings Flink SQL into dbt, the framework that has become the industry standard for building and managing data pipelines. The practical implication is that teams who have already invested in dbt workflows can extend those workflows into real-time streaming without adopting an entirely new toolchain. That kind of compatibility is often underrated in technology announcements but tends to be what drives actual adoption.

Private connectivity and the hybrid AI architecture

The third pillar of the release addresses network architecture. Support for Azure Private Link means that Flink jobs can now connect to Azure-hosted services, including Azure OpenAI, Azure SQL, and Cosmos DB, over Microsoft's private backbone rather than the public internet. For large enterprises, keeping AI workloads off the public internet is not a preference but a requirement, and the absence of private connectivity options has been a genuine barrier to moving certain workloads forward.

The broader model support announced alongside these features is also worth noting. Confluent now supports TimesFM models for anomaly detection within stream processing workflows, and has added support for Anthropic and Fireworks AI models that developers can use directly inside Flink. The direction here reflects a broader industry trend: rather than forcing organisations to choose a single AI provider, the infrastructure layer becomes the neutral foundation on which multiple models can operate.

The IBM context and what it suggests about direction

Confluent's position as an IBM company adds a dimension to this announcement that is worth reading carefully. The capabilities unveiled this week extend earlier announcements from IBM Think, where Confluent Cloud was integrated more deeply into IBM's watsonx.data platform. The framing is one of a real-time context layer that feeds AI applications across hybrid environments, meaning organisations that run workloads across on-premise infrastructure and multiple cloud providers.

That positioning speaks to a specific tier of enterprise customer: large, regulated, operating complex hybrid architectures, and currently stuck between an AI mandate from leadership and a data infrastructure that was not built for it. The Real-Time Context Engine, which Confluent highlights as generally available, is the piece that continuously delivers fresh and governed context into AI applications. The emphasis on "governed" is deliberate. It is the word that moves conversations from the innovation team to the security and compliance teams, and those teams are, in many organisations, where AI projects currently stop.

What Confluent is assembling, taken as a whole, is less a collection of features than a coherent argument about where enterprise AI needs to be built. The model layer, as the company's head of AI observed, is not where the problem lives. The problem lives in the infrastructure that feeds the model, governs what reaches it, and makes it practical for developers to work with at speed. That argument has been made before. The difference, with this release, is that there are now specific, shippable answers attached to it.

Related Articles:

The Internet Is Being Colonized by AI Bots. Two Tech Giants Think They Have the Fix

IBM Announces Plans with AWS to Fuel Innovation and Cloud Adoption Across the Middle East

Confluent Unlocks Scalable Real-Time Agentic AI With Streaming Agents