Databricks Revamps the Data Stack for Agentic AI

At its Data + AI Summit in San Francisco on June 16, Databricks unveiled a major extension to its lakehouse platform in an effort to reduce one of the most persistent sources of complexity in enterprise data management. The company’s ambition is to consolidate operational databases, analytical warehouses, duplicated databases, and the pipelines that surround them.

Data management teams have traditionally treated operational databases and analytical platforms as separate worlds. This has proven a major barrier to the broader operationalization of agentic AI, because it forces agents to work with disjointed environments that result in fragmented context, slower execution, and a growing risk of governance gaps as they scale.

In these situations, transactional databases are optimized for frequent, small updates, while analytical platforms are optimized for large queries across historical data. Companies then use pipelines and duplicated copies to connect the two. For instance, an ecommerce system might record a sale in one database, before copying it over to an analytical platform and later feeding it into a dashboard or AI application. There are sound technical reasons for that separation, but it also creates substantial operational overhead when multiple systems have to maintain up-to-date versions of the same information. That can lead to delayed business reporting, inconsistent records across departments and cost overruns due to duplicate storage expenses.

Databricks’ new Lake Transactional/Analytical Processing, or LTAP, provides a more consolidated approach that aims to bring transactions, analytics and streaming workloads into a single governed copy of data. In the same summit, they also introduced Lakehouse//RT, a real-time query engine that is now available in beta. Databricks said that LTAP relies on open table formats that are compatible with PostgreSQL, which could potentially reduce the amount of application rewriting for software teams as well.

Lakehouse//RT addresses a closely related challenge. Existing analytical lakehouses might be economical and scalable, but they are rarely fast enough for customer-facing applications or live operational decisions. According to Databricks’ own benchmarks, the new real-time query engine offers response times below 100 milliseconds at 12,000 queries per second, with performance gains of up to 16 times. Nonetheless, real-world results will vary depending on workloads and how they are configured.

The potential business outcomes of the new architecture are promising. After all, data pipelines require monitoring, they tend to break when schemas change, and they usually involve separate cloud services with specialist teams for each. A unified approach could reduce some of the costs involved in coordinating those otherwise disparate functions, while also enhancing productivity among data engineering teams.

Databricks’ announcement comes at a time when enterprises across the board are trying to operationalize agentic AI. The shift has rarely been straightforward, considering that AI agents require current data rather than historical training datasets alone. Agents also need to write information back into operational systems, so having multiple copies results in delayed synchronization and reduced output reliability. For example, an assistant tasked with producing a quarterly summary might tolerate some delay, but if an AI agent is tasked with something like approving an order or checking inventory, it will almost certainly require current transactional information to be genuinely useful.

For software leaders, the significance of Databrick’s new architecture will ultimately depend on whether it can provide agents with that reliable, real-time access to governed data without introducing a new layer of platform complexity.