Top insights from the Databricks Data + AI Summit 2026

LTAP: Killing the ETL pipeline once and for all

The headline announcement from Ali Ghodsi’s keynote was LTAP, or Lake Transactional/Analytical Processing. It is a new data processing architecture that Databricks positions as the successor to everything that has been built to bridge OLTP to OLAP for the past four decades.

The problem LTAP is solving is a familiar one. Transactional systems (databases powering applications) and analytical systems (data warehouses powering reporting) have always lived apart. Bridging them meant building CDC pipelines, maintaining replicas, and accepting that your data was always slightly stale by the time it reached analysts or AI models. Previous attempts to unify the two, most notably HTAP, ended up collapsing workload isolation and left organizations with expensive proprietary infrastructure. Zero ETL approaches hid the pipeline but never eliminated it.

LTAP takes a fundamentally different approach by unifying data at the storage layer rather than at the engine layer. The foundation of LTAP is Lakebase, Databricks’ serverless Postgres product that now stores data directly in Unity Catalog using the same open formats (Delta and Iceberg) as the rest of the Lakehouse. It means that there is no longer any latency issue when storing the transactional data in a columnar format, as opposed to the classic row format that has been the standard for the OLTP databases. The result is that the data of your ingesting applications are stored in the same way as the data you would normally use for analytical purposes, eliminating the need for ETL and CDC pipelines. Transactional and analytical workloads scale independently, but they read from the same single copy of data. No pipelines, no replicas, no dual governance models to keep in sync.

For data teams that have spent years managing brittle ETL jobs and CDC pipelines, this is a genuinely interesting architectural shift. Databricks reports that Lakebase already handles 12 million database launches per day, which suggests meaningful production adoption. New capabilities announced at the summit include cross-cloud disaster recovery, git-style branching and snapshots for safe experimentation against production data, and autonomous database operations where agents can monitor health and propose index changes.

LTAP is coming soon as part of Lakebase. The demo was also fairly impressive. They ran the same query 3 times, each time with a different engine. The classic Postgres compute took 74 seconds to finish the query, the CDC data (so after a pipeline was run) about 13 seconds and the new way of working (with the Reyden compute engine, more on that later) in 0.08 seconds (!). A big shift for large companies and a very interesting development to keep an eye out for.

Genie One, Genie Agents and Genie Ontology: A real AI coworker for business users

In 2024, Databricks introduced Genie as a conversational analytics assistant built on top of AI/BI. This year they took a significant step forward with Genie One, evolving Genie from a standard chatbot into an actual data-smart AI coworker for business users.

The key insight behind Genie One is that the main reason enterprise AI assistants underperform is not the model itself, but the context. Business knowledge is scattered across dashboards, queries, data sources, wikis, Slack threads, and email. When AI cannot find the information it needs, it fills the gap with inference, and the result is answers that are generic or outdated at best and wrong at worst. Genie One uses connectors to get insights from across the entire data estate of your organization.

Databricks addresses this with Genie Ontology, an automatic context layer that extracts and organizes knowledge from tables, queries, dashboards, pipelines, and connected applications into a living graph of how a company actually works. Metric definitions, business terms, the relationships between concepts and teams, Genie Ontology captures all of it automatically, without requiring anyone to hand-curate a knowledge base. It even uses OntoRank, a PageRank-style approach, to determine which source to trust when multiple definitions exist, weighing the authority of the author, the freshness of the asset, and how widely it is used across the organization. Furthermore, it enforces the permissions of each source, ensuring users only see the data and assets they are explicitly authorized to access.

The benchmark numbers Databricks shared are striking. Genie answered 84.5% of real-world enterprise data analysis questions correctly on the first attempt, compared to 52.4% for the strongest general-purpose coding agent and as low as 25% for the weakest. It also delivered those answers at twice the speed while minimizing token usage and overall cost.

Genie Agents takes the concept further by letting teams turn prompts into shareable, domain-specific agents. Previously Databricks customers built “Genie Spaces” as curated, governed chat experiences scoped to specific topics. Genie Agents evolves this concept to allow agents to take autonomous action: writing to external systems, generating documents, scheduling tasks, and reasoning over both structured and unstructured data. Creating an agent is as simple as describing what you want in plain language.

To integrate Genie One better in your current organization tools, it is now embedded natively in Slack and Microsoft Teams, and comes with iOS and Android mobile apps. For organizations that already have their own AI assistants, there is a Genie MCP App that allows them to plug in Genie without changing their existing workflows. It is a practical approach to adoption that without disrupting or replacing your existing tools.

Lakehouse//RT: Millisecond query performance without a separate serving layer

For anyone who has ever had to maintain a dedicated real-time serving layer alongside their Lakehouse, Lakehouse//RT is the most practically impactful announcement of the summit.

The problem it solves is well known in the industry. As organizations push more dashboards, embedded analytics, and AI applications through their data platform, the demand for low-latency query responses grows. The traditional answer has been to copy data into a specialized serving layer like Druid, Pinot, or ClickHouse. Fast for reads, yes, but at the cost of data duplication, a second governance model to maintain, and an entire engineering team dedicated to keeping the sync from breaking.

Lakehouse//RT is a new real-time data warehouse built directly on the Lakehouse, powered by a new query engine called Reyden. The positioning is clear: remove the need for a side-stack entirely by bringing millisecond performance to the same governed copy of data your analytics and AI already use. Preview customers report up to 16x better performance compared to dedicated real-time serving layers, with response times as low as 10ms on smaller datasets and sub-100ms on larger ones. On standard benchmarks, Lakehouse//RT delivers sub-100ms latency at 12,000 queries per second.

What makes the technical story compelling is not just the raw speed, but the consistency. Real-time serving layers typically hold up for simple lookups but collapse under complex analytical queries involving joins, window functions, or large table scans. The benchmark comparisons Databricks shared show Lakehouse//RT maintaining low latency across all three stress dimensions: under high concurrency, at large data volumes, and on complex queries. Multiple competing solutions failed to complete at the largest scale factors at all.

It is not only interesting for companies with large volumes of data, but also for the companies that are not working with terabytes worth of data. The older Spark engine could sometimes be a bit of overkill for working with smaller datasets (overhead, setup). Reyden eliminates this by providing extremely fast query speeds and more tailored size. It is truly a competitor for DuckDB or Polars, which were up until now more tailored for smaller datasets.

This was exemplified by a demo they gave. A query was run on the NYC taxi dataset: with the classic SQL compute, it took about 1 second to complete, with the new Reyden engine; the results came back after 0.007 seconds. If you add up all those little query runtimes, you are looking at a spectacular decrease of runtime, which in itself decreases the costs you will pay.

On the pricing and operations side, Lakehouse//RT removes the cluster-sizing exercise entirely. An auto-sizing feature determines the right baseline compute automatically, and incremental autoscaling adds or removes individual nodes as load changes rather than doubling capacity in one expensive step. For anyone running a separate serving layer today, this is worth evaluating now.

Lakeflow: Agentic data engineering is here

Lakeflow was already introduced at the 2024 summit as Databricks’ answer to unified data pipeline management, offering a platform from ingestion to transformation and orchestration integrated with Unity Catalog. This year’s update represents a massive architectural leap, signaling a complete shift toward agentic development and operations across the entire data lifecycle.

Genie Code is now deeply embedded within the Lakeflow environment. Rather than operating as an isolated autocomplete tool, it features end-to-end awareness of your ingestion setups, transformation logic, and orchestration dependencies simultaneously. You can leverage it to spin up custom ingestion connectors, author complex Python or SQL pipelines, and configure jobs with precise triggers.

To help bridge the gap between engineering teams and business stakeholders, Databricks announced the general availability of Lakeflow Designer. It is a visual, no-code interface where business analysts can build ETL pipelines using a drag-and-drop canvas and natural language prompts. Because the interface builds directly on Spark Declarative Pipelines, there is no translation layer or messy handoff code. Engineers can jump right into the canvas to review, refine, and optimize the underlying code without switching contexts.

On the ingestion side, Lakeflow Connect now covers more than 100 native managed connectors. New additions of particular note include Jira, GitHub, Confluence, SharePoint, Google Drive, Outlook, Meta Ads, TikTok Ads, Google Ads, and HubSpot.

For massive, high-volume event streams, Zerobus Ingest continues to scale out its capabilities. A new Kafka-Compatible API allows existing Kafka producers to route data straight to Databricks via a simple configuration change. It delivers near real-time data ingestion in less than 5 seconds with a high throughput of up to 100 MB/s (and over 10 GB/s table throughput) and highly concurrent workloads. This architecture seamlessly handles thousands of concurrent clients writing to the same table simultaneously without compromising performance or reliability.

Meanwhile, workflow management sees a boost as Lakeflow Jobs expands its footprint outside the boundaries of Databricks, boasting over 40 external integrations. Powered by an open operator framework, teams can now seamlessly trigger remote Snowflake jobs or manage downstream Slack notifications within a single DAG.

Genie ZeroOps is the operational counterpart to Genie Code. It reduces the time data engineers spend maintaining pipelines and fighting fires, freeing up more time to focus on building new pipelines. Genie ZeroOps is a background AI agent that monitors data and AI assets in production, detects failures, performs root-cause analysis using data quality metrics, error logs, and Unity Catalog lineage, and then generates proposed fixes that are validated in a sandboxed environment before a human approves them. The agent validates its proposed fixes by creating a shallow clone inside an isolated sandbox. To guarantee safety, it applies strict network isolation and runtime guardrails, while strictly obeying Unity Catalog permissions to ensure it only interacts with data you have access to. The human-in-the-loop model here is a sensible design choice: the agent does the diagnostic heavy lifting, but a person still decides when to apply the fix. If you give Genie ZeroOps permissions, it can also scan tables to detect PII and look where it is unwillingly exposed and give a Unity Catalog proposed fix that you can apply.

Conclusion

The common thread running through all of this year’s announcements is the elimination of architectural tax. LTAP kills the ETL pipeline between transactional and analytical systems. Lakehouse//RT kills the need for a separate real-time serving layer. Genie Ontology eliminates the manual knowledge curation that made enterprise AI assistants unreliable. Lakeflow ZeroOps reduces the operational overhead of running data pipelines in production. Databricks is making a clear bet that the complexity enterprises have accepted as normal is not inevitable, and that AI agents need a much cleaner foundation to do useful work. We believe that the new announcements could be a big game changer, leading to even more companies adopting Databricks to further democratize their data estate, and, more importantly, enable companies to quickly get the info they need.

This blog post only covered the initial keynote, but there were a lot more interesting sessions. They will become available online, and you can find more info on the official Data + AI Summit website. And for any questions about how these announcements affect your Databricks environment or roadmap, feel free to reach out to us.

Sources:

LTAP press release: https://www.databricks.com/company/newsroom/press-releases/databricks-launches-ltap-first-lake-transactionalanalytical

Genie One blog: https://www.databricks.com/blog/introducing-genie-one-genie-ontology-and-genie-agents

Lakehouse//RT blog: https://www.databricks.com/blog/introducing-lakehousert-real-time-performance-unified-lakehouse

Lakeflow blog: https://www.databricks.com/blog/lakeflow-new-era-agentic-data-engineering

Written by