Databricks Lakebridge: what is it and how to use it to migrate your legacy data warehouse?

Databricks Lakebridge is a free and open data migration tool to migrate legacy data warehouses to Databricks SQL. Databricks announced Lakebridge in June of 2025. It provides support in the migration process, including profiling, assessment, SQL conversion, validation and reconciliation.

Many companies still use various data warehouses that have been created over the years. Often there is a lack of documentation, and it is unclear which tables and views are really in use and what business logic is applied. For one of our customers, the Intellus Group facilitated the migration from their warehouse to Databricks. In this blog post we will describe in detail what Lakebridge is, the benefits it can offer, the key components and the different steps we have undertaken to make the migration successful.

What is Databricks Lakebridge?

Databricks describes Lakebridge as an AI-power tool for fast, predictable migrations to modernize your data warehouse. It is a free tool and provides support in three processes: assessment, transpile and reconcile. These processes will be described in more detail below. The main goal of Lakebridge is to make the migration from a data warehouse to Databricks much faster. There could be many reasons why companies would want to move from a traditional data warehouse towards Databricks, a few reasons are cost efficiencies, integrated security and data management, improvements on scalability and performance and AI & ML capabilities. And Lakebridge is one way to achieve this goal.

Without Lakebridge, a migration of such complexity takes a lot of time. All the data and processes in the data warehouse must be analysed, and data engineers had to understand the ETL process to then start migrating this logic manually. Afterwards, the actual migration can start. This can take many months to complete. All the SQL queries would have to be migrated towards Databricks SQL which is different from the legacy data warehouse. Databricks Lakebridge fills this gap by allowing the migration to be faster and more reliable.

Databricks says that Lakebridge can help with achieving automating 80% of these otherwise manual tasks which simplifies migration drastically. This decreases the delivery time of the project and reduces costs in many ways. Lakebridge cannot take away all manual tasks and a thorough analysis and execution by data engineers still need to take place. In the project we have completed it saved a lot of hours of writing SQL code.

At the time of writing, Lakebridge supports the following systems that it can analyse the data from. Databricks announced that in the future, more technologies will be supported.

We used Lakebridge because we had to do a migration from an Azure Microsoft SQL server for one of our clients. The  system was created many years ago and over time, had been further developed resulting in an explosion of tables, views and stored procedures. The first task was to analyse the system together with the business users and clearly understand what this warehouse is used for and what business logic was applied in the ETL.

To start with Lakebridge and before converting your queries to Databricks supported SQL, Lakebridge needs to be installed. After installation, we exported all the SQL files from the data warehouse that needed to be migrated as SQL source files. It is possible to make a direct connection to your data warehouse, however this is not required for the analyzer function to work. The connection to the data warehouse can be used for the reconcile function. This function reads source data and compares it to the new Databricks tables. The different functions of Lakebridge are explained later in this blog post in more detail.

The analyzer function needs a local directory path to where the SQL scripts are stored. This made it easier to create different sub-folders and folder hierarchies to maintain an overview of the different tables, views and stored procedures that need to be migrated. After the export of the SQL files, a new project in an Azure Devops repository was created and linked to a Databricks workspace. In Azure DevOps, all the raw files were in a folder. After connecting this repository to a local visual studio code, the necessary installation steps can be executed. After completing the pre-requisites, the following CLI command can be run:

databricks auth login

Now you will be asked to give a Databricks profile name and will be redirected to the Databricks environment. To install Lakebridge, simply run:

databricks labs install lakebridge

Databricks Lakebridge is now installed and active. To verify it is correctly installed, run the CLI command:

databricks labs lakebridge –help

Analyzer

The first of three steps of Databricks Lakebridge is the Analyzer function. This is an optional step but definitely recommended. It performs a detailed assessment of the legacy data warehouse. It does so by going through all the SQL code and metadata. Output can be produced by running the following command:

databricks labs lakebridge analyze

In this output, Lakebridge will provide an overview of the complexity of the ETL and SQL jobs that are built. At first, the full path to the source directory needs to be specified and the report name that will be exported. Finally, a source technology needs to be chosen. There are 34 options:

Lakebridge then produces an Excel file with various sheets going from the SQL script categories, the number of loops & cursors being used, the number of functions and the complexity of the SQL queries, etc. The summary pages give an immediate overview of the number of scripts, tables, views, lines of code, procedures and many more that are present. It also gives an indication of the complexity of the SQL queries. The analyzer that we ran completed in no more than 9 minutes and 29 seconds for more than 3500 SQL files! The analyzer can give you some insights into the difficulty of the migration that is being taken on and can be used for budgeting the costs and time of the project. It can help to identify possible bottlenecks and challenges earlier.

Transpile

Transpile is by far the most important and powerful step of Databricks Lakebridge. When starting this, Lakebridge will automatically convert all the original SQL files from an input source to an output target. With the output target being Spark SQL. These SQL files can then be used in Databricks jobs and workflows to trigger the ETL process. At first this command needs to run:

databricks labs lakebridge install–transpile

There are continuously new updates available for Lakebridge so it is very likely that Lakebridge will ask you to update. Just do so by doing:

databricks labs upgrade lakebridge

To start the conversion of the original SQL files to Spark SQL, run the following:

databricks labs lakebridge transpile

— source-dialect mssql

— input source C:UsersUserNameyour_project_nameyourinputlocationfiles

— output-folder C:UsersUserNameyour_project_nameyouroutputlocationfiles

— error-file-path C:UsersUserNameyour_project_nameerror_file_path.log

Change the source dialect accordingly to your source. The input folder in our case were all the raw files from the data warehouse that we created earlier. More options for the transpile function can be found when running databricks labs lakebridge transpile –help. A screenshot can be found below.A suggestion is to first run the transpile command on several files and validate the output before transpiling all the SQL files. Lakebridge remains an AI tool and can still make mistakes. It is likely that it does not transpile certain SQL syntax correctly. We encountered this multiple times and it is important to identify these cases early. If so, config files can be provided to the transpile CLI command to solve these problems.

Reconcile

The final function Databricks Lakebridge provides is Reconcile. Databricks describes it as an automated tool designed to streamline the reconciliation process between source data and target data. At the time of writing, it currently supports Snowflake, Oracle and other Databricks tables as input source. Here a connection to the data warehouse can be made. This function does not convert SQL files to Databricks, it reads source data and compares it to the newly created Databricks tables. Different validations between these systems are then executed. Reconcile provides four options where data between the source data warehouse and Databricks can be compared. The first one is the row option, which points out the rows missing or being different in either the target or the source. It does so by generating a hash for each row in the source and target tables and then comparing them. Then there are schema comparisons to make sure no columns are being forgotten or the datatypes still align. Thirdly there is the data option. It has the same features as the row option but adds a check for mismatched data, which is sample data with mismatches captured at each column and row level. It can also identify mismatched columns which holds a consolidated list of columns that has mismatches in them. It therefore operates at both the row and column level. It uses the join columns in the table, which are the primary keys to compare rows between source and target and then compares each column. This is different from the row function as the join columns are not necessary there to run this check. Finally, there is an option to combine the data and schema options.

Closing

Databricks Lakebridge has truly helped the migration of a data warehouse to Databricks much faster. It is an incredibly powerful tool that works relatively easy once everything is configured. The speed and easiness in which it transpiled the SQL files was much faster than doing it manually. This resulted in a lot of cost and time savings. It is worth pointing out that a good analysis always needs to be executed first before starting the actual migration. Lakebridge can still make mistakes so definitely validate a subset of files first before transpiling everything. It is worth to identify these mistakes early and test and fix, rather than reworking the whole process after everything is transpiled. The transpile function can be run multiple times.

In general, this tool is recommended and useful, especially if you already know the logic and processes in your data warehouse well. More information about Lakebridge can be found on the Databricks website. A detailed guideline on the different CLI command for the three functions can be found here.