Building a Databricks Environment on Azure: A step-by-step practical guide

1.    Why Databricks On Azure

Databricks has become the go-to data platform for a lot of companies of different sizes and different industries. The evolution from a managed-Spark platform to a fully fledged platform with Governance, AI, ML, BI solutions is nothing short of remarkable. This whitepaper aims to be a guide for novel users, aiding in the creation of a Databricks workspace on Azure.

What Readers will accomplish:

• Setup your own Databricks workspace on Azure
• Make connection to a Unity Catalog Metastore
• Make a secure connection to your Azure Storage Accounts
• Kickstart your development with some sample code

Prerequisites

• An Azure Subscription* + ability to create Resources
• Access to the Databricks Admin Console (Azure Databricks account admin in Azure)

*Start your free trial for Azure by signing up here

2.   Key Concepts and Components

Before diving into the implementation steps, it’s essential to understand the fundamental building blocks of the Azure Databricks platform:

Azure serves as the foundational cloud platform on which your Databricks environment operates. It provides the infrastructure, networking, storage, and security services that support Databricks deployments. Key Azure components such as Resource Groups, Virtual Networks, and Storage Accounts play critical roles in organizing and securing your Databricks resources. When you create an Azure Databricks workspace, it resides within a specific subscription and region, inheriting Azure’s governance and compliance features. Properly structuring your Azure resources — for example, aligning them with your organization’s management groups and policies — ensures scalability, cost control, and compliance from the very start of your Databricks journey. A lot of the concepts of this whitepaper are closely tied to Azure, which is the Cloud Platform of Microsoft, but a lot of the design choices for Databricks itself are portable to other Cloud Platforms like AWS (by Amazon) or GCP (by Google).

Workspaces are where you will operate most of the time in your Databricks environment. A workspace is where your teams collaborate, housing notebooks, libraries, dashboards, and experiments. Each workspace is linked to an Azure Databricks resource in your Azure subscription and comes with its own URL for access. Organizations typically create separate workspaces for different teams, projects, or environments (development, staging, production) to maintain appropriate isolation and governance.

Unity Catalog represents a paradigm shift in how organizations manage data governance in Databricks. Introduced as the unified governance solution for all data and AI assets, Unity Catalog provides centralized access control, auditing, lineage, and data discovery across all your workspaces. Unlike traditional metastores that were workspace-specific, Unity Catalog operates at the account level, enabling consistent governance across your entire Databricks deployment. Its three-level namespace (catalog.schema.table) provides intuitive organization while supporting fine-grained access controls at every level. Unity Catalog is not just a metadata repository; it’s a comprehensive governance layer that tracks data lineage from source to consumption, enforces attribute-based access controls, and provides audit logs for compliance requirements.

Clusters are the computational engines that power your Databricks workloads. A cluster is a set of virtual machines that work together to execute your code in parallel. Databricks offers two primary cluster types: all-purpose clusters for interactive development and exploration, and job clusters that are created to run specific automated tasks and terminate upon completion. Understanding when to use each type is crucial for cost optimization. Clusters can be configured with specific Databricks Runtime versions, which bundle Apache Spark with optimized libraries and performance enhancements. Since 2024, you can also opt for the Serverless cluster. This ready-to-go cluster is provisioned by Databricks itself. Key benefits of the Serverless Compute are minimal start-up times and appropriate scaling, without the need to tweak the different settings on your cluster.

Notebooks provide the interactive development environment where data engineers, analysts, and scientists write and execute their code. Databricks notebooks support multiple languages (Python, SQL, Scala, and R) and allow you to mix languages within a single notebook using magic commands. Beyond simple code execution, notebooks support rich visualizations, collaborative commenting, version control integration, and can be scheduled as production jobs. They serve as both development tools and living documentation for your data workflows.

Delta Lake forms the storage foundation for the lakehouse architecture that Databricks champions. Built on top of standard data formats like Parquet, Delta Lake adds ACID transactions, schema enforcement, time travel capabilities, and efficient upserts to your data lake. Every table you create in Unity Catalog is, by default, a Delta table, providing reliability guarantees that were traditionally only available in data warehouses. This technology enables you to build both streaming and batch processing pipelines on the same data with consistent results.

Metastore in the Unity Catalog context refers to the top-level container for metadata about your data assets. A single metastore can be assigned to multiple workspaces, enabling data sharing and consistent governance across teams. The metastore contains catalogs, which in turn contain schemas (databases), which contain tables, views, and functions. This hierarchy provides flexibility in organizing your data while maintaining clear ownership and access boundaries.

Architecture Overview

The Azure Databricks architecture follows a control plane and data plane model. The control plane, managed by Databricks in the Azure cloud, handles workspace management, notebook storage, cluster configuration, and job scheduling. The data plane, which runs in your Azure subscription, contains the actual compute clusters that process your data. This separation ensures that your data never leaves your Azure environment while benefiting from Databricks’ managed control plane services.

When you create an Azure Databricks workspace, Azure automatically provisions a managed resource group containing the virtual networks, network security groups, and storage accounts required for cluster operation. Your compute clusters run as Azure Virtual Machines within this infrastructure, accessing data stored in your Azure Data Lake Storage or other Azure data services.

Unity Catalog adds another layer to this architecture. The Unity Catalog metastore exists at the Azure Databricks account level (above individual workspace) and stores metadata about your data assets, access policies, and lineage information. When users query data through Unity Catalog, the system checks their permissions, logs the access for auditing, and tracks lineage automatically, all while the actual data processing happens on your compute clusters in the data plane.

The connection to Azure Storage is established through Unity Catalog’s external locations and storage credentials. Rather than hardcoding access credentials in your notebooks or mounting storage with legacy approaches, Unity Catalog enables you to define reusable storage credentials (like service principals or managed identities) and external locations that reference specific storage paths. This approach separates the concern of authentication from data access, making your code more secure, portable, and maintainable.

The Modern Databricks Approach

This whitepaper focuses on the modern, Unity Catalog-first approach to Azure Databricks development. While legacy patterns like mounting storage accounts directly to workspaces are still possible, Unity Catalog represents the recommended path forward for several compelling reasons:

First, it provides centralized governance that scales with your organization. Rather than managing permissions separately in each workspace and storage account, Unity Catalog gives you a single pane of glass for access control. Second, it enables secure data sharing between teams and workspaces without duplicating data or managing complex permission structures. Third, it provides automatic lineage tracking that shows how data flows through your organization, which is invaluable for debugging, compliance, and impact analysis.

By following the Unity Catalog approach from the beginning, you’ll build on a solid foundation that supports growth, maintains security, and simplifies governance as your data platform evolves.

3.   Azure Considerations and Prerequisites

Having the correct permissions in Azure is key to a swift and joyful implementation of Databricks. It’s not just clicking the “Create Workspace” button, there are a lot of architectural choices to consider when deploying. This section will guide you through the process of setting everything right in your Azure account.

Azure Subscription Requirements

A key consideration that often gets overlooked when deploying a Databricks workspace is that you should have sufficient quota on your subscription. If you choose to run the standard Compute clusters, Databricks will spin up Azure VMs in your own subscription. It is thus important to check that the quota you currently have are sufficient. Standard deployments typically use D-series or F-series VMs for general-purpose workloads, while memory-intensive jobs may require E-series VMs. If you anticipate running large clusters or multiple concurrent workloads, verify your vCPU quota ahead of time through the Azure Portal’s “Usage + quotas” section. Requesting quota increases can take several business days, so address this early.

It’s also important to take into consideration the cost pattern of Databricks. This consists out of two components: the underlying Azure infrastructure costs (VMs, storage, networking) and the Databricks Units (DBUs), which represent the processing capability of your clusters.

Another choice you need to make is whether to use a Standard or Premium tier workspace. The Standard workspace gives you access to a managed Spark compute, but lacks all the capabilities of Unity Catalog and most of the new products released by Databricks. They also communicated that the Standard Tier workspace is end-of-life and all remaining workspaces must be converted to Premium by October 1, 2026. For the rest of this guide, we will be looking at the Premium Tier workspace.

If a user wants to create a workspace on Azure, they should have at least a Contributor role on the resource group where the workspace will be deployed.

Unity Catalog Setup

To create a metastore for Unity Catalog, the user must be an Account Admin, with access to the Account Admin console. Databricks on Azure has a two-level administrative hierarchy: one for the account which sits above all workspaces, and one on workspace level.

Networking Considerations

Azure Databricks offer 2 primary ways to implement networking for your workspace.

Standard Deployment: Databricks manages all the virtual network infrastructure within a (automatically created) managed resource group. This is the simplest approach and recommend if you are currently at the beginning of your cloud journey. Clusters can access other Azure services through Azure Private Link or service endpoints.

VNet Injection: this allows you to deploy Databricks in your own Azure Virtual Network, giving you complete control over the implementation of your networking. Discussing this is beyond the scope of this guide. This set-up is often preferred by clients in strict security frameworks.

Storage Strategy and Data Lake Planning

If you are already on Azure, chances are big you are using Azure Data Lake Storage Gen 2 for storing all your data. It is a good idea to think about how to organize your datalake storage. The organization of your Storage Accounts can also be reflected in Unity Catalog, so the better this is from the start, the more user friendly your Databricks Platform will become across your organization. Some common patterns include:

Environment-based: Separate storage accounts for development, staging and production. This provides clear boundaries and simplifies access control

Domain-based: Separate based on your business domains. Also helps to set clear boundaries and scales very well.

A hybrid approach is of course also a very good possibility. It’s key to check which setup is currently available at your company and verify the integration possibilities with your Databricks workspace.

4.   Step 1: Creating your Azure Databricks Workspace

Login to the Azure Portal and search for the Databricks resource. Click create and you are prompted to the following screen:

Select the subscription on which you have sufficient privileges and select the Resource group of your choice. You can also choose to create a new one.

Next, give your workspace an appropriate name. For this guide, I chose a generic name, it could be the case that you separate your development/staging/production workspaces. It is then a good idea to add a tag like ‘DEV’ or ‘PRD’ to the name of your workspace, just to make things as clear as possible.

Finally, select the Region, Pricing Tier and give a name to the Resource Group that will automatically be created by Databricks to handle the infrastructure needed to run Databricks on your Azure account. Azure deploys the actual compute and networking components into this separate managed resource group.

After the deployment has been done successfully, you can check out your newly created workspace by going to the resource in Azure and clicking the URL in the top right

5.   Step 2: Unity Catalog Setup

Unity Catalog is an unmissable part of your Databricks setup. It is the one stop shop for all your governance, lineage, auditing and general overview of your data estate and data products. Every workspace must be assigned to a metastore, while a metastore can be linked to multiple workspaces. Currently, you only have the ability to create 1 metastore per region.

Go to the Databricks Admin Account Console and look for the ‘Catalog’ tab.

When you click “Create metastore”, a window will pop up asking you for the following settings:

• Name: the name of your metastore
• Region: where you want to deploy your metastore
• ADLS Gen 2 path: the path to a storage account to store all managed tables created in the metastore. Although it is optional, it is considered a best practice to provide some storage on metastore level for the following reasons:
  • Managed tables will remain in your Azure environment
  • Data will persist even if the metastore is deleted
  • You maintain ownership over your own data and have enough visibility
• Access connector ID
  • This is the managed identity you will use to access the storage account
  • You can find this in the Resource ID field once you have created the Managed Identity

Creating a Managed Identity

Managed identities are the best way to manage connections. An important benefit to using Managed Identities as opposed to Service Principals is that you don’t need to maintain credentials or rotate secrets. You can use them for the initial setup to connect the metastore to your storage account, but also later on to register external locations in Unity Catalog, but more on that later.

To create your managed identity for Databricks, search on the Azure portal for ‘Access Connector for Azure Databricks’.

You will see that the creation of your Databricks workspace already has created a connector for you, in the managed resource group that you had to specify during the creation process.

As an example, we will create a new one.

Click the create button and the next screen will appear

On the next page, you can select the System assigned managed identity. Click Review + Create and as easy as that, you have created your Managed Identity

The next step is assigning the correct privileges to your managed identity on the resources it needs to access. For this example, we will have a look at our storage account which we will use for our workspace. The managed identity needs a Storage Blob Data Contributor role to be able to read and write data to the storage account. You can also opt for a more limited role, and set the permissions on Container level, but since we will use this connector later on, we will set the permissions on the whole storage account.

Go to the Storage Account of your liking and go to Acces Control (IAM) and click “Add Role Assignment”. Note: you should be the owner of the Storage Account in order to give new roles to Managed Identities or people.

Select the Storage Blob Data Contributor role and select Managed Identities. On the right, a new tab will open. There you can select the Access Connector for Azure Databricks, and you should see your previously created managed identity pop up there.

Review and assign and your managed identity now has access to your Storage account.

Once you did that, it’s time to go back to the Unity Catalog Metastore page and fill in the correct details to make a connection.

On the next page, you can assign your workspace to your newly created metastore.

Testing the connection to Unity Catalog

A lot of the hard work is done now. By now, you should have a connection between Databricks and your Storage Account using the Managed Identity, you should have a metastore linked to your workspace and you should have access to your workspace in the first place. To check if all these things are true, you can go to the Query Editor to try and create a new catalog and schema. If you can execute these commands successfully, the link between your workspace and metastore is correctly provisioned.

Below you can find an example query to check the connection. The result of these queries should be ‘OK’ and you should see your catalog and schema appear in the Catalog Explorer.

A final test for your Unity Catalog is to check whether managed tables are actually being saved to the location you specified when creating the metastore. You can run following notebook code to read and write some sample data from Databricks to your Unity Catalog (and connected Storage Account)

Once you go to the Storage Account, you will see a Delta table that has been created under a somewhat peculiar path.

This verifies that Databricks was able to write the data to the correct storage account, and that Databricks itself looked after the path on the storage account, resulting in a Managed table. This path also shows one of the most important downsides of Managed tables, in the sense that you have no control over the path where it is stored, making it hard to have a good overview of the structure of your Storage Account. In the following chapter, we will see how you can manage this yourself by creating an External Location and adding a specific path to your write statements, making it clearer where you store your data.

6.   Step 3: Connecting to Azure Storage

Connecting Azure Databricks to your Azure Storage accounts is a critical step that enables your data platform to access raw data sources, persist processed results, and share data across teams. In the Unity Catalog paradigm, this connection is established through a secure, governed approach that separates authentication from data access, eliminates credential sprawl, and provides fine-grained access control. This section will guide you through the modern approach to Azure Storage connectivity using Unity Catalog’s storage credentials and external locations.

Implementation in Unity Catalog

Before Unity Catalog, connection to your Azure Storage Accounts had to be done using one of the following legacy approaches:

• Mounting storage with embedded credentials in
• Configuring cluster-level credentials
• Using account keys or SAS tokens

All these approaches are tedious and create security risks, governance challenges and operational complexity (changing when credentials expire,…) Unity Catalog revolutionizes this by introducing two key concepts that work together to provide secure, governed storage access:

Storage Credentials: Define how to authenticate to Azure Storage (the authentication mechanism)

External Locations: Define where data resides in storage and which storage credential to use (the authorization layer)

This leads to several advantages:

• Credentials are never exposed in the code
• Access can be granted at different granularities
• Access is audited through UC
• Permissions can be delegated: you can grant access to creating tables, without needing to give them access to the storage account itself

Storage Credentials

The recommended way of authenticating to Azure storage is by making use of Managed Identities, the same way as we saw before with the Unity Catalog implementation. Some advantages of the Managed Identity are:

• No credentials to manage: no secrets, keys or passwords
• Automatic credential rotation: Azure handles all lifecycle management
• Reduced security risk: no possibility of credential leak in code or logs
• Simplified operations: no expiration dates to track or renewal processes to implement

Of course, this implementation works optimal if you follow the best practices

• Use separate Access Connectors for different environments/workspaces
• Apply least-privilege RBAC roles

To create a Storage Credential in the Databricks UI, go to the Catalog Explorer and search for the “Create a credential” tab.

Once you are there, a new tab will open in which you can specify the details of your Storage credential:

• Credential Type: we will chose the Azure Managed Identity as per best practice
• Credential Name: choose a distinct name, as it will be used later on to
• The Access Connector ID: the Resource ID of your connector
• Comment: some more info on your Storage Credential

You can also do this in SQL with the following query:

CREATE STORAGE CREDENTIAL `databricksguide-credential` WITH DATABRICKS_MANAGED_IDENTITY DATABRICKS_AZURE_RESOURCE_ID ‘<access-connector-resource-id>’ COMMENT Credential to connect to the databricksguide storage account;

External Locations

To create external locations, there are some prerequisites. You must have the Storage Credential already created, the storage account and container must already exist, and the credential must have the appropriate privileges on the storage path. To configure the external location in your Databricks workspace, go to the Catalog Explorer again and choose Create an external location

You again will be prompted to fill in the details of your external location:

• External Location name
• Storage Type: for now, we will select the Azure Data Lake Storage
• URL: the url of your container, the format will be “abfss://<container_name>@<storage_account_name>.dfs.core.windows.net”
• Storage Credential: select the Storage Credential you just created before
• Comment: give some extra info on your external location

Of course, an external location is of no use for your users if they do not have the right privileges. Luckily, you can easily set them using SQL in the following format:

GRANT <PRIVILEGE> ON EXTERNAL LOCATION <EXTERNAL LOCATION NAME>

TO ‘USERS OR GROUPS’

The privilege can be one of the following:

• CREATE EXTERNAL TABLE: allows creating external tables on the specified location
• READ FILES: allows reading files directly from the external location
• WRITE FILES: allows writing files directly to the external location

Permissions only need to be set on this External Location, not on the Managed Identity or the Storage Account directly.

Once everything is set up correctly, you can start to create External Tables in your Unity Catalog, based on data in your Azure Storage Account. Compared to the Managed Tables we saw before, this approach allows more control over your data. It gives you the ability to create a clear folder structure, dropping the table does not drop the underlying files itself. So currently, this is the way to go when ingesting data and creating tables in your Unity Catalog. You still get all the benefits of registering your table in UC, but still have the full control over the underlying data.

7.   Testing your setup

For testing purposes, we will develop a notebook that checks if everything has been set up correctly. We will read in a csv file from the storage account that we registered as an external location. After some random modifications, we will save it again on the storage account, but at the same time we will register it as an External Table in Unity Catalog.

You can just use a path to your code, no need to incorporate any credentials. This is handled by the external location and storage credential we just set up. If you see the data, you have successfully set up your external location.

base_path = abfss://<container_name>@<storage_account_name>.dfs.core.windows.net/"
base_path = "abfss://data@databricksguide.dfs.core.windows.net/"
path_source = base_path + "movie/imdb_top_1000.csv"
df_movie = spark.read.option("header", True).csv(path_source)
display(df_movie)

We will do some small transformations and write it using a custom function which incorporates both the path on the storage account as well as the structure we want in unity catalog, and ties them together by reusing the catalog, schema and table name in the path on the storage account. You can find the function as an annex to this whitepaper.

import pyspark.sql.functions as F
df_best_movies = (df_movie.withColumn("IMDB_Rating",F.col("IMDB_Rating").cast("double"))
                          .filter(F.col("IMDB_Rating")>8.5))
write_to_unity_catalog(df=df_best_movies, 
                       catalog = "movie_catalog",
                       schema = "movie_schema",
                       table_name = "best_movies",
                       base_path = “abfss://<container_name>@<storage_account_name>.dfs.core.windows.net/")

Once that ran successfully, we can head over to the Catalog Explorer to see our newly created table. If we look at the details, we can indeed see that it is saved on the storage account on the path of our liking, making use of the catalog, schema and table name to create the proper structure.

8.   Conclusion

It can be hard to find your way around the implementation guides of Microsoft and Databricks to properly set up your environment. This guide aims to piece everything together, from creating resources in Azure to making the correct connections between the resources and more specifically your Databricks Workspace and your Azure Storage, which is of course key to set up a data platform in the correct way.

In conclusion, implementing Databricks on Azure following modern Unity Catalog standards empowers organizations to manage their data efficiently and securely. By leveraging managed identities, external locations, and proper workspace configuration, teams can build a scalable, compliant, and future-proof data environment. This whitepaper should serve as a foundational reference for teams embarking on their Databricks journey.

References

https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/azure-managed-identities

https://learn.microsoft.com/en-us/azure/databricks

Write Function

def write_to_unity_catalog(df, catalog, schema, table_name, base_path):
    """
    Write a Spark DataFrame to Unity Catalog as a managed or external table.

    This function:
    1. Ensures that the specified catalog and schema exist.
    2. Drops the table if it already exists.
    3. Writes the given DataFrame as a table registered in Unity Catalog.
    4. Saves the data at the specified storage path.

    Parameters
    ----------
    df : pyspark.sql.DataFrame
        The DataFrame to write to Unity Catalog.
    catalog : str
        The Unity Catalog name (top-level namespace).
    schema : str
        The schema (database) name inside the catalog.
    table_name : str
        The target table name to create in Unity Catalog.
    base_path : str
        The base storage path where table data should be stored.

    Returns
    -------
    str
        A confirmation message with the table path and Unity Catalog reference.

    Example
    -------
    >>> write_to_unity_catalog(
    ...     df=df_movies,
    ...     catalog="main",
    ...     schema="analytics",
    ...     table_name="best_movies",
    ...     base_path="/mnt/delta/"
    ... )
    'DataFrame written to /mnt/delta/main/analytics/best_movies and registered in Unity Catalog as main.analytics.best_movies'
    """
    # Ensure catalog and schema exist
    spark.sql(f"CREATE CATALOG IF NOT EXISTS {catalog}")
    spark.sql(f"CREATE SCHEMA IF NOT EXISTS {catalog}.{schema}")

    # Drop existing table if it exists
    full_table_name = f"{catalog}.{schema}.{table_name}"
    spark.sql(f"DROP TABLE IF EXISTS {full_table_name}")

    # Construct target path safely
    if not base_path.endswith("/"):
        base_path += "/"
    target_path = f"{base_path}{catalog}/{schema}/{table_name}"

    # Write DataFrame to Unity Catalog
    df.write.saveAsTable(full_table_name, path=target_path)

    msg = f"DataFrame written to {target_path} and registered in Unity Catalog as {full_table_name}"
    print(msg)
    return msg

---