Why you should start using Unity Catalog 

We noticed that there are still lots of organizations that use Databricks but do not leverage Unity Catalog. Hopefully, this blog can show the added value and give these organizations an extra boost. Firstly, we briefly explain the basis of Unity Catalog. Secondly, some examples are given to increase data governance in Databricks by using simple SQL queries. Thirdly, we display how lineage makes it possible to visually retrace your data to the source. Fourthly, the need for data exploration is stressed, enhancing the searchability of your data. In addition, by limiting the amount of searchable data, it is shown how Unity Catalog can help facilitate self-service. Lastly, we deep dive into one of the most recent features called Delta Sharing, which makes data sharing across organizations a walk in the park. 

Unity Catalog 

If you are already familiar with Unity Catalog this part can be skipped. However, we could not deep dive into some features without covering the basics. The Unity Catalog metastore consists of three levels, as depicted in Figure 1 below. The metastore is the top container for metadata. Typically, when you start a Databricks workspace for the first time in an Azure region, a metastore is automatically created. 

Figure 1: Unity Catalog schema 

The catalogs are situated on the first level. This is the top level and represents typically an organizational unit. Next, we have schemas as a second level. These are also called databases, which contain tables, views, volumes, functions and AI models. Schemas typically represent a project or a layer of the medallion architecture. The third and last level shows references to all your data products which can be secured with fine grained SQL access. 

Increase governance 

An important part of increasing governance is having a clear view on the security of all your data assets. It is especially important to know which data is sensitive and even more important to know who has access. Based on the Unity Catalog hierarchy you can easily give access to each level, even to the metastore itself. The access is also hierarchical, receiving access to a catalogue gives access to all its underlying schemas. In Figure 2, an example shows how specific access can be given to a specific schema with SQL. Apart from SQL you can also use a click-wise approach with catalog explorer, the Databricks CLI or REST APIs to set permissions. 

Figure 2: SQL script to grant hr specific schema and table access 

Best practice is to almost never give access to individual people in your organization. Making use of strictly defined security groups that dynamically change when people arrive or leave the company keeps the access dynamic and clean. It also pushes the access responsibility to each business unit individually. 

In addition to granting access to metastores, catalogs, schemas, and tables you can also set security on row level. In Figure 3 we show an example which limits the access of the European human resources department to only the European salary data. 

Figure 3: SQL script to limit access of ‘hr_europe’ to the European region 

But what about column level security? Well, even this is easily possible in Unity Catalog. To be complete we also show an example of this in Figure 4. In this SQL code a function is created to mask the ‘date_of_birth’ column for everyone except the hr department. Next, an employee table is created where the mask is applied upon. 

Figure 4: SQL code to mask the birthdate for everyone except the hr department 

Lastly, it is important to mention that Unity Catalog tracks an audit log of actions performed against the metastore. This makes it possible for admins to verify who accessed a specific dataset and what actions they took. You can also check all the access that was historically given to your data catolog via an easy SQL statement: SHOW GRANTS ON CATALOG my_catalog_name. 

Lineage 

Tracing back your pipeline to look for an error can quickly become a tedious task. What if there was a simple solution to this problem? Let us introduce the lineage option in Databricks! Lineage creates a beautiful trace of where your data comes from in a visual way. In Figure 5 a lineage example is depicted. Here you can clearly see that the production sales model uses data from three main datasets: Audi worldwide sales, Honda UK sales and BMW worldwide sales. 

Figure 5: Lineage example from Databricks 

This makes it possible for even non-technical users to derive the source data from tables used in reporting tools. You may wonder how Databricks creates these lineage visualizations from scratch. This is done by scanning all underlying spark code written in the notebooks to identify which tables depend on each other. The lineage graphs also follow the same permission model of Unity Catalog. Meaning that a user cannot check the lineage of a table if they do not have BROWSE or SELECT permissions on that table. 

Data exploration 

Data can be scattered throughout an organization, with numerous data teams working on numerous projects. For people who need to be onboarded on a team, or people from the business who want to start exploring data, it can be a big block on the road to knowledge. With Unity Catalog, all the assets are categorized, allowing for quick access to the data, and a quick automatic overview of where the data is being used. A user can check which tables make up a certain fact table (e.g., sales), a user can check which dimensions are frequently joined on this fact. All from an intuitive UI, which also allows for quick queries to further explore the data. 

All these features should accelerate the onboarding of new people on data teams, reduce the issues of a scattered data platform and allow for a vaster collaboration and a quicker development cycle. Mainly because little time is lost in explaining and establishing the current landscape and the links between the different data products.  

Self service 

Closely related to the data exploration part, is the self-service part. Combined with the fine-grained access that can be set up in Unity Catalog, administrators can prepare a specific set of data for a certain group of end users. These finely defined tables or models can then be made available for those end users to explore. End users are often overwhelmed by the sheer amount of available data assets that they do not know where to start when it comes to self-service. Hiding a lot of unnecessary data assets can lower the initial difficulty. Having a smaller scope in the beginning allows end-users to really master the data they are interested in. Because of the ease of use of Unity Catalog, these datasets can easily be expanded, such that end-users can slowly but steadily build up their knowledge when it comes to the available data in the company. Combining the greater knowledge on business level with self-service and exploration on data level can generate new ideas on how to optimize certain business processes but can also help to discover hidden weaknesses. As such, the at first sight “simple” way of setting access rules can have a big impact on the company as a whole.

Unifying data (Delta Sharing) 

Delta Sharing is an open protocol developed by Databricks to securely share data with other organizations, regardless of the computing platform they use. In this blog we will solely focus on the Databricks-to-Databricks protocol, as this has the most features and is also very closely related to Unity Catalog. 

Figure 6: example architecture Delta Sharing 

Concept: Sharing without Boundaries 

The idea of Delta Sharing is that other Databricks workspaces have read access on all kinds of data objects. Currently, it is possible to add tables, views, volumes, machine learning models and even notebooks to a share. The protocol allows data providers to share data assets directly from their data lake or Databricks Workspace. As already mentioned, this protocol can be accessed regardless of which data platform someone is using. A benefit of using Unity Catalog is that the setup is simpler than the setup with other data platforms. 

Security Integration with Unity Catalog 

One of the standout features of Delta Sharing in a Databricks-to-Databricks scenario is its integration with Unity Catalog. This integration allows for fine-grained access control, similar to how security is managed for internal users. For example, you can restrict access to specific partitions of a table, enforce row-level security, or apply even more complex access rules. The beauty of this system is that you can configure security for external parties in the same way you would for internal teams, providing a consistent and reliable security model across the board.  

No Data Replication 

One of the key benefits of Delta Sharing is that it eliminates the need for data replication. Traditionally, sharing data with external organizations often required creating copies of the data and transferring it through various channels, such as FTP servers or APIs. This not only increased storage costs but also introduced potential data synchronization issues. With Delta Sharing, there is no need for such replication. Data remains in its original location, and access is granted through secure sharing mechanisms. This approach ensures that recipients always have access to the latest data, without the need for complex data management workflows. 

Delta Sharing Use Case 

Delta Sharing is particularly useful in scenarios where organizations need to provide data to external stakeholders such as customers, partners, and regulators. For instance, a company that sells data products can use Delta Sharing to distribute data directly to clients, allowing them to query the data in real-time, rather than receiving static data files. This improves the timeliness and relevance of the data provided. 

Another use case could be in a very large company, with multiple departments or regional branches. It is highly likely that each of these branches has its own instance of a data platform. If the data platforms can use Delta Sharing, collaboration becomes much easier. It again reduces the need for vast ETL or data sharing mechanisms and reduces data duplication. 

Conclusion 

Clearly, Unity Catalog offers much more than a traditional data catalog. It is a robust, all-encompassing platform that brings together your entire data landscape, fortifies governance protocols, and equips your teams with the tools they need for data exploration and sharing. In today’s data-driven world, having a well-structured, secure, and readily accessible data catalog is no longer just an option, it is a necessity. So why longer hesitate? If you have any questions about this topic, feel free to contact us to help you in your journey.  

Written by