Microsoft Fabric: How useful is it for me?

Microsoft Fabric provides an all-in-one data workflow environment that addresses a variety of demands. Regardless of whether you are working on streaming analytics, operating a data warehouse or making informed data visualizations, Fabric commits to a single platform for all these activities. This eliminates the difficulty of working with multiple tools.

The real question on everybody’s mind is; is it already useful for me? The answer is (probably) yes, but it all depends on your needs. In this blog post we describe the available tools, and what you need to keep in mind.

Since its private preview launch in May 2023, Microsoft Fabric has made several big steps in closing the gap between data visualization (Power BI) and data engineering (former Synapse). Quite literally! You can now store your data layers side-by-side with your semantic models and dashboards. If you have multiple groups that need to access specific data, you can even work with multiple workspaces.

Preview features

Since November 2023, Fabric is generally available, but whenever new features arrive, they are first released in public preview mode. The extensive list can be found on the “What’s new in Microsoft Fabric” website.

These features can be compared to the infants of the Microsoft Fabric framework. They’re still learning to walk, are not that stable yet on their own feet, and aren’t that resistant to potential bugs. However, the list of viable components in preview is getting quite extensive, and some are worth checking out. Mainly because they simplify your work and save you time.

Work-enhancing preview components

Nobody likes stressing about version control, having a hard time dividing all the workload, or spending hours on something that could be fixed with a little automation/shortcut. If you’re looking for something to shorten your development time, these things might come in handy;

OneLake shortcuts

Do you have a lot of data in different services (Amazon, Google, Azure)? Then a OneLake shortcut might be just for you. Think about it as if you’re creating a shortcut on your own PC, pointing to another location on your machine without really copying or moving the data.

It gives you the idea your data is available inside Microsoft Fabric without moving it from the external source. Naturally, this changes once you start making actual copies via a pipeline, but you get the point.

The current connectors to external data sources are limited for now. You can only connect to other Lakehouses, AWS, ADLS Gen2, Dataverse, and GCP. If your data is stored on other platforms, you might have to wait a bit. Big spoiler alert: At the time of writing, Microsoft has already announced other upcoming connectors such as Oracle database connectors, MySQL Databases, Snowflake, and more.

Git, AI and API

Connectors aren’t the only tools that are going to become your next best friend;

– Git integration: Git is your go-to CTRL+Z companion and branching buddy. This creates a perfect opportunity for rollbacks, working together, version control and more. We will save you the lecture on how it is best practice anyway.

Side note: It is currently only supported for Azure Git Repos within the same tenant as your Fabric Capacity.

– AI-tools: The AI-based tools won’t steal your job, but they will make your job easier. You currently have different AI-based tools in preview mode. Look at AutoML or Copilot, for example. Those are already integrated and can be tested.

Side note: If your data is located outside of the US or France, your Tenant administrator must enable certain features to make the data accessible for Copilot. Also, note that Copilot Is only accessible from an F64/P1 or higher.

– API’s: Taking a closer look at Microsoft Learn, we see that almost all APIs are still in preview mode. The only exception being the Fabric Core API. They look very promising, however! You’ll come across REST APIs for Fabric Data Factory Pipelines, OneLake Shortcuts API, GraphQL APIs, and a few more. They’re quite handy when you’re looking to automate certain processes.

The list goes on and on, and will continue to evolve. Focusing on no-/low-code, MS Fabric will continue to launch components In public preview. So If you didn’t see what you were looking for, take a look at the roadmap to check for any other upcoming useful components.

General available features

Since we are comparing our components with humans, let’s also use the analogy here. General available features are fully matured adults. They might have their own flaws and maybe a bit of anxiety, but overall they function as well as possible.

Mr. Security

The adult you can rely on the most, is Mr. Security. When using new technology, you’ll always wonder if it’s secure enough. Let us assure you: don’t worry too much about Microsoft Fabric. You can safely access all your data in other Azure services, Fabric itself, Managed VNets or behind firewalls. If you’re wondering how, take a look at;

– Microsoft Fabric Private Links: A secure way to access your sensitive data in Microsoft Fabric by providing network isolation. More or less the same as a private endpoint, but you can connect with multiple private endpoints to a single private link service (managed by Microsoft). A private endpoint, however, connects to one private linked service.

– VNet Gateways for Dataflow Gen2: Gone are the days where you would need an on-premises data gateway. The VNet data gateways helps you to connect from Fabric to other Azure data services.

– Managed private endpoints: If you have put everything behind a firewall, the managed private endpoint is something for you. You can now access your data sources without exposing them to the whole internet. They can be configured for every fabric capacity as of august 2024.

– Trusted workspace access: Working with a lot of Azure storage accounts? Are they enabled with a firewall? Then don’t look further. Trusted workspace access provides a secure way to your firewall-enabled Azure Data Lake Storage Gen2 account. After connecting your data pipeline to your ADLS Gen2 account, you can start analyzing your data.

Extra security features such as row-level-security and object-level-security are also available. The side note with this is that using them will only work in the warehouse component and SQL-endpoint. In Power BI this will result in defaulting to Direct Query, which could have implications on performance or latency (depending on the scenario). Nonetheless, if you have a lot of data that is secured behind firewalls, Microsoft Fabric has some great solutions to access them.

Ms. Connectivity

With proper security, comes great connection. If we don’t have a solid security base, creating a connection becomes risky and unsafe. Hopefully, by now It’s clear that Microsoft Fabric has a stable security base, so naturally we can also safely create connections.

Microsoft Fabric has made sure you can safely establish a connection to your workspace components via the following general available features;

– Fabric’s CORE API: The basic principle of every API: making your processes available externally and usable in other environments. You can access your data, create items via endpoints, the right id’s and an access token.

Side note: Using a service principal with the Fabric Core API is currently limited. You’ll have no issues with your GET-calls, but POST-calls require other tokens. If you want to automate certain tasks, make sure you know how to get the right access token.

– Eventhousing: An eventhouse offers a solution to manage and analyze large amounts of real-time data. It’s also a new, dynamic way of hosting multiple KQL-databases. To add on top of that, Microsoft even made sure that you could connect with other (external) KQL-databases in a Delta Lake format.

– Power Query in VSC: The best friend of every Power BI-user is now also available inside of Visual Studio Code. You’ll be able to handle data transformations and preparations outside of the Power BI apps.

Side note: The Power Query SDK for Visual Studio is deprecated since June 30, 2024. You can try the new Power Query SDK in Visual Studio Code.

– Semantic link: Exploring your tables inside of the semantic model is now made more accessible. A semantic link allows you to establish a connection between your semantic model and Synapse Data Science. This means that you can now use your semantic model from Inside of your Notebook, saving your data analyst a lot of time. As of Spark 3.4, it is available in default runtime. So there’s no need to pip install it anymore.

Uncle environment

Safely tucked away Inside of your workspace, you’ll find another general available component; the environment. Contrary to the global environment all over the world, this is something we do have full control over.

– Environment item in Data Engineering: The heart of your whole data engineering operation. The environment is a centralized item where you can configure your required Spark job settings. Import custom or public libraries, select the spark version that works best for you and link it to your notebook.

Conclusion

It’s safe to say that Microsoft Fabric has a bright future ahead. It’s a platform for a wide range of profiles; from the seasoned data engineer to the IT-savvy business analyst… nobody Is being forgotten. In our opinion, Microsoft Fabric can already be used for the more standard use cases.

If you have some basic ETL needs and don’t have a lot of coding expertise, Microsoft Fabric is the tool for you. Before implementing Microsoft Fabric however, look at the complexity of your systems. You might come across some limitations if you don’t work (yet) with Azure storage accounts, AWS, …

As Microsoft Fabric is rapidly evolving, solutions are on their way, but it takes time. If your company has a very complex system, make sure to reach out to us first so we can properly help you.

Have you missed anything you would like us to talk about, check out, do we have you hooked to order a live demo? Or do you have some other important components that you want to share with us that you couldn’t live without anymore? Let’s connect!

Written by