Developing and deploying my ML models in MS Fabric – a user review

MS Fabric is on the market for some time now and where many of you have used it probably as an extension on Power BI, Fabric offers also some other interesting features.

In this blog we investigate how well MS Fabric can be used to develop and host Machine Learning (ML) models. As a test, we train a regression model to predict a house price based on certain characteristics. We will test how easy it is to develop the model and then deploy the model and use it on MS Fabric.

Our approach

As an approach, we use a typical data science process starting with defining the problem we want to solve: The goal is to get an estimation for house prices based on certain features by creating a regression model based on historical data.

Then we look at the data. This data can easily be loaded in as a file and then be transformed to a table in a Lakehouse. This allows us to have a first look at the data and check for inconsistencies. The SQL analytics endpoint allow us to write some more advanced analytical queries to look for tendencies or empty/missing values in the data. This can also be done in a notebook for the PySpark users. The notebook also allows us to use visualization techniques to see relationships and trends between columns or other python libraries. So far, so good.

Model set-up

Once we have our data ready, we write the code for training and executing the model and track its performances in the tools provided by MS Fabric.

We train the model with 2 different algorithms with a set of predefined parameters to evaluate the different results we get. We chose a random forest regressor and extreme gradient boosting algorithm. Each of these models have different parameters that the model requires to be trained with.

16 different versions of the model will be created in total, 8 for each algorithm with its parameters. The random forest regressor will use the “max_depth”, “min_samples_split” and “n_estimators” parameters, each of these parameters has 2 options, resulting in 8 different combinations. The same logic applies for the extreme gradient boosting algorithm using “learning_rate”, “max_depth” and “gamma” as parameters. Other and more parameters and algorithms can be used but for this case we kept it compact.

Setting up an Experiment

Now the set-up is determined, we run the model in an Experiment object. Like the name says, this component is used to Experiment with different options to find the best possible one. An Experiment keeps track of data from different runs. This data is then gathered in a centralized place where we can easily track their progress and evolutions.

An experiment has its own properties with general information. When writing the code for the model we can choose details to track for each model, these can be model performance metrics, model parameters, tags and the input and output schema of the model.

Properties

Each run has its own Properties with information about the run. It contains a run name, start date, duration, status to specify if a run was successful or failed, a run ID, created by to specify which user executed the run, a source that shows what notebook was used to execute the runs, the experiment name and the version (if the run is linked to a stored model with multiple versions).

Metrics

A metric is used to evaluate the performance of a run, this can be fully specified in the code using a dictionary with key-value pairs.

Parameters

Run parameters represent the model architecture that was used in the run. By tracking how the metrics evolve based on the parameters used, a calculated decision can be made of what model to use for a certain use case. They have to be set the same way as the metrics using a dictionary.

Tags

If you want to add extra information to a run, you can use tags. It is stored as a key-value pair and can tell something more about the object it relates to. Tags are completely customizable.

Input & output schema

The input and output schema shows us what the structure of the input was when training the model and what output it generated. This will also be the input it needs to make a prediction and the result it will generate. To get this information a signature has to be created about the model that will be logged.

Run details

In the run details view of the experiment all executed runs are shown on the left side, the properties and details of the selected run that have to be tracked are displayed at the center of the view.

Run list

The real power in Experiments comes from the interface that allows us to visualize the evolution of runs based on the details and properties from it. As developers we can select what runs we want to evaluate and apply filters on the details and properties to select a certain subset of runs. Then we can select the data about each run we want to get in a listed view and create a visual to evaluate each performance.

In the below example we wanted to see information about the duration of a run, the “train_mean_squared_error” metric and “used_model_name” parameter. A filter was applied to only see the runs where the value of “used_model_name” is equal to “XGBRegressor”. In the metrics comparison the evolution of the metrics “train_mean_squared_error” and “test_mean_squared_error” are now easy to compare for each run.

To wrap up the experiment phase: Fabric ML Experiments makes building and improving machine learning models a whole lot easier. The ability to track past runs, including parameters and results, means you can quickly see what works and what doesn’t. The built-in tuning options, and version control help keep everything organized and easy to manage. By giving a clear view of model performance and making it simple to compare different approaches, Fabric ML Experiments helps experiment faster and make calculated decisions.

Deploying the ML models

Once you have your model ready, it’s time to deploy. Once the model is stored and trained and ready to be used for predictions or classifications, it becomes an “ML model”. The structure of these models is a lot like the Experiments. Different model versions can be created and model properties as well as details are available (metrics, parameters, tags, input & output schema) and customizable.

In an ideal case an ML model will be created from the best run found in an Experiment. When this process iterates a few times multiple versions of a model will exist. To evaluate these different versions a list and visual comparison can be created using properties, metrics, parameters and tags. This will look and work the same as with the Experiment except for the absence of a filter option to select certain model versions.

We will have to choose the right version from the ML model and can then apply this version. There are 2 options for this.

A wizard can be used to define the input and output that the model has to use. Then a notebook will be generated with the code to use the model based on these configurations. This is a good option when no prerequisites exist.

The other option will give a basic code snippet where the developer still has to specify information about the input and output of the model. Here a few details have to be entered by code but it is easy to insert this into an already existing project notebook.

Conclusion

Once you’ve fine-tuned the model through Experiments, creating and managing ML models in fabric is straightforward. The ability to store different model versions and track their performance makes it easy to evaluate and choose the best one for deployment. The structured comparison of model properties, metrics and parameters helps making informed decisions when selecting the right version. With the option to create a notebook to use the model using a wizard or a code snippet into existing projects the deployment becomes flexible. This makes for a good user experience and overall a more pleasant experience when working on difficult tasks.