If your organization has landed on Microsoft Fabric as its data warehouse and dbt as its transformation tool, you’ll eventually hit the same question every dbt-on-Fabric team hits: how do I actually develop this locally, in VS Code, against a real Fabric Warehouse or Lakehouse — without a slow CI round-trip for every SQL change? […]
Tag: data engineering
Materialized Lake Views in Microsoft Fabric: your medallion pipeline in a few lines of SQL
Data engineering on a lakehouse used to mean a lot of moving parts. A notebook for each transformation layer. A Data Factory pipeline to run them in the right order. Custom validation logic across scripts. Schedules to maintain and a Monitor Hub to watch when things break. Five different surfaces and five different things to […]
Kimball in a data lake? Come again?
Most companies are already familiar with data modelling (be it Kimball or any other modelling technique) and data warehousing with a classical ETL (Extract-Transform-Load) flow. In the age of big data, an increasing number of companies are moving towards a data lake using Spark to store massive amounts of data. However, we often see that […]
Transfer learning in Spark for image recognition
Transfer learning in Spark demystified in less than 3 minutes reading Businesses that want to classify a huge set of images in batch per day can do this by leveraging the parallel processing power of PySpark and the accuracy of models trained on a huge set of images using transfer learning. Let’s first explain the […]
The Journey of attaining the Azure Data Engineer certificate
On February 23, 2021, Microsoft released a new beta certification exam, Exam DP-203: Data Engineering on Microsoft Azure. It is replacing the exams DP-200: Implementing an Azure Data Solution, and DP-201: Designing an Azure Data Solution. These previous exams DP-200 and DP-201 will retire on June 30, 2021. When passing the two old exams or […]
The Journey of attaining the Azure Data Engineer certificate
On February 23, 2021, Microsoft released a new beta certification exam, Exam DP-203: Data Engineering on Microsoft Azure. It is replacing the exams DP-200: Implementing an Azure Data Solution, and DP-201: Designing an Azure Data Solution. These previous exams DP-200 and DP-201 will retire on June 30, 2021. When passing the two old exams or […]
