In today’s fast-paced, data-driven world, businesses generate vast amounts of information daily. With over 328.77 million terabytes of data being created every day, the need for scalable, efficient data pipelines has never been more critical. Many organizations struggle with the complexity of traditional ETL (Extract, Transform, Load) processes—where moving, cleaning, and transforming data can be […]
Tag: python
What orchestrator to use for your ETL jobs?
In today’s data-driven world, organizations are facing the ever-increasing complexity of data pipelines. The need to handle diverse data formats, sizes, sources and processing requirements can result in the need of different types of transformations blocks, ranging from SQL to PySpark to low/no-code solutions. When complex table dependencies also enter the picture, you get high, […]
CI/CD of Synapse Serverless Part 2/2
Introduction In the first part we’ve explored the reasons why using CI/CD is important in projects. Additionally, we’ve defined three core concepts that need to be implemented as best-practices in Synapse Serverless databases. In this part, we’ll explore how we can continue using our beloved technical tooling ‘Visual Studio’ together with Git and Python in […]
Feature Store
Everyone who has already come in touch with data science, has already heard of features used in such models. One aspect that can become quite challenging, is reusing features in a consistent way, across several team members, projects and in environments. In this article, I will explain the most commonly used way to resolve these […]
Pandas, Koalas and PySpark in Python
If you landed on this page to learn more about animals, I have to disappoint you. Pandas, Koalas and PySpark are all packages that serve a similar purpose in the programming language Python. Python has increasingly gained traction over the past years, as illustrated in the Stack Overflow trends. Originally designed as a general purpose […]
Pandas, Koalas and PySpark in Python
If you landed on this page to learn more about animals, I have to disappoint you. Pandas, Koalas and PySpark are all packages that serve a similar purpose in the programming language Python. Python has increasingly gained traction over the past years, as illustrated in the Stack Overflow trends. Originally designed as a general purpose […]
Data Services Python Script Call
In the magical world of data extraction Data Services offers great out of the box functionality to extract data from databases like SQL Server, MySQL, PostgreSQL, Oracle… and from SAP ECC. However, to extract data from REST APIs it lacks some functionality. Therefore, we will use one of the most currently popular programming/scripting language: Python. […]
Process Mining: Understanding Simple Process Discovery Techniques using Python
Hi and welcome to this blog on process mining Process mining is a set of techniques used in the field of process management and improvement which supports the analysis of processes based on event logs. Process Mining is able to fire different algorithms on a certain event log to identify patterns and trends in your […]
Process Mining: Understanding Simple Process Discovery Techniques using Python
Hi and welcome to this blog on process mining Process mining is a set of techniques used in the field of process management and improvement which supports the analysis of processes based on event logs. Process Mining is able to fire different algorithms on a certain event log to identify patterns and trends in your […]
XML With Python
DOM (DOCUMENT OBJECT MODEL). We load the entire file in memory because then we can access the content like it’s an object. Which results in faster processing and also giving it the possibility to make adjustments in the file. SAX (SIMPLE API FOR XML) Reads the file bit by bit and processes the XML on […]
