Introduction to Data Science in Microsoft Fabric: A hands-on approach

Getting started with Data Science within the Fabric environment: A step-by-step process

Within the Fabric environment, data scientists can efficiently manage data, notebooks, experiments, and models while easily accessing data from across the organization and collaborating with their fellow data professionals.

Ideally, data science process consists of the following stages:

• Problem formulation and ideation

• Data discovery and pre-processing

• Experimentation and modeling

• Enrich and operationalize

• Gain insights

 

Stage 1: Problem formulation and ideation

Microsoft Fabric streamlines the data science process by eliminating the need for separate platforms for different roles. Data scientists and data analysts work within the same ecosystem, enabling effortless data sharing and collaboration. Data science practitioners can easily share Power BI reports and datasets, significantly improving efficiency during the problem formulation stage.

Stage 2: Data discovery and pre-processing

Microsoft Fabric users can engage with data stored in OneLake through the Lakehouse feature. Lakehouse seamlessly integrates with Notebooks, allowing users to browse and interact with data effortlessly. Users can effortlessly import data from Lakehouse directly into a Pandas data frame, enabling smooth data exploration from OneLake. Microsoft Fabric seamlessly integrates robust toolkit for data ingestion and orchestration pipelines, including data integration pipelines. With easy-to-create data pipelines, users can access and transform data into a format suitable for machine learning consumption.

• Data exploration

Understanding and visualizing data are essential elements of the machine learning process. Microsoft Fabric provides an array of tools tailored to explore and prepare data for analytical and machine learning purposes, depending on its storage location. Notebooks emerge as one of the most efficient avenues to initiate data exploration.

• Data preparation using Apache Spark and Python

Within Microsoft Fabric, users can transform, prepare, and delve into their data on a massive scale. Through Spark, users can utilize PySpark/Python, Scala, and SparkR/SparklyR tools for large-scale data preprocessing. Additionally, advanced open-source visualization libraries enrich the data exploration experience, enabling users to gain comprehensive insights from their data.

• Seamless data cleansing through data wrangler

A new feature has been introduced in the Microsoft Fabric Notebook interface. It allows users to utilize Data Wrangler, a coding tool, to streamline data preparation and automatically produce Python code. This enhancement streamlines the execution of time-consuming tasks, such as data cleansing, while promoting the creation of repeatable and automated processes through the generated code.

Stage 3: Experimentation and Machine Learning model

Notebooks equipped with tools such as PySpark/Python and SparklyR/R can manage the training of machine learning models. Machine learning algorithms and libraries play a pivotal role in model training. Library management tools facilitate the installation of these libraries and algorithms. This allows users to utilize various popular machine learning libraries to conduct model training within Microsoft Fabric. Moreover, widely used libraries like Scikit Learn can also be utilized for model development. Tracking of Machine Learning model training is facilitated through MLflow experiments and runs. Also, Microsoft Fabric incorporates a built-in MLflow experience, enabling users to interact with it for logging experiments and models.

• Synapse ML

The SynapseML library, formerly recognized as MMLSpark, is an open-source resource developed and managed by Microsoft. It aims to streamline the creation of massively scalable machine learning pipelines. Moreover, as an integral component of the tool ecosystem, SynapseML extends the capabilities of the Apache Spark framework in various innovative directions. By integrating multiple existing machine learning frameworks and novel Microsoft algorithms into a unified, scalable API, SynapseML offers a comprehensive solution. This open-source library comprises a diverse array of machine learning tools, facilitating the development of predictive models and the utilization of pre-trained AI models from Azure AI services.

Stage 4: Enrich and operationalize

Notebooks can conduct batch scoring for machine learning models using open-source libraries tailored for prediction tasks. Alternatively, users may choose to employ the Microsoft Fabric scalable universal Spark Predict function, which supports MLflow packaged models accessible in the Microsoft Fabric model registry.

Stage 5: Gain insights

In Microsoft Fabric, users can effortlessly write values into OneLake and seamlessly incorporate them into Power BI reports using the Power BI Direct Lake mode. This simplifies the process for data science practitioners to communicate their findings to stakeholders while streamlining operational processes.