Microsoft Fabric is an all-in-one toolbox for crunching numbers and making sense of data. It’s got a bunch of tools, like moving data around, storing it in lakes, doing engineering work on it, mixing it together, using it for science stuff, keeping an eye on it in real-time, and helping you make smart business decisions. And the best part is, it’s got a strong foundation that makes sure your data is safe, plays by the rules, and follows all the important guidelines. In this article, we will learn how to create data pipeline to copy data from azure blob and ingest into a Lakehouse in Microsoft Fabric. Let’s get started.
What is Azure Blob Storage?
Azure Blob Storage is a cloud-based object storage service provided by Microsoft Azure. It’s designed to store and manage unstructured data such as documents, images, videos, backups, and more.
Data Pipeline to Copy Data From Azure Blob and Ingest Into Lakehouse in Microsoft Fabric
The first thing we want to do is to create a workplace which is typical a container or organizing structure that allows us to collaborate on and manage content, such as reports, dashboards, datasets, and more.
To create the workspace, click on Workspaces and Click on New Workspace.
Provide name for the workspace. In this article, DataPipelineFromAzureBlob is given.
- Next, In the Data Factory platform, select Data pipeline.
- In the New Pipeline box, we provided Azure Blob Data as seen below.
- Click Create
- In the Start building your data pipeline, select Add pipeline activity and select Copy data
- In the Name box of the General tab, we provided Customer Data as seen below:
- In the Source tab, select External for the Data store type.
- Select New to create a new connection.
- In the New Connection box, search and select Azure Blog Storage
- Click Continue
In the Account name or URL of the Connection Settings, we provided the following:
In the Connection credentials tab, select Create new connection in the dropdown for the Connection.
- We provided: Wide World Importers Public Sample as the Connection Name.
- The Authentication kind is set to Anonymous.
- At the bottom left, click on Create.
To access the .parquet files in https://azuresynapsestorage.blob.core.windows.net/sampledata/WideWorldImportersDW/parquet/full/dimension_city/*.parquet:
- In the File path text boxes, we provided:
- Container: sampledata
- File path – Directory: WideWorldImportersDW/tables
- File path – File name: dimension_customer.parquet
- In the File format drop down, choose Parquet.
- Select Preview data next to the File path setting.
- Click on Cancel to close the Preview Data window.
- In the Destination tab, select Workspace for the Data store type.
- In the Workspace data store type dropdown, select Lakehouse.
- In the Lakehouse option, select New and provided CustomerData
- Click Create
- The root folder should be set to Table.
In Table Name, select New and provided BlogCustomerInformation (you can choose whatever name you want).
- Click Create
- Click on Run at the top of the pipeline tab. You will be required to Save. Go ahead and save and run the pipeline.
In the screenshot below, we can see that the data pipeline was successful.
Next, we need to check the data in the DataPipelineFromAzureBlog workspace we created initially. Click on the workspace.
In the screenshot below, we have the CustomerData in the Lakehouse and the SQL endpoint.
When we click on the CustomerData with the SQL endpoint, we can begin to write queries against the data.
In the screenshot below, we executed a query, and everything is working fine
See you in the next video