Add a Power BI Dataflow or Dataflow Gen 2 into a Pipeline with Microsoft Fabric

Video transcription:

Hi, my name is Adam McGinnis, I work for 3cloud in our data and analytics practice. In today’s video, we will be using a Data Flow Gen 2 to copy data into our lakehouse and we will be invoking that data flow through the use of a data pipeline.

To set it up the things that we will need, we first need a lakehouse, which we have already created called lakehouse products demo. And Microsoft fabric automatically creates the underlying SQL endpoint as well as the default data set. Now, we’ve already gone ahead and created a Data Flow Gen 2, and we’ve added the data destination as lakehouse. I’ll show you that configuration. And we have a data pipeline to invoke that data flow. So let’s first explore the lakehouse.

So here’s our lakehouse, you can see by the name of top here and if I expand tables, do you see that we have no tables, our lakehouse is completely empty, just created. And then moving on to our Data Flow X. Gen 2 itself, you can see that we’re using Power Query, and here is our sample data and our data destination has already been configured. If we hover over it, we can see the configuration values that we are loading it into the LH underscore products underscore demo with a type of table called products.

As you saw from the screen above, or before, we do not have this Products table created yet, this will be created the first time that this data flow is run and then here is going to be our pipeline itself. You can see this is an empty pipeline. So the first thing that we are going to do to copy data into is going to hook everything up together. So we’re going to choose an activity and center of our pipeline. Now that this has already been configured, we can close out of this, and we can come back over to our pipeline, we will click data flow. And let’s go and give this name. So data flow, copy, copy, product to lakehouse and under the settings, we will go ahead and select the data flow that we’ve already created. Now once this is done, it’s just a matter of saving it and then running the pipeline.

Let’s go and run this pipeline and we can watch the progress here as it gets kicked off and coming over to our lakehouse. Once this is done, we’ll be able to explore the data inside of Microsoft Fabric. Now because we do have a sequel endpoint, we were able to copy the sequel connection string and create a connection inside of Azure Data Studio. You can see here, we have the same thing when we run this we have no tables currently created and we are connected to the lakehouse. So we’re going to come over here to our pipeline and just let this finish running.

Okay, our pipeline just finished running and now if we come back over to our lakehouse, and if we click the refresh under lakehouse, and now we have our products table, we can click on the products table to load the preview. And we can see the products data that we had initially created has been copied over and here’s we can see it as well and if we wanted to view the table files, you can see we are creating a Delta lakehouse we have our delta log back by our parquet and then to finish the demo up we can come through to our SQL endpoint and now we are returning a table and we can invoke the same select star and get the the table back in Azure Data Studio.

Thank you if you would like any any further help with Microsoft Fabric or any other tools in Azure, then feel free to contact 3Cloud.