Blog & Company Updates

Creating New Azure Data Factory Data Flow File Data Sets

Are you looking to reuse file connections you have set up in Azure Data Factory prior to using Data Flows? Maybe you have a case where you’re building some pipelines and you’re pulling data from files, etc. and you want to use reuse those for a Data Flow. When I tried this I discovered that you cannot reuse these existing text file data sets, but you must recreate them. In today’s post, I want to share a quick tip on how to recreate these file connections to use in Data Flows.

Step 1 (Screenshot below): Create a new Data Flow in Azure Data Factory using your work canvas. Under the settings pick a data set and point it towards the file that you have previously set up. On the left side, you should see your previously made data sets. I have a couple previously created that are still there, but not available as selections in my Source Data set list within the Data Flow, so they will need to be recreated.

ADFDataFlowFileSource_02

 

Step 2:  When I create a  a new data set and point it towards my Azure Data Lake linked service, I’ll now see the menu shown in the screenshot below. There is an option for delimited text, which I was using in my case. I choose delimited text and set up the data set. If you look at the JSON beneath it, you can see they have made some changes in the formatting.

ADFDataFlowFileSource_03

When I move back to the Data Flow (see below), you will see the drop-down for source data and the newly created data set is now an option available to be used in my Data Flow. Note the files you previously made will not be available, but only the ones you recreate as new are available for selection.

ADFDataFlowFileSource_04

While you may not be able to reuse previously existing connections, I hope this quick tip will make your process of recreating these files easier. 

Need further help? Our expert team and solution offerings can help your business with any Azure product or service, including Managed Services offerings. Contact us at 888-8AZURE or  [email protected].

Author

  • Bob has broad experience designing and implementing Business Intelligence and Data Warehouse solutions on the Azure and Microsoft SQL Server platforms. He has spoken at SQL Saturdays and has presented webinars on various cloud architecture and data management topics.

Bob RubockiCreating New Azure Data Factory Data Flow File Data Sets