If you’re using Azure Data Factory and make use of a ForEach activity in your data pipeline, in this post I’d like to tell you about a simple but useful feature in Azure Data Factory. This tip will give you the ability if you’re using a ForEach activity within a data pipeline to decide whether to process each item in your ForEach loop sequentially or in parallel.
In the screenshot below, you’ll see I’m in a pipeline and I’ve configured a ForEach loop. Notice the box with the sequential option. If you leave that box unchecked, Azure Data Factory will process each item in the ForEach loop in parallel up to the limits of the Data Factory engine.
In most cases where we have a looping mechanism, including tools like SSIS, each item in the loop was processed in sequence and in a certain order. At times this can be pretty handy, and you can check the box and have that option.
Other times, the order doesn’t matter and in fact, it may be better to spread the workload and do multiple things at one time.
I’ve used this quick but powerful feature and wish we had this option before in SSIS, but I’m glad to see it now in Azure Data Factory.
So, again, a quick feature tip that I hope you’ll find handy and helpful.
Need further help? Our expert team and solution offerings can help your business with any Azure product or service, including Managed Services offerings. Contact us at 888-8AZURE or [email protected].
