HDInsight

Confession: I put a lot of subtexts in this blog post in an attempt to catch how people may be describing their move from SSIS to ADF, from SQL DBs, to SQL DWs or from scheduled to event-based data ingestion.  The purpose of this post is to give you a visual picture of how our well loved “traditional” tools of on-prem SQL Databases, SSIS, SSAS and SSRS are being replaced by the Azure tool stack.  If you are moving form “Traditional Microsoft” to “Azure Microsoft” and need a road map, this post is for you.

Summary of the Matter: If you only read one thing, please read this: transitioning to Azure is absolutely “doable”, but do not let anyone sell you “lift and shift”.  Azure data architecture is a new way of thinking.  Decide to think differently.

First Determine Added Value:  Below are snippets from a slide deck I shared during Pragmatic Work’s 2018 Azure Data Week.  (You can still sign up for the minimal cost of $29 and watch all 40 recorded sessions, just click here.)  However, before we begin, let’s have a little chat.  Why in the world would anyone take on an Azure migration if their on-prem SQL database(s) and SSIS packages are humming along with optimum efficiency?  The first five reasons given below are my personal favorites.

1. Cost (scale up, scale down)
2. Event Based File Ingestion
3. File based history (SCD2 equivalent but in your Azure Data Lake)
4. Support for Near Real Time Requirements
5. Support for Unstructured Data
6. Large Data Volumes
7. Offset Limited Local IT Resources
8. Data Science Capabilities
9. Development Time to Production
10. Support for large audiences
11. Mobile
12. Collaboration

Each of the reasons given above are a minimum one hour working session on their own, but I’m sharing my thoughts in brief in an effort to help you to get started compiling our own list.  Please also look at the following diagram (Figure 1) and note two things: a.) the coinciding “traditional” components and b.) the value add boxed in red.