Azure Data Lake

Technical Tips for a Successful Azure Data and Analytic Implementation

azure-icon-250x250This is my second attempt to articulate lessons learned from a recent global Azure implementation for data and analytics.  My first blog post became project management centered.  You can find project management tips here.  While writing it, it became apparent to me that the tool stack isn’t the most important thing.  I’m pretty hard-core Microsoft, but in the end, success was determined by how well we coached and trained the team — not what field we played on.

Turning my attention in this blog post to technical success points, please allow me to start off by saying that with over twenty countries dropping data into a shared Azure Data Lake, my use of “global” (above) is no exaggeration.  I am truly not making this all up by compiling theories from Microsoft Docs.  Second, the most frequent question people ask me is “what tools did you use?”, because migrating from on-prem, Microsoft SSIS or Informatica to the cloud can feel like jumping off the high dive at the community pool for the first time.  Consequently, I’m going to provide the tool stack list right out of the gate.  You can find a supporting diagram for this data architecture here.

Microsoft Azure Tool Stack for Data & Analytics
Hold on, your eyes have already skipped to the list, but before you make  an “every man for himself” move and bolt out of the Microsoft community pool, please read my migration blog post.  There are options!  For example, I just stood up an Azure for Data & Analytics solution that has no logic apps, Azure function, event hub, blob storage, databricks, HDI, or data lake.  The solution is not event-driven and takes an ELT (extract, load, and then transform) approach.  It reads from sources via Azure Data Factory and writes to an Azure Database logging the ELT activities in an Azure Database as well.  Now, how simple is that?  Kindly, you don’t have to build the Taj MaSolution to be successful.  You do have to fully understand your customer’s reporting and analysis requirements, and who will be maintaining the solution on a long-term basis.

If you still wish to swan dive into the pool, here’s the list!

Tom WardTechnical Tips for a Successful Azure Data and Analytic Implementation
Read More

Transitioning from Traditional to Azure Data Architectures

Confession: I put a lot of subtexts in this blog post in an attempt to catch how people may be describing their move from SSIS to ADF, from SQL DBs, to SQL DWs or from scheduled to event-based data ingestion.  The purpose of this post is to give you a visual picture of how our well loved “traditional” tools of on-prem SQL Databases, SSIS, SSAS and SSRS are being replaced by the Azure tool stack.  If you are moving form “Traditional Microsoft” to “Azure Microsoft” and need a road map, this post is for you.

Summary of the Matter: If you only read one thing, please read this: transitioning to Azure is absolutely “doable”, but do not let anyone sell you “lift and shift”.  Azure data architecture is a new way of thinking.  Decide to think differently.

First Determine Added Value:  Below are snippets from a slide deck I shared during Pragmatic Work’s 2018 Azure Data Week.  (You can still sign up for the minimal cost of $29 and watch all 40 recorded sessions, just click here.)  However, before we begin, let’s have a little chat.  Why in the world would anyone take on an Azure migration if their on-prem SQL database(s) and SSIS packages are humming along with optimum efficiency?  The first five reasons given below are my personal favorites.

  1. Cost (scale up, scale down)
  2. Event Based File Ingestion
  3. File based history (SCD2 equivalent but in your Azure Data Lake)
  4. Support for Near Real Time Requirements
  5. Support for Unstructured Data
  6. Large Data Volumes
  7. Offset Limited Local IT Resources
  8. Data Science Capabilities
  9. Development Time to Production
  10. Support for large audiences
  11. Mobile
  12. Collaboration

Each of the reasons given above are a minimum one hour working session on their own, but I’m sharing my thoughts in brief in an effort to help you to get started compiling our own list.  Please also look at the following diagram (Figure 1) and note two things: a.) the coinciding “traditional” components and b.) the value add boxed in red.

Tom WardTransitioning from Traditional to Azure Data Architectures
Read More

Azure Data Week – Modern Data Warehouse Design Patterns

BannerFinal Azure Data WeekIn his Azure Data Week session, Modern Data Warehouse Design Patterns, Bob Rubocki  gave an overview of modern cloud-based data warehousing and data flow patterns based on Azure technologies including Azure Data Factory, Azure Logic Apps, Azure Data Lake Store, and Azure SQL DB.  

There were many questions he was unable to answer during his session and we’re happy to share them with you now. If you missed Bob’s session or the entire week, you can still purchase access to the recordings by visiting azuredataweek.com.

Tom WardAzure Data Week – Modern Data Warehouse Design Patterns
Read More

Hybrid Cloud Strategies and Management

free training with Chris SeferlisAre you running a hybrid environment between on-premises and Azure? Do you want to be? In a recent webinar, Sr. Principal Architect, Chris Seferlis, answered the question: How can my organization begin using hybrid cloud today? In this webinar, he defines the four key pillars of true hybrid development, identity, security, data platform and development, and shows actionable resources to help get you started down the hybrid road.

Tom WardHybrid Cloud Strategies and Management
Read More