Leslie Andrews

Senior Consultant. I am an Information Technology Professional who likes to solve puzzles - I am a solution finder! I am always looking to learn new things and the right way to create and implement the things I learn - again with a focus of being efficient and productive.

An Intro to Azure Purview

Have you heard about one of the newest additions to Azure services, Azure Purview? This new addition to Azure services is currently in Preview and is a unified data governance tool that lets you easily create a map of your data landscape. I’m here to tell you more about Purview and share one of my favorite features.

In my video included in this post, I’ll walk you through a demo of how to utilize Azure Purview.

  • I’ll start with a view of a fictional data map. On that map, I have data centers and can group my data centers together.
  • I can also add many different data sources such as SQL Servers, Teradata, Hive Metastore, SAP, Azure Data Lake Storage Gen2, Azure SQL Databases, Power BI, as well as others, and the list is growing every day.
  • It allows for classification of sensitive data and there are built in systems classifications, as well as create custom classifications. So, if you store bank routing numbers, for example, Purview already knows what that is and how it should be formatted. You can then apply this classification to that field in the data map.
  • An example for a custom classification might be in a Police Department, where you may want to have a classification for the computer-aided dispatch and case numbers. And almost every organization has a custom employee ID. You can create those classifications and find them in multiple systems by using the automated data discovery process.
  • Purview creates a catalog that is easily searchable by your business users. They simple enter in a term and the interface will show matches and suggestions for different assets and places they can find them.
  • In my opinion, the coolest feature is the data lineage. Here’s how it works:
    • In my demo I have an example of a Power BI Campaign Analytics dashboard. It we start at the dashboard and work our way back, we’ll see there are two visualizations, campaign revenue and digital campaigns.
    • We can see that those come from a dataset that is embedded in the Power BI workspace. It also shows that both datasets come from some final data that had a prep and transform associated to it, along with showing the five different data sources.
    • The benefit here is being able to show our users where the data comes from, what happens to it, where it ends up and how it gets used can help them to understand the complexity of the data.

I am a strong proponent of data governance, but few organizations do it as it seems like a giant undertaking. I feel it’s one of the most foundational activities an organization can take to increase their data literacy and IQ. That’s why Purview is so exciting! It allows businesses to easily start cataloging their data and will show immediate value to the stakeholders in the organization.

Need further help? Our expert team and solution offerings can help your business with any Azure product or service, including Managed Services offerings. Contact us at 888-8AZURE or  [email protected].

 

Leslie AndrewsAn Intro to Azure Purview
Read More

Azure Data Factory 101

Are you just starting out with Azure Data Factory? In this post, I’ll give you an introduction to Azure Data Factory, covering what it is, how it works and how to set it up. Within the video included this post is a short demo of how to create and access the platform.

What is Data Factory?

  • Here is a clear definition that I found from the Cloud Academy. Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation.

How Data Factory works?

  • The key components of Data Factory are pipelines, activities, and datasets.
  • Pipelines are made up of activities. There are 3 types of activities:
    • Movement – the copy activity
    • Transformation – including Azure Functions, HD Insight, Stored Procedures, Spark and Databricks
    • Control – ForEachLoops, If Condition Branching, Wait Times, and Validations
  • Datasets represent the inputs and the outputs of the activities.
  • Linked Services – these are the connection strings and authentication for all types of sources for the data sets.
  • Data Flows – are the results of the datasets where you can apply logic and transform the data.
    • Mapping Data Flows are graphical with drag and drop functionality.
    • Wrangling Data Flows are more like using Power Query or M.
  • Integration Runtime – allows you to do data integration across different network environments. There are three types of runtimes: Azure, Self-hosted, and Azure SSIS. Depending on where the data is that you need to copy will determine which of these is appropriate for the use case.

In the video below, I provide a brief walk through of how to access and create in Azure Data Factory. Please check it out, as I think it is a good resource for those just starting out.

Our Azure Every Day series is another great resource. 3Cloud consultants posts weekly blogs and videos around all things Azure. You’ll find all current and past blogs on our website or by clicking here.

Need further help? Our expert team and solution offerings can help your business with any Azure product or service, including Managed Services offerings. Contact us at 888-8AZURE or  [email protected].

Leslie AndrewsAzure Data Factory 101
Read More

How to Create an Azure Key Vault in Databricks

Welcome to another edition of our Azure Every Day mini-series on Databricks. In this post, I’ll walk you through creating a key vault and setting it up to work with Databricks. I’ve created a video demo where I will show you how to: set up a Key Vault, create a notebook, connect to a database, and run a query.

Leslie AndrewsHow to Create an Azure Key Vault in Databricks
Read More

Getting Started with Azure Databricks

As part of our ongoing series on Azure Databricks, I’ll walk you through getting started by creating your own Databricks Service and Databricks cluster. First off, it’s important to know that Databricks is not available with an Azure free subscription, you must have an Azure pay as you go account. However, there is a free 14-day premium trial available.

Leslie AndrewsGetting Started with Azure Databricks
Read More