Cortana Intelligence Suite Tour

This is the second post in the series Modernize your Business Intelligence Architecture with the
Cortana Intelligence Suite. To see the first post, follow this link.

The Cortana Intelligence Suite is a collection of independent, but fully integrated data and analytic platform tools offered by Microsoft Azure. Customers can create complete streaming or batch-loaded Business Intelligence and Advanced Analytics solutions in these platforms without having to deploy or manage a single server. Each tool provides unique capabilities (with some overlap) and integrates together seamlessly. Customers can even mix and match tools in the suite to meet use case requirements without having to pay for each one because they are all considered to be pay-as-you-go cloud services.

Comprehensive Architecture

The architecture diagram below provides an example of how the Cortana Intelligence Suite services can be used to create a comprehensive real-time and batch solution. Starting at the top left of the diagram, code can be added to apps and IoT devices to send events in real time to Event Hubs. Stream Analytics can then be used to consume and publish the data from Event Hubs and simultaneously write a copy to Data Lake Store, perform analysis on the data and send the results to Power BI, and even run the data through a machine learning model.

On the batch side, Data Factory can be used to orchestrate the load of on-premises and cloud-born data into Data Lake Store and SQL DW. Aggregates from SQL DW can be stored in Analysis Services for a rich semantic-layer model for consumption from BI tools. Finally, Power BI can be used to create real-time streaming dashboards, or blended data models that mash up stored streaming data with batch data from on-premises. If that wasn’t enough, Power BI can also be used to create alerts. A closer look at each tool is highlighted in the sections below.

Event Hubs – Ingest It

Azure Event Hubs is a highly scalable data streaming platform and event ingestion service. It can be scaled up or down and can accept millions of events per second. It also provides robust security for letting apps, devices, and software connect to Azure to send data, events, and telemetry. Once loaded, the data stays in the Hub for a specified retention period so it can be consumed using other applications. This is the initiation point of getting streaming data into Azure for analysis.

Stream Analytics – Stream It

Stream Analytics is an event processing service that permits powerful, real-time analysis of your data. It allows you to write SQL code against streaming data which is executed in real time. Using Stream Analytics, you can integrate the data from Event Hubs to multiple tools simultaneously.

For example, one Stream Analytics job could be copying the data to Data Lake Store while another could be sending the results of a SQL query to Power BI in real time. Stream Analytics uses a familiar SQL language providing common analytic SQL techniques such as Joins, CTEs, Where, Group By, Having, and Aggregations. Another consideration is the concept of time-series functions, which let you look forward or backward on the data. One of the coolest aspects of Stream Analytics is the ability to add machine learning code as a Function and process machine learning models on the data in real time.

Data Lake Store – Store It

Data Lake Store is a fully managed Big Data store in the cloud. It is essentially the Hadoop Distributed File System (HDFS) as a cloud service. It provides all the benefits of cheap HDFS storage without the need to deploy and manage a persistent Hadoop cluster. There are no file size limits and you can scale your data storage into petabytes. It also integrates with Azure Active Directory for easy access and security, plus it’s accessible via all your Big Data compute tools such as HD Insight.

SQL DW – Relate It

SQL Data Warehouse is a SQL Server in the cloud for massive scale. It uses a database cluster-based Massively Parallel Processing Architecture (MPP) to allow customers to scale from a few terabytes to one petabyte of data. It’s highly elastic and scalable – companies can scale their cluster up and down in minutes. It can even be paused to eliminate compute charges. An important feature is its ability to communicate with Blob Storage or Data Lake Store using PolyBase. With this integration link, storage directories can be mounted to SQL DW as an eternal table and queried just like a regular SQL table.

Analysis Services – Speed It

Azure Analysis Services is a proven analysis engine that has existed on-premises with SQL Server since the 2012 version of the product, and is the same engine used by Power BI. Through Azure Analysis Services, customers can deploy blazing fast in-memory analytic models through a platform service without having to manage hardware, SQL Server configuration, and licensing. It is a scalable service that allows companies to shrink or grow with their needs. Users can create blended data models using familiar tools through Visual Studio and SQL Server Data Tools and deploy them to the cloud. Additionally, if a customer’s Power BI model grows in popularity or size beyond Power BI’s size limits, models can be migrated to Analysis Services which can handle the elevated requirements.

Power BI – Visualize It

Power BI is an industry leading BI platform that allows you to create stunning visualizations with all your data, whether it is on-premises or in the cloud. You can use it to deploy corporate and self-service capabilities to your organization, create your own open-source visualizations, and create alerts on your data to generate actionable dashboards.

ML – Learn It

Azure Machine Learning is a fully managed machine learning service. It gives companies the ability to create machine learning models using a drag and drop interface, which in turn reduces the data science learning curve significantly. Additionally, it allows you to deploy machine learning code as a web service so that it can be used by all your applications. If your organization has already invested in machine learning code within open-source tools such as R or Python, Azure ML allows you to deploy your native open-source code and leverage it in your Cortana solutions.

Data Factory – Move It

Azure Data Factory is a fully managed data orchestration tool that allows for the creation of workflows/pipelines that ingest, prepare, transform, analyze, and publish data from on-premises or cloud sources to any of the tools available in the Cortana Intelligence Suite. It also provides monitoring and management of these pipelines through a rich portal interface. To the data practitioner, Data Factory is not SSIS in the cloud, but closely resembles the control flow of SSIS.

HDInsight – Hadoop It