Microsoft Fabric is about building bridges – bridges between disciplines and bridges between professionals with different needs and perspectives.
Portland, Oregon is called “The City of Bridges”. There are 14 bridges that cross the two large rivers running through our city, and if you want to get anywhere around the Portland area, you must use a few bridges. This has us thinking that bridges exist because perhaps we live in different places, we do different things and maybe we have different ways to do our thing – whatever that thing is.
Bridges for IT Data Projects
Traditionally, if an organization needs a new reporting platform, they call in a database expert and a data engineering team to acquire, transform and land all their data. The analytic reporting developers wait while the data warehouse is created, ETL pipelines are developed, and data is transformed into the right shape for reporting. The timing of all this important work is never as expected so these professionals either sit and wait for each other or work impossible hours to meet deadlines. There is inevitable cleanup work, a few remaining problems to solve and discrepancies to resolve. Database professionals were taught to normalize data structures while Business Intelligence practitioners learned to de-normalize and simplify. Ironing-out the differences between these disciplines often requires patience and iterative collaboration. This is work that must be done in nearly every data and BI project. Even agreeing on simple things like naming conventions are often sticking points that need to be negotiated.
Why Can’t We All Just Get Along?
Development teams can certainly work together but communication and correlation can be cumbersome when you just need to get the job done. Sometimes it is easier to just roll-up your sleeves and do the work. Wouldn’t it be amazing to have one set of tools that can design the whole end-to-end data movement process, and work at-scale? Once the vision is realized, how amazing would it be to bring-in a team of specialized developers to enhance each component and expand the solution? To do this in one platform would be pretty cool.
Bridges for Business Centric Reporting
Modern BI tools exist partially because business professionals need to do a lot of their own data analysis. To have convenient reporting tools in the hands of business professionals is empowering they will use them differently than an IT developer. Technology professionals are mindful of data governance, security, version control and build processing whereas business data analysts just want to get the job done quickly and easily. Can a business authored solution be retrofitted to work at-scale and in accordance with organizational data governance? Perhaps by working closely with the business, we finally achieve both objectives.
Goals of Fabric, Simplified
Bridges are harder to build when things are far apart. Bringing data into one storage location makes it much easier to work with, particularly if data can exist in a universal and flexible storage format. Further, having a universal method to access and query the data would be not only be wicked awesome but it would be uber simple. Reducing copies of the data also helps so if we can reliably and efficiently connect to that data without making unnecessary copies, would reduce the overall complexity. Having the ability to allocate and manage the compute resources and cost would keep everybody happy. Oh, and finally having the ability to do just about everything imaginable in one place with one set of simple tools would be the icing on the cake. These are the goals of Microsoft Fabric.
Fabric Overview
There is a lot going on inside Microsoft Fabric, but all the power and sophistication is simplified by a collection of integrated assets. Following are just a few of dozens of these components used to perform data related functions and tasks. Each of these replaces what years ago would have been an entire product or application. Fabric starts with a Workspace.
A workspace is the container for all assets – data and services. These may include a variety of things like pipelines, notebooks, ML experiments and databases.
If you have worked with the Power BI service, you know about workspaces. You put all your stuff in a workspace, assign it a capacity to manage shared compute power and cloud system resources, developer and user access. Access can also be managed on the individual assets as well.
The fundamental native storage object in Fabric is called a Lakehouse which is a core part of Microsoft’s OneLake vision. In short, it’s one place to store absolutely everything: curated data, raw files, documents and development objects. A Lakehouse is really an Azure Datalake storage object under the covers, but it is also easier to configure and more flexible than that. A Lakehouse includes both file storage and data storage. You can create a folder structure and drop any kind of files you like into it. The Data part of a Lakehouse also stores files, but it has some built-in magic. Data is automatically stored in a collection of Parquet files with data access and processing orchestrated by a Spark compute cluster and Delta change tracking. Delta Lake handles operations in a similar fashion to sophisticated relational database products, and provides capabilities such as:
- Columnar Storage
- Space Efficiency
- Schema Evolution
- Complex Data Types
- Integration with systems and components
- Splitable file partitioning
- High performance
What makes Fabric a unique game-changer is its expansive sophistication masked by its simplicity. In short, Fabric is: a Software-as-a-service (SaaS) platform with simplified billing & management. The browser-based portal is a single pane of glass surfaced through the Fabric portal. The admin, developer and user portal is based on tried-and-true Power BI service portal that allows members of each persona to view only the objects and assets needed to perform their jobs.
Many Personas
Simply choose a persona and the portal shows you only relevant assets. For example, if you are a Power BI practitioner, you will see data shaping, modeling and reporting tools. If you are a Data Engineer, you will see Notebooks, Pipelines and Spark jobs. If you are a Data Scientist, you will see Notebooks and Machine Learning Experiments.
Fabric is inclusive. Shortcuts simplify and enable connectivity to existing data storage and services that enable secure access to data where it currently lives, with no need to make siloed copies.
Bridging Connections with Shortcuts
Fabric introduces new connector objects called shortcuts that are part of the OneLake experience.
We all know that in Windows, a shortcut is just a simple link to a file located somewhere else. In Fabric, the effect is the same, but a shortcut is much more than a simple address or link because it handles connectivity and security while abstracting away complex details.
We can’t stress enough how significant this simple feature is. Having the ability to leave source data where it resides, in the existing data lake, is a huge benefit. OneLake oversees all access permissions and credentials, eliminating the need for individual configuration of each Fabric experience to link with different data sources. Furthermore, by utilizing shortcuts, you can bypass the creation of edge data copies, thereby diminishing the delays often encountered with data duplication and staging processes. Rather than copying data into a Lakehouse to stage it for transformation, a shortcut allows it to remain in native storage until we need to move it. This reduces data movement en masse and minimizes unnecessary data silos. Shortcuts bridge data to Fabric supporting a hybrid data estate.
Conclusion
We are just tickled pink to see Fabric released. It has been a long time coming and 3Cloud has been involved in testing and providing feedback to the Microsoft product teams for nearly two years, since it was introduced to the MVP and partner communities, and throughout the private preview.
There has never been anything quite like Fabric and it has great promise to enhance your data estate by becoming the foundation for future enhancements.
By bridging new and existing data assets, Fabric enables simplified billing & configuration through a SaaS platform that is approachable and easy to manage. Developers and Data Analysts have a unified and customized experience so they can focus on their tasks and areas of expertise. Finally, shortcuts enable hassle-free data connectivity so you can work with data where it resides, working with master, system-of-record data that can easily be moved and transformed only when necessary.
Partner with 3Cloud
We look forward to working with all of you to expand your data estate to include Fabric capabilities and look forward to all the goodness Microsoft has in the future that can be used to enhance Data Analytics. The 3Cloud Data & AI Delivery Engineering team is responsible for guiding customers and our data teams with compelling technologies like Fabric.
The following team members contributed to this article: Steve Hughes, Paul Turley, Dan King, and Christian Alcaria.
If you’re interested in working with 3Cloud more closely on your next Fabric project, please reach out to us today.
