A common approach to assess ROI on any BI tool deployment is through adoption metrics. Tool usage is a tangible way to ensure business users are extracting value from the investment. Industry organizations like The Data Warehousing Institute (TDWI) often refer to highly adopted BI deployments as “Pervasive BI” – something many companies strive for but many struggle to achieve.
Actively monitoring usage is a key component of Adoption Management – a critical factor in Power BI deployment success. Power BI offers a way to do this via Audit Logs, a capability available to tenant administrators.
Working with your log data
To access audit logs, you must first turn on this capability on the Admin Portal by going to your settings, as shown below.
Note that to query the logs, you will be redirected to the O365 Admin Center Security & Compliance portal. From there, you can manually retrieve information regarding Power BI activities – for example:
- Who viewed a Power BI dashboard, and when
- Who is sharing dashboards
- When was a specific dataset deleted from your Power BI service
- When was a Power BI group created
When manually searching the log, you can download usage data in CSV format. You can also automate the downloads via PowerShell using a specified login through Exchange Online.
While loading the CSV file in Power BI, make note that log data is in JSON format. To serialize into a table, you can use a Query Editor transform:
As you expand the columns, ensure you click on “load more” within the column selector. Given that by default Power BI will not load columns unless explicitly selected, simply clicking “OK” will not expand the “ItemName” field which gives the Power BI asset name (report/dashboard/etc.) that we are tracking activities for.
What kind of questions can I answer via Audit Logs?
Currently, log data is centered around item usage (as opposed to by user type: consumer vs. power user, for example). As such, questions related to dashboard or report popularity can make for great uses of log data. Increased popularity can be used as a direct measurement of Power BI adoption. For example, you can see the 5 most popular dashboards by creating a simple running total DAX calculation, as shown below:
If sharing is done via direct dashboard shares, one could also analyze which drilling path (from dashboard to report) is the most popular. This is because reports do not show up on the user menu when a dashboard has been shared directly (only the dashboard shows up), yet users can still drill into the report from a dashboard tile.
Consumer behavior
By enhancing audit log reporting with a few calculations, we can compare the adoption behavior of consumers within specific lines of business.
By comparing individual audit logs on a specific report or dashboard, we can often see some consumers that – within a single line of business – display higher adoption rates than others. This is not necessarily intuitive, as you would think all business users with the same role would have a similar adoption profile.
It is recommended to define an ideal usage profile, by user role. With it, lots of insights could be gained by comparing the actual usage versus target usage behavior:
Note that higher usage (beyond target) may not necessarily translate into desired adoption behavior. For example, a user that accesses a report on a daily basis even when an automated data refresh is scheduled weekly might be experiencing issues consuming the report (for instance, they may have trouble retaining information due to a highly busy report layout).
Audit logs provide a starting point from which we can have an informed conversation with the business regarding their satisfaction with the report consumption experience.
Prototyping vs. Consumption
It is common for Super Users and/or Power Users to build reports on behalf of consumers. In this scenario, picture an adoption profile in which the Super/Power User will have usage activity spike during tactical prototyping, followed by a decline once the report is published to production (via Content Packs or direct dashboard shares). While their usage declines, you would expect increasing consumer usage signaling a positive adoption rate, as shown below:
Although audit logs do not provide information to separate user types (consumers versus power users), a Power BI report using audit data can easily be enhanced to categorize users based on known profiles.
Cycles of prototyping should normally be followed by spikes in adoption. Differing usage patterns can be a signal of adoption problems. For example, the usage pattern below could emerge when Super Users modify the model and/or report too frequently, which can confuse consumers who may have shared some initial excitement, but are no longer sure what to expect next time they access the report.
Audit logs do not dictate a method for tracking usage or adoption. However, as you plan for deployment, you likely have assumptions on how you intend super users, power users and consumers to leverage Power BI reporting. By explicitly defining expected usage patterns, the BI team can corroborate adoption, and proactively take corrective measures when log data indicate user behavior that does not conform with expectations.
If you have questions about BI or want to know more about how we might be able to help, contact us!
