How to Configure the Power BI Gateway to use Dataset Connection Parameters

A service provider or vendor might want to publish multiple copies of a report that should connect to different database servers or databases.  In a true multitenant service solution, we would have a singe database with row-level user mapping tables that filter data by the logged in user.  True multitenant solutions require quite a lot of planning and development work to implement.  In smaller-scale or interim solutions, copies of a report can be deployed to different workspaces and then queries can be parameterized to use different database connections.

In this post, I’ll demonstrate deploying and configuring such a solution where the server name and database name have been parameterized and setup to use the on-premises gateway to connect and refresh data.  I’ll also setup scheduled refresh.  The full video walk-through is below but I’ll do a quick review to set the stage.

This is the Power Query Editor in Power BI Desktop.  I have two parameters that are used to specify the ServerName and DatabaseName for each SQL Server query:

Power BI Gateway with Parameters (Time 0_00_57;24)

Once deployed to a workspace in the service, the gateway must be configured with a data source for every possible server name and database combination.  In my example, I can connect to my local server using the NetBIOS name, IP address or LocalHost.  These all are acceptable methods but a data source must be added to the gateway configuration for each so the that the connection strings match exactly.  Remember that the connection is from the on-prem gateway to the database server so names like LocalHost or an internal IP address will work just fine.  In my example, I’m using the IP address of my local loopback adaptor on the local machine to connect to a local instance of SQL Server over the TCP connection.

Power BI Gateway with Parameters (Time 0_04_31;15)

In the workspace, the dataset is bound to the gateway.  Click the ellipsis and choose Settings.

Power BI Gateway with Parameters (Time 0_02_44;22)

To bind the gateway to the dataset, click to select the radio button next to the gateway.  This flips the switch titled “Use a data gateway”.  Apply the setting and then you can refresh the data or schedule refresh.

Power BI Gateway with Parameters (Time 0_05_47;00)Power BI Gateway with Parameters (Time 0_06_09;27)Finally, the parameters can be updated right here in the dataset settings.

Power BI Gateway with Parameters (Time 0_06_46;21)

 

Video Demonstration

 

Pareto, Burn-down & Accumulating Trend Charts in Power BI

I’m managing an Agile team project using Microsoft Teams – the new project management platform integrated with Office 365.  Teams is a simple and useful project management tool but it’s new and light on features.  Using Power BI, we want to show the hourly task burn-down for each two-week sprint.  In JIRA and some other more mature project management platforms, the burn-down chart is a standard feature in the tool that shows the number of hours or story points remaining, compared to the estimated number for the sprint.  Just as I began working on that, a client asked for some help creating a Pareto chart and it occurred to me that burn-down and Pareto charts are very similar variations of the same type of chart presentation.  These are not so much chart types as they are a set of calculations and techniques for displaying a desired result.

Project Hours Burn-Down Chart

Here’s the Burn-down chart showing days of the sprint on the X-axis and the hours for all resources as columns.  The burn-down line represents the number of estimated hours remaining for the duration of the sprint as a percentage of the total estimated hours for all resources.

imageHere’s another variation of the burn-down chart using stacked columns for each project resource, in my example; developers named Marta, Rob and Vivek.  Again, the burn-down line shows the daily percentage of estimated hours remaining compared to the total hours that have been “burned” for all days up to and including the current day.

image

The columns in the chart are simply the sum of hours reported.  I didn’t even create a measure for this value.  It’s just a summable column in the ‘Project Hours’ table named “Hours”.  Here’s a look at the data, including the measures used in the chart:

image

…and finally, here is the DAX measure code used for the line part of the column/line combination chart:

Project Hours Remaining % =
VAR DayAccumHours =
CALCULATE( SUM( ‘Project Hours'[Hours] ),
FILTER( ALLSELECTED( ‘Day’ ),
‘Day'[Day Number] <= MIN( ‘Day'[Day Number] )
)
)
VAR ProjectHoursRemaining = SUM( ‘Sprint Estimates'[Est Hours] ) – DayAccumHours

RETURN
DIVIDE( ProjectHoursRemaining, SUM( ‘Sprint Estimates'[Est Hours] ) )

Pareto Chart

This style of chart is very similar but the key differences are that the columns are ordered by the contribution value in descending order.   You can see that my sample dataset just uses numbers for the categories (e.g. “8”, “2”, “4”, etc.) but these could just as easily be names of resources, sales people, customers or products.  The columns are in descending order of aggregate value rather a time series of the axis field value.  To order by measure or value (my aggregate field is literally called “Value”), set the Type property for the X-axis to “Categorical”.

SNAGHTMLc49dd9d

Here’s the Pareto chart.  The contribution line shows the percentage of the total contribution that a category item and all of it’s predecessors in ranked descending order have made to that point.  For example, the first four top-ranking categories (8, 2, 4 and 5) account for about 50% of the total.

image

The DAX measure code for the accumulating percentage value used for the line, is as follows:

Accum Value % All =
VAR AccumValue =
CALCULATE( SUM(‘Pareto Series'[Value] ),
FILTER( ALLSELECTED( ‘Pareto Series’ ), ‘Pareto Series'[Value] >= MIN( ‘Pareto Series'[Value] ) )
)
VAR AllSeriesTotal = CALCULATE( SUM( ‘Pareto Series'[Value] ), ALLSELECTED( ‘Pareto Series’ ) )

RETURN
DIVIDE( AccumValue, AllSeriesTotal )

Accumulating Trend Series or Trend Chart

For good measure, I have one more example.  I actually started experimenting with calculation techniques using this chart.  It is essentially the burn-down chart example with the line presenting the opposite of the burn-down, so maybe this is a “burn-up chart” – I don’t know.  This may be useful when you want to visualize a time or other continuous series on the X-axis and the accumulating contribution as a line along with the actual contribution values as columns.

image

…and the measure DAX code used for the trend line:

Accum Series Value % All =
VAR AccumSeriesValue =
CALCULATE( SUM(‘Pareto Series'[Value] ),
FILTER( ALLSELECTED( ‘Pareto Series'[Period] ), ‘Pareto Series'[Period] <= MIN( ‘Pareto Series'[Period] ) )
)
VAR AllSeriesTotal = CALCULATE( SUM(‘Pareto Series'[Value] ), ALLSELECTED(‘Pareto Series’ ) )

RETURN
DIVIDE( AccumSeriesValue, AllSeriesTotal )

I’m hopeful that this will be useful and save you some time and energy.  As always, please leave comments and reach-out if you need some help.

You can download the example Power BI Desktop file here: Simple Pareto & Burn-down Chart Example.zip

Power BI for Grownups

The message from the Microsoft Business Applications Summit this week was very clear: Power BI is growing up.

We have known for a while that Power BI is a great front-end tool for enterprise-scale tabular and multidimensional models.  Power BI Premium will now allow almost limitless data scale using extremely large datasets by leveraging composite models that employ both DirectQuery and VertiPaq cached mode tables.  This feature shatters the previous barriers where data models could only support either DirectQuery or cached mode (imported data).  Additionally, Microsoft publicly unveiled plans to add IT developer centric features to Power BI Desktop for managing large models, version control and application development lifecycle.

These announcements leave many open questions about how Microsoft will continue to support self-service and enterprise customers with one tool and about the long-term future of SSAS and Visual Studio-based SSDT. At this point, none of these tools are on the chopping block, but it is clear that James Phillips and his team have significant momentum and are not slowing down to wait for other products to catch-up.

In this interview with Christian Wade, he describes the benefits of this new capability.  Christian is Group Program Manager on the Power BI and Analysis Services product team with focus on data model development and scalability features.

Composite Models & Aggregations

In the opening keynote and again in his sessions, Christian demonstrated Power BI reports on the taxi driver activity database with over a trillion rows of raw data.  The larger dataset was in a Spark cluster, accessed using DirectQuery.  Aggregated tables were stored in the in-memory model using the new composite model feature.  As the data was explored in report visuals, the Power BI engine would seamlessly switch from tabular in-memory aggregate tables to DirectQuery source data in order to return low-level details.  Composite models will allow mashing-up imported database and file-based data with an DirectQuery.

There are limits and complexities with these new features.  You cannot mashup imported tables in a Power BI model based in a direct connection to SSAS, but enterprise-scale features in Power BI arguably may not steer a solution architect to select SSAS over Power BI for serious data modeling.  With incremental data refresh, large model support, row-level security and many other “big kid” features, Power BI might be a preferable choice.  I’m not ready to rule-out Analysis Services as the better option for most enterprise solutions – at least not in the near future, but Power BI is definitely heading in that direction.

Power BI Desktop Features for Developers

News Flash: Power BI Desktop will be the primary tool for serious, enterprise data model development.  A bunch of features are in the works and will arrive soon for preview.  The obvious question is what will happen to SSDT for Visual Studio and the answer is that it’s not going away but the product team is putting their energy into enhancing Power BI Desktop for now.  It will likely have some kind of “Beginner/Advanced” switch to simplify things for the novice user and isolate advanced or complicated, developer-centric features.

Here are a few screen captures of the forthcoming features:

Schema Compare & Model Integration

Like the BISM Normalizer tool created by Christian Wade, there will be a fully-supported compare and merge tool baked into Desktop.  The early build is branded with components from MAQ Software.

image

Relationship Diagram Designer, Properties Pane and Application Lifecycle Management Features

A new-and-improved diagram designer is imminent.  It will allow large models to be more effectively managed and displayed.  You will have the ability to save individual views of the diagram with selected, subject-area tables.

image

Like in Visual Studio, properties can be set and managed in a Properties sheet window.  Support for standard SSAS features like display folders is coming and can be managed here.

image

Business Applications Release Notes

With the velocity of features and updates, a comprehensive set of release notes are available here.  This document will be updated frequently with the revised roadmap, feature history and announcements.

APIS and Powershell CmdLets for Power BI Administration

At the MVP Summit in March Kay Unkroth promised API endpoints and Powershell CmdLets for managing all aspects of Power BI.  He even did a dance and sang a song to make the announcement memorable.  This week, the product team delivered what they promised, with the announcement official here.

The following features and capabilities are on the delivery short list coming in the next 2-3 months:

Power BI Service

The new, “modern workspace” eventually replacing current workspaces integrated with Office 365 groups.  The new app workspace will no longer be bound to an Office 365 Group. The feature will be optional at first, but will replace current workspaces soon.

Now, users can be: Member, Admin, Contributor

Can add groups to workspace

Premium capacity tenants can have nodes in different regions

Admins will have visibility and access to all objects and resources

Microsoft Business Applications Summit

Keynotes and Featured Sessions from the Microsoft Business Applications Summit are here.

SQL or M? – SSAS Partitions in Power Query/M

This is a continuation of this post

In the data platform industry, we have been working with SQL for decades.  It’s a powerful language and over many years, we’ve learned to work with it’s strengths and to understand and work around it’s idiosyncrasies.  M is a considerably more modern and flexible query language.  Some best practices have evolved but many are still learning the basic patterns of effective query design.  Reaching that stage with a technology often takes years of trial-and-error design and a community willing to share their learnings.  I will continue to share mine and appreciate so many in the community who share theirs.

Why Use M Instead of SQL?

For database professionals using SQL Server as the sole source of data for an SSAS or Power BI data model, there is a solid argument to be made in favor of encapsulating the query logic in database objects.  DBAs need to manage access to important databases.  A comment posted in an earlier post on this topic mentioned that SQL Server views can implement schema binding – which doesn’t allow a table or any other dependent object to be altered in such a way that it would break the view.  This is a good design pattern that should be followed if you are the database owner, have the necessary permission and flexibility to manage database objects as part of your BI solution design.  Ultimately, this is an organizational decision.  If the BI solution developer is not the DBA, you may have limited options.  If you don’t have control over the source database objects, if you are not using SQL Server or otherwise prefer to manage everything in the SSAS or Power BI project, Power Query is probably the right place to manage all the query logic.

In my earlier post, I used a table-valued user-defined function to manage the partition filtering logic in SQL Server.  Rather than using SQL and database objects, we’ll use Power Query alone.  The working M script is shown below.

For brevity, I’m starting by showing the solution but I will show you the steps we went through to get there a bit later.

Updating the Partition Definitions

The three partitions defined in the earlier example are replaced using the following M script, which returns exactly the same columns and rows as before.

image

Here is the M script for each partition:

“This week” partition:

let
Source = #”SQL/localhost;ContosoDW”,
SalesData = Source{[Schema=”dbo”,Item=”FactSalesCompleteDates”]}[Data],
#”Filtered Rows” =
Table.SelectRows( SalesData, each
[DateKey] >=
Date.StartOfWeek( DateTime.FixedLocalNow() )
)

in
#”Filtered Rows”

“This month before this week” partition:

let
Source = #”SQL/localhost;ContosoDW”,
SalesData = Source{[Schema=”dbo”,Item=”FactSalesCompleteDates”]}[Data],
#”Filtered Rows” =
Table.SelectRows( SalesData, each
[DateKey] >=
Date.StartOfMonth( DateTime.FixedLocalNow() )
and
[DateKey] <
Date.StartOfWeek( DateTime.FixedLocalNow() )
)

in
#”Filtered Rows”

“Before this month” partition:

let
Source = #”SQL/localhost;ContosoDW”,
SalesData = Source{[Schema=”dbo”,Item=”FactSalesCompleteDates”]}[Data],
#”Filtered Rows” =
Table.SelectRows( SalesData, each
[DateKey] <
Date.StartOfMonth( DateTime.FixedLocalNow() )
)

in
#”Filtered Rows”

The Power Query Litmus Test: Query Folding

When connected to an enterprise data source like SQL Server, Power Query should be able to pass an important test.  Use the Design… button to view the Power Query Editor.  Select the last query step and then right-click to show the menu.

image

If the View Native Query menu option is enabled, you are good.  This means the the query is being folded – and that’s a good thing.  Query folding converts the query steps into a native query for the database engine to execute.  This is the resulting T-SQL query script generated by Power Query:

select [_].[SalesKey],
[_].[DateKey],
[_].[channelKey],
[_].[StoreKey],
[_].[ProductKey],
[_].[PromotionKey],
[_].[CurrencyKey],
[_].[UnitCost],
[_].[UnitPrice],
[_].[SalesQuantity],
[_].[ReturnQuantity],
[_].[ReturnAmount],
[_].[DiscountQuantity],
[_].[DiscountAmount],
[_].[TotalCost],
[_].[SalesAmount],
[_].[ETLLoadID],
[_].[LoadDate],
[_].[UpdateDate]

from [dbo].[FactSalesCompleteDates] as [_]

where [_].[DateKey] >= convert(datetime2, ‘2018-07-01 00:00:00’) and [_].[DateKey] < convert(datetime2, ‘2018-07-15 00:00:00’)

You don’t need to do anything with this information.  It’s just good to know.  End of story.

And Now… The Rest of The Story

Power Query is an awesome tool that does some amazingly smart things with the simple data transformation steps you create in the designer.  However, it is important to make sure Power Query produces efficient queries.  During the development of this solution, I created an early prototype that didn’t produce a query that would fold into T-SQL.  Thanks to Brian Grant, who is an absolute genius with Power Query and M, for figuring this out (BTW, you can visit Brian’s YouTube tutorial collection here).

In my original design which I prototyped in Power BI Desktop, I thought it would make sense to create custom columns for each of the date parts needed to filter the partitions.  Here’s the prototype query for the query I originally named “This Month Thru Last Week”:

let
Source = FactSales,

#”Add DateTimeNow” = Table.AddColumn(Source, “DateTimeNow”, each DateTime.LocalNow()),
#”Change Type DateTime” = Table.TransformColumnTypes(#”Add DateTimeNow”,{{“DateTimeNow”, type datetime}}),
#”Add StartOfThisWeek” = Table.AddColumn(#”Change Type DateTime”, “StartOfThisWeek”, each Date.StartOfWeek([DateTimeNow]), type date),
#”Add StartOfThisMonth” = Table.AddColumn(#”Add StartOfThisWeek”, “StartOfThisMonth”, each Date.StartOfMonth([DateTimeNow]), type date),
#”Add StartOfPreviousMonth” = Table.AddColumn(#”Add StartOfThisMonth”, “StartOfPreviousMonth”, each Date.StartOfMonth(Date.AddMonths([DateTimeNow], -1)), type date),
#”Partition Filter” = Table.SelectRows(#”Add StartOfPreviousMonth”, each ([DateKey] >= [StartOfThisMonth] and [DateKey] < [StartOfThisWeek]) ),
#”Removed Columns” = Table.RemoveColumns(#”Partition Filter”,{“ETLLoadID”, “LoadDate”, “UpdateDate”, “DateTimeNow”, “StartOfThisWeek”, “StartOfThisMonth”, “StartOfPreviousMonth”})

in
#”Removed Columns”

As you can see, I created separate columns using Transform menu options based on the current date and time, stored in a custom column named “DateTimeNow”:

  • StartOfThisWeek
  • StartOfThisMonth
  • StartOfPreviousMonth

The rest was simple, I just added filters using these columns and then removed the custom columns from the query in the last step.  All good with one small exception… it didn’t work.  We learned that Power Query can’t use custom column values to build a foldable filter expression.  The filters just won’t translate into a T-SQL WHERE clause.

Checking the last query step with a right-click shows that the “View Native Query” menu option is grayed-out so No Folding For You!

image

Simple lesson: When query folding doesn’t work, do something else.  In this case, we just had to put the date comparison logic in the filter steps and not in custom columns.

SQL or M? – SSAS Partitions Using SQL Server Table-Valued Functions (UDFs)

[ Related posts in this series: SQL, M or DAX: When Does it Matter? SQL, M or Dax? – part 2 SQL or M? – SSAS Partitions Using SQL Server Table-Valued Functions (UDFs) SQL or M? – SSAS Partitions in Power Query/M ]

In SQL Server Analysis Services tabular projects, as of SQL Server Data Tools 2017, you can define table partitions using Power Query.  Of course, we still have the option to use SQL Server database objects like views or user-defined functions.  So, which of these two option makes most sense?  The same concepts and decision points apply to Power BI data models although the design experience is quite a bit different.

The following steps will bring us to a question: Using the new SSDT partition design method for SSAS 2017, should I define partition filtering logic in SQL or in Power Query/M?

The objective is to define three partitions in the data model for the Sales fact table in the ContosoDW database:

  • New transactions added in the current week
  • Adjusting entries for the current month
  • Historic records prior to the current month

New sales transactions in the source database needs to be refreshed in the data model every hour for reporting.  Reprocessing only the records since the beginning of the current week takes seconds to minutes.  If we schedule that partition to refresh every hour, users can have up-to-date reports throughout the day.  In addition to new transactions, adjusting records are made weekly but only to records in the current month before the end-of-month closing of the books.  Records in the current month that are older than the current week might be updated on occasion but changes don’t need to be available until the weekend.  Records older than a month rarely change and don’t need to be refreshed but once a month.  By scheduling only the first or second partition to process, data can be updated without requiring tens of millions of historical records to be reloaded.

Partitioning with a SQL User-Defined Function

I’ll step through the more conventional method we’ve been using for many years.  I’ve written the following T-SQL table-valued User-Defined Function named fnSalesPartitionForPeriod.  Three possible input parameter values allow the function to return rows for the past week, for the past month (up to the past week) or for all dates previous to the current month.

Here is the T-SQL script for a table-valued user-defined function created in SQL Server.  Passing in one of three parameter values will cause it to return the desired records.

/******************************
User-defined function used to partition Sales fact table in SSAS tabular model
@Period values:
PriorToThisMonth
ThisMonthPriorToThisWeek
ThisWeek

*******************************/

create function dbo.fnSalesPartitionForPeriod
( @Period varchar(100) )

returns table

return
select * from [ContosoDW].[dbo].[FactSalesCompleteDates]
where
(@Period = ‘BeforeThisMonth’
and
[DateKey] < dateadd(month, datediff(month, 0, getdate()), 0) ) or (@Period = ‘ThisWeek’ and [DateKey] >= dateadd(week, datediff(week, 0, getdate()), 0)
)
or
(@Period = ‘ThisMonthBeforeThisWeek’
and
[DateKey] >= dateadd(month, datediff(month, 0, getdate()), 0)
and
[DateKey] < dateadd(week, datediff(week, 0, getdate()), 0)
)

;

go

To create the three Sales table partitions using this UDF, I start by importing one table.  Here’s the Import Table dialog for the new Sales table in the data model.  I’ve selected the new UDF and entered the parameter value ‘BeforeThisMonth’ to define the first partition.

image

This part gets tricky and quite honestly, I rarely get the steps right the first time through.  I haven’t quite decided yet if my routine struggle with the SSDT Power Query editor is because I expect it work like it does in Power BI Desktop or if it truly has some quirks that catch me off guard.  Some of each, I think.  Regardless, I’m cautious to save copies of my work and if something doesn’t work, I delete the query and repeat the steps.

The query editor was smart enough to create an M function from the UDF query and this function needs to be invoked to generate the new Sales table.  Enter the parameter value once again and click the Invoke button.

image

Change the name of the new query to “Sales” and make sure that the query is set to “Create New Table”, then click the Import button on the toolbar.

image

After the table is imported, click the Partitions button on the SSDT toolbar.  As you can see, the Power Query “M” script for the Sales table calls the function and passes the parameter value I had set.  This default partition should be renamed and the other two partitions should be added using different parameter values.

image

Updating and adding the partitions is fairly simple, using these steps:

  1. Copy the original partition
  2. Rename the new partition
  3. Change the function parameter value

Rename the current partition with a friendly name.   Clicking the Copy button twice gives me two copies of the parameter.  You can see that I’ve commented the code with the valid UDF parameter values.

SNAGHTML3791b15

Now the table can be refreshed incrementally and only new transaction records for the current week or month can be updated during schedule refresh cycles.  Seeing green on this dialog is always a welcome sight:

SSAS Partition Process 2018-07-15_18-31-30

Partitioning with Power Query

No matter what the data source is; whether you use table-valued UDFs, views or in-line SQL, you are still using Power Query to define tables – so why not just use Power Query without creating database objects?

In another post, I’ll repeat the exercise using only Power Query to define the same partitions.

SQL or M? – SSAS Partitions in Power Query/M

Hands-on Workshops at the Pacific Northwest Power BI Symposium

Please join Power BI authors and community leaders for an afternoon and evening of deep learning.  Featured presenters include “Guy In A Cube” Adam Saxton and international author and Excel MVP, Matt Allington.  Deepen your skills with Power BI and get a recap from the Microsoft Business Applications Summit held in Seattle earlier in the week.

Jul 26, 2018 from 3:00 PM to 8:30 PM Portland, OR

Associated with  Portland Power BI User Group

Event Image

AGENDA –

3:00 ; check-in and registration

3:45 ; guide guests to workshops

4:00 ; workshops commence, ONLY REGISTER FOR ONE:

workshop 1 – TRISTAN MALHERBE (AZEO) – Power Query to Create a Calendar Table

register here: https://bit.ly/2K3U0lT

workshop 2 – PAUL TURLEY (CSG Pro) – Model and Visualize Financial and Accounting Data with Power BI and Excel

register here: https://bit.ly/2JOJy5v

5:00 ; workshops wrap up, seating in main event area (dinner served / networking)

5:35 ; presentation intro – RON ELLIS GAUT (CSG Pro / Portland Power BI User Group Leader)

6:00 ; presentation 1 – BRIAN GRANT (CSG Pro) – Shining a New Light on Calculate

6:45 ; presentation 2 – ADAM SAXTON (Microsoft / Guy in a Cube) – Business Applications Summit Recap

7:30 ; presentation 3 – MATT ALLINGTON (Excelerator BI) – DAX as a Query Language

8:15 ; closing remarks

8:30 PM ; event ends

Questions?  Contact Gregory Petrossian: gregp@csgpro.com for sponsorship options

SQL, M or DAX: When Does it Matter?

[ Related posts in this series: SQL, M or DAX: When Does it Matter? SQL, M or Dax? – part 2 SQL or M? – SSAS Partitions Using SQL Server Table-Valued Functions (UDFs) SQL or M? – SSAS Partitions in Power Query/M ]

Column-based calculations are part of every BI project. Some of the most common examples include building a street address column from individual fields, concatenating a person’s full name from First, Middle and Last Name fields; or creating a location string from City, State and Country fields. More complex examples might require a lookup or join operation to get a reference value used in a calculation that is then stored as a column on each entity record.  Keep in mind that we are strictly talking about calculated column values that are stored for each row and not dynamic calculations that run in the context of filters and slicers.  Those are measures and that is a separate topic.

It Depends, or It Matters (one or the other)

If the data source is a relational database that supports queries, should you perform these calculations in SQL, Power Query “M”, or in a DAX calculated column? The standard tongue-in-cheek answer from a consultant is usually “it depends”. That was the answer I grew up with, but apparently popular language has changed in the last generation. My kids, who are all now young adults, say “it matters”. Back in the day, if I said “Hey, Dad. Can we get ice cream on the way to the store?” He would say, “It depends on whether you get your chores done.” My kids would say “it matters whether I get back from the beach on time.” So, it either depends or it matters, I guess.

Self-service BI is all about having the freedom to create reports that make an impact and bring important value to business users and leaders.  When importing, shaping and modeling data; if we can get simple and mundane tasks out of the way, this leaves time and energy to move on to more important things.  If you can just get the core table structures in-place; with unique keys, calculated columns, and numeric columns for summaries and aggregate measures, you can design the more impactful bits of the solution to support the report design.

Many BI projects start out the same way, with aspirations to import data from several different sources, to work-out the complexities of cleansing and matching records in various tables to create a nice uniform data model used to build all kinds of beautiful dashboards and interactive reports.  Our optimism about making quick progress at the beginning of the project is often squelched when we realize that the data source for a lookup table isn’t reliable, and that the system of record is an application controlled by a different business group in some remote corner of the organization. The data is in a different format, access is restricted and the person in charge of managing it is on extended leave. We get caught-up in the complications of just getting essential data into the model and then deliver far less than expected. I can’t tell you how common this scenario is – especially in larger projects.

For calculated columns that end up stored in a data model table, there is rarely a difference in performance, storage or report query speed based on the technique used to calculate the column value. In cases where there is a technical advantage, the decision should be clear – use the most optimal method that is feasible. In the majority of cases where there is no strong technical argument for one method or the other, use the method that simplifies development and maintenance, and offers more control.

You should have a standard method for managing calculated columns, so you know how to maintain them down the road. This might seem trivial so why does it matter so much?

The data model schema is the foundation for your reporting solution and making changes after the rest of the solution is designed can be catastrophic if you don’t plan and manage future changes. A semantic data model is literally a house of cards. Deleting, renaming or changing the data type for a column could break every calculation and the report visual referencing that column. Whether you should create these calculated values in a source query using hand-written SQL or a database object like a view or user-defined function, in Power Query or as a calculated column using DAX – that will depend on who maintains the Power BI or SSAS model and who should manage the design in the future.

SQL and Database Objects

As a general rule of thumb, in formal SSAS projects built on a relational data mart or data warehouse that is managed by the same project team as the BI data model, I typically recommend that every table in the model import data from a corresponding view or UDF stored and managed in the relational database. Keep in mind that is the way we’ve been designing Microsoft BI projects for several years. Performing simple tasks like renaming columns in the 2012 or 2014 SSAS data model designer was slow and cumbersome. Performing this part of the data prep in T-SQL was much easier than in SSDT. With the recent advent of Power Query in SQL Server Data Tools 2017, there is a good argument to be made for managing those transformations but the tool is still new and frankly I’m still testing the water. Again, keep changes in one place for future maintenance.

Do your absolute best to avoid writing complex SQL query logic that cannot be traced back to the sources. Complicated queries can become a black box – and a Pandora’s box if they aren’t documented, annotated and easy to decipher.

Power Query/M

For less formal projects in Power BI data models, Power Query is king. We’ve never had a tool so flexible and easy to use. If I’m importing data from multiple sources into a single model, you bet I’m going to use Power Query instead of SQL queries because I’ll know where to find and manage all the query definitions.

DAX Calculated Columns

Why not use DAX calculated columns? There is a good argument for using DAX. It’s quick and easy, and sometimes more convenient. If I add a custom column to a multi-million row table defined in Power Query, I have to re-process the table to see the new column. If I use DAX, I don’t have to wait. If the calculation relies on a DAX calculation residing in the rest of the model, DAX is the clear winner. These cases are less common though. Once again, I’ll make the argument to manage calculations, as much as possible, in one place.

IT Process, Business Culture, Team Dynamics, Rules & Restrictions

Now that you have some clear criteria for always implementing column calculation in either SQL, Power Query or DAX; let me inject some reality back into the “it depends or it matters” equation.

I do my best to put cynicism aside and focus on what it takes to get IT projects over the finish line. I’ve found that good BI project practitioners are positive, optimistic and tough skinned; although there are many forces at work to change this disposition. If you know what I mean, no further explanation is needed. If you don’t, you will. Wherever you choose to work, just do what you can to maintain your perspective throughout your career.  There is much insight to be had when you learn how projects work and don’t work in different organizations.

I’ll give one example that represents situations I’ve encountered on several larger, formal BI projects over the years.  Defensively, I’ll say that this story is “inspired by true events” in my experience:

The BI project Architect, Database Administrator, Lead Developer and IT Director all agree that any schema dependencies on the data warehouse or data mart should be managed using database views and this becomes a paramount rule in the solution architecture. The SSAS data model and Power BI data model developers should import tables from these views. The team is using a pure Agile mythology and will use JIRA to manage and assign tasks performed in two-week team sprints.

Based on high-level report requirements documented by the Business Systems Analyst, the Lead Architect creates views in the database build script. The ETL developer must stage the source data and then the data warehouse ETL developer populates the dimension table before the view can be created, which takes 3 sprints or six weeks. The Power BI data model developer adds the table to the model in the 4th sprint. After a prototype report is created, the BSA gets feedback from a stakeholder user who tells us that customer names should be in a single column rather than separate first name and last name columns. A task is added in JIRA to modify the view with another task to refresh the data model, so it takes two weeks to add the column.

After the data warehouse is in production, the Power BI report developer gets word that the customer city, state and zip code need to be concatenated into a single column and marked as a geographical location, so they can be used in a map visual. The data warehouse is in production and managed by an offshore DBA group. A support ticket is created to request that the view used to populate the customer table be altered and a CustomerLocation column be added. Three days later, the contracted help desk determines this is not in their area of responsibility and closes the ticket as “completed” while the model developer continues to wait for a call or email. The email goes to the IT Director, who happily dismisses it since it was marked as “completed” while the developer sits and waits. Two weeks later, the issue resurfaces and the Project Manager organizes a meeting with the IT Director, BI Lead Architect, BI Lead Developer, Database Developer, In-house DBA and Help Desk Contractor Liaison to resolve the issue. In the meantime, users have exported their report to an offline Excel file and are working around the issue using a copy of the data.

A month after the request, the Power BI Developer spends 2 minutes creating a DAX calculated column and then creates the map report.