Someone recently asked me through LinkedIn how much it would cost to implement a BI project with Microsoft tools. This is a big question that can’t easily be answered without understanding the variables of scale, complexity, user and business requirements. My knee jerk reaction to a question like this is to stand by the “it depends” answer but this is a very good question. In this new age of self-service BI, can BI work really be more affordable? At the PASS Business Analytics Conference in San Jose last week I spoke with data analysts (not just IT developers) who were actually building BI solutions by themselves for groups of business professionals to run their companies using tools like Power Pivot and Excel. The monthly subscription cost for SharePoint Online and Power BI is $40 per person. Considering the power of these tools, if this meets their needs, that’s pretty cheap.
For substantive BI projects, some cost factors include the number of users, volumes of data, number of reports and specific components of the solution. Honestly, a small to moderate BI solution can realistically cost somewhere between $1,000 and $1,000,000 depending on many factors. If the source data quality is very good and data is conformed across the business, costs will be lower and if not; estimating is more difficult. I nearly always advise an assessment project followed by a proof of concept.
It is true that a single developer could use Power BI & Excel to deliver a small solution for a few hundred dollars in subscription fees that could be usable in a couple of weeks. A larger, more formal project employing 5 full-time developers in 6 months using the full suite of SQL Server BI tools might cost a million dollars to deliver.
I like to respond to questions with real answers and so I thought about the cost of some of the smaller BI projects I’ve worked on. From that experience, here is a realistic example for a small, 2-3 month project that might serve a hundred or so users:
Server hardware: $50,000
Software licensing (SQL Server & SharePoint): $100,000
External resource labor: $100,000
Internal resource reassigned labor: $100,000
Training & other misc costs: $50,000
______________________________
Rough guestimate total: $400,000
This estimate would be for a team that have done this kind of work successfully a few times and understand the cadence of BI project work. If you have not gone through this process, you could multiply the estimate by 2-3x. Every one of these cost components depends on factors like scale and complexity, changing requirements and data quality issues. For example, SharePoint server licensing cost increases when you add users so a solution for 100 users might cost $400,000 and a solution for 2000 users might cost $2 million.
As a consultant, I’m naturally inclined to recommend hiring a seasoned expert to manage things and steer the team through the project maze. The honest fact is that BI work is something we get good at only with a lot of experience. It takes time and energy to fully understand the dynamics of data, development teams and the unique challenges involved in connecting business, technology and information as we do with BI.
There are really no close parallels to BI projects so until you have a chance to go through the process, it’s difficult to anticipate costly elements like data quality problems that don’t show up until late in the project, learning to use semantic modeling tools and query languages like MDX and DAX, error handling and configuration management in ETL packages and reporting tools that seem to be easy to use until you take on complex visualization challenges. Compared with other types of software projects, in BI the easy stuff is easier and the hard stuff can be a lot harder. This makes it increasingly difficult to estimate the time and cost to complete a project so the next time a BI consultant answers your request for an estimate with an answer like “it depends” and “oh, somewhere between a thousand and a million dollars, please understand that we really mean it. Sometime we laugh a little because we know how hard it is to take this seriously but this is the truth.
Estimating this type of work is inherently hard. Several cost estimating formulas have been devised in the industry but for the most part, they’re all B.S. and exist simply because bean counters and executives demand hard numbers. An estimating formula taught in prominent project management curriculum is to add an arbitrary best-case estimate, a median or most likely estimate times four, and worst-case estimate and then divide that result by six. That’s it. It’s a little like the secret recipe for Kentucky Fried Chicken used by hundreds of professionally-trained project estimators who know the inside secret. Again, the super-secret formula is:
( [Best-case estimate] + ( [Median estimate] x 4 ) + [Worst-case estimate] ) / 6
Don’t get me wrong, I realize that it’s not acceptable in business to say “I don’t know what it’s going to cost. Just give me an open budget and we’ll do our best to deliver some results.” I once worked for a small consulting shop who serviced some big, well-known clients with very strict requirements for their vendors. We put a guy in charge of our estimating process. He analyzed dozens of projects and sought the advice of many experienced project managers and developers. He reviewed the final cost of several finished projects and came up with formulas to valuate the cost of application modules, database objects and individual lines of code. In the end, the best formula was to ask the developer to estimate their time to write and debug their work, apply a standard hourly bill rate, double that and add a third. I’m serious… that was our method – and it worked!
I worked for a larger, more formal consulting firm and they had an estimating model that was meticulously engineered down to every single item and unit of work that was recorded using piles of forms and templates. It took more effort to estimate the project then to actually build the solution. We had big clients with deep pockets who were accustomed to paying high hourly bill rates and covering even higher rates for Vice President and Director level people to come in for meetings and dinner shin digs. Nearly every project finished on-time and within budget! The funny thing is that the reason they were so good at estimating the project cost so accurately was that the estimate became a self-fulfilling prophesy. They built all this overhead into the estimate and then used it up when as they were flying all the consultants across the country every week to stay in fancy hotels and spend per diems. The work was executed as planned with plenty of management and administrative staff to track every single hour and deliverable. The best BI expert consultants I know are pretty responsible with their clients’ travel budgets and expenses. Their hourly bill rate is on the high side but they are worth every penny when you consider the return-on-investment for a successful project. In the end, the business gains value and thrives because business leaders can make informed decisions.
BI work is as much art as it is science. The science part provides methods and disciplines based on standards and industry best practices but the artistic element of data modeling and visualization is a path to both innovation and distraction. Self-service BI provides interesting opportunities to explore data and discover patterns but new ideas can also take a project off path. The great challenge is to envision a solution and deliver it within the defined scope and estimated budget. After delivering what was promised, we can build on the success of the project and be more creative after building on the foundation of a well-crafted BI solution.
