Measuring BI performance
A question often posed by professionals wanting to know how they can tell if their BI solution is performing well : How do you Measure Business Intelligence Performance? There are of course many ways to measure performance (take a look at our previous post, 30 Business Performance Indicators you can (and should) Measure") which can often boil down to tracking the quantified monetary benefits against the system cost. It is also worth noting that each industry is different and there are measurement parameters that are often industry or function specific.
Some performance measures
Generally, the most common way of measuring business performance requires measurement of some type of goal or KPI, whether it is an initial or newly formed one. This normally ties in nicely with measuring ROI. Measurement parameters could include, but are not limited to: scalability, usability, reliability, quality and adoption/usage.
Before implementing a BI solution there are normally some key organizational objectives and KPIs that have been put in place to work towards. Measuring business intelligence performance based on successful completion of these goals alone is perhaps a little too rigid, but certainly using them as indicators of BI performance can be helpful. ROI, often closely related to an organization's goals, can be used as an easily quantifiable measure of BI success, making it perhpas the "strongest" measure available.
Adoption/usage has the potential be a really good indicator, if able to be correctly quantified. A common problem with this type of parameter is that it's sometimes quantified too late on in the measurement process to be able to make any constructive changes - for example, if very few people are using your BI system, but you don't know why, it's more a case of too little too late since people have already lost interest.
In terms of usability, people suggest that as long as the system is being used and is easy enough to use, then it is performing. But just because a team is using BI a lot, does this mean it performs better than a team that only use it occasionally? Or if your system is particularly simple to use, does this make it a better than a system that requires help from the IT department?
Quality could refer to the quality of data used with the system, the quality of the system itself or the quality of the results derived from the BI analysis. Data quality can highly affect the performance of the BI system as well as the outcome, placing it as a strong measure of business intelligence performance.
Reliability - everyone wants a reliable BI solution. Something that will give consistent and accurate results. A BI solution could be considered as well-performing if it always delivers what the user wants. But this measure alone is probably not enough to quantify the performance level of a business intelligence system...
It has been suggested that at the end of the day, the success of a BI initiative should be measured in terms of value creation : whether the BI solution gives the business the tools to get as much value out of their data as possible, while at the same time providing valuable insight that can be applied to leverage enough money to cover the cost of implementing the BI solution and make a profit. What we drew from our research into this topic is that there is no standard way of measuring BI performance : each organization has different ways of measuring performance and quantifying success. It is highly likely that organizations use a combination of several of the above measures, as well as others not mentioned here, to draw conclusions about the performance levels of their BI solution.