Business Intelligence Best Practices -

Collaboration. Communication. Community.

 E-mail to friend
Ten Mistakes to Avoid When Estimating ROI for Business Intelligence

by Evan Levy
The most common missteps managers make when financially justifying their business intelligence (BI) and data warehouse programs.

In the late 1990s, rampant overspending on IT was standard practice for companies who were flush with funding and escalating equities. Today’s more sober economic climate has changed everything. Now executives are being held accountable for new capital expenditures, sometimes to the penny.

They’re also being asked to justify existing IT projects that a few years ago seemed bulletproof. Managers who thought their CRM, supply chain management, and data warehouse initiatives were safe have had a rude awakening. They are being asked to justify these initiatives or risk losing them to newer technology programs. This has meant going back to the business case drawing board and calculating return on investment, often for the first time.

As the ROI discussion grows louder, the mistakes also become more apparent. This booklet outlines what we’ve seen to be the most common missteps managers make in financially justifying their business intelligence (BI) and data warehouse programs. Hopefully, it can serve as a list of lessons learned if you too are being called upon to build a BI business case. Looked at another way, this is a list of worst practices to avoid at all costs. The future your company’s information asset may hang in the balance.

Mistake 1: Underestimating Existing Capabilities
A manager at a large wireless telecommunications firm recently called for help with her budget. She’d “taken a stab” at quantifying the ROI for the company’s data warehouse and wanted me to review her calculations. Her worksheet—which included a rigorous mix of hardware, software licenses, maintenance, burdened staff, and consultant costs, etc.—made clear that she’d forgotten an important step: understanding how current IT infrastructure and resources could be leveraged to reduce cost estimates and make a project more attractive to business executives.

With the conversation over the need for ROI quantification growing, many managers try calculating ROI prematurely. They throw everything but the kitchen sink into ROI models without understanding make-or-break issues such as:


· IT skills inventory
· Data source system inventory
· ETL inventory
· Data warehouse/mart inventory
· Software inventory
· Application inventory
· End-user communities

Overestimating costs in cases where resources or technologies already exist can sabotage a project before it starts. Even one manager who perceives startup costs to be too high or the development timeline to be too long can jeopardize an entire business case.

The inverse problem is even more widespread: exaggerated optimism that a new cutting-edge technology project can be completed on time and on budget. Decreasing funding for strategic programs like business intelligence compete with pervasive failure statistics—sometimes quoted at above 50 percent—making it even more of a mandate to determine current capabilities.

Recently, a customer sought to reengineer an outdated order entry system. The IT department wanted to upgrade its old character-based database application to one with a browser front-end and Web-based application server. The success of this team had traditionally been based on their knowledge of both business process and technology implementation. They were convinced that the browser-based application could improve order-entry productivity and dramatically reduce ongoing application maintenance costs, arguing that newer technology would reduce the ongoing cost of application maintenance.

In reviewing the development and maintenance alternatives, it became clear that such a strategy would likely fail. The issue of ROI calculation was reasonable—the numbers reflected improved business productivity and lower maintenance costs. However, the likelihood that their staff could take over the maintenance of the new technology without significant retraining and possibly re-staffing was a financial showstopper. The time necessary to either train new staff on the business, or retrain the current staff on new technology was too long. While staff training was part of the ROI calculation, the time and productivity impact for retraining was too costly. The ROI made sense—the project didn’t.

Data warehouse managers should conduct research before launching an ROI effort. (Talking to end users to obtain updated business requirements, reviewing the development pipeline, identifying IT’s skills and available platform resources, and knowing where critical data is in its evolution can all add weight to a business case and avoid the tabula rasa assumption that each project must start from scratch.)

My telco manager client realized that she had some homework do. She embarked on an informal, three-week-long application inventory in order to understand existing BI capabilities that could be leveraged for the new projects she was being asked to justify. She found that many existing applications provided data to the data warehouse in a closed-loop processing environment—data she could re-use for some of her own information reporting needs. She redid her ROI worksheet to factor in the existence of key data and processing, reducing her startup cost estimates, and accomplishing her mission: the company funded the BI portfolio through 2005.

Mistake 2: Failing to Get the Business Team Involved
No one wants to own ROI. Quantifying ROI can be incredibly difficult; understanding all the required skill and resource components can challenge even the best manager. When the planning phase of a project occurs, most IT managers take it upon themselves to calculate ROI on their own. However, determining baseline costs for both IT and non-IT resources can’t fall onto a single person. It’s important to ensure that baseline ROI numbers are acceptable to both IT and the business stakeholders.

Traditionally, ROI for technology investments has been the purview of the IT group. After all, who better to understand line item costs, licensing fees, ongoing maintenance and skill-set expenses, and all the other gritty details that translate into hard cost items?

However, business stakeholders should be involved. For one thing, they are better poised to explain the benefits of launching a new project, or continuing an existing one. Plus, nine times out of ten, they’re the ones begging for new functionality or better data. Business managers need to be ready to step up to the costbenefit plate, especially where the benefits are concerned.

The challenge for ROI calculation is the merging of both IT and business side numbers. Consider the pending Customer Value Modeling project at a regional bank. IT understands the data that will be necessary to calculate customer value, how much of that data needs to be sourced, which application platform is likely to process the calculations, the network speed necessary to “push” these scores to end users, whether additional processing power needs to be purchased, and the cost of the expertise needed to implement and maintain the system.

This is a big, complex, and expensive project. And if it’s all that management sees, it’s likely that Customer Value Modeling will remain a vision instead of a reality. An ROI calculation is only as effective as the explanation of return. This means that the success of a business case is usually directly proportional to its ability to explain the tangible benefits of an IT project.

Although the IT team is best suited to understand and estimate the costs of implementing technical capabilities, business stakeholder involvement is critical. When a seasoned marketing executive can make a case for differentiating customer treatment based on this value score, Customer Value Modeling becomes more than just another technology project. After all, understanding low-scoring customer attributes and converting them to higher-scoring customers usually means additional profitability. Pushing lower-value customers to automated support systems can cut costs. And understanding value segments can foster targeted communications, saving money in mailing costs while potentially generating additional revenues. Data analysis supporting business action is the goal of business intelligence, and the objective can’t be reached without having both business and IT involvement.

Combining costs and benefits into quantifiable numbers is the holy grail of ROI. To do this requires mutual cooperation between IT and business groups. The result is a project that loosens the purse strings in today’s tightened economic climate.

Mistake 3: Skepticism over ROI Viability
Getting management to understand and believe a project’s ROI is tremendously difficult. Even after all the research and number crunching there will inevitably be nay sayers who have problems with the numbers. The potential audience of cynics shouldn’t be a surprise; a project with calculated ROI typically threatens the very existence of other projects. Given that most companies have finite budget pool, ROI raises the bar on project justification. Once the first project is able to identify ROI, everyone else will be expected to illustrate ROI. This can be difficult, and will likely threaten the existence of many other projects. For every project that is funded, resources are usually taken away from other projects. The advent of ROI challenges the very existence of a project. While everyone has talked up the “business value” of their projects, few have been able to illustrate numbers.

It doesn’t matter whether it’s a product, a person, or a presentation. As the famous Six Sigma saying goes, “If it moves, measure it.” The challenge of determining the benefits of BI before the first application is developed or the first database is loaded can be fairly intimidating. When there’s skepticism about the value of a new BI initiative, it’s important to focus on estimating business benefits (financial as well as non-financial), and realize that cynicism is healthy. No project should be blindly approved, and every project should include a sober and realistic cost-benefit analysis.

The value of data analysis was never considered practical prior to the late 80s by the retail industry. The focus of retail operations was on cost; regardless of circumstances, cost always mattered. From reusing paper clips, to sharing desks and office equipment, nothing was sacred.

So when investing in staff productivity and business process improvement started, the idea of investing in a data warehouse to support the analysis of integrated customer purchase details, product inventory, distribution, and pricing seemed ridiculous. The cost for storing that information was measured in millions of dollars. This scale of investment had never been acceptable, “…no employee is worth that much…” However, the business paradigm was about to shift; the goal wasn’t just cost, it was also about business value.

Investing in staff productivity improvement isn’t new. However, as companies grow and become more complex, the value of intellectual capital increases many-fold. At large, general merchandise retailers like Sears and Wal-Mart, a merchandiser may control $5 billion or more of inventory—and that could be 10,000 products and dozens of suppliers. Clearly the complexity of the job is significant; and a minor mistake could easily cost the company $25 million (less than 1 percent of their inventory responsibility).

The impact of ROI in this particular circumstance was really in the merchandise staff member’s ability to better manage inventory and vendor relationships. An improvement of 1 percent or 2 percent in their purchasing responsibilities wasn’t unreasonable with better analysis and tools. So, while the cynics had argued that investment costs were prohibitively expensive, the business folks focused on the ROI opportunities for the company.

Mistake 4: Forgetting that Change Is Constant
There’s a reason that executives use the terms “forecast “ and “estimate” in many of their communications: they’re not 100 percent sure of the results on a pre-facto basis. Likewise, business expectation for ROI shouldn’t focus on perfection, and project approval should never be based solely on ROI estimates.

A regional airline—one of the few airlines that has experienced growth, profitability, and increasing customer satisfaction is one example of how focusing on numbers is only part of the story. The airline industry has undergone extensive challenges: skyrocketing fuel costs, reduced passenger travel, and a health epidemic that crippled the Asian travel market. One reason that this airline has been successful is that they focus on the numbers but at the same time realize that their business isn’t bulletproof.

For instance, the airline couldn’t possibly have foreseen the SARS outbreak. While a well-planned project should identify most foreseeable business issues, planning for every possible situation isn’t practical—it’s also likely to be a waste of time and money. Companies like GE don’t focus solely on their ability to accurately forecast business trends; they rely on their skills to respond to change. They make this happen through the use of BI and studying the details of both historical and current business data.

Those airlines that survived the surprise of the SARS outbreak didn’t succeed because they continued to focus on their financial estimates. They realized that the world had changed and their estimates were wrong. The old lyrics “Pick yourself up, dust yourself off, and start all over again,” became the plan. This airline continually analyzed their forward bookings data to understand how to better deploy their equipment and their staff. They shut down unprofitable markets and focused more resources towards other business opportunities.

Regardless of the accuracy of the initial project estimates, the numbers should be updated and communicated to reflect reality. With the facts, most project teams can make the adjustments necessary to address new challenges. The ability to succeed doesn’t lie with a perfect financial forecast—it focuses on a team’s ability to nimbly respond to a variety of circumstances.

Mistake 5: Not Converting Soft ROI into Financial Metrics
Ideally, projects not only promise financial payback, but “intangible” improvements as well. Improved customer and employee satisfaction, enhanced brand awareness, or better internal communication are all promising byproducts of new IT initiatives. Such improvements, where existing costs are difficult or impossible to quantify, are called “intangible” or “soft” ROI. They are often strategic, but not usually enough to pitch or justify a business intelligence project. Unless soft ROI issues can be converted to financial metrics (like revenue, cost, and profit), soft ROI will be ignored.

The allure of using “soft” ROI metrics is that they are typically easier to describe and conceptualize than their “hard” ROI brethren. Distributing a survey or polling customers is a simple way to determine and measure the customer experience. In fact, there are a number of sophisticated software tools that help in the development of questionnaires and even the analysis and segmentation of the results. However, the problem with soft ROI is that it isn’t a financial metric. You can’t write a check with employee satisfaction or brand awareness. More than ever, executives want financial results given that their jobs depend on delivering financial gains. When it comes to ROI, financial metrics win every time.

This isn’t to say that “soft” benefits like improved employee communication aren’t valuable. Indeed, translating such soft goals into hard measures is a practical and worthy activity. It can be as simple as establishing processes that are measurable and refine-able. Measuring the resulting efficiencies and attaching financial value to them allow the conversion of soft benefits into financial payback.

Customer satisfaction surveys are a good example. A company may do a good job of surveying its customers and storing the resulting feedback in its customer database as part of the customer profile. Storing historical customer satisfaction scores give a company the ability to measure them over time. If the company also stores product revenues, it may be able to correlate a rise or fall in customer satisfaction with corresponding revenue numbers, making the case for improved service being linked to increasing profitability.

The good news is the bad news: business intelligence is a great place to find “soft” ROI. So managers take up valuable page space in their BI business plans discussing how faster data deployment can result in “more engaged” and “more productive” employees. While valid and valuable, this point stops short of quantifying the value of that engagement and that productivity. For instance:


  • Higher staff involvement has been shown to decrease staff turnover, resulting in decreased costs. With a cost of $100,000 per customer service rep, simply reducing employee attrition by 5 percent can mean an additional $1.25 million in saved hiring and training costs—annually! (A true story!)
  • By providing on-demand reports to salespeople in the field, the sales staff was able to double their acquisition rates for new accounts. In the first year, this represented an additional $23 million in sales revenues.

The trouble is that most of what has been written on the topic of ROI stops short of these types of quantifications and focuses almost exclusively on the aforementioned “soft” measures. The business potential of closer colleague relationships are probably measurable somewhere down the line. The problem is in the proof: without financial measures, proving them is tricky at best. But proving them is required to get your BI project funded.

It’s important to remember that almost all executives are compensated based on how the company performs financially, thus their decisions—and the decisions of their staff—will continue to focus on hard financial return. So unless “soft” ROI can be quantified, it’s simply icing on the cake of tangible financial payback.

Mistake 6: Differentiating Cost Savings and Revenue Generation
ROI is usually only one of many IT success metrics. Business impact, customer satisfaction, and competitive parity are valid metrics for starting or continuing a strategic IT initiative. ROI, however, implies financial payback, which suggests understanding the value of money. While many business cases are ripe with financial details, they often confuse management because they don’t separate the financial details into cost and revenue areas. With every new CRM or BI project opportunity, it’s important to identify both revenue and cost issues associated with the business case. There are usually three basic financial metrics most companies work with: cost, revenue, and profit.

They’re simple enough terms, but when applied to ROI models they can vex even experienced financial managers.

When Citigroup generates a dollar of revenue, they’re able to pay all their bills (staff compensation, building maintenance, new product development, executive perks, etc.) and keep about 16 cents in profit. This isn’t a terribly complex idea; however, when project teams start developing business plans, they seem to be so focused on numbers, they never separate these three different numbers. There’s nothing more dangerous than a spreadsheet replete with misleading numbers.

There’s a simple equation that we like to use:

Costs + Profits = Revenue

Usually when management thinks “ROI” they think “cost reduction.” Every project is going to cost something; it’s important to understand the financial issues. You can see from our equation that if revenues stay constant, any decrease in costs will ultimately result in increased profits.

There’s also value in identifying projects (and their associated ROI) that drive new revenues. Although there’s a limit to cost and productivity improvement—costs can’t be zero—there is no limit to a company’s revenue upside. Often BI projects are justified based on cost savings (e.g., data integration) rather than with revenue generation (e.g., more targeted marketing campaigns driving increased sales).We strongly encourage IT departments to include ROI metrics and revenue generation details in their project prioritization process.

Revenue generation opportunities are often found with new activities associated with sales and marketing areas. New CRM projects delivering target marketing, new product packages, or product promotions are usually created to generate new income. The business case should include the revenue opportunities and the associated costs for IT implementation and business deployment.

Mistake 7: Over-Emphasizing TCO
Something happens as IT systems age. There’s a belief that the systems have to last forever. When software development projects are in their infancy, a great deal of energy is focused on developing a solution that provides the greatest benefit for the lowest possible cost. Most IT organizations are very good at charting the initial development costs and the associated ongoing maintenance costs. Unfortunately, there isn’t as much rigor involved in tracking the incremental and ongoing business value of projects and systems. It’s not uncommon for IT organizations to uncover old applications and tools that continue to be licensed and maintained but nevertheless have been abandoned by business users. While the ongoing cost of ownership has been well managed, the business value (or ROI) has been ignored.

The traditional hardware and software vendor paradigm is increase software and hardware maintenance costs to ensure that systems are upgraded and/or replaced on a three-year cycle. If you look at everything from mainframes to UNIX servers, and operating systems to application software, you’ll notice that a three-year cycle is a very common method because of accounting rules (and the reasonable life expectancy of the technology). Many companies have fallen into a trap of managing their IT budgets toward a three-year cycle. Total cost of ownership (TCO) has emerged as a top-of-mind issue in every IT organization as a means to focus on the reduction of ongoing system costs and expenses.

TCO has taken center stage with IT organizations as a means to focus on cost reduction. The TCO-premise is simple: what are the costs to manage and maintain a system over time? IT organizations have focused a tremendous amount of energy looking for alternatives to the current software and hardware cost structure and the results have been significant. Suddenly IT organizations are reviewing operational leases, outsourcing contracts, and application maintenance costs in order to include TCO as a metric. However, because of the current economic challenges that most companies are experiencing, a significant paradigm shift has occurred when project budgets are reviewed. In tough economic times, companies are prepared to exit markets, terminate projects, layoff staff, and eliminate product lines. Along with these decisions, those same business users are now asking if some IT infrastructure can be eliminated if it is no longer needed.

ROI has stepped up to the stage with TCO as a means to better illustrate the business value of a project. The premise of ROI is also quite simple: what’s the return on my investment. Where TCO identifies the ongoing cost of a project, ROI will determine the business value of a project.

While it’s not very often that companies challenge the value of improved communications (we’ve not seen anyone focused on removing desktop telephones or e-mail), there is a great deal of interest around the deployment of instant messaging. In this particular example, a TCO value of $83/year per user didn’t illustrate the value of deployment to IT or business users. When the ROI benefits of improved staff productivity and information distribution are included (a value that exceeded TCO by nearly 10 times), the decision for deployment becomes clear.

Everyone’s initial reaction to TCO estimates is to question the validity of the numbers.We recommend that focus include an identification of the overarching ROI for the system. When the question of system elimination occurs—the answer can force a decision based not on business cost, but business value.

Mistake 8: Ignoring the "Do Nothing" Alternative
Once ROI activities are successful, an assumption is always made that the project will be or has been funded and the business is committed to the initiative. As the old saying goes, nothing is certain other than death and taxes. But with every new project comes risk, and sometimes the perception of risk outweighs the value of a project, irrespective of ROI. The “do nothing” alternative is usually a response to the (real or perceived) high cost and risk of the project.

When modeling ROI metrics for a BI project, it’s very important to work through the potential outcomes for at least three possible scenarios: impact of failure on the project, the value of a successful project, and the impact of doing nothing. While most managers never publicly acknowledge the impact of a failed project, we encourage estimating this as part of a risk management approach. Identifying the business costs associated with delayed or failed projects isn’t complex. For existing systems that are being replaced (or upgraded), the impacts are likely to be the costs associated with the current system’s extended production life as well as the additional costs associated with the development extensions. For new systems, the impact will be the delay in delivering the financial benefits to the business. Because a BI project risks cost overruns, the likelihood of doing nothing actually increases proportionally to the price. The past 10 years have been littered with the wreckage of ERP, CRM, and data warehousing projects that were too big and too cumbersome to succeed. (One could argue that they were over-scoped, or that their sponsors hadn’t yet registered the best practice of incremental delivery.)

In any case, the details were all the same—dozens (or hundreds) of staff members, millions of dollars in new hardware and software costs, and 12 months before the delivery of any measurable business benefit. When management is considering a large expenditure, knowing the track record of the industry’s IT projects can turn “doing nothing” into a viable option.

Mistake 9: Failing to Measure Ongoing Value
So you’ve gone down the path of finding ROI. You’ve obtained business sponsorship, IT resource support, project management, and even funding. While in the past BI success was fulfilled by the deployment of the first release of the application, things have changed. ROI has become management’s mantra; executives want to make certain that BI and data warehousing continue to be worthwhile. Even after deployment, management wants to make sure that the values and benefits identified in the business case stay valid. Consequently, another aspect of project success is the tracking of ROI metrics over time.

During the initial development of the business case, the BI stakeholders should discuss the cost benefit and the associated assumptions. A change in the assumptions is usually the root of ROI surprises. The only way to address these issues is to measure and update ROI estimates regularly once the project is deployed.

Ongoing ROI measurement doesn’t need to be a complex undertaking—it should focus on the fundamentals: costs and benefits, both “hard” and “soft.” Success metrics should have been identified during the business case and project justification activities that occurred during the preliminary phases of the project. Unless there’s already an ROI reporting template in place (alas, most companies don’t have one), a simple spreadsheet with the financial numbers and the associated improvements reported monthly can go a long way to illustrate the value of ROI. In the event that the ROI numbers fall short, a monthly report will actually provide a jump-start to analyzing (and resolving) problem.

Thus, ROI reporting should be viewed as a valuable tool to ensure project success and the ongoing business value of BI. We’ve said it about our data warehouses for the past decade and the same is true for ROI: it’s not a one-time-only activity, it’s an ongoing process.

Mistake 10: Assuming ROI Is the Only Decision Metric

During the yearly budget review and IT project approval process that’s common with many of our customers, we’re frequently asked to help work through the planning details of both maintenance and development projects. Most IT shops have a set of standard project definition templates that must be completed to submit a project for budget allocation and approval. The metrics include business and functional needs, staff resource requirements, hardware and software tools, scheduling details, complexity, and cost. While a project’s financial details are almost always discussed—it’s not necessarily true to assume that they are the only deciding factor.

While knowing ROI can enable a business user to compare and contrast disparate activities, it won’t always determine if a project should move ahead. A company has to respond to outside pressures: competitive issues, regulations, customer needs, or changing markets (to name a few). The telecommunications industry has been undergoing significant change in product pricing because of competition; the price of gasoline skyrocketed in the late 70s because of deregulation; and big box retailers have destroyed the big department stores because of changing customer interests. In these instances, ROI wasn’t the driving force of change. There were other more pressing matters.

It’s critical to understand the business drivers when reviewing project’s value. When IT projects are closely aligned with business operational needs, the decision process for a project becomes very simple. When new products and services are introduced, enhancements to the billing systems may be necessary. When products are discontinued or customer markets are exited, modification to the customer support system may be needed. IT organizations almost always have to respond to changes to the business operating environment. Decision metrics such as ROI or costs aren’t the only factors in determining if IT should move forward. If the business changes, IT must respond.

When a new project or initiative isn’t aligned to a particular business need, the actual prioritization or valuation process can still be important. It’s not uncommon for technology-only or IT-centric projects to fall into this category. When a client recently selected a new ETL tool, the decision metrics included ROI, but they were also based on the maturity of the technology, the staff’s ability to learn the tool, and the vendor’s customer references. Once again, ROI was an important decision metric, but it was by no means the only one.

Recent articles by Evan Levy

Evan Levy -

Evan is a partner and co-founder of Baseline Consulting, a professional services firm concentrating on enterprise data issues. In addition to his executive management responsibilities at Baseline, Evan is actively involved in managing project delivery teams and guiding client solution delivery. He also advises vendors and VC firms on new and emerging product strategies. Considered an industry leader on the topic of data integration and management, Evan is a faculty member of The Data Warehousing Institute. He is co-author of the new book, Customer Data Integration: Reaching a Single Version of the Truth (John Wiley and Sons, 2006).