Business Intelligence Best Practices -

Collaboration. Communication. Community.

 E-mail to friend
Compliance as a Trojan Horse: Funding Your Enterprise Data Warehouse

by Stephen Brobst
Current IT spending in support of regulatory and compliance projects is significant, and it is expected to increase in coming years.

Current IT spending in support of regulatory and compliance projects is significant, and it is expected to increase in coming years. Gartner figures indicate that 12 percent of current IT project budgeting is directly related to compliance, and that this figure will likely exceed 15 percent in 2007. At first glance, this may appear to be a huge burden on an organization's ability to innovate and provide competitive differentiation from within fixed (and often shrinking) IT budgets. However, innovative organizations are turning this apparent compliance liability into an asset through the leveraging of enterprise data warehousing infrastructure.

Much of the regulatory compliance burden that organizations are under today focuses on the delivery of reports that require access to significant amounts of detailed historical data. For example, Basel II standards for financial risk reporting typically require a minimum of 7 years of detailed data, and will likely require 11 years in the near future. One bank I worked with recently has a Basel II risk reporting requirement for 25 years of history (due to the extended economic cycles associated with real estate and its book of business, which is heavily weighted toward long-term mortgages). Regulatory compliance guidelines for financial reporting according to the Sarbanes-Oxley Act in the U.S. (SOX), its equivalent in Japan (J-SOX), and International Accounting Standards (IAS) also require increasing amounts of detailed data. Summary data is no longer good enough. Moreover, significant retention of historical data is required for trend reporting and audit purposes.

The key to exploiting the regulatory dividend is to leverage the costs associated with compliance into an asset that delivers competitive advantage to the enterprise. By reusing the data required for regulatory reporting to provide analytics that drive better decision making within an organization, significant value may be exploited—beyond simply completing an auditor’s compliance checklist. However, obtaining this value depends on a well-architected data warehouse solution.

The Wrong Approach

In the rush to "check the boxes" for regulatory compliance, it’s easy for an organization to begin deploying data marts without architecting for the potential reuse of data assets. The primary principle of data mart deployment is to design to the specific needs of a particular business process. Thus, the information content and organization in a data mart is optimized for performance and usability aligned to a specific purpose. This is sometimes appropriate for analytic applications with extreme performance requirements, but when physical structures or content are overly customized, it can inhibit the reuse of data.

The danger is that each regulatory requirement for reporting may result in a new data mart. When this happens, information that is needed across regulatory requirements is not reused, nor are the data marts leveraged to provide analytic capabilities outside of the specific regulatory mandates for which they were built. It is not unusual to observe overlap as high as 70 percent in obligatory data across regulatory reporting requirements. This then causes higher IT investment than is necessary in terms of redundant systems, storage, data movement, resources required for data cleansing and integration, and the people required to manage the overall environment.

Even worse, the multi-mart deployment can create multiple sources of truth for regulatory reporting within an organization. So not only does the duplication of data create increased infrastructure costs within IT; it also increases organizational costs in terms of the effort required to reconcile numbers across reports. An enterprise that has multiple sources of regulatory reporting is likely to incur the higher auditing and compliance costs associated with inconsistent data sources.

Moreover, the deployment of specialized data marts for regulatory reporting makes it difficult to leverage the information required for compliance into other uses within an organization. The ability to parlay investments in data management toward competitive advantage is significantly reduced when a data mart is deployed without adequate consideration given to reuse of the information assets.

The Right Approach

The high-leverage strategy for regulatory reporting is to take an enterprise approach toward data warehouse deployment, meaning that compliance requirements and advanced analytics for driving business differentiation share information assets. An enterprise data warehouse treats information as a reusable asset. Its underlying data model is not specific to a particular reporting or analytic requirement. Instead of focusing on a process-oriented design, the underlying repository design is modeled based on data inter-relationships that are fundamental to the business across processes.

While some summarization and other denormalizations may be undertaken for performance reasons, an enterprise data warehouse design makes best efforts to maintain history at a detailed level with all business relationships intact. This allows for more effective reuse of data, because any summarization or denormalization performed in a repository design must assume some prior knowledge of business requirements. By keeping history at a detailed level, the information in the enterprise data warehouse can easily be reused for other analytic purposes (some of which may not even be known at the time of deployment for regulatory compliance).

The design goal is to source data into the enterprise data warehouse once, and use it for multiple purposes. (Store once, use multiple times.) In this way, regulatory compliance requirements can be used to fund data provisioning into the data warehouse to support industry-mandated reporting—but the information becomes available for other analytic purposes as a by-product of these efforts. A fundamental prerequisite to success is having an enterprise logical data model (ELDM) in place as a blueprint for organizing information in the enterprise data warehouse. The ELDM ensures that the organization of information is fit for analytic purposes across multiple domains.

Remember that the ELDM is a blueprint, not a detailed design. Creation of the ELDM should be time-boxed to four months or (ideally) less. For maximum efficiency, use packaged data model assets specific to a given industry (available from many third-party vendors). These will need to be modified by 10–20 percent, but this approach is much more productive than starting from scratch. Detailed design for the ELDM should be undertaken incrementally using specific projects (such as compliance reporting) to provide the funding and detailed requirements for deployment. Using this blueprint as a starting point ensures that the detailed requirements for each specific project will be realized in a way that creates reusable data assets.


There is constant pressure within IT organizations to do more with less. One very powerful means of achieving this goal is to reuse information assets. An enterprise data warehouse approach provides the means to store data once, but use it for multiple purposes. Data cleansing, integration, and provisioning into an information repository typically accounts for 50–70 percent of the cost of its construction. But by sourcing data into a reusable information asset in the form of a well-designed warehouse, these costs are incurred just once for the enterprise, rather than multiple times in the case of multi-mart deployments. When regulatory compliance requirements are satisfied via single-purpose data mart deployment, it incurs a significant cost burden to the organization, and no competitive advantage is obtained. On the other hand, with an enterprise data warehouse approach, the regulatory compliance investment is leveraged into an enterprise information asset that can be used for a variety of strategic purposes.

There should be little doubt as to the most effective investment approach when pursuing regulatory compliance. However, this is much easier said than done. An enterprise approach requires an enterprise data strategy and appropriate data governance within the organization. To be successful, organizations must think at an enterprise level rather than in departmental silos. Success requires a scalable approach to deployment of the information repository, rather than a data mart mentality with point solutions. As usual, the technology is the easy part; political considerations will be the biggest challenge to success.

Stephen Brobst -

Stephen Brobst specializes in the design and construction of DW solutions for Fortune 500 companies in the U.S. and internationally. Stephen performed his graduate work in computer science at MIT where his master's and PhD research focused on high-performance parallel processing. He also completed an MBA with joint course and thesis work at the Harvard Business School and the MIT Sloan School of Management. Stephen has been on the TDWI faculty since 1996.