Ten common mistakes companies make in data integration

One of the most critical IT management responsibilities is to ensure that the business has access to trusted information. This is actually a very challenging goal for many companies because the data needed to support business decision making is often inconsistent, redundant and of poor quality. Company data sources have become increasingly complex, often trapped in a complex tangle of disparate data stores and technology systems.

Large enterprises have typically approached the management of information in a siloed way. Each division or line of business within an organization such as finance, sales and marketing, operations, or a specific product line has been treated as a unique entity. Each entity requires different business applications and each of those applications has been tightly linked with its own data store. This siloed approach no longer meets the need of business users who need to understand and make decisions across the enterprise as a whole.

Why not? Each business application may need similar data, such as customer, product, or pricing data, but the definitions of these data may vary across departments. In addition, the data from the various data stores may have different structures, different interfaces, and even different semantics. The data on customers, products and services are often tied into a specific line of business. What happens when you want to cross sell across product lines. Are the definitions of the customer the same?

Creating a company wide environment of trusted information requires some data integration. This process requires a well-thought out architectural approach that will provide information about the business as a service to everyone who needs it. This architectural approach will typically require technology for ETL (extract-transform-load), quality management, the creation of a metadata layer, and a strategy for master data management (MDM).

Companies can often identify why data integration is required, but then fall short on implementing the technology in a way that maximizes the benefit to the business. It can be very challenging for companies to manage the data integration process successfully. Some of the biggest problems stem from a lack of understanding about the needs of the business. The process of data integration needs to be considered as part of an overall information management strategy for the business and within the context of the business strategy and priorities. It is important to consider the rules, strategies, and goals of the business as part of the process. Does this approach make sense to the business? Does this approach satisfy the requirements of the business?

If you follow a strictly technical approach to data integration you are likely to make some mistakes and fall short of reaching your desired goal. Successful companies look at information management holistically with an ultimate goal of providing trusted and consistent information about the business to everyone one from the CEO to customer service representatives to external partners and suppliers.

The following are ten common mistakes that should be avoided when planning for data integration.

1. Following a “fire-drill” approach to data integration. It is short sighted to use ETL technology as a tool to solve a one-time data integration problem rather than using this technology as part of a comprehensive approach to information management.
2. Not thinking about data as a shared and reusable resource. It is easier to budget based on getting a single task done. However, it is much more efficient and cost effective to be able to reuse data resources once the second, third and future projects are initiated.
3. Thinking tactically about data integration and missing out on opportunities to improve business process. Companies often implement data integration technology to eliminate time-consuming and labor-intensive processes that have been required to gain a consolidated view across business units. However, it is a mistake to focus on reducing head count and saving time in the data integration process, without also considering a broader strategic view towards improving overall business processes.
4. Not establishing an architectural framework with the capability of providing reusable information services. Once the data is decoupled from the business application, you need to develop a methodology that supports reuse so the data can be shared in different ways as needed. The information as a service approach is designed to ensure that business services are able to consume and deliver the data they need in a trusted, controlled, consistent, and flexible way across the enterprise.
5. Using software code to adjust for differences in definitions about customers, products, and other data types on a one-off project basis. In order to deliver information as a service, there needs to be repeatable way to manage complex processes without the expense and time required for recoding. This can be accomplished with the support of a metadata infrastructure.
6. Integrating data without placing a high priority on data quality. It is critically important that companies establish processes to cleanse and correct data as part of the overall data integration process. Creating standardized and consistent information will ensure that business users are more confident about business information and in a better position to grow the business and remain competitive.
7. Not creating a standardized way to handle data that is common to the various disparate IT systems and business groups. Companies need to understand the commonalities across different data types. This can best be achieved by developing a master data management (MDM) strategy to serve as the system of record for the consuming systems and applications.
8. The technical integration team and the business experts do not communicate effectively. There needs to be a shared and common language describing business processes to enhance communication between business and IT management. The business is more likely to have good quality information they can count on if the IT and the business establish an efficient process for sharing knowledge and requirements.
9. Business owners are reluctant to give up ownership of data. In order to gain the efficiency and accuracy in the data integration process, it is important to establish a consensus among the various data owners regarding data terminology and definitions, and there needs to be a clear understanding of the data lineage and who is responsible for these data over time. This often requires a significant cultural change because individual business experts often have a long history of managing data for their line of business or department as if it was a stand-alone entity. Companies need to find a way to balance the need for individual business experts to maintain control over their own data with the need for centralized management of data within a metadata environment.
10. Trying to do too much in one project. When data is integrated across departmental data silos, previously inaccessible data becomes available to business users. Companies can take on projects that would have been impossible before because of the enormous amounts of hand coding and manual data collection that would have been required. However, these benefits can be lost if companies try to tackle too much at once. Enterprise-wide information management projects must be approached in an incremental way so that there is time to evaluate and improve data quality, understand the needs of the business, and establish repeatable methodologies and processes.

 

What happens when your BI vendor gets acquired?

With all of the acquisitions happening in the business intelligence space, customers are in a state of confusion. What do all of these changes mean — both short term and long term? In my view, it is best to take a pragmatic approach to this changing market landscape. This acquisition spree in BI is impacting as many as 80,000 customers who use business intelligence software from Hyperion, Business Objects, or Cognos. What should customers be asking their vendors about the future directions of their products? All three BI vendors have either just been acquired or are soon to be acquired by three large IT companies – Oracle, SAP, and IBM, respectively. Is it business as usual or do the IT managers need to alter their business relationships and plans for business intelligence implementations now that a wave of consolidation is underway?

Customer concerns about the re-alignment in the business intelligence area tend to fall into three categories: impact on legacy environments, impact on future buying decisions, and technical innovation. Management issues will vary with how business intelligence software has been deployed at their companies and how well integrated this software has been with other information management software.

Legacy environments. Many companies tend to use BI tools for traditional management reporting. In general, they like to stick with what they know and what their users are trained to use. Therefore, they want to be assured that they can continue using the familiar reporting tools in the same way. While most managers assume that once an acquisition is complete, there will be new operating efficiencies. What they don’t always know is whether those efficiencies will translate to savings. They also would like to understand how the company might benefit from the fact that a larger company with more resources has bought the company they have been dealing with. How will my company benefit from the acquisition? Will there be any changes in the sales and service teams? What about pricing issues and planned upgrades?

Typically, it will take a while before changes are evident. For example, if you are getting good value from your use of Crystal Reports—the Business Objects’ reporting solution designed to support business modeling, analysis, and decision making – it really may not matter to you which company purchased Business Objects.

Ironically, some of the most important issues that might emerge happen when customers merge with each other. For example, one company might have standardized on Cognos for its analysis and reporting while the company being acquired uses Business Objects as a standard. Since large enterprises rely on business intelligence software to gain insight into production, sales, revenue, or other data across divisions and subsidiaries, too many tools may make decision making more difficult.

It is interesting that these acquisitions are hitting the market at the same time that companies are trying to move from a departmental view of data to an enterprise perspective. A unified and standardized approach to information management across the enterprise is becoming a top priority. Companies that have accumulated many different BI vendor software may use this time of change to re-evaluate a BI strategy.

Typically companies are used to managing multiple software components from multiple vendors. The expectation is that the consolidation of two or more of a single vendor will lead to benefits resulting from the tighter integration of the products. This is often the best outcome in terms of support, training, and management. However, this is typically a multi-year effort by the vendors building a unified portfolio based on acquired software. As businesses move from a traditional siloed single purpose data warehouse to information integration and analytics, having one vendor to call is often a welcome change as companies try to simplify the management of its infrastructure.

Impact on Future Buying Decisions

The situation may be a little different if a customer is in the middle of a proof of concept (POC) for a project designed to develop a single view of a company’s customer base. How will the recent acquisitions in the business intelligence market impact how businesses to move forward?

Consider the example of a large bank that had recently made a significant acquisition. The bank wanted to understand its most profitable customers across the newly merged company. Customer history and sales data was retained in a siloed manner by line of business and there was no integration between the data for the two companies. This company used Cognos for many of their executive level reports, but now management had raised concerns about data quality.

In order to develop a single view of customer they needed to look for incompatibilities and inconsistencies in the disparate data sources. These data sources needed to be integrated and IT needed to assure the business that the information was accurate, complete, and trust worthy. The bank selected Informatica to provide the software needed to help with the integration and to improve the quality of its data.

Informatica has a comprehensive solution for data integration and data quality. In addition, just a few months ago Informatica and Cognos announced an expansion of their strategic relationship. This partnership fit well into the CIO’s priority to consolidate all the many disconnected vendors in use in IT and to ensure that the integrations between different applications are as tightly integrated as possible. Now, suddenly Cognos is part of IBM and a direct competitor to Informatica. What should the CIO do? Certainly IBM is committed to supporting all of Cognos’ existing partnerships. You should not have to change your plans because of the acquisition, however there are questions to ask about how things will change in the future.

Innovation

All the BI vendors mentioned above have well-established partner relationships with emerging information management software companies. There has been a lot of customer support for the creation of tighter integrations between the information management infrastructure and the business intelligence layer. Companies have recognized that the reporting structure becomes meaningless if the supporting data cannot be trusted.

The partnerships have been important because much of the important innovation comes from small emerging companies. Partnerships with major players makes it easier for the company to leverage innovation with lower risk. The established players can provide the integration with the analytics and reporting technology. As companies attempt to unlock much of the data that has been previously unreachable at the enterprise level, it will become much more important to have a unified approach to information quality, information integration, and business intelligence. Market consolidation will help ensure that innovation is better utilized in a predictable manner.