Looking for the Small in Big Data

While the buzz in the big data market is around massive amounts of data and what you can do with that data, in truth companies are more interested in the small stuff. Let me explain. One of the greatest benefits of getting hold of lots of data is that a company can see patterns and anomalies that would have gone undetected with a smaller set of data.

Companies are interested in big data not so much because of what that data represents in terms of volume, variety and velocity, but what the analysis of that data can do for the business. As business decision makers begin to recognize the potential of big data, they realize that the biggest insights and hidden gems of knowledge are found when big data actually becomes quite small. However, the analysis must start with very large volumes of data. Exploring the world of big data takes business leaders beyond the data found in their organization’s traditional databases. Researchers, data scientists, and marketers, are analyzing very large volumes of unstructured data from previously untapped sources such as e-mails, customer service records, sensor data, and security logs.

Now that companies have the ability to deal with so much more data, they are free to incorporate a greater variety of data into the mix. For example, they are incorporating social media, mobile phone location, traffic, and weather data into more traditional analysis. But what is the goal? Add more data and more elements so that those patterns emerge. Armed with the insight from analysis, business leaders can take a more precise set of data and compare it to data from a data warehouse or a system of record. As a result, the research becomes more targeted and directed to fit in with the context of the business.

Why do companies need to take this targeted approach to leveraging big data? In essence, companies want to use big data analysis to make a personalized offer that is just right for the customer when that customer is ready to buy. In some cases, the answer may be related not to selling but to diagnosing problems with a manufacturing system or a patient with an unexplained illness.

How do you move from big data analysis to small data insights and personalized action? You need to consider three elements: defining your business problem, defining and analyzing your data sources, and integrating and incorporating your big data analysis with your operational data.

Defining your business problem. Companies are beginning to ask the traditional questions about customers, products, and partners in new ways. For example, are you looking to manage your customer interactions armed with in-depth and customized knowledge about each individual customer? Companies with a focus on driving continuous improvement in customer service are asking, “How can I delight this customer and anticipate their specific needs?” These are important goals for businesses competing in today’s fast-paced, mobile-driven market.

Defining and analyzing your data sources. What information do you need to make the right offer to your buyer when he deciding on a purchase? What information can you glean from outside sources such as social media data? What big data sources do you have available internally that were previously underutilized? For example, can you use text analytics to gain new insight about customers from call center notes, emails, and voice recordings? One important goal with big data analytics is to look for patterns and relationships that apply to your business and narrow down the data set based on business context. Your big data analysis will help you find the small treasures of information in your big data.

Integrating and incorporating the analysis of you big data with your operational data.
After your big data analysis is complete, you need an approach that will allow you to integrate or incorporate the results of your big data analysis into your business process and real-time business actions. This will require some adjustment to the conventional notion of data integration. In order to bring your big data environments and enterprise data environments together, you will need to incorporate new methods of integration that support Hadoop and other nontraditional big data environments. In addition, if you want to incorporate the results of very fast streaming data into your business process, you will need advanced technology that enables you to make decisions in real-time.

Ultimately, if you want to make good decisions based on the results of your big data analysis you need to deliver information at the right time and with the right context. In order to make the results of your analysis actionable, you need to focus more on the small – targeted and personalized results of big data – than on the large data volumes.

IBMs Vision for Analytics in the Midmarket: gaining deeper business insight

I recently attended an IBM analyst meeting focused on solutions for the midmarket.  What caught my attention was the focus on analytics as an important and growing revenue opportunity for IBM.  In fact, IBM mentioned during the meeting that 70 percent of midsized firms are looking for analytics solutions.  It is clear from this meeting that IBM wants to bring a comprehensive set of analytical tools to the midsize companies.  Unlike some of IBM’s packaging, analytics tools are being packaged specifically for the midmarket so that they can be more consumable and affordable.

Analytics is fast becoming a high priority for companies as a result of the explosion in the variety, velocity, and volume of data with a potential impact on business decision-making. Much of this data is unstructured, such as the text included in customer service records, customer sentiment data in social media, or streams of data from instrumented devices.  Making good business decisions  often requires analysis across multiple sources and types of data.  Companies often have independent systems designed to manage business processes ranging from order/inventory to point of sales, marketing research, and customer relationship management. The challenge for many of these companies is that answering the most urgent questions about the business requires analysis across all of these independent systems. Even a small company with a few hundred employees may have a dozen systems are are disconnected and keep the company from having a full picture of the business.

Therefore, it is not surprising that some midsize companies are finding they can benefit from business analytics solutions. Yet, while some midsize companies are finding ways to get the answers they want using analytics, the word needs to spread to other companies still struggling with manual spreadsheet analysis that doesn’t go deep enough.

IBM is going to market through its business partners that typically support midsize companies with a variety of solutions. These business partners are being asked by their clients in the midmarket to help them implement technology solutions that will enable them to make smarter business decisions. They want to find new ways to deeper their understanding of customer expectations and priorities. For example, a midsize retailer might be trying to figure out why certain products are returned while others sell well.  The analytics market offers huge opportunity for IBM and its partners.

The approach IBM is taking with analytics for the mid-market is to offer its partners a pre-configuration of hardware and software into a single system at a price point  that is both affordable for midsized companies, but also has enough of a margin to make it attractive to a partner channel.

However, the challenge for partners is to change the traditional way they have gone to market.  Many partners that have built successful businesses by specializing in hardware sales or a specific category of software such as IBM  Rational find that they need to meet a broader set of client requirements.  They now need to both learn the new analytics products and be ready to sell and implement solutions differently.  Selling analytics to the mid market requires much more than a technical sell. Partners need to have a thorough understanding of the business context in which the analytics will be used to help customer visualize the potential business value.

One of IBM’s offerings that partners should be looking at is  the IBM Smart Analytics System 5710, which is a database appliance for business intelligence and data analytics targeted at the SMB market. The IBM Smart Analytics System 5710 is based on IBM System x, runs Linux, and includes InfoSphere Warehouse Departmental Edition and Cognos 10 Business Intelligence Reporting and Query.  The system is designed to enable partners to get their clients up and running very quickly a broad set of  analytics and business intelligence capabilities. I expect that you will see a lot more of this type of packaging from IBM with collaboration from its solution business partners.

Why you need an information governance strategy for 2010

You say you already have a plan in place to guard your company’s data? Are you sure it has you adequately protected? While you certainly understand the need for data security – your sales challenges are tough enough without exposing your customer’s credit card information to a security breech, for example – the chances are good that in 2010 you will consider various options for improving the security of your data.  If you are going to protect your company’s most valuable asset — your data —  you will begin to view data security as a component of a more comprehensive information governance strategy.

The risks of internal or external threats to your company’s data are becoming more complex as the depth and breadth of your information expands rapidly and your data is shared with business partners, suppliers, and customers.  In addition, as companies begin to take advantage of cloud services for some of their workloads, additional complexity is added to the multitude of security concerns. Many companies have deployed a disjointed approach to securing, controlling, and managing its data making it hard to anticipate and prepare for constantly changing security risks. There are lots of different ways that unauthorized users may enter your network or otherwise steal your data.  Many companies typically have a distinct solution to combat each one individually and typcially can’t each of themprotect against all of them and. For example, access control, data encryption, network traffic monitoring, vulnerability testing, and auditing may all be monitored with independent applications.

There is a good reason why many companies find they need to deploy lots of different solutions to effectively govern its information. Some of the most innovative solutions have come from emerging companies who have built a niche around a particular vertical market or some segment of the information security market. So you deploy the best solution for you biggest challenges and move on. However, as you begin to think more holistically about your needs for information governance, you will want to ensure that information security solutions are well integrated. This is one reason why emerging companies with an information security solution have become desirable acquisition candidates for larger software vendors.

Guardium, a privately-held company based in Massachusetts, is one of the most recent examples of this trend. When IBM announced its acquisition of the company in the last week of November, Guardium moved from a fast growing startup to one of the pillars of the IBM information governance strategy.The company’s technology helps clients with some of the most challenging issues around unauthorized access to critical data. Their solutions provide secure access to enterprise data – across many different database environments such as IBM, Oracle, Microsoft, Teradata and others.  In addition, customers can reduce operational costs by automating regulatory compliance tasks. While many companies may have the ability to monitor one database at a time, Guardium brings added value by enabling companies with complex environments to monitor databases across their organization.

This acquisition aligns well with IBM’s strategy to provide customers with a well-integrated and comprehensive approach to information management. IBM has spent in the range of $12 Billion over the past five years to add software assets that will help companies to make more intelligent decisions and realize more business value from their information.

Do you have an analytics strategy and why should you care?

After just returning from IBM’s Information on Demand (IOD) Conference in Las Vegas, I would like to take this opportunity to virtually whisper just one word in the ear of a current day Benjamin Braddock, “analytics”. Many businesses have spent the past 25 years or so automating and streamlining business processes in order to drive improvements in efficiency and productivity.  But now, it is becoming apparent that these businesses expect their future success will increasingly depend on how skillfully they manage, govern, and analyze information. Businesses are applying analytical techniques to business information to help reduce risk and increase the certainty that they are making the right decisions.

IBM has, in fact, spent $12 Billion in software investments (both organic multiple acquisitions like SPSS, Cognos, Filenet, iPhrase, and Ascential Software, just to name a few) over the past 4-5 years to ensure it will be able to support its customers in their quest to unlock the business value of information. In addition, in April of 2009 IBM announced a new organization comprised of 4000 consultants focused on advanced business analytics and business optimization – teams with skills in applying business intelligence technologies like mathematical modeling, simulation, data analytics, and optimization techniques.

In an era of intense competition, tight credit, and cost concerns across global and vertical markets, this focus on getting the most value from the information you have makes a lot of sense. Companies find they are processing more information than ever before, but less of this information is being accurately and adequately used.  The quantity of available data that a business needs to manage and understand has skyrocketed along with the increase in instrumented and intelligent products. For example, RFID tags that are embedded in manufactured products,  plants and animals generate an enormous amount of data in efforts to control inventories and improve security and safety.  Trying to make decisions with inadequate,  inaccurate, or untimely  information is like driving a fast sports car down the highway with a very large blind spot impeding your view of the truck approaching on your side. You need to know about the obstacles that might appear in  your pathway before you try to make a “real-time” correction and steer your car (or your business) of a cliff.  So, students and business leaders alike please take note, I see some “analytics” in your future.

The battle to grab customers in a down market

I attended the HP Analyst Meeting in Boston a few weeks ago and had several discussions with the business intelligence (BI) group. It is clear to me that HP is struggling to try to figure out the best way to sell in this type of down market. Obviously, this isn’t easy for anyone.  HP’s approach to solving this problem for its BI, data warehousing, and analytics solutions was to create a BI solutions group consisting of consulting (based on the 2006 Knightsbridge acquisition) and technology (Neoview, an integrated hardware and software platform for enterprise data warehousing).  One of the best articulations of the approach HP has adopted came from a discussion by a sales executive who is on the front lines of trying to convince customers to part with cash. What struck me was the coordinated effort that was necessary to sell to a large global organization with huge data management challenges.  This got me thinking about what it takes to sell in a very tough market.

To be successful, this sales person went all out. The Neoview product team and BI consultants all pulled to together to provide the right solution for the customer. Over the course of about ten weeks they conducted at least 100 interviews with the company to build strategy, roadmaps, and ROI estimates. The  team worked with the customer to create a master plan that showed how HP could help the company with its goal of transforming its business.

The HP sales team leveraged many different resources to make sure they had an excellent understanding of the customer’s needs and that the company understood how HP could help. HP made sure to get executive sponsors in key leadership positions at the customer organization. They also brought in some of HP’s top thought leaders and made sure that happy customers were available to discuss their experiences. In addition, HP leveraged its partner network (including Ab Initio, SAS, and SAP Business Objects) to provide a complete solution.

Neoview was a good fit for the customer’s data management challenges. Problems with inconsistent customer information and disconnected IT systems were so hard to manage that it was becoming impossible for IT to adequately support the business. This customer’s top priorities were to regain control of its existing analytics data store and revamp enterprise customer intelligence and enterprise risk management. They liked the way Neoview was built. It is based on HP’s NonStop engineering expertise that has been used for over 30 years in industries such as financial services (stock exchanges) and telecommunication (switching) where the management of vast amounts of data is essential. Neoview is designed to support hundreds of terabytes of data and over one thousand processors. The customer also had some concerns about issues like getting its team up to speed on the product. HP stepped up to meet their concern by offering training and help with Neoview’s operation to ensure a smooth transition..

As I stepped back from this discussion, it occurred to me that successful technology sales in this type of complex market is incredibly challenging. It is simply not enough to make an announcement and hope for the best. On the one hand,  the ingredients for a successful sale  sound pretty simple – you need to understand the customer’s pain and provide the right solution at the right price. Easy? Try telling that to a team that just implemented a full-scale, coordinated sales push, made all the right moves, and beat out formidable competitors to win the sale. It’s not so easy, particularly with a complicated IT solution in a market where  the business demands fast and cost effective results.And, if it takes such a coordinated effort to win one sale, how realistic is it to expect to sustain these efforts over the long term?

There are three main requirements for selling IT products and solutions in today’s market:

Get the basics right.You need to provide good technology at the right price. Your marketing plan needs to be based on clear and concise messages and your sales team needs to be able to articulate  those messages in a way  that is just right for your customer. This sounds like a good plan in any market. The difference today is that you can’t count on some of the sales that might previously have been considered “low-hanging fruit”. For example, assume you are preparing a proof-of-concept for a prospective customer and several of this company’s developers know you and your product from their work at other companies. Although it is helpful to have strong supporters of your product on the prospect’s IT team, their support can not overcome a product or solution that doesn’t solve real business problems. Today, the business is demanding more from IT – more business value, more trusted data, more control over costs, and more control over project time lines.

Understand your customer’s needs. You need to understand your customer’s challenges and expectations in order to make sure your product/solution is a good fit One of the most common mistakes that software product marketing teams make when preparing marketing materials is to focus on the outstanding and differentiating features of their product from a product-centric instead of a customer-centric point of view. You need to understand what problem you are solving for your customer and how your solution will solve this problem faster-cheaper-or with more flexibility for future changes than your competitors products. An understand of customer needs should happen at two levels. First, it helps to look at customers in a vertical market so your solution and  marketing strategy account for industry specific complexities and challenges. Second, you need to understand the requirements of the prospect at hand -within the context of its industry as well as unique situations such as a recent acquisition or internal changes that may impact their sales decision.

Develop a coordinated and well organized approach. The difference between a good sales effort and a great one is in the way internal teams and business partners collaborate to put knowledge about customer needs and product/solution capabilities together to find the right fit for the customer. For many software vendors, the services or consulting team (internal team or external partnership) has a large role to play to help close the deal. For example, regulatory requirements in industries such as  health care and financial services industries are continuing to change making significant demands on IT environments for companies in these industries. Although, you may have a great solution for handling vast amounts of data and improving data security and governance, you may not have the opportunity to prove it to your prospect without a coordinated sales effort. By bringing your product and sales teams together with a consulting team with deep experience in the regulatory and data requirements for health care and financial services you will be better equipped to show the business value of your solution.

As you try to finalize your deal, it often comes down to very similar issues across different types of customers.  They are all looking for quick, inexpensive fixes to hard problems! The reality is that there are no easy solutions to closing deals in this economy. Understanding what the customer needs is hard, but  what is even harder is making all of this scalable.

Ten common mistakes companies make in data integration

One of the most critical IT management responsibilities is to ensure that the business has access to trusted information. This is actually a very challenging goal for many companies because the data needed to support business decision making is often inconsistent, redundant and of poor quality. Company data sources have become increasingly complex, often trapped in a complex tangle of disparate data stores and technology systems.

Large enterprises have typically approached the management of information in a siloed way. Each division or line of business within an organization such as finance, sales and marketing, operations, or a specific product line has been treated as a unique entity. Each entity requires different business applications and each of those applications has been tightly linked with its own data store. This siloed approach no longer meets the need of business users who need to understand and make decisions across the enterprise as a whole.

Why not? Each business application may need similar data, such as customer, product, or pricing data, but the definitions of these data may vary across departments. In addition, the data from the various data stores may have different structures, different interfaces, and even different semantics. The data on customers, products and services are often tied into a specific line of business. What happens when you want to cross sell across product lines. Are the definitions of the customer the same?

Creating a company wide environment of trusted information requires some data integration. This process requires a well-thought out architectural approach that will provide information about the business as a service to everyone who needs it. This architectural approach will typically require technology for ETL (extract-transform-load), quality management, the creation of a metadata layer, and a strategy for master data management (MDM).

Companies can often identify why data integration is required, but then fall short on implementing the technology in a way that maximizes the benefit to the business. It can be very challenging for companies to manage the data integration process successfully. Some of the biggest problems stem from a lack of understanding about the needs of the business. The process of data integration needs to be considered as part of an overall information management strategy for the business and within the context of the business strategy and priorities. It is important to consider the rules, strategies, and goals of the business as part of the process. Does this approach make sense to the business? Does this approach satisfy the requirements of the business?

If you follow a strictly technical approach to data integration you are likely to make some mistakes and fall short of reaching your desired goal. Successful companies look at information management holistically with an ultimate goal of providing trusted and consistent information about the business to everyone one from the CEO to customer service representatives to external partners and suppliers.

The following are ten common mistakes that should be avoided when planning for data integration.

1. Following a “fire-drill” approach to data integration. It is short sighted to use ETL technology as a tool to solve a one-time data integration problem rather than using this technology as part of a comprehensive approach to information management.
2. Not thinking about data as a shared and reusable resource. It is easier to budget based on getting a single task done. However, it is much more efficient and cost effective to be able to reuse data resources once the second, third and future projects are initiated.
3. Thinking tactically about data integration and missing out on opportunities to improve business process. Companies often implement data integration technology to eliminate time-consuming and labor-intensive processes that have been required to gain a consolidated view across business units. However, it is a mistake to focus on reducing head count and saving time in the data integration process, without also considering a broader strategic view towards improving overall business processes.
4. Not establishing an architectural framework with the capability of providing reusable information services. Once the data is decoupled from the business application, you need to develop a methodology that supports reuse so the data can be shared in different ways as needed. The information as a service approach is designed to ensure that business services are able to consume and deliver the data they need in a trusted, controlled, consistent, and flexible way across the enterprise.
5. Using software code to adjust for differences in definitions about customers, products, and other data types on a one-off project basis. In order to deliver information as a service, there needs to be repeatable way to manage complex processes without the expense and time required for recoding. This can be accomplished with the support of a metadata infrastructure.
6. Integrating data without placing a high priority on data quality. It is critically important that companies establish processes to cleanse and correct data as part of the overall data integration process. Creating standardized and consistent information will ensure that business users are more confident about business information and in a better position to grow the business and remain competitive.
7. Not creating a standardized way to handle data that is common to the various disparate IT systems and business groups. Companies need to understand the commonalities across different data types. This can best be achieved by developing a master data management (MDM) strategy to serve as the system of record for the consuming systems and applications.
8. The technical integration team and the business experts do not communicate effectively. There needs to be a shared and common language describing business processes to enhance communication between business and IT management. The business is more likely to have good quality information they can count on if the IT and the business establish an efficient process for sharing knowledge and requirements.
9. Business owners are reluctant to give up ownership of data. In order to gain the efficiency and accuracy in the data integration process, it is important to establish a consensus among the various data owners regarding data terminology and definitions, and there needs to be a clear understanding of the data lineage and who is responsible for these data over time. This often requires a significant cultural change because individual business experts often have a long history of managing data for their line of business or department as if it was a stand-alone entity. Companies need to find a way to balance the need for individual business experts to maintain control over their own data with the need for centralized management of data within a metadata environment.
10. Trying to do too much in one project. When data is integrated across departmental data silos, previously inaccessible data becomes available to business users. Companies can take on projects that would have been impossible before because of the enormous amounts of hand coding and manual data collection that would have been required. However, these benefits can be lost if companies try to tackle too much at once. Enterprise-wide information management projects must be approached in an incremental way so that there is time to evaluate and improve data quality, understand the needs of the business, and establish repeatable methodologies and processes.