Data Mining 101

Dateline: 08/06/00

Data Mining, Knowledge Discovery in Databases, Data Warehousing. Chances are that you've stumbled across these buzzwords many times in the past few years. Data mining and warehousing are related concepts - data warehouse gathers historical data and is used to obtain perspectives on the business trends. But what exactly is data mining? To put it formally, data mining is defined as automated or semi-automated extraction of hidden predictive and descriptive information from large databases. Another definition says it is the non-trivial extraction of implicit, previously unknown, and potentially useful information from data. But let's start from the beginning and see what can be expected from the current tools and technologies.

You've heard that data mining technology can be an important weapon in a company's arsenal, uncovering critical information you can transform into a competitive advantage. As any other new technology that enjoys commercial attention, it is still surrounded by a great deal of hype. There is no magic in data mining, but considering the information overload we are dealing with, it is well worth to learn more about the technology behind commercial products available today. Cynics would say that data mining equals statistics plus marketing, but cynicism stops when you remember that the amount of data stored in world's databases doubles every twenty months. This novel mixture of statistics, Artificial Intelligence (especially machine learning), and database research was conceived back in the 1980s when researchers began to worry about extracting information from massive collections of data. The explosive growth in the amount of data available electronically brings it to the forefront of applied AI, aiming to close the gap between the generation of data and our understanding of it.

The main aim of all data mining tools is discovering understandable patterns in data. This technology tells users what they didn't know and had not even thought of asking. Main processes included here involve association (correlating one event to another), sequences (discovering when one event leads to another later event), classification (recognition of patterns resulting in new organizations of data), clustering (finding groups of facts not previously known), regression (learning a function that maps a data item to a real-valued prediction variable), etc. The most popular methods for data mining include decision trees and rules, nonlinear regression and classification (including neural networks), example-based methods (nearest-neighbor classification, case-based reasoning), inductive logic programming and others. If you are beginner in the field, these terms may sound scary, but don't let neat GUIs and marketing materials fool you: data mining still requires knowledge about statistics and the algorithms working behind the scene. Without clean data and good models, there is no hidden power that will transform the garbage at the input to the valuable information at the output. Vendors of complimentary technologies sometimes contribute to the confusion, marketing reporting and OLAP (On-Line Analytical Processing) tools as data mining products. Of course, these can be used to discover useful information, but you are the one doing all the dirty work in this case - if you don't suspect a pattern is there, you will never find it. Remember, genuine data mining tool must support (semi-)automatic discovery of patterns.

The number of potential application areas for this technology is huge. The finance and insurance industries have traditionally recognized the benefits of data mining. Possible applications include detecting patterns of fraudulent credit card usage, finding hidden correlations between different financial indicators, identifying stocks trading rules from historical market data, identifying behavior patterns of risky customers, claims analysis, etc. However, the number of other applications in retail, marketing, transportation, medicine, and many other areas (see how NBA coaches use DM tools) grows steadily. Almost all of our decisions and habits are recorded somewhere and data mining becomes our only hope for extracting meaningful patterns. Of course, privacy issues related to the "proper" use of DM tools will continue to have serious ethical implications, but I'll leave this part of the story for our forum discussions.

Data mining software can be costly, especially if we are talking about enterprise-wide packages that run on parallel servers and mainframes. But as vendors attempt to simplify their tools and offer industry-specific applications, some tools can fit even cut-down budgets. Be aware that these tools are not universally useful, because they offer only a single algorithm may or may not work for your specific business application. Others offer a full suite of algorithms, but cost up to several hundred thousands of dollars and sometime require external help with setup and larger projects. To see what's going on in this field, here are several sites offering a full range of DM tools:

• Agnoss KnowledgeSTUDIO, KnowledgeSEEKER, KnowledgeAccess, KnowledgeExcellerator
• Azmy SuperQuery
• Business Objects BusinessMiner
• Cognos Cognos Scenario, Cognos 4Thought, Cognos Visualizer
• IBM Intelligent Miner family
• WizSoft WizWhy, WizRule