Web Usage Mining With SQL Server 2000
|
Web Usage Mining With SQL Server 2000 |
 |
Design and implementation of a Data Mining framework that can be used as an addition to the IBuySpy Portal or any other portal/document management framework and dynamic Web site.
|
|
|
|
|
|
|
Elsewhere on the Web
|
• Preparing and Mining Data with Microsoft SQL Server 2000 and Analysis Services
• Implementation details/code for Web usage mining systems
|
|
|
| |
|
Data mining (DM) and data warehousing (DW) are related concepts - a data warehouse gathers historical data and is used to obtain perspectives on the business trends and patterns. As any other new technology that enjoys commercial attention, it is still surrounded by a great deal of hype. There is no magic in data mining, but considering the information overload we are dealing with, it is well worth learning more about the technology behind commercial products available today.
The main aim of all data mining tools is discovering understandable patterns in data. This technology tells users what they didn't know and had not even thought of asking. The rough taxonomy of DM methodologies would include association (correlating one event or item to another), classification (recognition of patterns resulting in new organizations of data), clustering (finding groups of facts not previously known), regression (learning a function that maps a data item to a real-valued prediction variable). These terms may sound scary, but don't let neat GUIs and marketing materials fool you: data mining still requires basic knowledge about statistics and the algorithms working behind the scene. Without clean data and good models, there is no hidden power that will transform the garbage at the input to the valuable information at the output. Vendors of complimentary technologies sometimes contribute to the confusion, marketing reporting and OLAP (On-Line Analytical Processing) tools as data mining products. Of course, these can be used to discover useful information, but you are the one doing all the dirty work in this case - if you don't suspect a pattern is there, you will never find it.
The number of potential application areas for this technology is huge. The finance and insurance industries have traditionally recognized the benefits of data mining. Possible applications include detecting patterns of fraudulent credit card usage, finding hidden correlations between different financial indicators, identifying stocks trading rules from historical market data, identifying behavior patterns of risky customers and claims analysis.
I have recently published a detailed article that explains the implementation of a web usage mining system that mines web access log files and other information that reveals user browsing and access patterns. It analyzes the navigational behavior of users in a web site and suggests other related pages or products. This is a variation of market basket analysis; a class of DM algorithms that examine a list of transactions in order to determine which items are most frequently purchased, viewed or accessed together. Such modeling techniques are based upon the theory that if you buy or view a certain item, you are more (or less) likely to buy or view another related item, depending on the shopping decisions made by previous buyers. The input to a market basket analysis algorithm usually contains a long list of sales transactions with information on a particular product and a sale or a customer, depending on whether the goal of the process is to find which items sell together at the same time, or to the same person.
On the other hand, clustering or grouping users according to their interests (looking from the pages they visit frequently) is a prerequisite for building sophisticated personalized web sites and collaborative filtering solutions. Clustering algorithms in general attempt to segment a population into one or more groups that have similar characteristics and are therefore expected to behave in a similar manner. There is typically no specific outcome or attribute that must be predicted.
More detailed introduction to Data Mining with SQL Server can be found in SQL Server 2000 Resource Kit, Chapter 24: Effective Strategies for Data Mining.
Data Mining and SQL Server 2000
SQL Server 2000 has introduced data mining features for the first time, allowing the SQL developers to leverage their existing skills and build scalable data mining solutions. The data mining functionality in SQL Server 2000 Analysis Services is tightly integrated with both relational and OLAP data sources. It includes two data-mining algorithms developed by Microsoft Research: Microsoft Decision Trees and Microsoft Clustering.
The Decision Trees algorithm is typically employed in classification and estimation tasks, as it focuses on providing information paths for rules and patterns within data. Web usage mining tasks usually employ apriori (Agrawal et al. 1993) or comparable algorithms to find association rules. Our experiments show that the Decision Trees algorithm performs quite satisfactory in the web usage mining environment. It also generates easily understandable rules, requiring only basic knowledge of DM techniques.
The Microsoft Clustering Algorithms provider uses an expectation-maximization algorithm to segment data into clusters based on the similarity of attributes within cases. We'll use it to segment web pages and web site users into clusters based on the history of user's interactions.
Microsoft's OLE DB for DM specification introduces a common interface for data mining that gives developers the opportunity to embed highly scalable data mining capabilities into their existing applications. Its goal is to provide an industry standard for data mining so that different data mining algorithms from different platforms can be easily interconnected.
Data mining model is a basic concept introduced in OLE DB for DM. It is an equivalent to the table in a relational DBMS in the sense that it has a list of input and/or predictable columns with the appropriate data types, but the similarity stops here. It is only a container, and it does not store raw data: instead, it stores the patterns that data mining algorithms have discovered. OLE DB for DM extends the table creation syntax from SQL for creating DM models. They can be created a number of ways in Analysis Services, depending on the location of the data mining model. Data mining models can only be created on the server through the Decision Support Objects (DSO) library. The easiest way is to use the mining model creation wizard. It generates data mining creation and training queries and sends these queries to data mining providers. We'll use this approach for DM model creation and training, although we could also write code to connect to the data mining provider through ADO or DSO, and then issue our queries in a similar way as we typically issue SQL queries.
Here is an overview of the whole process:
The rest of the materials, including a step-by-step tutorial on creating the data mining framework can be found at ASP Today. Another excellent resource for practical data mining with SQL Server is Microsoft's online book "Preparing and Mining Data with Microsoft SQL Server 2000 and Analysis Services".