As with any emerging research area, there is still no established vocabulary for data/text/Web mining, which can lead to confusion when comparing results and techniques. To clarify this, let's just say that text mining takes unstructured textual information and examines it in an attempt to discover structure and implicit meanings buried within the text. On the other hand, data mining is the process of extracting valid, previously unknown and comprehensible information from conventional databases and using it to make crucial business decisions. It takes only structured data on the input, while text mining works with information stored in an unstructured collection of text documents - and the Web is the biggest collection of them all. You shouldn't mix it with Web usage mining, the process of mining Web access log files or other user information that reveals user browsing and access patterns on one or more Web localities. This type of applications are little beyond the scope of this article, but one good example can be seen at Humboldt University Berlin where their Web Utilization Miner analyzes the navigational behaviour of users in a Web site, or performs sequential pattern discovery in any type of log. It discovers patterns comprised of not necessarily adjacent events and satisfying user-specific criteria.

Most of the systems using content-based approach for information retrieval use bag-of-words representation, where a document is represented as a vector of features, with a specific frequency assigned to the each feature (this frequency corresponds to the number of times each stemmed word occurs in the document). More in-depth discussion and technical details on stemming, stop words, TF-IDF, n-grams (word tuples), etc. can be found in the previous article. I'll briefly present several new commercial or academic solutions for text and Web mining here.

IBM Intelligent Miner for Text turns unstructured information into business knowledge for organizations of any size. It is probably the most complete solution for information retrieval/knowledge acquisition from the Web, including text analysis tools such as a feature extraction tool, clustering tools, a summarization tool , and a categorization tool. It also incorporates the IBM Text Search Engine, NetQuestion Solution, an Internet/intranet text-search solution, and the IBM Web Crawler Package. It is running on several operating systems, including Windows NT, AIX, OS/390 and Solaris.

Semio Corporation offers several solutions based on SEMIOLEX technology, developed during years of research by Dr. Claude Vogel. Semio's linguistic processing of text does not require a rich understanding of text such as would be required with traditional linguistic processing. Instead, this technology is based upon pioneering research in the field of computational Semiotics. It automatically creates browsable, searchable directories for intranets and Web sites, and then categorizes the content in a way that's appropriate for people at various levels of your organization. Semio's Taxonomy engine creates a multi-level directory structure that includes thesaurus-like links. These links cross-reference related parts of the directory to help match thought patterns both hierarchically and across concepts.

iCrossReader from InsightSoft-M lets users to automatically build an on-demand survey from excerpts of documents without the need to first search for the texts and conducting a visual review. This concept abandons the convention of presenting full documents (as search engines do) or their summaries (as summarizers do). Instead, the user is provided with excerpts that follow the specified personal interest. It is available as a ready-to-use application or a customized solution following specific needs of the user.

Attar Software offers a wide range of solutions for capturing knowledge from various sources based on graphical decision trees with optional knowledge capture using example tables, truth tables, exception trees and pattern rules. Application structure maps and central object dictionary are used for clarity and ease of maintenance. Developers can use an extensive command language for procedural programming and connectivity.

Another interesting product, TextAnalyst from Megaputer Intelligence, is available as a standalone application or as a set of COM-based modules implementing individual analytical functions. It implements a variety of analysis functions based on automatically created semantic network of the investigated text, including textbase navigation, topic structure extraction, concept tagging, clustering, etc. Currently the development team of TextAnalyst is working on implementing automated classification of documents. Measuring the similarity of individual texts is another future feature under consideration. Further information about the system and an evaluation copy of TextAnalyst can be found at Megaputer's Web site.

TextSmart fro SPSS is a survey research tool that makes analysis of responses to open-ended questions fast and affordable. It can be used across a wide range of applications, including customer satisfaction surveys, brand and image surveys, production surveys, social and political research, health and lifestyle research, consulting research and similar fields.

Instinct Software's Text-o-Scope offers a new approach focuses on knowledge acquisition to help users handle large collections of documents. It automatically identifies patterns, correlations, key terms and features. Text-o-Scope will present the user with visual maps of information. The user can drill-down to the relevant documents in any of these maps.

Readware from Management Information Technologies, Inc. offers a product line that is comprised of a central ConceptBase, and a few independent functional algorithms or software agents (a text compiler, a survey analyst and a query processor) that utilize the Readware ConceptBase to process textual information from text files. Company claims it is filled with fundamental concepts of knowledge that are independent of any language or any given text. Thereby it can automatically and accurately recognize and classify the concepts, terms, names, numbers and other elements of the text of any binary-encoded message or document. The results of these unsupervised algorithmic processes are stored as a sort of a conceptual and mathematical meta-map. Readware is commercially available as a toolkit that includes a Readware C++ Class Library. This is delivered as an open-source, object-oriented encapsulation of all important Readware functions. Developers can use the encapsulation to easily integrate Readware techniques into the next generation search engines, document management systems, text collections and information and research systems.

Programmers interested in building custom solutions for Web mining will surely benefit from W4F, World Wide Web Wrapper Factory. This is a toolkit for generation of wrappers for Web sources. It consists of a retrieval language to identify Web sources, a declarative extraction language (HEL) to express robust extraction rules and a mapping interface to export the extracted information into some user-defined data-structures. To assist the user and make the creation of wrappers rapid and easy, the toolkit offers some WYSIWYG support via wizards. It has been successfully used to generate wrappers for database systems and software agents, making the content of Web sources easily accessible to any kind of application. Another good programmer's tool is PROTAN (for PROTocol ANalyzer), a computer-aided content analysis system composed of 30 programs. The Natural Language Playground at Carnegie Mellon is a great collection of interactive language tools, and similar tools can be found at Columbia University.

   Site(s) of interest:

• AltaVista Discovery
  Free software that provide users with powerful content-retrieval capabilities and makes it possible to synthesize different paradigms of Web and desktop searches.
• IR - IE and Statistical Learning Methods
  Applications of Machine Learning techniques to Text-Mining, Information Extraction and Information Retrieval, including problems such as routing and filtering, information dissemination, highlighting, clustering, etc.
• Finding Out About
  "Search Engine Technologies from a Cognitive Perspective", very interesting new book by Richard K. Belew.
• InfoSpiders
  An artificial life - inspired system using local selection and internalization for autonomous, scalable information retrieval in large, dynamic, distributed, heterogeneous databases such as the Web.
• WebMiner
  An older resource and knowledge discovery system for the Internet.
• Web Mining: Information and Pattern Discovery on the World Wide Web
  Two important and active areas of current research are data mining and the World Wide Web. A natural combination of the two areas, sometimes referred to as Web mining, has been the focus of several recent research projects and papers.
• Web IR & IE
  This site is a collection of online resources for research in the field of information retrieval and information extraction from the web.
• Machine Learning Applied to Information Retrieval
  An exhaustive list of papers on this topic.
• Turning Information Into Knowledge
  White paper describing IBM Intelligent Miner for Text.
• WHIRL
  WHIRL is a system that's designed to integrate information from various places on the Web. Rather than exploring a number of sites to get the information you want, WHIRL enables you to simultaneously query a number of related sites using a database-like interface. Technically, the main innovation in WHIRL is a new sort of "soft" database system, which uses text-matching methods developed in information retrieval to approximately match names.
• Learning for Document Categorization
  Good resource for links on related topics.
• Text Mining: Beyond Search Technology
  The rise of text mining is taking search engine capabilities to the next level. Here's how to reap the advantages while avoiding the pitfalls of implementing this new technology within your organization.
• Text Mining, Web Mining, Information Retrieval and Extraction from the WWW References
  Probably the most comprehensive source of links in this field.
• Information Retrieval
  Classic book by C. J. van Rijsbergen in HTML format.
• Online Text Mining
  This site will serve as a clearinghouse of information on the topic of Online Text Mining, with links to many articles and vendor sites relevant to this topic.
• Personal Web Watcher
  Personal WebWatcher is a "personal" agent that accompanies you from page to page as you browse the web, highlighting hyperlinks that it believes will be of interest. Its strategy for giving advice is learned from feedback from earlier tours. Excellent source of information and research papers.
• Andreas S. Weigend
  Extracting knowledge from (possibly quite large) data sets, applying time series analysis, statistical artificial intelligence and neural networks to problems in business, marketing and finance, as a part of the Knowledge Discovery, Data Mining and Modeling Group at NYU/Stern.
• Dragomir R. Radev
  Great source of links related to natural language understanding and text summarization.
• ML meets Text
  Projects and research at University of Dortmund.
• CNLP
  Connectionist Natural Language Processing uses neural networks to process language and/or speech. Adherents believe it represents a more flexible method than more traditional AI approaches.
• WEBSOM
  Connectionist document exploration tool that builds an ordered map of the information space where similar documents lie near each other on the map.
• Marti Hearst
  Research focusing on user interfaces and robust language analysis to build information access systems. Great source of information, including LINDI, a text data mining system for automated discovery of new information from large text collections; TileBars, an interface is an attempt to show the user, graphically, the relationship between the words in the query and the documents retrieved; and Cat-a-Cone, a novel user interface that integrates search and browsing of very large category hierarchies with their associated text collections.
• The Big Picture
  Visual browsing in Web and non-Web databases - projects, research, products and services.