Information Extraction
|
Behind the Scenes of aboutAI.net: Information Extraction |
 |
How to extract information from semi-structured information sources: a short tutorial with pointers to source code samples.
This article is the first in the series that will explain the use of various AI-related techniques at this site. The solutions described here could be easily deployed at your site(s), depending on the platforms and development tools you are using. aboutAI.net uses Microsoft .NET platform with SQL Server 2000 as the main DMBS. While some of the techniques will be platform-specific, we will try to explain the most universal concepts, pointing you to other sites with more details and source code.
We will start with the short tutorial on techniques used for information extraction. The main idea was to implement a functionality that would allow visitors to use this portal as a starting point for further exploration of AI-related topics on the Web. Juggling with dozens of news sources, search engines, USENET newsgroups and other resources is time consuming and frustrating process - there are simply too many Web sites with too much content out there. In order to simplify this process, we designed a framework for extracting interesting and fresh AI information from other sites. The basic idea was to "scrape" the target pages automatically, several times a day, insert the extracted data to a database, and than include the preformatted data into the standard portal module layout. The final result can be seen in action at our home page in the "Job Hunt" section that lists the AI-related job posts from the major job boards. The same technique is deployed at the Discussion page, listing fresh forum threads from other sites. A slightly modified approach is taken for extracting discussion threads from Usenet newsgroups. These ideas can be applied to comparison shopping engines, "intelligent" search agents and assistants, business intelligence solutions, specialized meta-search engines and similar services.
In this article we focus on building an extensible VB.NET application framework for Web extraction tasks from heterogeneous Web sources. It uses advanced techniques for converting ill-formed HTML to well-formed XML, with XSLT transformations provided by the powerful XPath specification. Best of all, there is no need for commercial third-party controls, as all of the necessary ingredients are already packaged into the VB.NET distribution.
The majority of HTML document available on the Web is poorly formated and describes only presentational details, while we would like to use well-formed content description. The structure of target document can change very often, especially in dynamic environments used for sophisticated banner advertising and content management. We could solve this problem by writing advanced regular expression scripts (available in almost all modern programming languages) that will extract only the "clean" content from the HTML markup. As it turns out, this is a very cumbersome and error-prone process, requiring excellent knowledge of various parsing techniques. To make things even worse, it is a maintenance nightmare, and every time the target document changes you'll have to re-code, re-test and re-compile the extraction procedures. Isn't there a better way?
You could always choose one of the commercial packages for information retrieval, data aggregation, content engineering and similar tasks. We already described several packages that will help you fight the "infoglut" in one of the previous articles. There are also several proposals of models and query languages for semi-structured data extraction that can enable easier development of information retrieval and information integration systems. Other systems are used to mine XML code to find relationships and predict values using fuzzy-logic rules, or to automatically generate Java wrapper programs for Web information sources.
However, one of the most elegant approaches for extracting semi-structured data from HTML is described in an article at IBM developerWorks: "Automatically extract information with HTML, XML, and Java" by J. Myllymaki and J. Jackson. They presented the coding tools necessary for enabling any Java developer to begin his or her own extraction work with a minimum amount of effort and extraction experience. While the example in this article focused on merely extracting weather information about Seattle, nearly all of the code is reusable for any data extraction task. By wisely choosing data sources that are reliable and picking anchors within those sources that are tied to content and not format, you can have a low-maintenance, reliable data extraction system in a matter of hours. We divided the whole process at aboutAI.net into several steps:
- Identification and retrieval of HTML source documents
- Preprocessing the HTML documents, removing the ill-formed tags that cannot be automatically removed
- Transforming the resulting HTML to XHTML
- Applying the XSLT stylesheet to convert the resulting XHTML to "data-centric" XML that is suitable for information extraction
- Saving the extraction results to a database
The detailed description of the process, along with the source code in VB.NET is now available at the ASP Today Web site.