Data Extractors

Special Report - Data Extractors

Dateline: 09/21/00

Finding the "right stuff" - fresh and useful AI-related information - is one of the tasks I do on a daily basis. Juggling with dozens of news sources, search engines, USENET newsgroups and other resources is time consuming and frustrating process - there are simply too many Web sites with too much content out there. One sentence from the NQL's site makes it painfully simple: "With Web content doubling every two months and human beings' ability to digest information a constant at .3 MB an hour, the need for automation is clear." I thought that tools for intelligent information retrieval, data aggregation, content engineering and other methods for dealing with infoglut are in still their adolescence. People are interested in developing comparison shopping engines, intelligent search agents and assistants, specialized meta-search engines and similar services, but often cannot find the right tool for the job. Java, Visual Basic, Perl...? Well, OK, you can use any of these, but be prepared to invest some time, money and serious commitment. However, situation has turned upside down in a past few months: now there are several affordable and easy-to-use languages/developments environments that can be used to process, manage and really take advantage of Web content. See below for a key facts and pointers to sites with more detailed information on each product.

Data Extracting Tools

DEXTR's Data Extractor by DogHouse Enterprises, Inc., starting from $99

PROs The market this tool is geared toward is "Data Extraction for Dummies". It can read and write to most of the popular databases as well as legacy, report, binary, text, and, of course, HTML files. Probably one of the simplest and cheapest (but still powerful) data integration, data extraction and data migration tools.

CONs Missing integration with other development tools/languages and Web server software, scalability issues, available only for Windows.

NQL by NQL Inc., starting from $2500 for Workstation Edition

PROs Reliable tool for rapid and simple development of intelligent agents, bots, spiders, middleware and scalable business to business content aggregation applications. Easy integration with other languages (VB, Java, C/C++), multi-threading/parallel processing, e-commerce and mobile device support. Only tool in this category that provides support for advanced AI techniques like fuzzy processing and neural networks.

CONs Pricey, available only for Microsoft platforms.

More information: Network Query Language, new network programming language for the rapid development of agent applications.

REBOL by REBOL Technologies, free for Core Edition, starting from $249 for REBOL/View, including graphical compositing system for user interfaces

PROs REBOL code and data can span more than 40 platforms without modification using ten built-in Internet protocols, with pieces of a program distributed over many systems. Excellent support and documentation. REBOL/Command is a new addition to the REBOL product family that extends the features of REBOL/Core by integrating access to operating system specific functionality such as dynamically loadable libraries, shell commands and applications, and ODBC-compliant data sources.

CONs Still maturing, with several additions launched last month.

More information: Talking to Your Database, a short introduction to the new scripting language.

Web Language by Compaq Corp., free

PROs Scripting, imperative, interpreted language for automating tasks on the World-Wide Web, with built-in support for common protocols like HTTP and FTP, and popular data types like HTML and XML. Large user base. Implemented in Java, the complete source code is freely available. The system design is that extensions (in the form of modules) are easy to add by Java programmers.

CONs Support issues - the site was down for a several months, with rumours that Compaq will commercialize the language. Quite difficult to use by less experienced programmers.

More information: WebL, scripting language for automating tasks on the Web.

World Wide Web Wrapper Factory by Tropea, Inc., free for educational purposes, otherwise $995 per copy

PROs Toolkit for rapid authoring of software components that integrate Web-published data into applications (wrappers). Both extraction and a mapping phase are guided by the structure of the data. Both involve programming with high-level descriptions and both are semi-automated by interaction with simple wizards.

CONs Small user base and code respository. Rather expensive for commercial use.

Of course, this short report doesn't include all products. The demand for this type of tools is growing, and I already heard some exciting announcements. For example, WebWrappers X will include runtime and development environment that allows you to create intelligent agents by using easy to use syntax and VBScript as the core programming language. Stay tuned for more info on this and other data extraction/information gathering tools.

*PROs*	The market this tool is geared toward is "Data Extraction for Dummies". It can read and write to most of the popular databases as well as legacy, report, binary, text, and, of course, HTML files. Probably one of the simplest and cheapest (but still powerful) data integration, data extraction and data migration tools.
*CONs*	Missing integration with other development tools/languages and Web server software, scalability issues, available only for Windows.

*PROs*	Reliable tool for rapid and simple development of intelligent agents, bots, spiders, middleware and scalable business to business content aggregation applications. Easy integration with other languages (VB, Java, C/C++), multi-threading/parallel processing, e-commerce and mobile device support. Only tool in this category that provides support for advanced AI techniques like fuzzy processing and neural networks.
*CONs*	Pricey, available only for Microsoft platforms.
More information: Network Query Language, new network programming language for the rapid development of agent applications.

*PROs*	REBOL code and data can span more than 40 platforms without modification using ten built-in Internet protocols, with pieces of a program distributed over many systems. Excellent support and documentation. REBOL/Command is a new addition to the REBOL product family that extends the features of REBOL/Core by integrating access to operating system specific functionality such as dynamically loadable libraries, shell commands and applications, and ODBC-compliant data sources.
*CONs*	Still maturing, with several additions launched last month.
More information: Talking to Your Database, a short introduction to the new scripting language.

*PROs*	Scripting, imperative, interpreted language for automating tasks on the World-Wide Web, with built-in support for common protocols like HTTP and FTP, and popular data types like HTML and XML. Large user base. Implemented in Java, the complete source code is freely available. The system design is that extensions (in the form of modules) are easy to add by Java programmers.
*CONs*	Support issues - the site was down for a several months, with rumours that Compaq will commercialize the language. Quite difficult to use by less experienced programmers.
More information: WebL, scripting language for automating tasks on the Web.

*PROs*	Toolkit for rapid authoring of software components that integrate Web-published data into applications (wrappers). Both extraction and a mapping phase are guided by the structure of the data. Both involve programming with high-level descriptions and both are semi-automated by interaction with simple wizards.
*CONs*	Small user base and code respository. Rather expensive for commercial use.