GNU Wget

Wget is a freely available network utility to retrieve files using HTTP and FTP, the two most widely used Internet protocols. It works non-interactively, thus enabling work in the background. It is available for the most brands of Unix, as well as MS DOS and Windows. Several Web sites offer a lot of related tips and scripts for setting up different Wget functions.

eSpider

eSpider is a Web spidering ActiveX control written in C++ using the Active Template Library (ATL) which creates extremely fast, and tight code (around 44Kb for this object). eSpider performs the crawl for you through a single method call. User can control how many pages it returns, the depth to which it crawls and specify unwanted. The pages are then returned in a completely structured way, making it easy to interpret the layout of the remote site. Any development environment which supports ActiveX can be used, including Visual Basic, Visual C++, Delphi, etc. However, this is the only commercial application in this review.

Harvest-NG

Harvest-NG is a set of open source tools for building a standards-compliant Web crawler. It is implemented in Perl, taking advantage of many of the existing tools. The development of Harvest-NG was started as an attempt to preserve the strong features of the Harvest architecture, but to allow more rapid developing and prototyping from a cleaner, better structured codebase. It employs an interesting summarization approach, using a standard format called SOIF (or Structured Object Interchange Format) that produces a "resource description" for any given document. This way, other tools can deal with the content without problems related to parsing multiple input file formats.

ht://Dig

The ht://Dig system is a complete Web indexing and searching system for a domain or intranet, covering the search needs for a single company, campus, or a particular sub section of a Web site. As opposed to some Web server based search engines, ht://Dig can easily span several web servers. The type of these different web servers doesn't matter as long as they understand common protocols like HTTP. It is one of the most powerful open-source packages in this category, offering a complete, multiplatform (you'll need to study Idiot's Guide to Installing ht://Dig on Win32 for non-Unix systems) search engine solution - the crawler part itself is appropriately called htsearch.

Java Web Crawler

SourceForge is a relatively new and very popular service for open source software developers. You can host your project on SourceForge servers and use many of their services like bug-tracking, forums, etc. Java Web Crawler is one of such open source projects aimed at developing a complete solution for retrieval of Web pages. It is not very active at the moment, but may attract programmers willing to learn more on this subject.

Larbin

Larbin is a multipurpose Web crawler, initially developed for fetching XML pages and storing them to the database of an XML-oriented search engine. Thanks to its origins, Larbin is very generalistic and easy to customize. It is freely available for Linux and other Unix flavors under the GPL license. Larbin is capable of fetching around 5,000,000 pages a day on a standard PC (of course, this figure heavily depends on the speed of your Internet connection). You can follow the crawler's progress through its internal Web server on a port 8081 and submit the URLs to crawl using the port 1976. Advanced users can write customized procedures by overriding standard functions in src/xinterf/output.cc. Another, more simple way to customize your crawler is by editing configuration files larbin.cong and include/types.h.

MacroBot

MacroBot is a free Win32 search agent that can be programmed to execute data retrieval analysis and retention functions. It uses Basic as a scripting language so users can plug-in macro scripts or create their own scripts. MacroBot comes with user documentation and SAX basic scripting language instructions. It can be used with Instant Index's FindMine to provide full text search capabilities.

Pavuk

Pavuk is another freely available UNIX application used to mirror contents of WWW documents or files. It transfers documents from HTTP, FTP, Gopher and optionally from HTTPS (HTTP over SSL) servers. Pavuk has optional GUI based on GTK+ widget set. It can be run in eight different modes, changing crawling strategies to best fit the your needs.

webbase

webbase is a crawler written in C++ for Linux OS. Besides basic crawling strategies, it can also build a full text database containing the retrieved information. The main goal of this project was to develop an application ready to efficiently crawl millions of URLs and store retrieved data in a MySQL database.

WebSPHINX

WebSPHINX (Website-Specific Processors for HTML INformation eXtraction) is a Java class library and interactive development environment for Web crawlers. It is geared towards developing crawlers that are site-specific, personally customized and relocatable. Crawler Workbench is a visual, interactive environment used for personal crawling tasks that require little or no programming. Site specific rules can be encapsulated in content analyzers, known as classifiers. Rules help the crawler decide which links to follow and which pages or parts of pages to process. The simplest crawler in WebSPHINX is implemented by overriding two basic callback methods, shouldVisit and visit, used together with a powerful pattern matching mechanism. While a crawler is running, the Crawler Workbench also displays the growing crawl graph, consisting of the pages and links it has encountered. Relocatable crawlers can be uploaded and executed on a remote Web servers.