WebL

Dateline: 09/22/98

I am searching for agent-related software products almost every day. Sometimes there are no news; sometimes it's just a new version of some well-known concept or product. And sometimes it is something really exciting and useful, so I have to forget about all other jobs, download all the available information immediately and stay up all night to learn about it as much as I can. This week I'll briefly describe WebL, new programming language for automating tasks on the World-Wide Web that kept me awake for several nights.

Writing page information extractors, meta-search engines, Web crawlers and similar programs can be a difficult task. I have used Java and Visual Basic for such tasks: can't say it was easy or fast. In contrast to other general purpose programming languages, WebL is specifically designed for automating tasks on the Web. Not only does the WebL language have a built-in knowledge of Web protocols, but it also knows how to process documents in plain text, HTML and XML format!

The flexible handling of structured text markup is the key feature of this language. WebL also supports features that simplify handling of communication failures, the exploitation of replicated documents on multiple web services for reliability, and performing multiple tasks in parallel, while providing traditional imperative programming language features like objects, modules, closures, control structures, etc.

WebL has several features that simplify web computations. These include:

The WebL interpreter is written in Java (it runs JDK versions 1.1 or later), and distributed under the open-source development model with a non-commercial use license. Free! And it is 100% pure Java, except for the Browser module on the Windows platforms, which needs a Windows DLL to display output inside your web browser. However, if you don't need this functionality (like me), scripts are portable.

Programming language syntax, although highly specialized, is very easy to understand. Want to build your custom Web crawler? Here is an example:

// A demonstration web crawler that restricts its
// crawl to a specific site.
import Str, WebCrawler;
var MyCrawler = Clone(WebCrawler_Crawler,
    [.
        Visit = meth(s, page)
            var title = Text(Elem(page, "title")[0]) ? "notitle";
            PrintLn(page.URL, " title=", title);
        end,
        ShouldVisit = meth(s, url)
            Str_StartsWith(url, `http://www.yahoo.com`)
            and
            Str_EndsWith(url, "(/)|(.html?)")
        end,
    .]);
MyCrawler.Start(2);
MyCrawler.Enqueue("http://www.yahoo.com/");
while !MyCrawler.farm.Idle() do Sleep(10000) end;

WebL module WebCrawler exports a single object called Crawler that implements a multi-threaded Web crawler. To use it, you should override the methods Visit and ShouldVisit: the language itself is object-oriented, and the [..] notation is used for constructing objects. The Visit method is called by the crawler each time a page is visited, and the ShouldVisit method returns true when a specific URL must be crawled. The crawler is activated by the Start method which takes as argument an integer specifying how many threads should perform the crawl. At this point the crawler has no pages to crawl yet. Pages are inserted into the crawler queue with the Enqueue method. As each page in the queue is processed, the crawler will extract all the anchor (<A>) tags in that page, and call the ShouldVisit method to determine if the page referred to by the anchor should be crawled or not. The Abort() method can be called at any time to terminate the crawl.

That's it! The only drawback is that WebL programs tend to run slowly sometimes, and their memory usage is quite high (WebL keeps everything in memory, including complete copies of pages). However, you'll write programs for the Web much easier now.

There is simply to much to say about this language that is simply beyond the scope of this article - WebL will be the main subject for more feature articles in the near future. In the meantime, visit the WebL Web site and download the alpha version. An excellent 135 page reference manual in PDF format is included in this distribution.
To subscribe to the mailing list, send a message to [email protected] for further instructions. You are also welcome to send e-mail to the author, Hannes Marais. He is especially interested in WebL success stories. There is no doubt that he will receive quite a number of such stories...