ThoughtTreasure
Dateline: 11/10/99
I often hear sarcastic comments on AI and related technologies, stating that there is too much hype and buzzwords added by the marketing wizards and no real progress in the field. Seems that "AI inside" label is simply used too often to lend the marketing appeal to traditional software products. On the other side, there are many impressive projects developed by small teams of programmers that could benefit from the broader public exposure. This time I will present ThoughtTreasure from Signiform, an ambitious attempt to create a freeware Natural Language Processing (NLP) engine backed by a solid ontology. Erik Mueller, a founder of Signiform, is known as the author of "Daydreaming in Humans and Machines", which proposes daydreaming as a useful capability for computer systems, improving their efficiency and creativity.
ThoughtTreasure is best described in the words of its creator as "a comprehensive platform for natural language processing (English and French) and commonsense reasoning". It runs on PCs and Unix and is available for free download.
It allows an application to obtain answers to questions easily answered by humans but enormously difficult for computers (and NLP researchers).
Mueller estimates the size of human commonsense "knowledge base" to over 100 million items, while ThoughtTreasure only contains around 100 thousand items (still very impressive for a project of such size). A database contains 20,000 concepts organized into a hierarchy, where each concept has one or more English and French approximate synonyms, for a total of 50,000 words and phrases. In addition, there are around 14,000 assertions about concepts: a green-pea is green, a green-pea is part of a pod-of-peas, and a pod-of-peas is found in a typical grocery store. To match the complexity of the real world and our natural language, ThoughtTreasure uses three different representation schemes:
This is one of the best tools for experimenting with natural language processing and commonsense reasoning, especially considering the availability of source code and an extensive manual describing advanced techniques like data extraction.
That's not all: some linguistic knowledge is hardcoded, and other more complex techniques are used for specific tasks, like understanding natural language texts or understanding emotions. For an excellent discussion of all technical details, see the overview of ThoughtTresure's capabilities.
|
The original Usenet post:
Article 5464 of rec.arts.movies.reviews: From: [email protected] (Jim Garnier) Newsgroups: rec.arts.movies.reviews Subject: Review of film "Emma" Date: 01 Sep 1996 15:01:02 GMT Douglas McGrath directed "Emma". The film is passionate. It's rated PG. Emma stars Gwyneth Paltrow as Emma Woodhouse. She's lovely. I give it a **** (out of four).
An internal representation of the extracted information:
Now you can ask ThoughtTreasure questions which can be answered based on the extracted information: > Who directed Emma? Douglas McGrath directs Emma. > Emma is rated what? Emma is rated PG. > Who stars in the film? Gwyneth Paltrow stars in Emma. |
ThoughtTreasure is written in C, with source code freely available (more than 70,000 lines). However, you don't have to master C programming in order to customize it. Any program in any language can communicate with a ThoughtTreasure server as long as it can establish a socket connection. The Java-based client API consists of a collection of Java classes that implement the ThoughtTreasure Server Protocol (TTSP).
Data extraction with ThoughtTreasure is one of my favorites. It is not a job for beginners, as it requires good understanding of the subject you are working on. You'll need to to figure out where the required concepts fall in the ontology and enter them if they are not already present. However, when all modifications are in place, ThoughtTreasure can parse texts from Usenet posts and similar sources, extract the information and answer questions about the specific subject.
I have to admit that ThoughtTreasure continues to surprise me with its capabilities even after several months of use. OK, it has several shortcomings, like memory leaks, slow loading and hardware requirements, but the overall impression is very good. Give it a try and let me know how you liked it!
On a side note, ThoughtTreasure is not only program of its kind. I have recently heard about The Meme Machine, a program to build a many-level knowledge from a text stream. It takes a piece of text as an input and builds up syllable, word and phrase knowledge from that text. An internal representation of the text is also constructed during that process. It can then use this knowledge to "comprehend" future texts, parsing the text into the internal language model constructed. The program operates in a connectionist neural fashion, with a virtual neural network storing 128 initial nodes, each representing one of the 128 basic ASCII characters making up most text files on the net. From here, the system has the ability to store new nodes made up of binary pairings of existing nodes. See this Web page for more information on the program, example output, it's methods of operation and philosophy. The author, Brian Smith, can be contacted at [email protected].
Site(s) of interest: