Search Engines: The Next Generation
|
Search Engines: The Next Generation
|
 |
AI-based solutions for capturing and indexing the vast amount of information available on the Web
AI-related technologies were always at the heart of every Internet search engine service. However, challenges presented to the current technology have grown explosively in the last few years. NEC Research Institute recently reported that nearly 7 million pages (!) are being added to the Web each day, making it even harder to search through the already existing 2 billion pages. Top-down, crawler based solutions simply cannot keep up with a system growing at such breakneck pace, so only a small percentage (20-30%) of Web content is captured and indexed by individual services. Same holds for directories operated by human operators and similar solutions.
Another problem is posed by the limited ability of search engine spiders to extract information from a wide variety of database-driven sites. Such systems typically require some sort of human interaction using search forms to fetch information, and form a large portion of cyberspace commonly referred to as the "invisible Web". A study by BrightPlanet puts the size of the invisible Web at 400 to 550 times larger than the "visible Web".
Some users rely on the use of proprietary AI algorithms buried inside client-side tools. One of the most comprehensive lists of Internet Searching Tools includes examples of such approach: one of them, AnswerChase PROve offers utilities like intelligence monitoring, content building, content streaming, and content sharing. Its search domain covers all topics but is specialized in business and finance. Intelligent agents are sometimes also used to enhance search engine performance.
At the server-based end of spectrum of "intelligent" search services, natural language processing (NLP) techniques and linguistic analysis are used to summarize content and identify relevant entities. Ask Jeeves was one of the first search engines using NLP approach that didn't require understanding complex rules for composing advanced queries. Another NLP search solution, Albert, uses proprietary natural language processing and machine-learning algorithms for which three U.S. patents are pending. Its greatest advantage is language independence: handles any language including each user's very own wording.
As for the linguistic analysis and other means for intelligent content summarization and recognition, several commercial products and services are using different techniques to achieve the same goal. Autonomy's core technology - Bayesian Inference - provides an automated infrastructure for the profiling of users, aggregation, categorization, hyperlinking, context-based searching and personalization of large volumes of unstructured information. LexiQuest solutions analyze text to understand the meaning of each word, the structure of each sentence, and their overall concepts. Over the last two decades, LexiQuest has developed a highly evolved underlying technology leveraging a 60,000-word dictionary and 500,000 links between more than 150,000 semantic concepts to boost the accuracy of various search engines. Cycorp recently founded a new division called e-Cyc to bring the power of Cyc, world's largest general-purpose knowledge base, to these environments. The e-Cyc Portal Toolkit is a special release of the Cyc Knowledge Server that allows Internet and intranet web sites to add common-sense intelligence to their environments. It could be described as an advanced type of Internet search engine that's being offered as a solution to the preponderance of time-consuming "false positives" and "false negatives" often produced in the process of browsing. Intelligenesis, a New York based company led by mathematician Ben Goertzel, is working on a system called WebMind that will incorporate search engine functionality, but will also involve visual exploration of information, and interactive processing whereby the software asks the user questions to clarify their intentions.
YellowBrix creates topic categories using neural networking techniques. Content and/or product inventory is then filtered, categorized and the relevant data is stored in information models. But we are not limited only to the textual formats: eTantrum's Music ID Service utilizes patented acoustical analysis technology which has the ability to identify an audio "work" based on its analogue waveform and acoustical characteristics. They are also offering the Songprint SDK, a free library that can automatically identify any given piece of music.
Seattle-based 360 Powered Corporation recently unveiled new search and indexing technology that is radically different than current search and indexing models. Their Push-Indexing technology offers automated, self-populating, bottom up, agent-based solution to capturing and indexing the vast amount of information available on the Web. CEO and founder of Cycorp, Doug Lenat, states that 360 provides the first new qualitative approach that makes a functional difference in how knowledge is indexed, how the Web is indexed and ultimately, how information is retrieved. They effectively reversed the traditional approach used by spidering search engines. Push-Indexing employs distributed agent technology to transmit indexed data (all formats, not only HTML) on a daily basis to the 360 Master Index. It will be refreshed every 24 hours to accurately represent the current content. 360 will encourage content providers to download free client-side agents. To maximize efficiency, software detects incremental updates and broadcasts only changes from the previous state. Two types of these agents are used: Host Agent for Web servers or Site Agent for individual Web sites. Host Agent is developed in Java for Web hosts who want to index all or a portion of the sites on their Web servers. It is obvious that many users will not have privileges to install and run this type of software. Site agent, on the other hand, runs on any Web server with support for CGI Perl.
According to company officials, this system will also extract concepts from the indexed documents to deliver more relevant search results. Index Cards allow users to explicitly describe their content with categorical, demographic, and geographic metadata. It is an XML-based open standard that will hopefully bring us a step closer to the "Semantic Web" everyone is talking about these days. Information contained inside present meta tag systems are notoriously difficult to comprehend for search engine spiders are still far away from standardization.
360 promises to overcome all known limitations of search engines by providing content and concept-based search, indexing the entire Web regardless of how large or how quickly it grows. It remains to be seen how exactly will they persuade host administrators and content providers to download and use client-side agents. Standardization of the push-indexing technology, integration with popular Web server software and cooperation with some of the major portal sites could be the keys to success. Another question is how will they protect the integrity of the master index from spammers. Anyway, if you're interested in joining the Early Adoption Program, fill out and submit the form available at their Web site.
And for the end, a new open source technology for search engines has been recently announced by the Gnutella Next Generation development team. gPulp will be using the basic protocols developed for Gnutella, popular distributed file exchange system, it will search for information on the Web in a real time.