For better results, search engines want your intelligence

Search engines are trying to get smarter by enlisting help from their users  | IslamOnline.net

Say you want to know how much salt there is in a pinch of salt. So you head to Google with your query. And, ever the generous, the search engine coughs back more than a million links.

And if you’ve left Google set to its default 10 results per page, you will be in no luck with the salt pinch query. The first link that may resemble an answer comes 11th (a Yahoo Answers page).

But you could also stop by Wolfram Alpha, a newcomer on the search engine scene. There you will get a smaller number of results: a single page that contains in neat tables all the information one would probably hope to know about the pinch of salt.
Between Google’s generosity on one end and Wolfram Alpha’s terseness on the other, a growing spectrum of search engine approaches is emerging, with the aim of making it easier for users to find what they want on the web.

Last May, Microsoft offered its updated search engine, named Bing. In Bing, Microsoft seems to have gotten over the number of results (that is, links) as the sole parameter of search quality. Instead, the
focus is on increasing the utility of a relatively smaller number of search results.

Bing does this by offering related search terms and search history on the left pane, as well as a quick view panel on the right for a sneak look at the link page without visiting it.

Another change in Microsoft’s engine is the attempt to incorporate more artificial intelligence, such as natural language processing (NLP), so that Bing is better able to discern what the user wants than
which pages are more popular than others. For example, if you query Google for “Huntington’s disease” you’ll get the results ranked by how popular they are (determined by how many links they receive, especially from popular sites). And these “popular” pages most often happen to be the most useful ones.

Bing, by contrast, aims to recognize if your query is about a historical figure, a disease, or a city among others, and tries to direct you straight to the websites most likely to “answer” your query.
So in the “Huntington’s disease” search, Bing would list on top of the results links to web encyclopedias, medical dictionaries, and medical research centers. And in the left pane you’ll see links to related terms.

Granted, Bing and Google may look alike in terms of the outcome — on the first page of search results. But the ways they arrive there are different.

Keywords Not Enough

“Yet increasingly search engines – including Google – are realizing the deficiencies of exclusive dependence on the algorithm and the need to augment artificial intelligence with human intelligence.”

Bing, of course, isn’t the first to try to outdo Google by offering more annotated search results. Clusty should probably get credit for being first to offer a pane on the left side of the page with groupings, clusters, of the search results. In our Huntington’s disease example, Clusty’s left pane offers keyword clusters (as in “genetic,” “society,” “people with Huntington’s disease”, etc.) as well as a “sources” grouping and a “sites” grouping.

And then came Ask engine (formerly AskJeeves). It too tries to shift the attention away from more results — Google’s unmatched advantage so far — to annotated results.

Another recent entrant is Cuil, a search engine developed by former Google engineers. Cuil offers a more eye-friendly search results page than Google’s. And its right-side column offers a wealth of related terms and sources.

Then there’s Wolfram Alpha. Wolfram Alpha, apparently, takes an entirely different approach to web search. Google, Yahoo, Bing, Cuil, Clusty and almost every other search engine use basically the same
model (and with varying degrees of success.) They index the web’s sites and pages constantly. When queried, they serve lists of links ranked according to some parameter.

Not so with Wolfram Alpha. It neither indexes zillions of pages nor offers results in lists of links. Instead, Wolfram Alpha (dubbed by its creators the “computational knowledge engine”) draws data from a number of public databases (a small number for now.) When you search Wolfram Alpha, the engine pieces together an “answer” to your query from those databases it had indexed.

So in place of the sea of blue links in an infinite horizon of white, what you always get on Wolfram is a single grayish, pinkish page, in which the information culled from different databases is presented in tables — information that you can also get as a single PDF file if you want.

Artificial Intelligence Not Enough, Either

Still Wolfram Alpha has its limitations. First, it relies exclusively on information in official databases and thus skips a great deal of sources, including newspapers and public-domain journals, among others. Second, while Wolfram Alpha takes pride in utilizing artificial intelligence, it does not seem particularly intelligent with spelling suggestions or alternative term suggestions as other search engines do.

Still, worthy of note about Wolfram Alpha is that it seems to reinforce two opposing trends in the search engines realm. On the one hand, it takes to an extreme the new approach of maximizing the utility of a manageable number of search results. So where Bing, Clusty and Ask try to afford users more information in a smaller number of search results (than Google), Wolfram Alpha tries to give the answer in a single page. If you are lucky, that is, because to many queries the answer is “Wolfram|Alpha isn’t sure what to do with your input.”

On the other hand, Wolfram Alpha is built around the notion that the search algorithm alone is sufficient to recognize what the user wants. Thus far, Google (whose success is owed to its PageRank algorithm) is the champion in this arena. PageRank turns up highly relevant web pages for any given keyword based on the relative popularity of each page.

Yet increasingly search engines — including Google — are realizing the deficiencies of exclusive dependence on the algorithm and the need to augment artificial intelligence with human intelligence. Last November Google launched its “search wiki” feature, allowing signed-in users to re-rank, add, delete, and annotate their own search results (and Google saves changes in the user’s account).

Yahoo seems to be adopting the same philosophy, but is taking a different path. In May 2008 Yahoo announced SearchMonkey, an open platform that allows outside developers to create apps that can be
plugged into the search engine to enhance search results. So in place of one monolithic piece of code, countless Firefox-like plug-ins will pitch in to smooth the search engine’s rough edges — or so hopes Yahoo.

Another initiative by Yahoo along the same lines is Yahoo Answers, launched in December 2005. Here ordinary users pose the questions that bug them and their peers answer them. And answers are ranked by their (perceived) usefulness.

Users’ Help Needed

Divergent as they may be, Google’s search wiki, Yahoo’s SearchMonkey, Microsoft’s Bing, and Vivisimo’s Clusty appear to have one thing in common: an understanding that as the web grows, search engines need to get smarter in helping users. And since technology has not grown at a pace corresponding to the web’s growth, search engines are enlisting help from their own users.

Google’s cofounder Sergey Brin admitted that in the company’s 2008 annual report. “Perfect search requires human-level artificial intelligence, which many of us believe is still quite distant.” (as reported in Technology Review, July/August 2009)

Where the future may lead is probably a model in which a search engine is able to tap into the collective intelligence of its users more efficiently. Google does this now indirectly by relying on back-links for ranking pages (the more links a page receives, especially from “popular” sites, the more popular it becomes). It’s likely you’ll see a more advanced utilization of users’ decisions embedded into the way search engines work.

Until search engines are more intelligent, however, we’ll have to take their results with a grain of salt — or maybe even a pinch of salt, which is exactly 380 milligrams, according to Wolfram Alpha.