Search Engines, Part 1 of 2

Last spring I saw a chart comparing search engines to celebrities (I think it was in Wired, but don't quote me on that). Yahoo!, it said, is the Angela Lansbury of search engines (reliable, predictable), while, at the other end of the scale, the Go Network is Dennis Rodman (who knows what to expect; anything could happen). A list of URLs for sites mentioned may be found at the end of this article. This column is the first of two that will look at search engines and how to make the best use of them.

In November, I attended Internet Librarian '99, and picked up some useful tips. Danny Sullivan, author of Search Engine Watch, asked "How do you find a needle in a haystack? Can you find it in half of a haystack? What if the needle is not in that half of the haystack? Will it be easier if the whole haystack is dumped on your head? Do you need more hay, or just a metal detector?"

There are literally THOUSANDS of search engines ("metal detectors") available on the World Wide Web. For a directory, visit Beaucoup, which offers links to over 2,000 sites, some very specific and some very broad. No one search engine searches the entire World Wide Web. Each one chooses a small portion of the Web for its database. Northern Light consistently comes out near the top in surveys of which has the largest database. At last count there were over 800 million Web pages, although everyone at Internet Librarian '99 agreed there are probably more and, at this writing, Northern Light claims to cover 197,970,728 of those.

There are three main kinds of searchers and many sites are using all three. The original search engine was a spider or crawler. HotBot is a good example. Type in the phrase "ice cream" in the search box, tell HotBot to find it as a phrase, and HotBot's computer reads through the pages in its database looking for ice cream as a phrase. The results could include Ben & Jerry's home page, a page about someone's birthday party where ice cream was served, and a page about ice cream headaches.

Early in the game, Yahoo! created the directory approach. Yahoo! employees (human beings) find and review pages about golf, choose the ones they believe Yahoo! users would find most useful, and when someone types in the word "golf," Yahoo! pulls up that list of pages from their directory.

Recently portals have started to pop up everywhere. A portal is a page where the user has an ID and a password, and dictates the information displayed. For example, I have created a page at My Snap that has links to headline news from sources I have chosen, weather forecasts for places of interest to me, art news, book reviews and health news. Sadly, I have no stocks, so I have eliminated the module that gives stock reports. Were I a sports fan, I could get current scores and information about my favorite teams. Portals are available at most search sites and at many other kinds of sites, the premise being that if I check My Snap daily, while I am there, I will use Snap's other products as well.

GoTo.com -- not to be confused with Go Network, which is owned by Disney -- offers paid placement for results. Thus, a site willing pay 12¢ will be listed higher in the results than a site only willing to pay 8¢, and both will be listed higher than a page unwilling to pay anything. In theory, this practice eliminates "optimizing" by webmasters. This is an example of optimizing given by Danny Sullivan. During the height of the impeachment hearings last year, he searched for "Monica Lewinsky." Among the returned results was a page titled "Monica Lewinsky Nude and Jukebox for Sale." The webmaster optimized this page by putting the words "Monica Lewinsky" and "nude" in the title so that if someone searched either of those terms, his jukebox page would be appear on the list of hits. Instead, the owners of the page could pay GoTo.com to be the first result listed for the query "jukebox." The controversy lies in whether this is advantageous to users. If the first 20 sites listed are paid placement, the user may miss a site that has exactly the information needed but that is not paying for placement.

Search sites change frequently. They are constantly updating, stream lining, and just generally tinkering with their pages. The thinking behind this is simple. If a user finds sufficient results every time they use a certain site, they will return to that site again and again, generating more advertising revenue for the site owners. When visiting a search engine for the first time, or revisiting one not used in a while, take the few minutes to read the search tips. There is no point wasting time adding quotation marks to a phrase if the search engine does not recognize that punctuation.

To stay abreast of things, visit Search Engine Watch, which offers all the ins, outs, and changes that are going on with search engines. Subscription to the free monthly newsletter is available at the site. Recent articles have included "The 'New' Alta Vista," "Who's the Biggest of Them All?," and "Google Gets Out of Beta."

A more specific discussion of individual search engines will appear in our first issue in the new millennium.

URLs of Interest Internet Librarian '99 proceedings overview can be found at http://www.infotoday.com/il99/presenta tions/default.htm

Search Engine Watch
http://searchenginewatch.com

Beaucoup
http://www.beaucoup.com

Northern Light
http://www.northernlight.com

HotBot
http://www.hotbot.com

Yahoo!
http://www.yahoo.com

Snap
http://www.snap.com

GoTo.com
http://www.goto.com

Go Network
http://www.go.com