November 29, 2004 - Essentials for Finding Information on the Hidden Web

weekly fodder for the flock...

Join our e-mail list!

Just type your e-mail address below and press submit.

Print
Essentials for Finding Information on the Hidden Web
By Shirley McDonald
November 29, 2004

If you are tired of thousands of links from your favorite Internet search engine, yet none of the sites prove relevant to your information needs, then this Essentials column is for you!

The most popular search engines (Google, MSN, AOL) only tap the surface of the information found on the Internet. Much information is not “visible” to these search engines for various reasons, some technical and some deliberate exclusions by either the host server or the host site. There are, however, search engines and directories that tap into the Hidden Web (also called Deep Web or Invisible Web), providing links to:

Public information that is 400-550 times larger (550 billion individual documents vs. one billion on the Surface Web)
Quality content that is 1,000 to 2,000 times greater than the Surface Web*

Here are some of my favorite directories (which means the links are categorized by subject) of information on the Hidden Web:

Digital Librarian: A librarian’s choice of the best of the Web. Okay, we all know librarians specialize in finding information! Includes categories from Activism to Yurts, Tipis, and Tents.
Librarian’s Index to the Internet: Information you can trust! Okay, after this one, no more librarian-made sites (but we librarians really do know how to find information!).
IncyWincy: The Invisible Web Search Engine. This one offers a direct search, a directory search, or a meta-search (several search engines at once). They claim “50 million pages spidered and indexed and hundreds of thousands of search engines indexed and searchable.” I like the cutesy name, too.
Open Directory Project: The largest human-edited directory of the Web.

Then there are the direct search engines of information on the Hidden Web. Here are just a few:

Search Adobe PDF Online: Whoever thought of a search engine that searches just Adobe files? These guys did. However, you can search more than a million summaries of PDF files on the Web and see the summaries before deciding to view the file.
AlltheWeb: Supports searching in 36 different languages and promises “up to the minute news…with hundreds of stories indexed every minute.”
Vivisimo: Clusters your results into categories with its “Clusty” feature (what’s with all these cutesy names?). I especially like the links clustered by topic.

Here are a couple of specialized databases that might be of interest:

SingingFish: calls itself the “world’s leading audio and video search engine.”
LookSmart’s Find Articles: need an article? LookSmart allows you to search 5.5 million articles from over 900 publications. It’s quite overwhelming!

And if you just do not know where to start searching, NoodleTools (where do they come up with their names?) provides a wonderful chart that connects your information need to the most efficient search strategy. I could spend all day just checking out all these search engines and directories!

The popular search engines are still valuable; I used Google to find some of my information on the Hidden Web. And since this field is evolving so rapidly, there is some overlap in the information offered. I am sure that the big, popular guys are working diligently to close that gap.

However, for now, if you need quality information, try an Invisible Web search engine or directory to get that needed extra nugget.

ninetyandnine.com

---------

Shirley McDonald’s busy life makes her especially thankful to have found the Invisible Web tools. Now she doesn’t have to spend so much of her valuable research minutes on Web searches that go nowhere.

* Bergman, Michael K. “The Deep Web: Surfacing Hidden Value.” www.beta.brightplanet.com/deepcontent/tutorials/DeepWeb/index.asp (8 November 2004).