Before I tell you about the remarkably useful and completely free information utility I've been using, I should describe why I needed it in the first place. But that requires, in addition, a lament about the current sorry state of the World Wide Web.
As it does almost everywhere, spring in my neighborhood means the commencement of long-delayed home repairs. In setting about on these projects, I naturally planned to use the Web as a resource to not only bone up on topics like roof repair, but also to find experienced and honest local trades people to hire. How lucky we are, thought I, to be able to do spring cleaning in the age of broadband.
What I actually found online was a different story. No matter what I searched for, I ended up with a distressingly large percentage of what might be called second-generation Web spam. (While mostly associated with email, "spam" is also used to describe undesirable search results.)
This is not the search spam of the early Internet, where you would search for "Disney" and instead get a sex page. Rather, it's new and improved spam: pseudo-useful pages that are usually just shells for ads.
In many cases, a page might at first glance seem like a guide to your topic. But after a minute or two, it becomes evident that the information is virtually useless but is surrounded by an ocean of ads. In other cases, you find "referral services" -- dozens of them -- that promise to put you in touch with reputable contractors. But these sites inspire little confidence that the contractors deemed reputable aren't simply those who have written the Web site operator a check. (A dead giveaway to these sites, by the way, is their abundant use of stock photos of improbably attractive, well-dressed people conferring enthusiastically on the telephone.)
Noted search watcher Danny Sullivan says there is no specific word for these kinds of sites. (Folks with an historical bent might refer to them as Potemkin Web sites because they are all facades.) What's behind them isn't anyone with any particular interest or experience in what you're searching for, but instead someone who is trying to make money from the simple fact that you have arrived at the page. Meanwhile, though, the sites are cluttering up the Internet and making it vastly less useful, certainly for what I was trying to use it for.
All of this is occurring because of a number of recent Web developments. The most important is that first Google and then Yahoo and everyone else have introduced advertising programs that make it easy and lucrative to sell ads on a Web site. If you can generate a lot of traffic to a roofing-related Web page, then you can easily make money off those eyeballs.
Thus, a kind of schizophrenia exists at search-engine companies. Half their engineering staff is busy trying to keep useless pages out of search results; the other half is busy coming up with tools that make it easier for people to create and profit from the useless pages in the first place.
The second development is that Web sites no longer use human beings much to help rank their search results. It's now largely done by software (even though some insiders say that Google and the rest use human editors a lot more then they let on). The search companies have reduced their reliance on humans in part because they are expensive, and in part because the Web lately has been enthralled by the success of Google's Page Rank algorithm, which ranks Web sites depending on who else links to them.
The problem, of course, is that spammers know about these algorithms and are constantly trying to trick them. Search engines respond by fiddling with the algorithm; spammers make their own adjustments; the beat goes on.
Human beings, though, can't be fooled as easily. Among researchers in academia, the pendulum is swinging back to getting people involved again, the way they were in the early days of search.
One example is the "TrustRank" approach co-developed by Zoltan Gyongyi, a Stanford University Ph.D. student. With this method, first human beings pick reputable, high-quality Web pages, and then software automatically finds other sites based on the pages the people-blessed sites link to.
With all the money involved, spammers will surely try to crack that system, too -- though Web users everywhere should be rooting for Mr. Gyongyi and the others to prevail. Future generations will marvel at the utter lawlessness of these early days of the Internet, with spam and phishing and viruses and what-have-you. Imagine a world where you could surf harmlessly and where things would be what they seemed. How cool would that be?
My new information utility isn't quite that ideal, but at least it does many things better than the current Web. It's neatly indexed, making it a snap to find something. It's awesome for locating nearby merchants. In many cases, you can even use it to find email addresses and URLs.
You have one of these information utilities, too, though there's a good chance you're using it now to elevate your computer monitor by three inches. It's called the Yellow Pages.