Saying the Web is useful is stating the obvious

Now that the novelty of surfing has gone (okay, so I'm off by seven years), we look for ways to actually use the Web to make our lives more comfortable. The "old ways" of the Web have certainly helped in some areas. Most of the books I buy come from Amazon, not from my local bookstore. (My local bookstore isn't even the second place on that list -- the one at the airport is.)

Another example for the utility of the Web (assuming that you've not been convinced of that even before starting to read this) is that information became accessible through the use of search engines. When I need to look up a technical issue today, I start at Google. I do that even when  I'm looking (for example) for a Win32 API. Considering that I'm employed by Microsoft and have access to its fully-indexed intranet (which is absolutely huge), this is a testimony to how a good search engine can make your work easier.

These examples, and many others I could bring here, show the utility of the "first-generation" Web. In the past, many people devoted a lot of thought to how "second-generation" Web technologies would look. Two technologies stood out as possible candidates: agents, and push technologies.

Agents

The idea of an agent is that you have some piece of code that represents you. In particular, it knows what your interests are, and can roam about the Web looking for stuff that interests you, skipping between machines and networks.

Clearly, agents have failed. More accurately, they never got their chance at success. Almost nobody is willing to let you run your agent on their machine, for good reasons. (a) They're afraid you'll compromise their security; (b) running agents costs money in terms of CPU power. There's also a technology barrier (namely, how are these agents are run; for example, what runtime do they require) that nobody bothered to solve.

So agents never really saw the light of day, and the world seem to have left this idea behind.

Push technologies

Now this looked like it was about to succeed. Companies such as PointCast raised lots of money. Microsoft stood behind it with Active Desktop. Content provides really wanted to send you their content (with a healthy doze of ads). Only there was nobody on the consumer (er... client) side that was interested. Push technologies today mean that you can get a newsletter from Slate once a day summarizing the headlines of several important newspapers in USA.

The failure here seems to be because there's a huge impedance mismatch between what content providers (people who want to sell you stuff) wanted people to get (ads) and what people wanted to get (information). I'm subscribed to Salon's newspaper, because it's much faster to read their daily summary then to go over all the newspapers myself. If they mixed-in other stuff into it, I wouldn't be reading it.

The next generation

What's the point of all this? To tell you that we bear witness to the "real" second-generation Web. Now is the time it's hapenning. There is a shift today towards automated Web surfers that you can use to bring you the information that you need, in the format that you want.

Like an iron bar cooling in a magnetic field, where all the magnetic dipoles suddenly align towards the same direction creating a strong magnet, the Web is re-aligning itself towards utility. Here are the key factors that I think play a hand in this next generation.

RSS. RSS is a family of standards (they are called "versions", but we recognize a good kicking-under-the-table situation when we see one) that describe how sources (called "channels") of self-contained information capsules (called "items") can identify themselves to consumers. The really neat thing about RSS is that it is simple and effective, so that almost any type of human process that generates information can do so in an RSS format (pages on the Web, an entry in a journal, mail, an IM message).

Similarly, you can consume RSS in a variety of ways. Most RSS consumers (called "aggregators") today simply display items in a list on a browser. But one can easily vision other ways to display such information to humans. For example, you could have an RSS aggregator send each items to you as a mail message, or have all new items from the same channel aggregated into the same message. (This would be great for people working in an environment like Microsoft's, a company run by Email.)

One thing to note here is that RSS feeds are mostly generated at the source. Most bloggers today publish their blogs both at their Web sites and as RSS feeds. You can then subscribe to their RSS feed, and never have to go visit their Web site again. (I say "subscribe", but notice that they are mostly unaware of this subscription -- it is my local RSS aggregator -- my agent replacement if you will -- that does the impedance matching between the source that posts information regardless of its clients and clients who periodically pull information in order to push it to their human consumers.)

RssDistiller

In the future, we can expect that the use of RSS will be much more widespread than it is today. In the meantime, however, we already have a killer application that makes RSS feeds on the client -- rather than the server -- side. It's called RssDistiller.

What RssDistiller does is rather simple: it periodically samples Web pages that you've indicated your interest in, looking for changes. If it finds any it scrapes these pages and converts these changes into RSS representation. This is possible because so many Web pages today are finally being sufficiently organized for an automated tool to scrape.

As an example, I use RssDistiller to scrape the MSDN site. In the past, I've had to manually navigate to MSDN to go over the new items posted there. Although MSDN has a subscription option (where you get an Email on changes), I don't use it. I'm already subscribed to so many other Email newsletters, all with a different format, all containing lots of "fluff" information I'm not interested in. (For example, why do I need that "stuff our lawyers make us say" in each message?)

Scraping MSDN using RssDistiller and then reading it using Radio's builtin aggregator is much more efficient for me. I'm also seriously considering investing time in tweaking the available source code of Aggie (another RSS aggregator), so that I could display information the way I want it. (For starters, I would make all channels display items in order of publish date, rather than the other way around.)

Summary

The point of all this is that we live in exciting times. I know this has been said alot about the Web. Personally, I think that this is the third major shift in how we do things on the Web (the first two being the Web itself -- HTML, HTTP, and URLs -- and search engines).

Something to take away

There has been discussions about whether one should publish complete information capsules using RSS, or just a short passage, leaving readers to jump to the source to get the entire capsule. I think it is clear from what I've written here that I think people should place entire capsules in their feeds. I'm interested in what people have to say, not in how they desgined their site to say it.

Copyright © Copyright 2002 Ziv Caspi.