Gary Robinson: Gary Robinson's Spam Rants

Thu, 02 Oct 2003 14:25:35 GMT

Ray Ozzie thinks email is going away:

Think about it. Think about the rate of increase of "noise" in email over the past two years, which is a very short time. Think about where we'll be in as short as five years. Can you imagine?

Right now, every major enterprise has a "content-scanning gateway" that processes every incoming email, looking for Dangerous Stuff. Many individuals do the same thing on their own computers. Some enterprises are beginning to quarantine incoming email for extended periods - sometimes an hour or more. Maybe you'll get too much junk, or maybe you won't get what you're supposed to get. Maybe you'll get it, but it'll be too late. It depends upon where they turn the knob on the software ... and it's insane.

If you're hoping for some super-duper neo-PIM to or super-filter or super-law to come along to make your life easier, spare yourself the agony and just think ahead: it is NOT a sustainable solution if it is still called "eMail". eMail is thirty years old, and we owe it a great debt of honor, but it has been pushed well beyond its design center and it's time to move on. Incrementally, progressively, but most definitely.

And his solution?

If you have work to do with others, online, try workspaces. There are many different types - from Groove if you like client-based mobility, to SharePoint if you like using Websites. No noise, no spam, tuned to save your time. Of course, you can't give up on eMail, and likely never will. As time goes on, though, you'll only visit eMail as a low-priority background task, much as you do when sorting through your physical mail at home. You'd never do important work through your home mailbox, would you?

The problem with working with others in Ozzie's vision of a workspace is that we all have to buy into a particular product, because they aren't compatible. So if I want to use workspace to work with my lawyer with have to bothhave Groove or SharePoint or whatever. The end of that path is either that everyone has a number of different workspace applications that they've paid for, learned how to use, and are willing to keep updating over time, or one company dominates the space ala Microsoft Office. I think it's clear that the latter is the more likely solution, for pretty much the same reason that most people in business use Office rather than a motley collection of premium word-processing tools and spreadsheets. Unfortunately for Ozzie, who runs Groove, I think such a solution is more likely to come from Microsoft than from Ozzie.

But in reality email has more legs than Ozzie thinks. In the quotes above, he is quite passionate about how rapidly email is degenerating. But you know what? I run SpamSieve and I don't have a spam problem any more. Yes, I get about 200 spams a day. But I don't see any but 1 or 2 of them. It's not a problem. And SpamSieve, like a number of other filters, is accurate enough that loss of legitimate mail ("false positives") is not a significant problem.

Now, as long as most people don't know about or don't make the effort to install such filters, there will still be enough profit in the spam business to motivate the sending of billions of spams per day. But one by one, email software manufacturers are adding spam filtering to their software -- most recently Eudora and Microsoft Outlook. (I'm happy to mention that Eudora uses math that is built on the mathematical suggestions of Paul Graham and myself, as does SpamSieve.)

Before long, enough people will have spam filtering -- whether they are aware of it or not -- that spam will become significantly less profitable than it is now. (And not only because of filtering, but also because even the most gullible and insecure among us can only buy so many penis enlargement products before they decide there are better things to do with their money.) Spamming will be less enticing to prospective spammers.

Added to that are increased legal risks due to laws such as recently passed in California, and perhaps most importantly, the likelihood of ISP's and and Internet-using corporations working together to divide the Internet into "two camps, those who can be held accountable for spam, and those whose status is unknown."

When everything is added together, it is unlikely that spamming will continue to make email harder to use. It is much more likely that the noise level of email will be diminishing steadily over the next few years.

Another factor will be the fact that sophisticated users -- those who are most likely to be on a lot of legitimate news mail lists from corporations they do business with -- will increasingly be savvy enough to choose RSS rather than email for most of that sort of news, further increasing email's signal/noise ratio.

So in the end, I think that Ozzie is indulging in a bit of wishful thinking to believe that email will soon be so useless that it will have to be replaced by the likes of Groove. Of course, things like Groove do have benefits; my company in fact has a Groove account, which we received as a benefit of our SBIR/NSF grant. However, we still use an open-source wiki for group document management. The fact that the wiki is free doesn't even play a factor in our choice; our Groove account is free (to us) too. The wiki just works; it's easy and convenient and there is really not much reason for us to use anything else.

While I think that there will advances over email and wikis in the coming years, my guess is that most people will be using products that are open-source and/or conform to open standards. Even though we have had little reason to use Groove, there is also little reason reason not to use a product like Groove if it were opened up so that it didn't lock up the user's data.

I forecast that Groove, and products like it, will be marginalized if they don't create open standards for freely sharing data with other products, but if they do so, compatible open-source products will emerge that make it hard to charge a lot for them. Microsoft would seem to be the company with (by far) the best shot of creating a proprietary, closed solution, but the world is a lot wiser now than when it fell for Office, and will be wary about choosing to get tied into another Microsoft monopoly. And open-source mechanisms for software development have gained power and acceptance in recent years. That's great for consumers of information technology, and less good for people that want to make a lot of money in the software industry.

Thu, 28 Aug 2003 20:07:31 GMT

"Dennis K. Berman offers a ray of hope to people frustrated and demoralized by the spread of spam: Spamming could eventually burn itself out by becoming a victim of its own proliferation, he muses." [From the ACM.org's summary of a Wall Street Journal article. The summary is worth checking out.]

Sat, 23 Aug 2003 13:06:35 GMT

Well I'm back in Maine. I actually got a fair amount of work done on my laptop on the drives to and from New York City.

Partly on the drive south, I developed and tested some changes to the chi-square approach to spam detectionand testing so far the changes seem to result in roughly a 5% improvement in accuracy. On the drive back I developed another possible improvement that I hope to find time to test over the weekend.

Mon, 11 Aug 2003 22:02:19 GMT

SpamBayes, which is based on math suggested by me as an improvement to the approach in Paul Graham's seminal "A Plan For Spam" article, and supplemented with a great observation from Tim Peters that the "Unsure" range of spam/ham classification is very useful, won this comparative testing against PopFile.

The test was slashdotted today.

John Graham-Cumming, who is responsible for PopFile, Tim Peters and I are all members of ActiveState's Anti-Spam Task Force.

I must admit I'm competitive enough to enjoy that sort of thing. ;)

(Note, John is doing a fantastic job at documenting specific tricks used by spammers.)

Tue, 29 Jul 2003 01:36:23 GMT

"'We’re going to see a lot more spam, instead of less,' said David Sorkin, who teaches a spam law course at the John Marshall Law School in Chicago. 'What most of the state statutes do is legitimize spam as long as you don’t also commit fraud.' [Stateline.org]

Tue, 29 Jul 2003 00:08:00 GMT

"Measuring the cost of spam, which takes into account such factors as lost productivity and wasted time, is an imprecise science, and estimates on its total toll vary: Ferris Research reckons that spam cost the United States $10 billion in 2003, while Nucleus Research pegs the U.S. cost around $87 billion." [New York Times (Free registration req'.)]

Tue, 01 Jul 2003 14:53:28 GMT

Lawmakers call for global action on spam. [News.com]

Fri, 27 Jun 2003 19:12:16 GMT

"Junk e-mailers are spreading viruses that let them send spam anonymously through home computers, according to an e-mail security firm.

The company, MessageLabs, operates servers that block spam and viruses for its clients. Its analysis of data shows that mass distributions of junk e-mail are increasingly coming from the Internet addresses of computers that have in the past sent out viruses as e-mail attachments. 'There is a high correlation,' said Matt Sergeant, senior antispam technologist for the New York-based company. 'About 30,000 machines have both open-proxy software and are responsible for sending viruses.'" [News.com]

Thu, 26 Jun 2003 02:16:42 GMT

"A new variant of the Sobig virus started spreading on Wednesday, raising the specter that spammers will have a host of new PCs to use as platforms for sending bulk e-mail.

Initial analysis by antivirus companies indicated that the mass-mailing computer worm, called Sobig.E, doesn't have a malicious payload. However, e-mail service provider MessageLabs believes spammers will use the virus's mail program on victims' computers to send anonymous messages.

'This is almost certainly being precipitated by a spammer that is trying to create more open relays to send spam,' said Mark Sunner, chief technology officer for the U.K.-based company." [news.com]

Tue, 24 Jun 2003 23:41:03 GMT

Bill Gates on how to end spam.

Wed, 14 May 2003 17:04:33 GMT

'Buffalo Spammer' arrested in New York. "On Wednesday, Attorney General Eliot Spitzer said that his office brought four felony and two misdemeanor charges against Buffalo, N.Y., resident Howard Carmack for alleged identity theft and forgery that allowed him to send more than 825 million e-mail messages through the Internet service provider EarthLink. " [News.com]

Wed, 14 May 2003 16:38:02 GMT

People are getting concerned about the potential for challenge-response anti-spam systems to cause problems for mail lists and other legitimate emailers who can't supply manpower to respond to the challenges. Today I noticed two such articles, on TidBits and taint.org.

One of the biggest worries: "Challenge-response technology is about to become significantly more widespread... with EarthLink about to test such a system for its 5 million customers."

Wed, 14 May 2003 16:13:35 GMT

" After spending years cracking down on spammers, two prominent organizations that list senders of junk e-mail are fending off an unorthodox legal challenge by e-mail marketers.

In a case that's raising the hackles of antispam activists, a group of anonymous e-mail marketers are charging that two blacklisting sites, Spamhaus and SPEWS.org (Spam Prevention Early Warning System), have published false, misleading and libelous information about their business practices." [Wired]

Tue, 29 Apr 2003 16:08:54 GMT

Brad Templeton's Reflections on the 25th Anniversary of Spam [hat tip to MacInTouch]

Brad's piece is an enjoyable and informative read, with such tidbits as Richard Stallman's response to the first spam 25 years ago -- he defended it! It also talks about the history and future of various approaches to combatting spam.

Wed, 23 Apr 2003 19:21:51 GMT

Spammers sue anti-spam groups. [SlashDot]

"The Register reports in a story today that spammers have banded together under the name EmarketersAmerica.org to sue various anti-spam groups- days before a large conference on spam hosted by the FTC(which will be attended by many spammers). Anti-spam groups think the timing is not by coincidence, but believe the move may backfire because they will be able to countersue and get access to spammer's internal documents."

Wed, 23 Apr 2003 17:36:42 GMT

An article about spam honeypots by the founder of the Honeynet Project. [ZDNet]

Tue, 22 Apr 2003 14:45:41 GMT

The NY Times has a front-page article about spam today.

Wed, 16 Apr 2003 13:02:16 GMT

The latest "anti-spam" product being marketed by means of spam is Silicon Paradise's Spam Remedy.

Sat, 12 Apr 2003 02:59:58 GMT

A "new bill would require online marketers to include valid return addresses in e-mail messages so users could block unwanted messages." [MSNBC]

Tue, 08 Apr 2003 01:19:31 GMT

Porn spam--legal minefield for employers [News.com]

Fri, 04 Apr 2003 19:03:01 GMT

"Antispam company Habeas is suing bulk e-mailers, accusing them of using its poetry without permission in an unusual use of trademark law to clamp down on spammers." [News.com]

Thu, 03 Apr 2003 02:40:35 GMT

Microsoft patent on "probabilistic classifiers".

Wed, 02 Apr 2003 20:13:18 GMT

A Microsoft Paper on Bayesian Spam Filtering.

Mon, 24 Mar 2003 15:46:20 GMT

Mailblocks, a new challenge/response anti-spam system. "CNET News.com's Charles Cooper says Phillip Goldman is either a rich guy with a death wish or a man on the verge of making tech history." [News.com]

Their web site mentions a patent. I was unable to find it in a quick search on the PTO's web site. If you find out anything about it, please make the info available here on the wecanstopspam.org wiki.

Mon, 24 Mar 2003 15:41:50 GMT

" Microsoft's MSN Hotmail, a free Web-based e-mail service, has tightened restrictions on daily outbound messages sent by subscribers, a tactic it says will help curb spam." [News.com]