Av rating:
Total votes: 25
Total comments: 12


Jesse Liberty
Breaking Spam
20 December 2006

Spam has finally gone from an inconvenience, to a significant annoyance, to an unmanageable burden.

Until recently, I was able to manage spam using software and it took only a few minutes out of each day, but with the stepping up of the amount of spam I’m receiving in my email (and apparently everyone else is too) it has become an international pestilence. What is worse, this is a problem fairly easily solved, and yet we are mired in argument about which 'scheme' to use. It is time to get off the dime and implement one.

Personally, I’d vote for stamps, a scheme that has been in the pipeline for years. I'd collapse two proposals into one, thereby overcoming the objections to both. Here is how it would work:

Stamps to Solve Spam

In order for my ISP to accept and pass on an email to me, it would need to have a 'stamp' on it. The stamp would have to match a hash-code algorithm (or an identification GUID) that the ISP would purchase from an international consortium, and that hash-code algorithm would be (a) secret, (b) free to the ISP, and (c) something that the ISP could process very fast. The cost to the ISP would be virtually zero – certainly less than the spam filtering being carried out today.

To generate such a stamp, the sender has two choices. They can 'buy' a stamp from their ISP, for $0.001USD, or they can pay for it by solving a hash-code problem expected to take about 1 second on a typical home computer. The ISP keeps the payment in exchange for the overhead of generating the stamps, and is permitted to sell them in bulk lots of ten (that is, a penny's worth at a time).

By the same regulatory conventions that cover domain names and other aspects of the Internet, recipients will not be allowed to 'demand' higher cost stamps be affixed to incoming mail (all stamps will be of a single type as far as ISPs are concerned) and ISPs are not allowed to increase the cost of stamps without international agreement.

To the consumer, the cost of sending email remains virtually unchanged. I can send 100 emails a day, every day for a month for less than the cost of one first class stamp. And who sends 100 or more emails a day anyway?

Well, spammers do. Indeed, to a spammer, the costs escalate pretty quickly. A million outbound emails just went from free, to $10,000. If that spammer wants to avoid the deci-penny cost, they’ll have to pay the computing cost, and one million seconds is two years of computing power. Pretty soon, it's time to buy more computers. In any case, spam stops being free.

We can, of course, get fancier, and have a sliding scale, in which stamps from any given ISP rise in cost during the course of a month. The first 100 stamps in a month are a tenth of a penny, the next 100 stamps are a penny each, each block of 100 stamps after that double in price (.02, .04, etc.). It quickly becomes prohibitive to buy from the same ISP, and spammers must spread their business across ISPs, creating more difficulties for them, more bookkeeping and greater overheads.

One objection, often raised, is that even twenty or thirty cents a month is prohibitive in some countries. There are many answers to this problem. First, it would be perfectly legal for an ISP to choose to deliver unstamped mail to anyone who wishes to receive it; this not only allows everyone in a given country to opt out, but international companies can also opt to receive unstamped mail from specific senders. Second, there is the computed hash-value alternative, which would be virtually unnoticeable for routine email.

The key question is whether spammers would be able to break the lock on the stamps themselves. If the stamps use the same level of technology currently used in Public Key Encryption, there is no reason to suspect that we can stop them cold. Yet if US, British and E.U. law enforces the stamp act, we should see a dramatic drop in inbox-stuffing junk email within days.

Personal Hacks

Until recently, I was happily using software called SpamCatcher, but the effort of sorting through the suspected spam became overwhelming with the volume of junk I was receiving. And SpamCatcher was actually an effective program! I may go back to SpamCatcher, but for now, I’ve been forced to stop taking email from all unknown addresses just to cut down on the time I’m spending dealing with email every day.

To do this, I’ve written a 'rule' in Outlook, that I run by hand twice or more every day that sends a polite note informing everyone who sends me email (whose address is not in my list of contacts) that their email was not read, but that they can reach me through my website. I then dispose of their email. This rule is shown in figure 1:

Figure 1. The Outlook Rule

(To create the reply template, use the Outlook editor, and save your email as a template rather than as an email).

To make this work, I had to create a program to accept email through my web site, as shown in figure 2:

Figure 2. Email Form

Having to cut out all incoming unknown mail turns out not to be all bad news. By being forced to change, I learned a few things along the way.

First, I was surprised to see how few folks objected to my polite note telling them that their email was not received and that to reach me they needed to log on to my site and send me email from there. This kind of reaction convinced me that I can't have been the only one suffering from spam overload.

Second, I had a real-world opportunity to play with AJAX when creating the form, showing me (as I will detail in an article for Simple-Talk very soon) how incredibly easy it is to create water-marks (text that disappears when you type in the text box) as shown in figure 3. I also learned how easy it is to make animations, as shown in figure 4.

Figure 3. Watermarks

Figure 4. Animation (frozen)

The form asks the user to put in his or her return email, a subject and the text of the email. When the user clicks send, the email is sent to me from a known, safe, email address using SMTP.

I use Outlook to route these emails to a particular folder and reply as quickly as possible, often adding the author to my safe senders list.

This has worked quite well, at least for now. I’ve cut down on handling junk email from an hour a day to a few minutes, a few times a day, and I’m pretty comfortable that if I’m trashing important email at least the sender is being notified and offered another way to reach me. That said, this is a very ad-hoc solution, and one that is very dependent on user-intervention.

The right answer, of course, is not to have each recipient write code (and not every recipient has a website to redirect potential senders to in the first place) but to solve it at the ISP level. If the stamp idea has rough edges to smooth out, I’m convinced there are smart people who can solve the problem, if they are sufficiently motivated. The problem, of course, is that spammers make money on what they do, and have resources to defend their industry. Recipients, however, have no organized voice.

As spam increases, however, the cost in wasted time and effort is growing, so I’m reasonably optimistic that the balance may be shifting, and perhaps not long after you read this, those smart folks who are competing for our high speed internet connection dollars will discover that low spam is a competitive edge.



This article has been viewed 2700 times.
Jesse Liberty

Author profile: Jesse Liberty

The president of Liberty Associates, Inc and a Microsoft MVP, Jesse Liberty is the author of the international best-selling Programming C#, Programming VB 2005, Programming ASP.NET and numerous other books, including the forthcoming Programming .NET 3. He has written dozens of articles for leading industry publications and has been a featured or keynote speaker at international industry events. Jesse’s biography is listed on Wikipedia and he maintains a political blog a technical blog , and a free private forum on which he provides support for all his writing.

Search for other articles by Jesse Liberty

Rate this article:   Avg rating: from a total of 25 votes.


Poor

OK

Good

Great

Must read
 
Have Your Say
Do you have an opinion on this article? Then add your comment below:
You must be logged in to post to this forum

Click here to log in.


Subject: Auto-replying to all email is negligent
Posted by: Anonymous (not signed in)
Posted on: Thursday, December 21, 2006 at 2:55 PM
Message: Jesse,

1) You start by stating that your spam filtering is not up to the job.

2) I will state that practically all spam arrives with a faked/spoofed email address.

Therefore, you are contributing to the tide of unwanted mail as valid email addresses that were spoofed will get an email from your autoresponder.

The invalid email addresses will cause a tie up on your outbound mail gateway as they fail to deliver. And then where do those undeliverables go? Back to your valid email address? If the domain is valid, but the mailbox part of the address is invalid, where does that undeliverable go? Again, back to your valid email address? If the domain is valid and the mailbox part is invalid and the recipient domain has a wildcard mailbox, they'll receive your junk too.

What happens when a bad guy sends mail to your autoresponder with a mailfrom or reply-to address that is in your domain, or is actually the mailbox that your autoresponder is listening on?

What happens when a bad guy notices that you blindly autorespond to every message? He will send you a very small junk message, and set the mailfrom or reply-to to each address he wants to annoy, and your autoresponder will happily send your message to each one, every time. Since the point of your autoresponder is to free you from monitoring the messages, you also won't monitor it. Will you get a notice from your ISP, or will you hope that one of the abused email addresses has someone that will take your autoresponse message at its word and try to contact you in good faith to point out that you are spamming them?

Subject: Got to disagree with your article big time!
Posted by: John A Thomson (not signed in)
Posted on: Thursday, December 21, 2006 at 6:02 PM
Message: It is no coincidence that the recent upsurge in spam happened about the same time as the one million Botnet came online! The majority of SPAM comes through this and other botnets. Getting rid of these Botnets will be the most effective measure in ridding the world of SPAM, well for the few minutes that it takes the spammers to find another way to do it anonymously, at zero cost with a level of untraceability that ensures they stay out of prison for a very long time!

Spammers will always find a way around the best antispam solutions and they will easily create an automated bot to spam you through your web interface if they do desire. Look how bad blog comment spam has become in recent times. Sure some of this is down to the blogging API, but the spammers also have automated bots that can populate web front ends.

Stamps aren't the solutions, just as domain keys and SPF have failed to stop the tide of scum-mails. All you're doing is raising the cost of operating on the Internet for legitimate users.

Both the US and the EU have laws and regulations in place that makes it illegal to SPAM and does that stop it... NO! We don't need more regulation of the Internet - even Amnesty International thinks that to be true: http://irrepressible.info/ . We need better enforcement of existing regulations and laws to make spamming less attractive.

The arguments in your article would appear to be ill-researched and some of your assertions are fundamentally flawed.

A layered approach to antispam protection, like every other security solution, offers the best results. I'm fairly lucky that I only get a couple of email getting through the protection on any day. Thanks SpamAssassin, BoxTrapper and SpamBayes.


Regards

John

Subject: Spam
Posted by: Ian Logan (view profile)
Posted on: Friday, December 22, 2006 at 3:23 AM
Message: I totally agree that the ISPs need to get their act together. Spam is such a pain. However I am not sure that the Stamp idea will work fully simply because the spammers also user their bot networks and so would spread the spam over 1,000s of computers - there are enough of them wide open out there!
If the ISPs actively worked together on White/Black lists and filtered the email then that would go a long way to reducing the bulk of it.

Subject: Attitude
Posted by: John Thorpe (not signed in)
Posted on: Friday, December 22, 2006 at 3:51 AM
Message: What really annoys me is the attitude of people that being blocked by your anti-spam system is your fault and not theirs. I have recently implemented a two stage anti-spam system that first checks the most popular black lists and then sends to a second level that checks the mail with a message center to see if a lot of these mails have been seen recently - in which case it is spam tagged and the SCL is set to 5 for your mailserver to deal with. It also checks SPF records, and has been so effective that my spam count has dropped from 1500 a month to about 30! So far very few false positives have occured and the reason for being blocked is genuine. Those that do get through are soon noticed by the second level and repeats don't make it even a few hours later.
Then you get a snotty message from someone that says YOUR system has blocked their legitimate mail and when you check you see their IP is on a blacklist for hitting a spam honey trap. You are then faced with either allowing their IP through on a white list (and if it is an ISP SMTP relay then you allow all the spammers on that network through), or basically being lectured on how your mailserver is causing them problems because it's bouncing their email. Too few people know how to set up mail servers properly (try blocking mail where the RDNS on the incoming mailserver IP doesn't match an A record and see how many big companies have poorly configured DNS on their mailservers!) and the attitude that spam is the receivers problem will never help to prevent it.
In the meantime I am of the opinion that my vastly reduced mailbox far outweighs the occasional problem with genuine mail being blocked because somebody is using an ISP that doesn't control it's outbound mail effectively. The upside of this method as well is that once configured maintenance is zero as the mailserver does all the work.
The only real breakthrough will come when every sender of email takes responsibility for doing it properly, and then the spam mail will be easy to shut out and will never cause a problem. Once it is understood that it is not getting through then there will be no point sending it, and then the capacity of the Internet will take a great leap forward. I won't hold by breath though....

Subject: The Tide of Spam
Posted by: Anonymous (not signed in)
Posted on: Friday, December 22, 2006 at 7:14 AM
Message: I read somewhere that 8 persons are responsible for 90% of all the spam sent today.

I'm being completely serious about this; have five of them killed.

I suspect the others and those who would emulate them would get the message.

Subject: Ending spam
Posted by: Hercules Gunter (view profile)
Posted on: Friday, December 22, 2006 at 9:41 PM
Message: One the suggestions I've read for dealing with spam involved having a charge for sending mail which is invoked only if the recipient chooses to do so, so that legitimate mail remains free but unwanted mail can be penalised. This sounded like a good idea to me, until I started receiving many, many delivery-failure notices for emails I hadn't sent - some spammer used my email address as the value in the sender field, even though I had nothing to do with it. I suspect that that address is now on assorted blacklists and I even had hate mail as a result. Conclusion: even this charge-the-culprit scheme can be abused and an innocent party bear the cost.

Another idea I've come across is for domain name servers to keep a list of IP addresses of mail servers valid for each domain. Mail-handling servers would then be able to match the sender's IP address to the domain name for acceptance, and otherwise reject the mail (for my money, without returning a failure message, thus not adding to the load on the network).

Implementation of this would have to be phased over a period, but is there any reason it would not work?

Subject: Ending spam
Posted by: Hercules Gunter (view profile)
Posted on: Friday, December 22, 2006 at 9:41 PM
Message: One the suggestions I've read for dealing with spam involved having a charge for sending mail which is invoked only if the recipient chooses to do so, so that legitimate mail remains free but unwanted mail can be penalised. This sounded like a good idea to me, until I started receiving many, many delivery-failure notices for emails I hadn't sent - some spammer used my email address as the value in the sender field, even though I had nothing to do with it. I suspect that that address is now on assorted blacklists and I even had hate mail as a result. Conclusion: even this charge-the-culprit scheme can be abused and an innocent party bear the cost.

Another idea I've come across is for domain name servers to keep a list of IP addresses of mail servers valid for each domain. Mail-handling servers would then be able to match the sender's IP address to the domain name for acceptance, and otherwise reject the mail (for my money, without returning a failure message, thus not adding to the load on the network).

Implementation of this would have to be phased over a period, but is there any reason it would not work?

Subject: Further to Hercule Gunter's post
Posted by: xxxxx (not signed in)
Posted on: Saturday, December 23, 2006 at 12:17 PM
Message: Hercule Gunter's suggestion of a list of valid email servers for each domain is already possible. The DNS MX record includes a list of servers for a domain authorised to receive mail for that domain.
All you have to do is turn that on its head - the first public IP address in the headers should be one of those listed in a DNS MX record lookup for the sender's domain. If not, divert to an Administrator mailbox for review (or discard).
Yes, it would involve a lot of MX record lookups, but the savings in storage and processing later on could justify the means in setting this up.

Subject: Quick response to a misunderstanding
Posted by: JesseLiberty (view profile)
Posted on: Sunday, December 24, 2006 at 2:22 PM
Message: First comment wrote:

>>What happens when a bad guy notices that you blindly autorespond to every message? <<

I never blindly respond to any message. I clean my junk mail of delivery errors, try not to blindly respond and to cut off endless bounces. But yes, to some degree there is always a risk in responding to any unknown sender. On the other hand, as a small business, I can neither afford to read through 1500 spam messages a day nor ignore potential customers.

This is a dilemma many small businesses face.


Subject: A good spam filter
Posted by: James Hebben (not signed in)
Posted on: Wednesday, December 27, 2006 at 4:51 AM
Message: I have used the Cloudmark (http://www.cloudmark.com/desktop/) anti-spam filtering software for over a year now. It is a community based model, has a cheap subscription, and leaves me virtually spam free. I get around 100 spams mails per day and Cloudmark usually successfully filters virtually all of them autoimatically (maybe one or two spams get through per day).

The only downside I see to this tool is that it runs on the desktop, so my mail client still has to download all the spam mail before processing it. However, they do also provide a server based solution, but I am yet to find an ISP that hosts this tool :-(((


Subject: Interesting article.
Posted by: Anonymous (not signed in)
Posted on: Tuesday, January 02, 2007 at 3:03 AM
Message: I quite like the stamping idea, although I'm sure there are bound to be some unforeseen technical difficulties. Also, judging by the amount of REAL junk mail I get through my letterbox I'm not entirely convinced this would put everyone off, although I'm sure it would cut the quantity significantly and make the problem more manageable.

Like you I'm totally fed up of the amount of spam I get, particularly to my personal email address, to the point where I'm seriously considering abandoning it entirely. I think the point a lot of people miss is that yes, a spam filter will get most of the stuff out of my inbox and straight into a separate folder. However, then I have to go through that separate folder searching for false positives and frankly I just don't have the time or the motivation to wade through the hundreds of messages that have been filtered out.

However, I'm not sure the auto-responder is the best idea for a couple of reasons:

(i) If the reply-to address is a fake, which is quite likely, you'll probably get a delivery failure notification back... so I'm assuming you have another rule that means you don't respond to the failure notification as well? If you are responding to this then you're probably annoying the hell out of your ISP's sysadmins.

(ii) If the reply-to address is real, which is unlikely, it's a bit like saying "hey there, my email address is valid" to the spammer, or rather to the software they use, so they'll just send you more. Of course, you'll never see this additional spam, but it'll mean your mail takes longer to download and means you'll be contributing to traffic/bandwidth problems.

But anyway, a perennial problem, and I *wish* there was a solution to it, other than abandoning email.


Cheers,
Bart

Subject: Botnets....
Posted by: Anonymous (not signed in)
Posted on: Tuesday, January 02, 2007 at 3:21 AM
Message: I've just read through some of the comments on this and I think actually Ian Logan is pretty much spot on. The problem is that of course spam isn't sent from the spammer's machine primarily, it's sent from hundreds or even thousands of pwn3d systems around the world in peoples' homes and offices. Now, if the mail originates from these machines then it will be these people who are paying for it rather than the spammers.

This causes two problems if your machine is part of a botnet. If you pre-pay for stamps you're never going to be able to send any email of your own since the spam-bots will use your quota as soon as you've bought it. On the other hand if you get a bill at the end of the month... well, you're going to get a pretty big bill.

Arguably it's the responsibility of users to ensure the security of their machines, however many users aren't IT types, don't have technical skills, and aren't interested in getting them because they have better things to do. This is the reality, and no matter how much geeks (hello Slashdot: I mean YOU, or at least a fair number of people who post) complain about Joe Sixpack consumer being an idiot and deserving whatever he gets this isn't going to change, and I'd have to say why should it? When I buy any other device I want it to just work. I don't want to spend hours configuring and messing around with it. Why *should* a PC be any different? Why *should* I need to constantly tinker with it? I'm a techie and this whole process bores and irritates even me when it applies to my home PC. To a lot of people it's just another piece of consumer electronics like a DVD player or an XBOX, and that's the way it's marketed to them.

So somehow we have to square that with the spam problem and come up with a solution that isn't going to penalise people just because they're technically ignorant, and I think that perhaps the stamp solution isn't it unfortunately.

 

















Level Playing Field
 The Federal Government in the States accepts tenders for their IT projects from a wide-range of... Read more...

Second Life: A Virtual World of Real Money
 As more and more people invest in alter egos to live a pseudo life online in Linden Labs' latest... Read more...

Andrew Tanenbaum: Geek of the Week
 Andrew Tanenbaum has had an immense influence on the way that operating systems are designed. He... Read more...

Risking your Reputation
 IT companies sometimes don't survive an incident that damages their reputation. Often, when... Read more...

Ross Anderson: Geek of the Week
 Professor Ross Anderson is one of the foremost experts in Computer Security in the world. He has... Read more...

Linus Torvalds, Geek of the Week
 Linus Torvalds is remarkable, not only for being the technical genius who wrote Linux, but for then... Read more...

Driving up software quality - the role of the tester
 Have you ever wondered what a software tester does? Helen Joyce, test engineer at Red Gate software... Read more...

Coming Out as a Cancer Survivor - A Guide for Software Developers
 A personal perspective on the responsibilities of a cancer-surviving software developer Read more...

The Computer that Swore
 Database Developers occasionally get crazy ideas into their heads. Phil Factor should know; He... Read more...

The Writing on the Wall
 Phil Factor offers an intriguing theory on why so many, hugely complex, government IT projects fail. Is... Read more...

Over 150,000 Microsoft professionals subscribe to the Simple-Talk technical journal. Join today, it's fast, simple, free and secure.

Join Simple Talk