Message Hygiene in Exchange Server 2007

Around four out of every five email messages are spam. Now that the nuisance threatens to engulf what has become the normal route of business communications worldwide, all the vendors of Mail Servers are taking more active steps to confront the problem. William Lefkovics explains how Microsoft Exchange 2007 is meeting the challenge of the rising tide of spam

Message Hygiene in Exchange Server 2007: A Defense-in-Depth Approach

Unsolicited Commercial E-mail (UCE), or spam, is still a huge problem. Different research firms place the total percentage of spam to be anywhere from 70 to 90% of the total volume of e-mail around the world. Different enterprises may see variances in either direction from this range. It is still too much.

Spam Botnets
 Most spam today is distributed by networks of unsuspecting workstations infected with malware. These bots are small applications capable of, among other things, forming and sending SMTP messages en mass by remote command. Spammers typically pay for use of botnets (short for “robot networks”) rather than maintain their own. In the United States, the Federal Bureau of Investigation (FBI) has been working through “Operation Bot Roast” which identified 1 million infected hosts in the US alone and has resulted in the arrest and conviction (or guilty pleas) of several botnet operators.   (http://www.us-cert.gov/ press_room/botroast_200711.html)

Microsoft, in response to this ongoing battle, improved upon their set of native tools in Exchange Server 2003 Service Pack 2. They provided a more complete system for administrators to eliminate known spam early in the SMTP conversation and give them more control in assessing spam levels on more questionable messages. Exchange Server 2007 expands and improves on this defense-in-depth approach giving administrators effective sequential tools without additional software cost.

There is no silver bullet that will prevent spam from reaching inboxes, but there are several steps Exchange administrators can take to reduce inbound spam significantly. With Exchange Server 2007, Microsoft introduced server roles. There are five roles in total – Mailbox, Client Access, Hub Transport, Unified Messaging, and Edge Transport. The Edge Transport role is intended to reside in a perimeter network and perform the majority of the inbound message hygiene functionality for the Exchange organization. The Edge Transport server is typically the gateway SMTP server for the enterprise. Assuming so, message hygiene begins here.

524-Figure1_MMG_RG_MessageFlow_AntiSpamA

Figure 1 shows the inbound flow of messages passing through the various anti-spam agent layers. Each layer implements a different test in the effort to validate messages. If 80% of messages arriving at your gateway are spam, then it certainly makes sense to drop those e-mails as early as possible in the SMTP conversation. The anti-spam tools available on an Exchange 2007 Edge server are shown in Figure 2.

524-Figure2_Edge_EMC.JPG

Figure 2 shows the anti-spam tools available on an Exchange 2007 Edge server. Double clicking or selecting the desired anti-spam feature in the middle pane, and clicking Properties in the Action pane, will open the configuration settings for that feature. The four IP lists, allow and block, make up the Connection Filtering Agent.

Connection Filtering Agent

 

The first line of defense, and probably the most prolific one in terms of the potential volume of effective message filtering, is connection filtering.

Feel the Power!
The Exchange Management Shell (EMS) is an extension of Windows PowerShell for managing Exchange 2007. The EMC is built upon the EMS. For all the actions taken in the EMC, there is an equivalent EMS command. There is a complete set of cmdlets to manage connection filtering from the EMS. For example, Get-IPBlockListConfig on the Edge Server will return the IP Block list configuration.

For all the Exchange Server cmdlets run Get-ExCommand in the EMS. Microsoft maintains a library of these cmdlets as well. The anti-spam cmdlets can be found at http://technet.microsoft.com/en-us/library/ bb124601(EXCHG.80).aspx.

The Connection Filtering Agent is concerned with the source IP address of the host that is connecting on TCP Port 25. The agent references the IP address of the host making the connection request against an IP Allow List and an IP Block List. In addition it can query an IP Block List Provider or an IP Allow List Provider. If the IP address is listed on a block list, either internal or at a block list provider, and not on an allow list, then the connection is dropped. The IP Block list is maintained by the administrator. IP Block List Providers, also called Real-time Block Lists (RBLs) or DNS Block Lists (DNSBLs), maintain databases of known spam hosts.

A good strategy is to select one or two of the well known DNSBLs to complement your manual efforts to maintain corporate IP Block and IP Allow lists. Over the last five years, Spamhaus has been a solid DNSBL for me. I have also used Spamcop over that same time period. Spamcop has shown some inconsistency but has performed well over the last year or two. If you choose just one, I recommend zen.spamhause.org. In my company, and at clients, this step alone blocks over 55% of inbound spam messages. These are messages that my Exchange system does not have to process any farther. An independent resource for DNSBL performance can be found at http://stats.dnsbl.com/.

Connection filtering is exceptional! That is, it allows for an Exception List. You can configure Exchange to allow messages addressed to specific SMTP addresses to pass through even if the source host IP address is on a block list. It might be a good idea to add postmaster@yourdomain.com to the Exception List.

Configuring Exchange 2007 to query a DNSBL is simple. From the Exchange Management Console (EMC), open the configuration settings for IP Block List Providers and select the Providers tab. The Add button opens an input box to enter the name of the DNSBL.

524-Figure3_MMG_RG_IPBlockListProviders.

Figure 3 shows the IP Block List Providers Properties window with two DNSBLs entered.

Sender Filtering Agent

 

If the message survives Connection Filtering, the administrator can have Exchange reference the sender SMTP address against a list of addresses to block. The sender address is available after the MAIL command in the SMTP conversation. Specific SMTP addresses can be blocked, or entire domains. Adding *.domain.com to the Sender Filtering list will also block all subdomains for domain.com. While the default action is to block messages, Sender Filtering can also stamp the result in the message header and allow it to pass through. Content Filtering will consider the stamp status in its rating of the message (more about that later).

The EMC has a simple interface to enter blocked senders in the Sender Filtering properties. Addresses can also be added at the EMS command line with a simple cmdlet such as:

 

Recipient Filtering Agent

The next command in the SMTP transaction is the RCPT command, where the intended recipient of the message is exposed. Here we can apply a filter based on the recipient information. It is possible to block incoming messages for specific addresses, but more importantly, it is also possible to have Exchange query ADAM (or Active Directory if the Anti-spam agents are run on the Hub Transport server) to validate the recipient address.

No Edge Server?  No problem!  –
In the absence of an Edge Transport server, as may be the case in smaller companies, the Hub Transport role can host the Anti-Spam agents. First, on the Hub Transport server, you must run the PowerShell script Install-AntispamAgents.ps1, found in the \scripts folder on the Exchange 2007 DVD or extracted download.
After the script has completed, the Exchange Transport Service must be restarted. There will now be an Anti-Spam tab for the Hub Transport Server node under the Organisation Configuration container in the Exchange Management Console. Exchange 2007 sp1 moves a couple of settings to the server container.

Selecting the checkbox by “Block messages sent to recipients not listed in the Global Address List” can reduce the workload of other anti-spam agents and the server in general. If Exchange Server accepts the message without validating the recipient address, and the address does not exist in the organization, then it will have to submit a Non-Delivery Report (NDR) back to the sending server.

Having the server check the directory for addresses does make it easier for spammers to perform address harvesting. The connecting server can test for a series of generic addresses in the same session. Exchange 2007 allows for a short delay when servers provide the RCPT address before replying with 550 5.1.1 to indicate user unknown. A delay of a few seconds can reduce the value of the connection to spammers while not reducing performance significantly on the Exchange side. This delay is called SMTP Tarpitting. It is still effective in some instances but, with the advent of spam bots, connecting hosts show little or no adverse reaction – they just seem to go on unabated. Individual enterprises should monitor Recipient Filtering to gauge its effectiveness.

Finally, The Recipient Filtering layer is also an easy place to prevent users from receiving any external e-mail, by including their address in the Recipient Filter block list. They will still be able to send to the internet. Again, the interface in the EMC is a basic address entry form with an equivalent cmdlet such as:

SenderID Filter Agent

The anonymous nature of the SMTP protocol allows senders to use incorrect or nonexistent domains in the source address of a message. SenderID is based on the Sender Policy Framework (SPF) and uses the same DNS record format. SPF, originally called “Sender Permitted From” is outlined at www.openspf.org and RFC 4408. With the SenderID Filter Agent enabled, Exchange Server executes a DNS lookup for an SPF record for the sending SMTP domain. If a published SPF record is available, Exchange will be able to determine if the IP address of the SMTP source is authorized to send e-mail for the sending SMTP domain. This anti-spoofing measure is described in RFC 4406 entitled Sender ID: Authenticating E-Mail.  As with other message hygiene layers, the administrator can conserve resources by configuring specific  exceptions. Messages from specific domains, or those addressed to certain recipients, can be configured to bypass SenderID queries.

SenderID tries to determine the correct sender address to validate against, which is not always obvious. The SenderID Filter Agent will parse the message headers applying a specific algorithm to arrive at the Purported Responsible Address (PRA). This algorithm is defined in RFC 4407 called Purported Responsible Address in E-Mail Messages. This also differentiates SenderID from basic SPF.

The SPF record query returns a status. SenderID only deletes or rejects a message when the status returned from the lookup is set to FAIL. This is configurable in the SenderID Properties window in the EMC. No action is taken for other status levels which include PASS, NEUTRAL, SOFT FAIL, TEMP ERROR, and PERM ERROR.

Confused with SPF record formatting?
There are a few resources online to assist in creating a properly formatted SPF record and validating an existing record. Microsoft maintains such a tool, affectionately called Sender ID Framework SPF Record Wizard found at http://www.microsoft.com/mscorp/safety/content/ technologies/senderid/wizard/.

Content Filtering Agent

In Exchange Server 2003, Microsoft made available a separate download called the Intelligent Message Filter (IMF) Version 1. The IMF was updated to version 2 and included in Exchange 2003 Service Pack 2. In Exchange Server 2007, the IMF is now referred to as the Content Filter Agent, which could be considered IMF v3.

Content Filtering is engaged after the previous layers have assessed the messages and after the DATA command is fulfilled in the SMTP conversation. Content Filtering is a little bit of a ‘black box’. However, with sufficiently large samples, people have worked backwards to compile reasonable algorithms that may be in play. Microsoft has a great deal of experience with spam. Hotmail.com, msn.com, and Microsoft.com have provided almost unlimited samples from which to create an effective content filter mechanism. Microsoft calls it SmartScreen technology and it forms the basis for content filtering in Outlook as well as Exchange Server. Exchange assesses messages and tries to quantify the likeliness that the message is spam. This measurement is called the Spam Confidence Level (SCL) and is stored as an attribute of the message. Content Filtering assigns a value between 0-9 and records this in an X-header. The higher the SCL, the greater the chances the message is spam. There is also an SCL value of -1 reserved for internal and authenticated messaging.

In the Content Filtering properties window, we can add custom terms to ensure that certain messages are either blocked or allowed to pass. A pharmaceutical distributor may want to whitelist the term Viagra, for example. In addition, recipient addresses can be listed as exceptions in the filtering process, such as postmaster@domain.com. New to Exchange 2007 is the ability to whitelist certain senders or domains bypassing content filtering. This is not available in the EMC, but the cmdlets are intuitive. To whitelist an SMTP address, the following can be run from the EMS:

Administrators can assign one of three actions to a message, based on its SCL value: delete, reject or quarantine.

524-Figure4_MMG_RG_ContentFilteringPrope

 Figure 4 shows the options in the EMC to assign actions, based on specific minimum SCL ratings. This also shows the current settings on my Edge Server. You will note in Figure 4 that messages with an SCL value of 7 will be quarantined to a separate mailbox. This means that SCL values of 6 or less will be have the rating appended to the header and the message passed through to the Hub Transport and on to the recipient.  These settings are not required and different values may be more appropriate in different companies. Administrators should test different threshold values to see what is most effective for their enterprise.

There is also a Junk Mail Threshold for SCL messages. This is configured through the EMS only and simply determines the SCL value at which messages are moved to the Junk Mail folder of the recipient’s mailbox.

The final thing I will mention is Sender Reputation. Sender Reputation is a new addition to the message hygiene efforts in Exchange Server. Exchange analyzes characteristics of SMTP sessions of unknown senders. After a sender has sent 20 messages, Exchange maintains a Sender Reputation Level (SRL) analogous to the SCL for content filtering. Sender Reputation, as implemented in Exchange 2007, is a Microsoft creation that is in its infancy and presents potentially compelling options for the future of anti-spam technologies.

Summary

Spam is a moving target. Currently there is no single tool to deploy in the enterprise that will solve this ongoing problem easily. With Exchange 2007, Microsoft provides, out of the box, a multi-faceted defense against UCE. The key to effective spam filtering, using the native Exchange Server 2007 tools, is to determine unwanted messages as early as possible in the message flow. When used together, the tools provide a competitive anti-spam solution at no additional software cost.

Tags: , ,

  • 26140 views

  • Rate
    [Total: 20    Average: 4.7/5]
  • Stu

    helpful
    William, your article was very helpful in clearing up the process exch 2k7 goes through to filter through email. Thanks and keep it up!

  • Jack Pannell

    Edge versus Hub Transport
    Thank you so much for the small box in your piece on Intelligent Message Filter entitled “No Edge Server, No problem”. I had looked everywhere for this info!

    Seems everyone assumes that every org no matter how small is going to set up 4 or 5 Exchange Server 2007 computers. The smaller customer will not.

    MS is too busy patting themselves on the back for Exchange 2007 and telling us all how way cool it is instead of well documenting the product. If your network is under 100 computers, you probably were not jumping up and down for all the scripting tools either.

  • AnnaL

    Anonymous comments disabled
    We have had to disable anonymous comments on this article due to relentless spamming. If you wish to comment on this article, please sign in.

  • Baljinder singh

    mail problem
    hi sir,
    My Name is baljinder singh,i have a problem in my exchange server,i send mail from my mail to gmail,yahoo mail,hotemail etc then i send mail not her inboxe,,going that mail spam….why?

    give me salution at my mail id:-baljinder.tech@gmail.com.

    i waitnin ur answer

  • montgoma

    Quarantine mailbox like in example – please help
    I created a quarantine mailbox like in the example above and set the thresholds to 9, 8, 7, also like in the screenshots which they were pretty close to mine already. I gave myself full rights on the quarantine mailbox so I can monitor it and it is steady filling up. Since implimenting it roughly an hour ago, I am up to 3,000 emails and it is steadily counting up. Is this normal? I work for a college/university and I am just wondering if there is a way to not have this mailbox fill to capacity in such a short time. It seems to be all SPAM, nothing legitimate in there so I assume my scores are ok so far. Please advise.