SMTP Configuration and e-mail deliverability

Prerequisite:

Server with Windows Server 2003 / 2008

Sending e-mails is quite straightforward; all modern programming languages have some built-in methods to achieve this task. Delivering an e-mail is a totally different matter.

Probably anyone that has ever had any dealing with websites sending e-mails has faced the problem where the e-mail is ending up in Spam/Junk box or not being delivered at all.

There are a few ways to improve e-mail deliverability. The path to achieve sensible e-mail deliverability can be long and bumpy. It includes:

1)      Choosing  the domain registrar

2)      Setting up the DNS

3)      SMTP service configuration

4)      Setting up sender authentication technologies

5)      Where to go next? (E-mail content, best practices, blacklists)

1   Choosing the domain registrar

Oddly enough – not all domain registrars allow creating / modifying TXT records. This is the absolute minimum for our DNS configuration – all e-mail authentication technologies require new TXT records in your domain.

This list captures domain registrars that support TXT records:

http://www.kitterman.com/spf/txt.html

123-reg is on that list, let’s try creating a domain there or better not, unless you want to spend hours stressing over why your sender’s authentication is not working.

As we can see 123-reg is not supporting DKIM which is weird since DKIM entry in DNS is just a normal TXT record. But sadly it’s not working … DKIM entries will not be found while using 123-reg. For most users it’s ok that they can only add SPF. But I decided to find a different domain registrar. I found quite few that are supporting DKIM entries (some of them directly through web-panel, some by support requests)

• Bluedomino
• DNS Made Easy
• GoDaddy
• Media Temple
• Network Solutions
• Pair
• Register.com
• Textdrive
• Melbourne IT

I’ve chosen GoDaddy, but feel free to pick any other provider or stick with your current one if you don’t need DKIM.

2   Setting up DNS

There are 2 important things to take into account while setting up your DNS:

• Your A record must point to your sending server’s IP address – this can be done from your DNS management panel.

• Your PTR record must point to your domain – to change PTR you need to have IP assigned to you by your hosting company, not all hosting companies allows you to change PTR record, but most of them will change it for you when you send a support ticket.

Now let’s have a look at our settings.

MX records are responsible for the specifying mail server that is in charge of accepting e-mails. While sending e-mails the recipient server should not look at MX records, but if you want to receive e-mails you need to add server’s domain names which can accept e-mails.

To test your DNS settings you can use a very useful tool provided by mxtoolbox :

http://www.mxtoolbox.com/SuperTool.aspx

With it you can easily type commands like “a:<your domain name>” or “ptr:<your IP address>” and check whether the A record is resolving to your IP and whether your IP is resolving to your domain name.

Below you can see the results from mxtoolbox check:

3  SMTP service configuration

I assume that you have IIS SMTP service. Please note that under IIS7 SMTP service is still available though IIS6 management console.

First let’s enable the connection with our SMTP service through localhost.

This can be done by going to Properties of our SMTP service. On the bottom of the second tab ( Access ) we need to open “Relay…” window and add 127.0.0.1 which is a localhost address.

Another step is to set a fully-qualified domain name for our SMTP service. This will be used for HELO and EHLO handshakes. Let’s go to 4^th tab ( Delivery ) and click “Advanced…”. We need to specify how our SMTP server should “introduce” itself.

Since we have everything ready let’s test our e-mail.

Posrt25 is providing us with an excellent tool for e-mail testing:

http://www.port25.com/domainkeys/

As you can see from the article above, we have 2 ways of e-mail testing:

• “If you wish to receive the results at the address in the “mail_from,” the sample message should be sent to check-auth@verifier.port25.com.”

• “If you wish to receive the results at the address in the “from” header, the sample message should be sent to check-auth2@verifier.port25.com.”

One of the ways to send an e-mail is to use telnet.
Using telnet can be useful for a simple e-mail tests but it can cause some errors later on. As for now let’s try using telnet.

Let’s quickly go through the screenshot.

• EHLO / HELO <your domain name>
  Response should be 250 OK

• Specifying sender e-mail address ( mail from ):
  mail from: <your sender e-mail address>
  Response : 250 OK

• rcpt to: <recipient e-mail address>
  Let’s send to check-auth2@verifier.port25.com
  Response : 250 OK

• command : data will enable you to write e-mail headers and body. You can now specify headers like ‘From display name’, ‘Subject’ and message itself (body). We are only interested in ‘From display name’ so we can put e-mail address
  type:
  From: <email address where you want to receive the report>
  press “Enter”, Type “.” and “Enter” again.

After a few seconds, up to a minute, you should get an e-mail with a report. It will probably look something like this:

This means that we neither have SPF record, DomainKeys, DKIM nor Sender-ID authentication. SpamAssassin marked our e-mail as “ham” which is opposite to spam so it has been categorised as non-harmful e-mail.

One more important thing while configuring SMTP service is that it cannot be open-relayed. Open mail relay means that anyone (including all sorts of spammers) can send e-mails through your server, which is a very bad situation and should be avoided at all costs.

There is plenty of on-line services that can check whether your site is open-relay but for our example let’s use mxtoolbox again.

You can open link from before (http://www.mxtoolbox.com/SuperTool.aspx ) and type command smtp:<your domain name>

Or you can also use this link: http://www.mxtoolbox.com/SuperTool.aspx?action=smtp:<your_domain_name_here>

Result should look similar to this:

4  Setting up sender authentication technologies

• SPF ( Sender Policy Framework ) and Sender-ID

These two are giving the option to specify which server is allowed to send e-mails for a given domain. This is done by modifying the domain DNS record.

For example you can add a new TXT record to your domain with a value “v=spf1 a ~all”.

which means that:

v=spf1 // we are using version 1 of SPF

a // domain must have A record that can be resolved to sender address

~all // (Soft Fail) Mail may possibly come from an IP address which does not match result from resolving A record (our example), but this kind of e-mail will have bigger chances to go to spam box.

Other common parameters are:

“v=spf1 a -all” // ‘-all’ mean that if an e-mail originates from address not present in domain’s A record it will go to junk or will be rejected

“v=spf1 a ip4:<sender server ip> ~all” // ip4:<sender server ip> ( please replace <sender server ip> with public IP address of your server ) means that every host in this IP4 range can send e-mails.

Let’s add sample TXT entry to our DNS:

There are 2 very good wizards that can guide you through the process of creating a correct record:

http://old.openspf.org/wizard.html

http://www.microsoft.com/mscorp/safety/content/technologies/senderid/wizard/

Sender ID is using the same sender validation method, syntax and they both need to be published as new TXT records in DNS. It doesn’t need to be published in your DNS since SPF is compatible with Sender ID, but it all depends on the receiving server.

Sender ID is not really popular, even Microsoft is using SPF in his “Sender ID wizard”, but you can always publish Sender ID record.
For example :

“v=spf1 a ip4:<sender server ip> -all”

Written as Sender ID record :

“spf2.0/mfrom ip4: <sender server ip> -all”

You can submit your SPF record to be added to Sender-ID program (again it looks like real Sender-ID record is not needed ) using this link :

https://support.msn.com/eform.aspx?productKey=senderid&page=support_senderid_options_form_byemail&ct=eformts&wa=wsignin1.0

After setting only SPF record let’s try to send e-mail to our port25 to validate our configuration. The results are:

SPF – passed, that’s good.

Sender ID – neutral? It’s because we were sending an e-mail using the same telnet technique as before with ‘From field’ set to different e-mail address.

From now on I would recommend sending e-mails from your own application. Some very simple C# code for sending e-mails:

using System;
using System.Net.Mail;
namespace Email
{
  class Program
  {
    static void Main(string[] args)
    {
      Console.WriteLine("Type from address (and hit Enter):");
      string fromAddress = Console.ReadLine();
      Console.WriteLine("Type from name (and hit Enter):");
      string fromDisplayName = Console.ReadLine();
      Console.WriteLine("Type Recipient address (and hit Enter):");
      string toAddress = Console.ReadLine();
      Console.WriteLine("Type subject (and hit Enter):");
      string subject = Console.ReadLine();
      Console.WriteLine("Type body (and hit Enter):");
      string body = Console.ReadLine();
      MailMessage Mail = new MailMessage();
      MailAddress ma = new MailAddress(fromAddress, fromDisplayName);
      Mail.From = ma;
      Mail.To.Add(toAddress);
      Mail.Subject = subject;
      Mail.Body = body;
      Mail.Headers.Add("Reply-To", fromAddress);
      try
      {
        SmtpClient smtpMailObj = new SmtpClient();
        smtpMailObj.Send(Mail);
        Console.WriteLine("Email sent, press enter to exit.");
      }
      catch (Exception ex)
      {
        Console.WriteLine(ex.Message);
        Console.WriteLine(ex.StackTrace);
      }
      Console.ReadLine();
    }
  }
}

We will also need app.config for our small Console Application :

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<system.net>
<mailSettings>
<smtp>
<network host="localhost" port="25"/>
</smtp>
</mailSettings>
</system.net>
</configuration>

This of course is not production code, this is simply code that you can use for test applications. Sender-ID will fail because of invalid ‘From address’. We will need a POP3 account on our server to set a proper ‘From address’ and receive e-mail report s back to our server.

After setting up POP3 and using our Console Application we will get result like this:

This is what we wanted.

We will discuss setting up POP3 in the next section.

• DomainKeys and DKIM

Another approach to authenticate sender is DKIM, which was created by DomainKeys (developed by Yahoo!) merged with IIM ( Cisco ). Both technologies are using cryptographic signature generated from an e-mail header and body. That signature is then passed along with the e-mail to the receiving server, which validates signature against published DNS information.

In this article we will try to implement only the DKIM. There is a small problem with Windows build-in SMTP service – it does not support DKIM signing.
What are our options then? You could manually add cryptographic key in C# code, or you could write an application that will be adding the signing and still use the built in SMTP service or finally you could use 3^rd party Mail server / plug-in. That last option is recommended and at the same time is the simplest. There are many commercial mail servers ( or plug-ins that integrate with existing Windows SMTP service ).

Commercial plug-in that will allow you to use your existing server configuration: http://www.emailarchitect.net/domainkeys/

Free mail server ( SMTP, POP3 and IMAP ) with DKIM support:

http://www.hmailserver.com/

Those are only a few options out of many available. Feel free to find most suitable solution for you. In this article we will go with hMailServer.

Before installation you should disable Windows SMTP service. An installation tutorial and basic configuration can be found here: http://www.hmailserver.com/documentation/latest/?page=howto_install

http://www.hmailserver.com/documentation/latest/?page=basic_configuration

Now we have everything set up: we are running our new SMTP service, we have POP3 service and our Consol Application available for tests. Let’s add DKIM.

There are many tools to generate public / private keys for DKIM. I’ve chosen a quick on-line way:

http://www.socketlabs.com/services/dkwiz

You will need to specify your domain name and selector which can be any word you’d like.

Your generated result should look similar to this:

Steps that we need to carry out now are quite simple:

• Add two new TXT records to your DNS as described on the page with generated DKIM keys. Create new private key file ( any text file ). File needs to include ‘begin tag’ (—–BEGIN RSA PRIVATE KEY—– ), key and ‘end tag’.
• Open hMailServer administration and navigate to your domain. You should be able to see DKIM on the 5^th tab. Add your private key file and then specify selector. Header and Body method should be set to “Relaxed”, Signing algorithm to SHA256.

Configuration should look similar to that:

We can easliy test whether our DKIM has been properly set in DNS by visiting those pages:

http://domainkeys.sourceforge.net/policycheck.html

http://domainkeys.sourceforge.net/selectorcheck.html

Results:

Note: Don’t forget to remove “t=y” from your TXT record after you finish testing DKIM (t=y means that domain is in test mode).

Since everything is set let’s send a test e-mail again. Let’s use our test application and since we have POP3 server configured let’s send an e-mail to check-auth@verifier.port25.com.

Note: all received e-mails can be found physically on the server by navigating to
<installation path>\hMailServer\Data\<your domain name>\<account address>

If everything is properly configured we should get this report:

This looks better now. Let’s see how Google is ‘seeing’ our e-mail.

Signed-by : saktos.info – that’s what we wanted.

Let’s take a closer look at e-mail source:

SPF, Sender-ID and DKIM have passed. One would think that that’s the end of our journey since mail server is configured. Well not really… even setting everything does not mean that we can send e-mails everywhere with any content and they will always go to the inbox…

5  Where to go next? (E-mail content, best practices, blacklists)

Blacklists are very important part of e-mail deliverability. It doesn’t matter what kind of authentication methods are being used if we appear on the blacklists. They should be monitored on a regular basis. We can again use mxtoolbox for that purpose: http://www.mxtoolbox.com/blacklists.aspx

All e-mail services that filter spam are not only based on authentication or blacklists, but also work by using some advanced content filters. Some very small lists of spam keywords can be found here: http://www.activewebhosting.com/faq/email-filterlist.html

Avoid sending e-mails containing lots of images and never send attachments with some potentially dangerous files like *.exe. If you need to send an attachment it’s better to provide the user with a link to your website than attaching a file.

If you want to do a marketing campaign never buy e-mail addresses. It’s much better to gain e-mail addresses through your own website. Every time a user clicks “Report Spam”, it’s being sent back to the server which is lowering your server reputation.

If your e-mails are still going to the spam box I suggest reading through some very helpful articles:

1)      Direct link to PDF file explaining how to improve deliverability to hotmail

http://download.microsoft.com/download/e/3/3/e3397e7c-17a6-497d-9693-78f80be272fb/enhance_deliver.pdf

2)      Still having problems with hotmail ? Check Hotmail postmaster services:

http://postmaster.msn.com/Default.aspx

3)      CAN – SPAM wiki page. This act contains many best – practices for sending e – mails.

http://en.wikipedia.org/wiki/CAN-SPAM_Act_of_2003

4)      Google bulk messaging guidelines

https://mail.google.com/support/bin/answer.py?answer=81126

This entry was posted on Tuesday, May 11th, 2010 at 2:22pm and is filed under Techy stuff. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.