Archive for the ‘Techy stuff’ Category

Prerequisite:

Server with Windows Server 2003 / 2008

Sending e-mails is quite straightforward; all modern programming languages have some built-in methods to achieve this task. Delivering an e-mail is a totally different matter.

Probably anyone that has ever had any dealing with websites sending e-mails has faced the problem where the e-mail is ending up in Spam/Junk box or not being delivered at all.

There are a few ways to improve e-mail deliverability. The path to achieve sensible e-mail deliverability can be long and bumpy. It includes:

1)      Choosing  the domain registrar

2)      Setting up the DNS

3)      SMTP service configuration

4)      Setting up sender authentication technologies

5)      Where to go next? (E-mail content, best practices, blacklists)

1   Choosing the domain registrar

Oddly enough – not all domain registrars allow creating / modifying TXT records. This is the absolute minimum for our DNS configuration – all e-mail authentication technologies require new TXT records in your domain.

This list captures domain registrars that support TXT records:

http://www.kitterman.com/spf/txt.html

123-reg is on that list, let’s try creating a domain there or better not, unless you want to spend hours stressing over why your sender’s authentication is not working.

As we can see 123-reg is not supporting DKIM which is weird since DKIM entry in DNS is just a normal TXT record. But sadly it’s not working … DKIM entries will not be found while using 123-reg. For most users it’s ok that they can only add SPF. But I decided to find a different domain registrar. I found quite few that are supporting DKIM entries (some of them directly through web-panel, some by support requests)

• Bluedomino
• DNS Made Easy
• GoDaddy
• Media Temple
• Network Solutions
• Pair
• Register.com
• Textdrive
• Melbourne IT

I’ve chosen GoDaddy, but feel free to pick any other provider or stick with your current one if you don’t need DKIM.

2   Setting up DNS

There are 2 important things to take into account while setting up your DNS:

• Your A record must point to your sending server’s IP address – this can be done from your DNS management panel.

• Your PTR record must point to your domain – to change PTR you need to have IP assigned to you by your hosting company, not all hosting companies allows you to change PTR record, but most of them will change it for you when you send a support ticket.

Now let’s have a look at our settings.

MX records are responsible for the specifying mail server that is in charge of accepting e-mails. While sending e-mails the recipient server should not look at MX records, but if you want to receive e-mails you need to add server’s domain names which can accept e-mails.

To test your DNS settings you can use a very useful tool provided by mxtoolbox :

http://www.mxtoolbox.com/SuperTool.aspx

With it you can easily type commands like “a:<your domain name>” or “ptr:<your IP address>” and check whether the A record is resolving to your IP and whether your IP is resolving to your domain name.

Below you can see the results from mxtoolbox check:

3  SMTP service configuration

I assume that you have IIS SMTP service. Please note that under IIS7 SMTP service is still available though IIS6 management console.

First let’s enable the connection with our SMTP service through localhost.

This can be done by going to Properties of our SMTP service. On the bottom of the second tab ( Access ) we need to open “Relay…” window and add 127.0.0.1 which is a localhost address.

Another step is to set a fully-qualified domain name for our SMTP service. This will be used for HELO and EHLO handshakes. Let’s go to 4^th tab ( Delivery ) and click “Advanced…”. We need to specify how our SMTP server should “introduce” itself.

Since we have everything ready let’s test our e-mail.

Posrt25 is providing us with an excellent tool for e-mail testing:

http://www.port25.com/domainkeys/

As you can see from the article above, we have 2 ways of e-mail testing:

• “If you wish to receive the results at the address in the “mail_from,” the sample message should be sent to check-auth@verifier.port25.com.”

• “If you wish to receive the results at the address in the “from” header, the sample message should be sent to check-auth2@verifier.port25.com.”

One of the ways to send an e-mail is to use telnet.
Using telnet can be useful for a simple e-mail tests but it can cause some errors later on. As for now let’s try using telnet.

Let’s quickly go through the screenshot.

• EHLO / HELO <your domain name>
  Response should be 250 OK

• Specifying sender e-mail address ( mail from ):
  mail from: <your sender e-mail address>
  Response : 250 OK

• rcpt to: <recipient e-mail address>
  Let’s send to check-auth2@verifier.port25.com
  Response : 250 OK

• command : data will enable you to write e-mail headers and body. You can now specify headers like ‘From display name’, ‘Subject’ and message itself (body). We are only interested in ‘From display name’ so we can put e-mail address
  type:
  From: <email address where you want to receive the report>
  press “Enter”, Type “.” and “Enter” again.

After a few seconds, up to a minute, you should get an e-mail with a report. It will probably look something like this:

This means that we neither have SPF record, DomainKeys, DKIM nor Sender-ID authentication. SpamAssassin marked our e-mail as “ham” which is opposite to spam so it has been categorised as non-harmful e-mail.

One more important thing while configuring SMTP service is that it cannot be open-relayed. Open mail relay means that anyone (including all sorts of spammers) can send e-mails through your server, which is a very bad situation and should be avoided at all costs.

There is plenty of on-line services that can check whether your site is open-relay but for our example let’s use mxtoolbox again.

You can open link from before (http://www.mxtoolbox.com/SuperTool.aspx ) and type command smtp:<your domain name>

Or you can also use this link: http://www.mxtoolbox.com/SuperTool.aspx?action=smtp:<your_domain_name_here>

Result should look similar to this:

4  Setting up sender authentication technologies

• SPF ( Sender Policy Framework ) and Sender-ID

These two are giving the option to specify which server is allowed to send e-mails for a given domain. This is done by modifying the domain DNS record.

For example you can add a new TXT record to your domain with a value “v=spf1 a ~all”.

which means that:

v=spf1 // we are using version 1 of SPF

a // domain must have A record that can be resolved to sender address

~all // (Soft Fail) Mail may possibly come from an IP address which does not match result from resolving A record (our example), but this kind of e-mail will have bigger chances to go to spam box.

Other common parameters are:

“v=spf1 a -all” // ‘-all’ mean that if an e-mail originates from address not present in domain’s A record it will go to junk or will be rejected

“v=spf1 a ip4:<sender server ip> ~all” // ip4:<sender server ip> ( please replace <sender server ip> with public IP address of your server ) means that every host in this IP4 range can send e-mails.

Let’s add sample TXT entry to our DNS:

There are 2 very good wizards that can guide you through the process of creating a correct record:

http://old.openspf.org/wizard.html

http://www.microsoft.com/mscorp/safety/content/technologies/senderid/wizard/

Sender ID is using the same sender validation method, syntax and they both need to be published as new TXT records in DNS. It doesn’t need to be published in your DNS since SPF is compatible with Sender ID, but it all depends on the receiving server.

Sender ID is not really popular, even Microsoft is using SPF in his “Sender ID wizard”, but you can always publish Sender ID record.
For example :

“v=spf1 a ip4:<sender server ip> -all”

Written as Sender ID record :

“spf2.0/mfrom ip4: <sender server ip> -all”

You can submit your SPF record to be added to Sender-ID program (again it looks like real Sender-ID record is not needed ) using this link :

https://support.msn.com/eform.aspx?productKey=senderid&page=support_senderid_options_form_byemail&ct=eformts&wa=wsignin1.0

After setting only SPF record let’s try to send e-mail to our port25 to validate our configuration. The results are:

SPF – passed, that’s good.

Sender ID – neutral? It’s because we were sending an e-mail using the same telnet technique as before with ‘From field’ set to different e-mail address.

From now on I would recommend sending e-mails from your own application. Some very simple C# code for sending e-mails:

using System;
using System.Net.Mail;
namespace Email
{
  class Program
  {
    static void Main(string[] args)
    {
      Console.WriteLine("Type from address (and hit Enter):");
      string fromAddress = Console.ReadLine();
      Console.WriteLine("Type from name (and hit Enter):");
      string fromDisplayName = Console.ReadLine();
      Console.WriteLine("Type Recipient address (and hit Enter):");
      string toAddress = Console.ReadLine();
      Console.WriteLine("Type subject (and hit Enter):");
      string subject = Console.ReadLine();
      Console.WriteLine("Type body (and hit Enter):");
      string body = Console.ReadLine();
      MailMessage Mail = new MailMessage();
      MailAddress ma = new MailAddress(fromAddress, fromDisplayName);
      Mail.From = ma;
      Mail.To.Add(toAddress);
      Mail.Subject = subject;
      Mail.Body = body;
      Mail.Headers.Add("Reply-To", fromAddress);
      try
      {
        SmtpClient smtpMailObj = new SmtpClient();
        smtpMailObj.Send(Mail);
        Console.WriteLine("Email sent, press enter to exit.");
      }
      catch (Exception ex)
      {
        Console.WriteLine(ex.Message);
        Console.WriteLine(ex.StackTrace);
      }
      Console.ReadLine();
    }
  }
}

We will also need app.config for our small Console Application :

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<system.net>
<mailSettings>
<smtp>
<network host="localhost" port="25"/>
</smtp>
</mailSettings>
</system.net>
</configuration>

This of course is not production code, this is simply code that you can use for test applications. Sender-ID will fail because of invalid ‘From address’. We will need a POP3 account on our server to set a proper ‘From address’ and receive e-mail report s back to our server.

After setting up POP3 and using our Console Application we will get result like this:

This is what we wanted.

We will discuss setting up POP3 in the next section.

• DomainKeys and DKIM

Another approach to authenticate sender is DKIM, which was created by DomainKeys (developed by Yahoo!) merged with IIM ( Cisco ). Both technologies are using cryptographic signature generated from an e-mail header and body. That signature is then passed along with the e-mail to the receiving server, which validates signature against published DNS information.

In this article we will try to implement only the DKIM. There is a small problem with Windows build-in SMTP service – it does not support DKIM signing.
What are our options then? You could manually add cryptographic key in C# code, or you could write an application that will be adding the signing and still use the built in SMTP service or finally you could use 3^rd party Mail server / plug-in. That last option is recommended and at the same time is the simplest. There are many commercial mail servers ( or plug-ins that integrate with existing Windows SMTP service ).

Commercial plug-in that will allow you to use your existing server configuration: http://www.emailarchitect.net/domainkeys/

Free mail server ( SMTP, POP3 and IMAP ) with DKIM support:

http://www.hmailserver.com/

Those are only a few options out of many available. Feel free to find most suitable solution for you. In this article we will go with hMailServer.

Before installation you should disable Windows SMTP service. An installation tutorial and basic configuration can be found here: http://www.hmailserver.com/documentation/latest/?page=howto_install

http://www.hmailserver.com/documentation/latest/?page=basic_configuration

Now we have everything set up: we are running our new SMTP service, we have POP3 service and our Consol Application available for tests. Let’s add DKIM.

There are many tools to generate public / private keys for DKIM. I’ve chosen a quick on-line way:

http://www.socketlabs.com/services/dkwiz

You will need to specify your domain name and selector which can be any word you’d like.

Your generated result should look similar to this:

Steps that we need to carry out now are quite simple:

• Add two new TXT records to your DNS as described on the page with generated DKIM keys. Create new private key file ( any text file ). File needs to include ‘begin tag’ (—–BEGIN RSA PRIVATE KEY—– ), key and ‘end tag’.
• Open hMailServer administration and navigate to your domain. You should be able to see DKIM on the 5^th tab. Add your private key file and then specify selector. Header and Body method should be set to “Relaxed”, Signing algorithm to SHA256.

Configuration should look similar to that:

We can easliy test whether our DKIM has been properly set in DNS by visiting those pages:

http://domainkeys.sourceforge.net/policycheck.html

http://domainkeys.sourceforge.net/selectorcheck.html

Results:

Note: Don’t forget to remove “t=y” from your TXT record after you finish testing DKIM (t=y means that domain is in test mode).

Since everything is set let’s send a test e-mail again. Let’s use our test application and since we have POP3 server configured let’s send an e-mail to check-auth@verifier.port25.com.

Note: all received e-mails can be found physically on the server by navigating to
<installation path>\hMailServer\Data\<your domain name>\<account address>

If everything is properly configured we should get this report:

This looks better now. Let’s see how Google is ‘seeing’ our e-mail.

Signed-by : saktos.info – that’s what we wanted.

Let’s take a closer look at e-mail source:

SPF, Sender-ID and DKIM have passed. One would think that that’s the end of our journey since mail server is configured. Well not really… even setting everything does not mean that we can send e-mails everywhere with any content and they will always go to the inbox…

5  Where to go next? (E-mail content, best practices, blacklists)

Blacklists are very important part of e-mail deliverability. It doesn’t matter what kind of authentication methods are being used if we appear on the blacklists. They should be monitored on a regular basis. We can again use mxtoolbox for that purpose: http://www.mxtoolbox.com/blacklists.aspx

All e-mail services that filter spam are not only based on authentication or blacklists, but also work by using some advanced content filters. Some very small lists of spam keywords can be found here: http://www.activewebhosting.com/faq/email-filterlist.html

Avoid sending e-mails containing lots of images and never send attachments with some potentially dangerous files like *.exe. If you need to send an attachment it’s better to provide the user with a link to your website than attaching a file.

If you want to do a marketing campaign never buy e-mail addresses. It’s much better to gain e-mail addresses through your own website. Every time a user clicks “Report Spam”, it’s being sent back to the server which is lowering your server reputation.

If your e-mails are still going to the spam box I suggest reading through some very helpful articles:

1)      Direct link to PDF file explaining how to improve deliverability to hotmail

http://download.microsoft.com/download/e/3/3/e3397e7c-17a6-497d-9693-78f80be272fb/enhance_deliver.pdf

2)      Still having problems with hotmail ? Check Hotmail postmaster services:

http://postmaster.msn.com/Default.aspx

3)      CAN – SPAM wiki page. This act contains many best – practices for sending e – mails.

http://en.wikipedia.org/wiki/CAN-SPAM_Act_of_2003

4)      Google bulk messaging guidelines

https://mail.google.com/support/bin/answer.py?answer=81126

Recently, we had to migrate an application that uses windows authentication from an old server to a new one. Of course, we did not want the users to lose their passwords in the migration. Also, as there were several hundred users, we did not want to recreate the user accounts by hand.

Migration can be divided into two parts:

1. Migrate the user accounts
2. Migrate the passwords

A. Migrate user accounts

Migrating user accounts (without passwords) can be done using the AddUsers tool from the Windows Resource Kit (see http://support.microsoft.com/kb/199878). It can be downloaded from http://www.petri.co.il/download_free_reskit_tools.htm

On the source server,

1. Download AddUsers
2. From the command prompt, run AddUsers /d users.txt , to dump user account information to a file with name users.txt. Note that this file will not contain passwords.
3. Open users.txt with a text editor. Users.txt has three sections: [Users], [Global] and [Local ]. The [Users] section has one line User Name,Full name, Password, Description, HomeDrive, Homepath, Profile for every user on the source server.
4. Remove the user entries that do not have to be migrated (e.g. the system accounts). Similarly, remove all groups that do not have to be migrated from the [global] and [local] sections.
5. Be weary of comma’s in Descriptions in users.txt, as they interfere with the comma delimitation. There is an option to use a different delimiter if needed.
6. The target server is unlikely to allow users to be created without a password. For each user, add a temporary Password (e.g. temp123).
7. Users are added to groups using the source machine’s name. Change this to the name of the target machine (e.g., globally replace the source machine’s name with the target machine’s name).
8. Copy users.txt to the target server.

On the target server,

1. Download AddUsers
2. From the command prompt, run AddUsers /c users.txt /p:e

This recreates all users and groups in users.txt on the target server, and adds the users to the right groups. The /p:e option ensures that passwords do not expire.

B. Migrate passwords

Migrating passwords can be done using the copypwd tool. See http://www.systemtools.com/free.htm for documentation and download. Warning: do not use the steps below if the data on the server is extremely sensitive.

1. Download copypwd on the source server.
2. In this step, we create a file copypwd.txt that contains every user name on the source server, and a hash that represents the password for that user.
   1. If you are not accessing the source server using a remote desktop connection, run copypwd DUMP > copypwd.txt.
   2. If you are accessing the source server using a remote desktop connection, copypwd cannot be run remotely. Take the following steps.
      1. Go to Control Panel/Administrative Tools/Component Services, Services. Start the Task Scheduler service if is stopped. If the service is stopped and the option to start is greyed out, then right-click it, select properties, and set startup type to manual. Then start the Task Scheduler.
      2. Create a batch file run.bat in a directory path that executes copypwd DUMP > copypwd.txt
      3. From the command prompt, run: at hh:mm “c:\path\run.bat” (where hh:mm is one minute in the future)
      4. After one minute, this creates the copypwd.txt file that contains the passwords.
      5. Stop the task manager if it was started in step 2a.
3. Copy copypwd.txt to the target server.
4. Update the passwords using copypwd SET. As in step 2, copypwd can’t be run using a remote connection.
   1. Make sure the Task Manager service is running on the target server, see step 2a.
   2. Create a batch file run.bat in a directory path that executes copypwd SET > done.txt
   3. From the command prompt, run: at hh:mm “c:\path\run.bat”
   4. After one minute, this will update all the users and create the file done.txt. Check done.txt to see if the update was succesfull.
   5. Stop the Task Scheduler if it was started in step 4a.
5. Delete the password info and the user account info on the source and target servers.

that’s it! and it only took only 25 simple steps…

Can you believe there’s no handy way of looking at the table sizes out of the box?

*sigh*

So here’s a chunk of SQL to do the job. I found it on the interweb so I can’t take credit but it deserves  more attention I think.

DECLARE
@id int,
@pages int,
@objname varchar(750)

SET NOCOUNT ON

CREATE TABLE #tblSize
(
Name varchar (100),
Rows varchar (100),
Reserved varchar (100),
Data varchar (100),
Index_Size varchar (100),
Unused varchar (100)
)

CREATE TABLE #spt_space
(
rows int null,
reserved dec(15) null,
data dec(15) null,
indexp dec(15) null,
unused dec(15) null
)

-- declare main cursor to get first user table name from sysobjects
DECLARE TabNameCur CURSOR FOR
SELECT id, name
FROM dbo.sysobjects
WHERE xtype = 'u'
ORDER BY name
OPEN TabNameCur
FETCH TabNameCur INTO @id, @objname

WHILE @@FETCH_STATUS = 0
BEGIN

TRUNCATE TABLE #spt_space

INSERT INTO #spt_space (reserved)
SELECT sum(reserved)
FROM sysindexes
WHERE indid in (0, 1, 255)
AND id = @id

SELECT @pages = sum(dpages)
FROM sysindexes
WHERE indid < 2
AND id = @id

SELECT @pages = @pages + isnull(sum(used), 0)
FROM sysindexes
WHERE indid = 255
AND id = @id

UPDATE #spt_space
SET data = @pages

UPDATE #spt_space
SET indexp = (SELECT sum(used)
FROM sysindexes
WHERE indid in (0, 1, 255)
AND id = @id) - data

UPDATE #spt_space
SET unused = reserved
- (SELECT sum(used)
FROM sysindexes
WHERE indid in (0, 1, 255)
AND id = @id)

UPDATE #spt_space
SET rows = i.rows
FROM sysindexes i
WHERE i.indid < 2
AND i.id = @id
--This step required as 'convert.../1000' cannot be used with varchars
INSERT INTO #tblSize
SELECT name = object_name(@id),
rows, --= convert(char(11), rows),
reserved = convert(decimal (8,2), (reserved * d.low / 1024.)/1000),
data = convert(decimal (8,2), (data * d.low / 1024.)/1000),
index_size = convert(decimal (8,2), (indexp * d.low / 1024.)/1000),
unused = convert(decimal (8,2), (unused * d.low / 1024.)/1000)
FROM #spt_space, master.dbo.spt_values d
WHERE d.number = 1
AND d.type = 'E'

FETCH NEXT FROM TabNameCur INTO @id, @objname
END

-- close & deallocate main cursor
CLOSE TabNameCur
DEALLOCATE TabNameCur

SELECT Name, Rows,
Reserved + ' MB' as Reserved,
Data + ' MB' as Data,
index_size + ' MB' as Index_Size,
unused + ' MB' as Unused
FROM #tblSize
ORDER BY
      Reserved DESC

DROP TABLE #tblSize
DROP TABLE #spt_space

Enjoy

One of our developers has been unfortunate enough to have developed a hospitalising condition. A capable guy, he was at the hub of a lot of things. It also meant that he had a lot of stuff checked out when he was struck down. The project has moved on somewhat and we need to branch the code. Exceedingly messy if there are file changes pending. TFS2008 has a command line remedy to this.

tf undo [/workspace:NAMEOFWORKSPACE [;USERNAME]] $/LOCATION /s:http://YOURSERVER:8080 /recursive

I will leave it to you good people to substitute out all of the necessary values for the capitalised placeholders.