Thursday, August 09, 2007

Importing old mail into GMail.... with the proper dates!

So, you've decided that you want to do all of your email management from GMail. Maybe you've looked around a little, and found Mark Lyon's helpful GMail Loader. While this is a great utility, the method that it uses to import your emails causes GMail to reset the dates. While this isn't a huge issue, really, it was enough to keep me from using it as a solution... I wanted it to be perfect, or not at all. Unfortunately, this was by far the only option for importing emails.

Until recently....


Google has started to allow users to check up to 5 external POP email accounts from within GMail, integrating the mail with GMail. You can use this new feature to your advantage--if you have just a bit of technical dexterity--to import your local emails into GMail.

All that you need to do is run your own POP server! While this might sound like a non-trivial undertaking, it's not as bad as you might think. Furthermore, if you have SSH access to the machine on which your existing POP server is hosted, you can skip the installation step altogether.

Here's the high-level process for importing your emails using a private POP server:

1. Set up a POP server (or find one where you have access to location of the mail spool in the filesystem)

2. Using your current email program, find all of the emails that you want to import, and export them in "mbox" format.

3. Take this newly created mbox file, and move it into the mail spool area of your POP sever, giving it the name of the username that you use to check your email. In many cases, this is simply the first part of your email address, though not always.

4. Set up a new GMail pop-checking account. Enter the IP address or domain name of your newly created (or played with) POP server. I'd suggest setting the importer to automatically archive the mail and give it some unique label, just for your own sanity.

5. Sit back and wait. It will take a long time.
POP Clients

For Linux, I recommend teapop since it took no time for me to setup (it's in the Ubuntu repository), and seems to work pretty well. Of course, there are a number of other options, like qpopper, or the UW mail programs, or exim, but they may require more configuration.

For Windows, I don't know any native POP servers. So you'll have to do your own investigating here. One option is to install Cygwin and use a Linux pop server. I haven't done this myself, so I can't offer any detailed advice, but if you need help, feel free to email me.
Further Notes

If you're going to use a pre-existing pop server, I suggest using one that you're not actively receiving mail on. Mucking around in the mail spool may cause you to lose an email or two, or other bizarre things.

If you set up a POP server on your machine at home, remember that you'll need to set up your home router to forward requests for port 110 (POP) to the machine carrying your email. If you want to use a secure connection, it will be a different port.

Additionally, if you are sending mail from a POP server you set up on your machine at home, you should set up a DynDNS name for your box. Since the process takes a long time, your dynamic IP address could change before GMail is done getting everything.

This process is VERY VERY slow. I've found that the GMail popper will only check 200 mails at a time. I'm not sure exactly how it decides the interval between checks (they mention that they use an algorithm to decide how frequently to check each pop account), but I've found that at steady state, it eventually slows to only checking once every 1-2 hours. (presumably some sort of rate-limiting behavior.) For an inbox of 18000 emails, it took about 3 days before everything was imported.