Your personal GMail-like mail system: fetching with getmail

Now your e-mail storage is converted and you are ready to go. Hold on, it is time to do some preparation work. While not strictly necessary, I suggest you you figure up and write down the following information:

  • Which user id will own the mail storage files. It will also run the mail retriever and (in future) the delivery agent. I suggest creating a dedicated UID for it.
  • Which top directories would contain your e-mails.

I decided to create the user mailman, and create the storage in its home directories under the root user:

# useradd -m mailman
# mkdir -p /home/mailman/storage/business
# mkdir -p /home/mailman/storage/personal
# cp -r <old business email storage path> /home/mailman/storage/business/
# cp -r <old personal email storage path> /home/mailman/storage/personal/
# chown -R mailman /home/mailman/storage/

As you see, the mail is stored in two different directories inside the storage directory, and all the files are owned by the mailman user.

Once you set up the storage and the user, you’re ready to setup the mail fetching with getmail.

Installing getmail

Getmail is a Python script which retrieves e-mails from your mail providers and stores them into the Maildir folders according to configuration. You can use procmail instead, but I found getmail easier to configure and its ability to deliver to Maildir mail boxes natively is very useful in the early stages of building the e-mail system.

Installing getmail is typically as easy as installing the relevant package from your Linux distribution. However if your distro doesn’t provide one, you can always go to the getmail website and download it. Unpack it, and build and install it as described in README under the regular user:

> cd getmail-4*
> python setup.py build
> sudo python setup.py install

Configuring getmail

Getmail stores its configuration together with the internal cache files in a directory. Which means for each account you want to retrieve the mail from you will create a directory with the getmail configuration file getmailrc in it. So for my three mail providers I create three directories:

# su - mailman
> mkdir -p getmail/personal
> mkdir -p getmail/business
> mkdir -p getmail/gmail

and in each of those directories I created the getmailrc file. Since all my providers support POP3 retriever over SSL, the content of the getmailrc file was very similar to each of them. Note that if your provider requires you to use IMAP, you need to use a different retriever so check the getmail configuration.

Here’s the getmailrc configuration I use to retrieve email from my gmail account, which is stored in the /home/mailman/getmail/gmail/getmailrc file:

[retriever]
type = SimplePOP3SSLRetriever
server = pop.gmail.com
username = <gmail username>
password = <gmail password>

[destination]
type = Maildir
path = /home/mailman/storage/personal/

[options]
read_all = 0
delete = 0
delete_after = 3

As you see, for the sites which support POP3 over SSL all you need is to change the server, username and password in the [retriever] section. If the e-mail from this specific site should be delivered into a different storage (i.e. business), you also need to change the path in [destination] section but that’s all.

If your site supports POP3 but does not support SSL, use the type=SimplePOP3Retriever instead of using SimplePOP3SSLRetriever. If the POP3 server listens on a non-standard port number like 11000, you can also add a port=11000 parameter into this section. See the getmail documentation.

The options I use is to keep the mail on the remote server for three days, then delete it. This ensures the remote mailbox is not likely to overflow with mail, and provides me with some safety net knowing I can still access my recent email directly.

Testing your configuration

To test your configuration run getmail with the –getmaildir argument followed by the path to your getmail directory under the mailman user which can write into the maildirs:

> /usr/bin/getmail  --getmaildir /home/mailman/getmail/gmail

and you will see the output similar to the one below:

Copyright (C) 1998-2009 Charles Cazabon. Licensed under the GNU GPL version 2.
SimplePOP3SSLRetriever:gmail_username@pop.gmail.com:995:
 0 messages (0 bytes) retrieved, 0 skipped

If you receive any errors, correct them. For gmail, for example, you may need to enable POP3 access in your Account Settings if you never done it before. If you get write errors, make sure that the user you’re running it under has the permissions to write both into the maildir storage and into the current directory.

Once the configuration was successfully tested, use it as a template. Copy the getmailrc file into different directories, edit the relevant fields and test them.

Running it regularly

Since you probably don’t want to run getmail manually, I suggest using crontab for this task. Type crontab -e under the mailman user and create the following entry for each account you want to retrieve the mail from:

*/3 * * * *  /usr/bin/getmail -q /usr/bin/getmail  --getmaildir /home/mailman/getmail/gmail >/dev/null 2>&1

this line starts getmail every three minutes, and keeps it completely quiet. If you decide to remove the /dev/null redirections and your Internet connection goes down for a few hours, you’re going to receive a bunch of e-mails every three minutes reporting those connection errors, so beware.

Now you have the mail storage with current mail, and it is time to configure the IMAP server to access it.

 

Email 1 Comment

Your personal GMail-like mail system: converting emails

As mentioned before, the email system uses Maildir format for storage. There are a lot of reasons to choose Maildir and I’m not going to debate about it although technically you can adopt this guide for MBOX storage as well. And since I already had the emails which I want to keep, and since I already had them in MBOX format from Thunderbird, I had to convert them back to Maildir format. If your e-mail storage is in Maildir format already, you can just copy it into two different storages (business maildir and personal maildir) and go on. If you however have MBOX, read on.

There are a lot of different scripts to convert MBOX to Maildir format. None of them frankly supports Thunderbird directly, so you’ll have to try them. I used mb2md.pl from this site, although I still had to modify it as it did not convert all the mail. I suggest you use this one as well unless you find something better which suits your needs.My modifications include better parsing of mboxrd which is the subset Thunderbird uses to store the messages, proper handling of Thunderbird subfolders and ignoring the indexes and empty folders. The modified script could be downloaded here: mb2md-3.20.pl

The conversion script uses the Date::Parse Perl modules which is likely to be missing from your default Perl installation. If you experience the message like that:

Can't locate Date/Parse.pm in @INC (@INC contains: ...
BEGIN failed--compilation aborted at mb2md-3.20.pl line 385.

Then grab your superuser account and install them:

myhost:~ # perl -MCPAN -e "install Date::Parse"
... a lot of output...
/usr/bin/make install -- OK
myhost:~#

The way you run the conversion script is the following:

perl mb2md.pl -R -s ~/.thunderbird/<profile dir>/<mail dir> -d /path/to/<destdir>

Use the absolute path for -d command line option, otherwise the script will create the destdir in your home directory – probably not what you wanted. This will take a while to finish if you have a lot of emails.

Make sure that all emails converted properly, and the number of files in each directory corresponds to the number of emails in the associated folder in Thunderbird. The command

ls -1 <directory> | wc -l

will show you the number of files in a directory. Note that Maildir stores emails in cur directory; new and tmp are only used during e-mail delivery.

You can move the directories around. For example, you can create the business directory and move there all the Maildir directories related to business, and move the rest of the e-mail directories into personal directory.

A side note if you did not convert your email storage because you used the native KMail storage. dovecot uses a different naming structure, so you need to rename your folders to keep the proper structure. For example, the following folder structure:

company/customers/Apple
company/customers/IBM
company/policies

will be represented in KMail Maildir format as:

.company.directory/.customers.directory/Apple/
.company.directory/.customers.directory/IBM/
.company.directory/.policies.directory/

while dovecot requires this structure to be presented as

company.customers.Apple/
company.customers.IBM/
company.policies/

As you see, there is no dot prefix nor .directory suffix in the Dovecot directory representation. And the whole directory structure is represented as a single directory with entries separated by a dot, which means there are no subdirectories deeper than 2nd level nesting. You will probably have to rename them manually as I’m not familiar with any application doing that. However it should be a simple task as there is no possibility of mail loss unless you delete a directory during the rename.

Now you got the e-mail storage, either converted or your native one, you can proceed with the next step which will be dedicated to the installation your IMAP server, dovecot.

Email Comments Off on Your personal GMail-like mail system: converting emails

ColorPreference class for Android

There are many Android color pickers used for preferences, but they all seem to be unnecessary complex for an ordinary user. If all you need is to let the user choose a single color from a set, you can use this simple class which is conveniently integrated into the Preferences scheme.

      

Continue reading »

android 2 Comments

Karaoke Lyrics Editor version 1.3 is released

Version 1.3 of Karaoke Lyrics Editor has been released with several major changes:

  • Audio player switched to FFMpeg/SDL instead of incredibly buggy and unreliable Phonon. No more “cannot play this file” error on Win32.
  • Added improved export in CD+G format with a preview dialog and easily configurable options.
  • Improved the CD+G tester which allows playing any CD+G file, not only the one generated internally. Therefore it is now possible to do several exports, test them all and see which one would suit you the best.
  • Added support for the video and image backgrounds (Edit/Insert picture and Edit/Insert video). Any lyrics can use multiple backgrounds, which will be switched at specified times. Note that all existing lyrics formats are based on text and do not support backgrounds (it will be ignored when exporting to, for example, LRC format). Therefore it is only relevant for CD+G and video export.
  • Added support for video export. You can now export your Karaoke song together with music, lyrics and the background into a video file!
  • Numerous improvements and bugfixes.

See the program page for details.

Karlyriceditor 29 Comments

FontPreference dialog for Android

Using the FontManager class it is possible to implement a simple and nice font selection dialog on Android platform. This is also one of the very few ListView implementation which implements the custom ListView by properly subclassing the DialogPreference the same way ListPreference does. It may also be interesting for anyone who is looking for a way to draw the custom items in the ListPreference style.

     

Anyway, here it is. Apache license.

Continue reading »

android 6 Comments

Your personal GMail-like mail system

My readers remember that I switched to Thunderbird back in December. This was quite difficult experience which included a lot of trial and errors, and although it didn’t result in any e-mail corruption or loss, this was definitely not kind of experience I’d like to go through in future. This made me wonder whether it is possible to avoid it altogether. I wanted a solution which would be:

  • Independent of the e-mail client I am using. If tomorrow I decide to switch to a different e-mail client it should take no more than 15 minutes and should not result in any e-mail conversion or loss.
  • Accessible and synchronized across multiple devices. I want to access all my email from my desktop e-mail client no matter which one of three OS I’m booted in. I want to access it from a Mutt running on my server, from my laptop, from my phone and from my tablet. And I want to see the consistent picture from all of them.
  • Accessible on a single IP/port no matter how many accounts I want to support. This makes it possible to access my e-mail from various Internet cafes, airports, hotels, hospitals and so on which often provide very limited connectivity. For example, blocking the SMTP port is very typical in those environments, and some only provide access to HTTP and HTTPS ports.
  • Flexible, easy to use and setup, working on Linux and backed up regularly on my schedule.

Continue reading »

Email Comments Off on Your personal GMail-like mail system

From KMail to Thunderbird: extra add-ons

The previous posts covered how to add the KMail functionality into Thunderbird. This post describes some other useful Thunderbird plugins, which add even more functionality. I deliberately missed some add-ons which are described everywhere as “must have” and only listed those I use myself and find valuable. Feel free to Google something like “top ten addons for thunderbird” to get the second opinion.

Folder view showing unread folders

Thunderbird supports the view which only shows the folders containing unread messages. However it doesn’t preserve the folder hierarchy in this view. The Advanced Unread Folders add-on adds an extra folder view “Unread (Hierarchy)” into the View-Folders menu. Could also be configured to always show some specific folders as well.
Continue reading »

Email Comments Off on From KMail to Thunderbird: extra add-ons

Enumerating the fonts on Android platform

Some Android applications may find it useful to obtain the full list of the fonts installed on the device together with the font names.They could be used to choose the best matching font for specific purposes, or to let the user choose the font. Unfortunately the Android platform provides no such API in Java.

It is still possible to enumerate all the fonts on the device using the file system operations. The fonts are stored in several directories, the /system/fonts being the most typical. However it would be nice to provide the font name as well, not just the file name. This could be achieved by parsing the TTF file format, which is quite a simple task.

This post provides the source code for the Android font enumerator which is used in the Karaoke player project. The method enumerateFonts() returns the HashMap of all the fonts installed on the device. Each entry in the map has the absolute path to the font as the key, and the font name as a value.
Continue reading »

android 2 Comments

From KMail to Thunderbird: hotkeys

If you have done everything mentioned before, you now brought your Thunderbird functionality on par with what you had in KMail. And probably the only annoying things still remained is the keyboard mapping. Probably more than once you pressed “A”expecting it to “Reply to all” and ended up with an archived message instead. No problem, here you go.
Continue reading »

Email , , Comments Off on From KMail to Thunderbird: hotkeys

From KMail to Thunderbird: system tray icon

One of my personal pet peeves during the first week of using Thunderbird was the lack of system tray nofitication about the new mail. Apparently this functionality is only available on Windows. And I really missed the KMail system tray functionality with the number of unread messages shown right on the system tray icon. And there seem to be no plugin extension which would provide this functionality…

Continue reading »

Email , , , Comments Off on From KMail to Thunderbird: system tray icon