Reverse-engineering the KaraFun file format. Part 2, the directory

In the first part we found out the header format, and that it does not provide us with the directory location. However we know there must be a directory, as the KaraFun application must know where exactly in a file the files are stored, and how large are they. At minimum there should be the directory offset and either the total size or the number of files. At the first thought the DIFW header value may contain the number of files, and the MUSL value contains the directory offset (its value is 0x11D which is after 0x117). However if we check other KaraFun files at the same page, we would see that for some files the MUSL value is less than header length. Therefore it cannot be the offset, and probably is the music length in seconds. Nor DIFW is the number of files. A quick search for the JPEG signature “JFIF” finds out at least three JPG files, so there are more than two files in this archive.

So where it is the directory? Since the header length varies (because it uses the strings with variable length), it could be in one of two places. Either it is at the end of the file (not the case as we saw above), or it is supposed to follow the header directly. Let’s look carefully at the bytes following the header:
Continue reading »

android, reverse engineering Leave a comment

Reverse-engineering the KaraFun file format. Part 1, the header

Several of our users have expressed disappointment that our Ulduzsoft Karaoke Player for Android does not support the popular KaraFun Karaoke format. This format seem to be very popular in some countries, and unfortunately there seem to be no player on Android capable of playing those files. Even the KaraFun Android application does not play those files which is unfortunate. Therefore we decided to add support for this format.

The main issue we had to overcome was lack of any documentation on Internet about this popular format. There is no free open-source software supporting this format either. Therefore to support this file format I had to reverse-engineer it. Fortunately I have the relevant experience, and it was not a very difficult task. Then I decided to document those efforts for the readers to better understand how the reverse engineers work as there seem to be a lot of misunderstanding about the process. All I ultimately needed was a few KaraFun karaoke files. I didn’t even download any KaraFun software, and there was no need to use the editor. The whole format, including the encryption, was reverse-engineered by just looking at the file content.

Hopefully this article would be useful for the people who would like to support KaraFun files in their projects, or just curious about how the reverse engineering of file formats is done.

Continue reading »

android, reverse engineering Leave a comment

Ulduzsoft Karaoke Player for Android is out of Beta now!

Today we released Ulduzsoft Karaoke Player for Android version 1.12. It has been a beta for close to nine months, and it is labor time. The application seem to be well-received, with a competitive functionality and the most bugs seem to be fixed. We had a long discussion about the app monetization and decided to go with a freemium model with the actual app remaining free and ad-supported through Google AdMob, with the option to disable the ads by purchasing the license key which is also available on Android Market.

So it is time for the application to become self-sustained, when any further development depends on the user feedback and the income generated by the application purchases and the ads. Let’s see how well it goes.

android 1 Comment

Android dialog to choose a directory or file based on AlertDialog

This dialog could be used to let the user choose a file or directory. Since it is based on AlertDialog it doesn’t have to be instantiated through startActivity() and therefore could be used, for example, in a PrederenceDialog subclass. The provided code only selects the directories, but it is easy to modify it to select the specific files as well. This code is used in the Ulduzsoft Karaoke Player. Apache license.

Continue reading »

Uncategorized 3 Comments

Parsing ID3v2 tags in the MP3 files

This simple tag parser is very useful when you just need to get the basic information about the MP3 files, such as the title and the artist. Of course it could be extended to extract more information if necessary.

Apache license.
Continue reading »

android 3 Comments

Your personal GMail-like mail system: Sieve support

Dovecot has built-in support for the Sieve mail filtering language. It is very useful to do the server-side email processing such as:

  • Removing unwanted e-mail messages before they are delivered to your inbox;
  • Copying or moving e-mail messages to different folders;
  • Creating vacation autoresponses or any other kind of autoresponses
  • Configuring the actions above depending on message sender, recipient, subject, body, and so on.

This useful functionality is provided by the Dovecot mail delivery agent, dovecot-lda. However since we do not use it for delivery, it is not enabled. So the first step would be to enable it.

Do you have the Sieve plugin?

The Sieve language support plugin is not part of Dovecot source code. It is provided as a separate source which may or may not be packaged by your Linux distro. OpenSUSE packagers did the great job and dovecot bundled with OpenSUSE contains both Sieve and ManageSieve extensions. You may be less lucky in which case you may have to build it yourself.

An easy way to check if you have it is to see whether you have the configuration file /etc/dovecot/conf.d/90-sieve.conf in your dovecot installation. This file only comes with the Sieve plugin, so if you have this file, this means you have the plugin as well.

Enabling the Sieve plugin

The Sieve plugin could be enabled in the Dovecot configuration by editing the /etc/dovecot/conf.d/15-lda.conf file and adding sieve into the list of those mail plugins loaded by default:

protocol lda {
 # Space separated list of plugins to load (default is global mail_plugins).
 mail_plugins = $mail_plugins sieve
}

Enabling the dovecot-lda authentication agent

Unlike the getmail, the dovecot-lda does not know the Maildir path where to deliver the mail. Instead it receives the virtual user name as the command-line parameter, and queries the path from the dovecot authentication process. This makes the configuration simpler, but this also means the authentication should be enabled in the dovecot configuration. This is done by the auth-userdb service specified in the /etc/dovecot/conf.d/10-master.conf file and it is enabled by default. However in our case it needs a little tweaking since we run getmail under our mailman user and it will start the dovecot-lda process under the same user. By default this user does not have access to the dovecot authentication service and will not be able to query the necessary information. Therefore we need to change its permission by editing the auth-userdb section and changing the user the socket permissions as following:

unix_listener auth-userdb {
 mode = 0600
 user = mailman
 #group =
}

Then restart dovecot. The dovecot-lda can now be used.

Reconfiguring getmail to use dovecot-lda for mail delivery

To use the dovecot-lda for mail delivery getmail needs to be reconfigured as it needs to be explicitly told to use the mail delivery agent instead of doing direct delivery. This is done by modifying the [destination] section in the getmailrc configuration file. The new section should look like the one below:

[destination]
type = MDA_external
path = /usr/lib/dovecot/dovecot-lda
arguments = ("-f", "%(sender)", "-d", "pers" )

As you see the path to the destination mailbox is now specified anymore. Instead the -d command-line option followed up by the virtual user name (which in our case is pers) is used to tell the dovecot-lda which user this email is for. The mail directory is looked up through the dovecot authentication agent.

Now send yourself the test e-mail and verify that it gets properly delivered. Check the /var/log/mail in case of any errors, which will likely be triggered by either the wrong virtual username or lack of permissions to access the dovecot authentication process socket.

Managing the Sieve scripts from your e-mail client

Bundled together with Sieve, Dovecot provides a nice way to manage the Sieve scripts right from your e-mail client assuming it has necessary capabilities. KMail can do it natively, Thunderbird can do it with the Sieve add-on.

This functionality must be enabled in Dovecot to be supported, but it is fairly easy to enable. Just edit the /etc/dovecot/conf.d/20-managesieve.conf file and uncomment the line started with protocols:

# Uncomment to enable managesieve protocol:
protocols = $protocols sieve

Then restart dovecot. That’s all.

Sample Sieve script

An example of Sieve script which does various tasks:

require ["fileinto", ["vacation"];
if header :is "From" "john.doe@high.profile.business.com"
{
    # Store business emails from john.doe in a Work/JohnCompany folder
    fileinto "Work.JohnCompany";
}
elsif header :is "From" "buyviagra@spam.com"
{
    # Just delete that spam
    discard;
    stop;
}
elsif header :contains "From" "john.doe@"
{
    # If this is email from john.doe sent from any other of his accounts, move it to Parties folder
    fileinto "Parties";
}
else
{
    # Tell everyone else I'm on vacation
    vacation

    # Reply at most once a day to a same sender
    :days 1

    # The auto-reply subject
    :subject "Out of country: vacation"

    # The reply text
    "I will be out of country for a week.
     Please call my mobile +X XXX XXXXXXXXXX for urgent matters.
Best regards
John Smith";
}

									
Email , , Leave a comment

Your personal GMail-like mail system: the web interface

So you got the system which is as good as GMail, but you also want to have the Web interface. This comes handy when you’re in the middle of changing e-mail clients, or you’re traveling without your laptop and want to check your e-mail from the public Internet cafe. So you want to have the Webmail access to your mail – which, obviously, should not disrupt your regular e-mail flow.

After trying several free solutions I decided to use the RoundCube. It was simple enough to install and configure, and it has the best Web interface I’ve seen so far around free webmail solutions.

This post assumes you have installed and configured the Apache with mod_php5 on your Linux web server, or you know how to do so.

Installing RoundCube

The installation itself is very simple and consist of pretty straightforward steps:

  • Download the latest stable RoundCube version from the project SourceForge page
  • Create a roundcube directory in your web server root and unpack the archive content there. It creates the roundcubemail-0.x directory.
  • Move all the files from there. Make sure you moved .htaccess file as well. Try rmdir roundcubemail-0.* – if you get “Directory not empty” error you forgot it. You should now have the whole installation in your /roundcube directory.
  • Password-protect the directory if you’re security-aware, just in case there are security issues with RoundCube which you won’t find in time about. This could be done by adding the following content to the .htaccess file:
AuthType Basic
AuthName "RoundCube install"
AuthUserFile user.passwd
AuthBasicProvider file
AuthUserFile /path/to/auth/file/outside/the/web/directory
Require valid-user

and create the auth file specified in the AuthUserFile by running htpasswd2. I also suggest making it readable by the web server only:

sudo htpasswd2 -c /path/to/auth/file/outside/the/web/directory roundcube
  • The line below adds the roundcube user with the password you specified. Remember it.
  • Make the roundcube temp and logs directories writable by the web server
<strong><span style="font-size: medium;">Creating the SQLite database</span></strong>

I suggest to use the SQLite database for roundcube. You’re not going to use it often and heavily so there is no need to run the heavy database daemon.

Type the following:

> mkdir sqlite
> sqlite -init SQL/sqlite.initial.sql sqlite/sqlite.db
> sudo chown -R wwwrun sqlite

You need this because to use SQLite the process requires write permission on the directory the database is stored (to create journal files).

The version of RoundCube I installed had an error in the SQL script. If you get the similar error message:

CREATE UNIQUE INDEX ix_searches_user_type_name (user_id, type, name);
SQL error: near "(": syntax error
SQLite version 2.8.17
Enter ".help" for instructions
sqlite>

Just press “Ctrl+D” to exit sqlite prompt, and edit the SQL/sqlite.initial.sql file. Find the following line:

CREATE UNIQUE INDEX ix_searches_user_type_name (user_id, type, name);

and edit it so it looks like that:

CREATE UNIQUE INDEX ix_searches_user_type_name <span style="background-color: #ffd700;">ON searches</span> (user_id, type, name);

then delete the sqlite/sqlite.db file and try again.

Configuring RoundCube

  • Open the browser and point it to http://<your site>/roundcube/installer/
  • Login as the user roundcube with the password you specified above when running the htpasswd2 line.
  • Press the “Start installation” button below.
  • The next screen verifies that you have everything which is needed to run RoundCube. If it lists missing modules, install them. For the “available databases” make sure it says “OK” near SQLite; ignore any error with other databases there. Press Next.
  • In the next screen scroll to the Database setup. Select SQLite in the listbox. Type the full path (including the file name) of the sqlite.db you just created in the previous step there. Leave other fields intact.
  • In the IMAP settings add your hostname/IP as ssl://your.address.com
  • Do the same in SMTP settings assuming you did set up a local SMTP server. For SMTP you have to specify the port, use 587. Leave the smtp_user/smtp_pass empty and check the “Use the current IMAP username and password for SMTP authentication” checkbox below.
  • The next screen will show you two configuration files and asks you to copy them to the server. Download them in your browser and scp them to the web server into the roundcube/config/ directory.
  • Scroll down the screen and find the Continue button somewhere in the middle (not on the bottom). Press it.
  • On this screen it verifies the configuration and database. Should be OK everywhere. You may also test your IMAP/SMTP there. Don’t worry if IMAP port is shown as 143, if you set it as ssl:// it will connect properly.
  • Once everything is fine, go back to your console and delete the installer folder from your roundcube installation.

Using RoundCube

Open the http://your.site.com/roundcube and login with your IMAP credentials. You should be able to login successfully. If RoundCube still says “login failed” and you’re sure your credentials are correct, most likely you forgot to change permissions on the sqlite directory in the database file. Both should be owned by the user your web server is run under.

Once you log in you should see the message list. Messages are opened using the double click. Enjoy!

Email Leave a comment

Your personal GMail-like mail system: SMTP server

In general you do not have to run your own SMTP server, and can use the SMTP servers of your mail providers to send the outgoing mail. However creating your own SMTP server makes your setup complete and much easier to use. You can use the same accounts with the same credentials you used for IMAP, which makes the automatic configuration in the mobile devices a breeze. Also if you send an email through your own SMTP server with Thunderbird it is sent much faster because the connection is local. And it may be important if you travel and find yourself in a place which provides free Internet access but doesn’t let you to connect to the port 25 so you cannot use your provider SMTP server.

I choose Postfix as my SMTP server due to its strong security record and it being installed by default on OpenSUSE 12.1. If it is not installed by default on your Linux distro, it should definitely be part of the setup and you should have no problem installing it using the system-specific package.

Configuring Postfix

Postfix keeps its configuration in the /etc/postfix directory, where we need to edit several configuration files.

/etc/postfix/main.cf

Before editing this file I suggest starting Postfix so OpenSUSE would have its chance to append its own configuration into main.cf. Then open this file and scroll down to the bottom. If you see a lot of parameters there, pay attention when you modify something above it in this file. Only the latest modification applies, and your early modification can be overriden later.

The following parameters need to be changed:

# Should be set to your fully-qualified hostname
myhostname = your.host.name

# Your public IP address the SMTP will be listen on.
inet_interfaces = localhost,your.public.IP.address

# The default relay host which will relay your email in case any other rules fail.
# I specified the SMTP server which is used for my personal correspondence
relayhost = smtp.gmail.com

# Enable authentication for incoming SMTP connections
smtpd_sasl_auth_enable = yes

# Use Dovecot authentication agent, so the same credentials could be used for Postfix
smtpd_sasl_type = dovecot

# Path to the authentication socket
smtpd_sasl_path = private/auth

# Disable anonymous logins even with SSL/TLS
smtpd_sasl_security_options = noanonymous
smtpd_sasl_tls_security_options = noanonymous

# Disable sending mail without authentication, and allow any destination if authenticated
smtpd_client_restrictions = permit_sasl_authenticated, reject_unauth_destination, reject
smtpd_sender_restrictions = permit_sasl_authenticated
smtpd_recipient_restrictions = permit_sasl_authenticated,reject_unauth_destination

# Enable and enforce using TLS for incoming SMTP connections
smtpd_use_tls = yes
smtpd_enforce_tls = yes

# Only allow authentication on secured connections
smtpd_tls_auth_only = yes

# Generated SSL certificates. You can reuse those from Dovecot or generate a different set
smtpd_tls_cert_file = /etc/postfix/ssl/postfix.pem
smtpd_tls_key_file = /etc/postfix/ssl/postfix.key

# Do not ask for a client cert
smtpd_tls_ask_ccert = no

# Do not include the information about connection status into headers
smtpd_tls_received_header = no

#
# Outgoing authentication
#
# Enable outgoing SMTP authentication (this assumes your mail provider
# requires you to authenticate when you send mail)
smtp_sasl_auth_enable = yes

# the password file for the outgoing connections (described later)
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd

# Do not try to send the mail without authentication
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous

# Try to use TLS for outgoing SMTP connections but send without it if the remote doesn't support it
smtp_tls_security_level = may

# Use different mail relays for the different logins
smtp_sender_dependent_authentication = yes
sender_dependent_relayhost_maps = hash:/etc/postfix/sender_relay

/etc/postfix/master.cf

We edit this file to enable Postfix to listen on submission port (587) and on SMTP-SSL port (465). We also disable listening on port 25.

# Comment out this line as well as any other line which starts with "smtp  inet" if any. 
# Do not connect the one which starts which "smtp unix"
#smtp inet n - n - - smtpd
#smtp inet n - n - 1 postscreen

# Add this line to listen on submission port for regular SMTP connections
 submission inet n - n - - smtpd
 -o smtpd_tls_security_level=encrypt
 -o smtpd_sasl_auth_enable=yes

# Add this line to listen on port 465 for SMTP connections over SSL
 urd inet n - n - - smtpd -o smtpd_tls_wrappermode=yes
 -o smtpd_tls_wrappermode=yes
 -o smtpd_sasl_auth_enable=yes

/etc/postfix/sender_relay

This file specifies which SMTP server Postfix will use to send the mail depending on who the e-mail is from. So let’s assume you want your mail which has the sender address bill@mybusiness.com to be sent through the smtp.mybusiness.com:587, mail which has bill@mypersonal.com through the smtp.mypersonal.com and everything else should be sent through your default relayhost. This could be achieved with the following file:

bill@mybusiness.com    [smtp.mybusiness.com]:587
bill@mypersonal.com    smtp.mypersonal.com

Only the mail servers are specified in this file, the authentication credentials are specified in a different file. After you modified this file you must execute the following command to make your changes visible to Postfix:

postmap /etc/postfix/sender_relay

/etc/postfix/sasl_passwd

This file specifies the authentication credentials for those outgoing mail servers which need it. Assume the e-mail sent from bill@mybusiness.com in the example above should use the smart:simple credentials to talk to the associated SMTP server. The bill@mypersonal.com uses myuser:mypassword and your default relayhost (in our case smtp.gmail.com) uses the user:12345 as the username:password you’ll create the following file:

# Per-account information
bill@mybusiness.com  smart:simple
bill@mypersonal.com  myuser:mypassword
# Login information for the default relayhost.
[smtp.gmail.com]     user:12345

After you modified this file you must execute the following command to make your changes visible to Postfix:

postmap /etc/postfix/sasl_passwd

Modifying Dovecot configuration

Dovecot configuration needs also to be modified to add support for Postfix authentication agent. However the modification is quite simple. Edit the file /etc/dovecot/10-master.conf and find the section service auth in this file, and add the following into this section:

# Postfix smtp-auth
unix_listener /var/spool/postfix/private/auth {
   mode = 0660
   user = postfix
   group = postfix
 }

Generating SSL certificates

Since we configured Postfix to use SSL we need to provide it with the certificates. You can use the same certificates you generated for Dovecot, or you can generate a different pair. Copy them into the directory specified in the main.cf configuration file above; defaults are /etc/postfix/ssl/postfix.pem and /etc/postfix/ssl/postfix.key

Then restart dovecot and Postfix, and try to send mail from the different addresses, and see in the logs whether the mail goes to the proper destination.

Email 1 Comment

Your personal GMail-like mail system: dovecot, the IMAP server

Now you have the e-mail storage with current up-to-date messages, and it is time to set up the access to it.

As stated in the design goals, the e-mail storage should be available via IMAP protocol. This allows proper concurrent access to the same e-mail storage while using multiple computers, laptops and the mobile devices. And since the IMAP server is local, the connection is very fast (it doesn’t feel like remote storage at all) and bandwidth is not an issue.

Installing dovecot

I reviewed and tried to install several different IMAP servers and decided to use dovecot. It supports Maildir storage natively, has an extensive set of features, is secure enough to be used by major ISPs and is easy to configure. Also it seems to be included in every major Linux distribution, saving you time and efforts. I installed dovecot20 rpm packaged with OpenSUSE 12.1, which provided the with the version 2.0.14. Although it also instals in tow three SQL backends, this  isn’t an issue as you don’t have to use them.

Before processing with dovecot configuration you need to generate the SSL certificate. Do it now.

Configuring dovecot

Find your dovecot configuration directory. Most likely it gonna be /etc/dovecot, but may be different in your case. It has the dovecot.conf file as well as conf.d directory with a bunch of other files which can on the first glance scare even an experienced system administrator. However it is fairly simple to configure, and I will explain the options you need to change.

Each section below explains which parameters were changed from their defaults, and why.

/etc/dovecot/dovecot.conf

Here we set up the supported protocols and IP addresses (but not port numbers) to listen on.

# Uncomment protocols as we only server IMAP, no POP3/LMTP
protocols = imap

# If you have a multi-homed host and want to listen only on specific IP addresses, 
# uncomment and fill up listen as well
listen = 192.168.0.1,1.2.3.4

/etc/dovecot/conf.d/10-auth.conf

Here we set up the authentication mechanisms we will use for the user authentication. We also enable some options to make the installation more secure.

# Uncomment it as we want to disable plaintext auth on non-encrypted connections
disable_plaintext_auth = yes

# Lowercase the username before doing the database lookup. Very helpful with
# mobile devices which tend to capitalize the username when it is entered
auth_username_format = %Lu

# We will use file-based authentication with virtual users, so uncomment this line
!include auth-passwdfile.conf.ext

/etc/dovecot/conf.d/10-logging.conf

Here we temporarily enable more detailed logging to help us debugging the various configuration problems. Keep the copy of this file so you can revert it back once the server is working and the issues are ironed out.

auth_verbose = yes
mail_debug = yes

/etc/dovecot/conf.d/10-master.conf

Here you can change the port number IMAP is listenin on (inet_listener imaps entry) if you do not like the default 993 port.

/etc/dovecot/conf.d/10-ssl.conf

Our SSL configuration for IMAP. Here we go!

# We require SSL
ssl = required

# SSL certificate and the private key we generated before
ssl_cert = </etc/dovecot/ssl/certificate.pem
ssl_key = </etc/dovecot/ssl/certificate.key

/etc/dovecot/conf.d/20-imap.conf

Here you can enable workarounds for some buggy clients. Howeveer as you use Maildir there are no necessary changes.

/etc/dovecot/conf.d/auth-passwdfile.conf.ext

Here we tell Dovecot that we store our virtual users in the file /etc/dovecot/users and we will use MD5 passwords

passdb {
 driver = passwd-file
 args = scheme=MD5 username_format=%u /etc/dovecot/users
}

userdb {
 driver = passwd-file
 args = username_format=%u /etc/dovecot/users
}

Setting up virtual users

Virtual user file format maps the virtual users authenticated by dovecot into system user IDs. Since we use the same system user and home directory for all our virtual users, we will return the same user ID but different mail directories. So let’s configure two users, pers with the password personal and busi with the password business, to access the personal and business mail storage respectively.

First we need to find the user and group ID of our mailman user:

# grep mailman /etc/passwd
mailman:x:1001:100::/home/mailman:/bin/bash

so the user ID is 1001 and group ID is 100. Then we need to encode the passwords. Since we specified the MD5 as password scheme, we need to store the MD5 hashes of our passwords. So let’s find out them:

# echo -n "personal" | md5sum -
0be5a6c82893ecaa8bb29bd36831e457 -
# echo -n "business" | md5sum -
f5d7e2532cc9ad16bc2a41222d76f269 -

Now we have all necessary information to create our /etc/dovecot/users file. Its format is similar to /etc/passwd, as it has the following fields separated by a colon:

  • Virtual username
  • MD5 password
  • User ID of the system user which is used to access the mailbox, in our case it is 1001
  • Group ID of the system user which is used to access the mailbox, in our case it is 100
  • An empty field
  • Home directory of the mail storage. When combined with the value of userdb_mail parameter should produce a complete path to the mail storage root.
  • An empty field
  • Other options. In our case we need to use the userdb_mail option to specify the directory where the mail is stored inside the home directory described above. We also specify the mail storage format maildir.

So we end up with the following file:

pers:0be5a6c82893ecaa8bb29bd36831e457:1001:100::/home/mailman/storage/::userdb_mail=maildir:~/personal
busi:f5d7e2532cc9ad16bc2a41222d76f269:1001:100::/home/mailman/storage/::userdb_mail=maildir:~/business

Now we can start dovecot, which is done via the way supported by your Linux distro. For RedHat, CentOS and OpenSUSE it is done by running the startup script:

/etc/init.d/dovecot start

and try to connect to your IMAP server using the credentials.

Troubleshooting

If the installation doesn’t work, the first thing to check is the system log file. Dovecot writes into the /var/log/mail file, which contains the valuable information about problems encountered by your IMAP server. Most common errors would likely be authentication failed (verify the MD5 password checksum and that you copied the whole string into the users file) and lack of permissions to access the mailbox (make sure the user ID specified in the users file belongs to the user which has permissions to access the mailbox)

Manual test

If the problem still persists, it may help to attempt to connect to the IMAP server manually. This may help to diagnose the problem better.

First use openssl to connect to the IMAP server:

# openssl s_client -connect my.ip.address:993
.. a lot of text output from openssl ...
* OK [CAPABILITY ...] Dovecot ready.

If the connection failed, either the IMAP server did not start up properly (check the log files), it doesn’t listen on the proper IP address (check the configuration), or the connection is firewalled and you need to open the port in the firewall.

Then we can try to log in using our credentials (note the first dot, it is required):

. LOGIN pers personal

. OK [CAPABILITY IMAP4rev1 ...other capabilties]

If login didn’t success, you have authentication problem – check the log files.

Now you can try to list the folders:

. LIST "" "*"

* LIST (\HasNoChildren) "." "Sent"
* LIST (\HasNoChildren) "." "Drafts"
* LIST (\HasNoChildren) "." "INBOX"
. OK List completed.

Your list may be different, but if the boxes are listed, your IMAP server is likely to work properly, and the problem is in your e-mail client configuration.

 

Email Leave a comment

Your personal GMail-like mail system: fetching with getmail

Now your e-mail storage is converted and you are ready to go. Hold on, it is time to do some preparation work. While not strictly necessary, I suggest you you figure up and write down the following information:

  • Which user id will own the mail storage files. It will also run the mail retriever and (in future) the delivery agent. I suggest creating a dedicated UID for it.
  • Which top directories would contain your e-mails.

I decided to create the user mailman, and create the storage in its home directories under the root user:

# useradd -m mailman
# mkdir -p /home/mailman/storage/business
# mkdir -p /home/mailman/storage/personal
# cp -r <old business email storage path> /home/mailman/storage/business/
# cp -r <old personal email storage path> /home/mailman/storage/personal/
# chown -R mailman /home/mailman/storage/

As you see, the mail is stored in two different directories inside the storage directory, and all the files are owned by the mailman user.

Once you set up the storage and the user, you’re ready to setup the mail fetching with getmail.

Installing getmail

Getmail is a Python script which retrieves e-mails from your mail providers and stores them into the Maildir folders according to configuration. You can use procmail instead, but I found getmail easier to configure and its ability to deliver to Maildir mail boxes natively is very useful in the early stages of building the e-mail system.

Installing getmail is typically as easy as installing the relevant package from your Linux distribution. However if your distro doesn’t provide one, you can always go to the getmail website and download it. Unpack it, and build and install it as described in README under the regular user:

> cd getmail-4*
> python setup.py build
> sudo python setup.py install

Configuring getmail

Getmail stores its configuration together with the internal cache files in a directory. Which means for each account you want to retrieve the mail from you will create a directory with the getmail configuration file getmailrc in it. So for my three mail providers I create three directories:

# su - mailman
> mkdir -p getmail/personal
> mkdir -p getmail/business
> mkdir -p getmail/gmail

and in each of those directories I created the getmailrc file. Since all my providers support POP3 retriever over SSL, the content of the getmailrc file was very similar to each of them. Note that if your provider requires you to use IMAP, you need to use a different retriever so check the getmail configuration.

Here’s the getmailrc configuration I use to retrieve email from my gmail account, which is stored in the /home/mailman/getmail/gmail/getmailrc file:

[retriever]
type = SimplePOP3SSLRetriever
server = pop.gmail.com
username = <gmail username>
password = <gmail password>

[destination]
type = Maildir
path = /home/mailman/storage/personal/

[options]
read_all = 0
delete = 0
delete_after = 3

As you see, for the sites which support POP3 over SSL all you need is to change the server, username and password in the [retriever] section. If the e-mail from this specific site should be delivered into a different storage (i.e. business), you also need to change the path in [destination] section but that’s all.

If your site supports POP3 but does not support SSL, use the type=SimplePOP3Retriever instead of using SimplePOP3SSLRetriever. If the POP3 server listens on a non-standard port number like 11000, you can also add a port=11000 parameter into this section. See the getmail documentation.

The options I use is to keep the mail on the remote server for three days, then delete it. This ensures the remote mailbox is not likely to overflow with mail, and provides me with some safety net knowing I can still access my recent email directly.

Testing your configuration

To test your configuration run getmail with the –getmaildir argument followed by the path to your getmail directory under the mailman user which can write into the maildirs:

> /usr/bin/getmail  --getmaildir /home/mailman/getmail/gmail

and you will see the output similar to the one below:

Copyright (C) 1998-2009 Charles Cazabon. Licensed under the GNU GPL version 2.
SimplePOP3SSLRetriever:gmail_username@pop.gmail.com:995:
 0 messages (0 bytes) retrieved, 0 skipped

If you receive any errors, correct them. For gmail, for example, you may need to enable POP3 access in your Account Settings if you never done it before. If you get write errors, make sure that the user you’re running it under has the permissions to write both into the maildir storage and into the current directory.

Once the configuration was successfully tested, use it as a template. Copy the getmailrc file into different directories, edit the relevant fields and test them.

Running it regularly

Since you probably don’t want to run getmail manually, I suggest using crontab for this task. Type crontab -e under the mailman user and create the following entry for each account you want to retrieve the mail from:

*/3 * * * *  /usr/bin/getmail -q /usr/bin/getmail  --getmaildir /home/mailman/getmail/gmail >/dev/null 2>&1

this line starts getmail every three minutes, and keeps it completely quiet. If you decide to remove the /dev/null redirections and your Internet connection goes down for a few hours, you’re going to receive a bunch of e-mails every three minutes reporting those connection errors, so beware.

Now you have the mail storage with current mail, and it is time to configure the IMAP server to access it.

 

Email 1 Comment