sieve filtering

Here is some basic sieve filtering (under dovecot, should be same for other imap server though).

I am using FC11, each account can have their own sieve filter.  The default is ${HOME}/.dovecot.sieve which is a text file.

Here is a snippet of my .dovecot.sieve file.

NOTE: you need the sieve plugins (envelope, subaddress, regex, relational)


require ["reject", "fileinto", "envelope", "subaddress", "regex", "relational"];
if  anyof (
header :contains ["Return-path"] "[email protected]",
header :contains ["List-Id"] "Firewall Wizards Security Mailing List",
header :contains ["List-Post"] "mailto:[email protected]") {
fileinto "lists.firewall";
} elsif anyof (
header :contains ["X-Mailing-List"] "pgsql-general",
header :contains ["List-ID"] "pgsql-general.postgresql.org",
header :contains ["X-Mailing-List"] "pgsql-performance",
header :contains ["List-ID"] "pgsql-performance.postgresql.org") {
fileinto "lists.postgresql";
} elsif anyof (
header :contains ["X-BeenThere"] "rubyonrails-talk",
header :contains ["List-ID"] "rubyonrails-talk.googlegroups.com") {
fileinto "lists.rails";
} else {
# The rest goes into INBOX
# default is "implicit keep", we do it explicitly here
keep;
}

Converting from mbox to maildir, procmail to deliver+dovecot+sieve

I’ve been wanting to do this for a while, but too busy, too lazy and finally too worry that I might lose my mail archive…. etc…etc…etc..

Over last weekend, I bit the bullet and did it.  Partly because my DSL line was down, so there was no network activities, no mail, so might as well take advantage of the down time.  My next project is to drop AT&T DSL, I am tired of their bs.

Not meaning to get off the subject, but I am one of the earliest DSL sub in the neighborhood.  Got mine in ’96 (only 768k down/256K up).  I am about ~14-15000 ft from nearest CO, limit of DSL line.  Back then, Pacbell promised they will upgrade facility in the area so I can get faster speed.  Fast forward to 2009, 14 years later, still same old… and still crappy service.

Ok, back to converting from mbox+procmail+dovecot to maildir+deliver+dovecot+sieve.  My mail archive goes back a long way, still have emails from 1987 (UUCP/Usenet mail).  I’ve always used mbox format, as that was the default then, and since I never used qmail….

I do separate my mail into folders (just another mbox file) of course, otherwise my main mailbox would be in the gigabyte range.  I use procmail to filter incoming email into these separate boxes, to automate dealing with requests, to bounce spam, etc.  My .procmailrc is around 30K in size with lots of recipes, and also pulling in dozens of include files.

I knew that mbox is inefficient and that procmail is killing my system I/O, but sheer inertia kept me where I am.

After some research, I found mb2md.pl — a Perl script — that will do most of the grunt work in converting an mbox file into maildir.   I am using Postfix for my MTA and dovecot for IMAP, so support for maildir is already there.  Supposedly, procmail supports maildir, but…

Here is the basic HOW-TO for converting mbox to maildir.

Basics: I am using Postfix, dovecot on FC11

1. Stop postfix and dovecot.

2. Install dovecot-sieve (and dovecot-managesieve if you have users beside yourself)

3. Edit /etc/postfix/main.cf and change mailbox_command to /usr/libexec/dovecot/deliver

mailbox_command = /usr/libexec/dovecot/deliver -a “$RECIPIENT”

The -a “$RECIPIENT” part is if you want to use [email protected]

4. Edit /etc/dovecot.conf

  • look for protocol lda { section and make sure this is there, add/uncomment/edit if not
  • # Support for dynamically loadable plugins. mail_plugins is a space separated
    # list of plugins to load.
    mail_plugins = sieve
    mail_plugin_dir = /usr/lib/dovecot/lda
  • look for plugin { section and look for these lines:
  • # Sieve plugin (http://wiki.dovecot.org/LDA/Sieve) and ManageSieve service
    #
    # Location of the active script. When ManageSieve is used this is actually
    # a symlink pointing to the active script in the sieve storage directory.
    sieve=~/.dovecot.sieve
    #
    # The path to the directory where the personal Sieve scripts are stored. For
    # ManageSieve this is where the uploaded scripts are stored.
    sieve_dir=~/sieve
  • Make sure /usr/libexec/dovecot/deliver exists and is executable.

5. If your mbox is in /var/spool/mail/yourname (or in /var/mail/yourname), i.e. mine is /var/spool/mail/tin, then you can run the following command to convert that mbox into maildir.

mb2md -s /var/spool/mail/tin

mb2md will automaticaly create a directory named Maildir in your home directory (i.e. it created /home/tin/Maildir for me).  The converted maildir files will go in there (from /var/spool/mail/tin).

My filtered incoming mail (via procmail) is in the mail directory of my home directory, i.e. /home/tin/mail.  There are multiple directories and multiple mbox files in there.

There are also directories such as .imap directory, created by dovecot.  Anything there that is not a valid mbox file will be ignored by mb2md.

So now run

mb2md -s /home/tin/mail -R

That should recursively travel into your mail folder and convert all mbox files there into maildir format in your Maildir directory.

I did found some mbox files that was ignored or failed to convert.  But manually running mb2md on those file worked, e.g. mb2md -s /home/tin/mail/missed-mbox-file.

To make sure there won’t be an accident, I renamed my mail folder to mail.old.  Then start up postfix, dovecot.

I’ll add more information about the .dovecot.sieve file later.

Updates on ODV

2/26/10 Walmart to acquire VUDU

This is so funny.  All of a sudden I got a bunch of IM pings and email from ex-VUDU people asking me if I had heard about “it” yet….  What was “it”?

That turned out to be the BIG news that Walmart is buying VUDU, lock, stock and barrel!  I guess VUDU will survive with Walmart’s muscle behind it… or will it?  I just don’t think Walmart is a risk taking company, nor do I think that it is innovative.  I would love to be proven wrong though.

There are so many companies now in this space.  The space itself is getting fragmented.  We have:

  • Server based video streaming over Internet – this is divided into those requiring a box/appliance vs those streaming to your computer.  Some provide services to both (Amazon, Blockbuster, Netflix, etc.).
    • Require Appliance
      1. Amazon
      2. Blockbuster
      3. Netflix
      4. TiVo
      5. VUDU
    • Stream to computer
      1. BBC
      2. Hulu
  • Video streaming OTA (Over The Air)
  • Video streaming over cable (traditional like Comcast, Charter, Time Warner, etc.)
  • Video streaming over satelite (Dish, DirecTV, etc.)

9/15/09

I believe VUDU is getting out of the hardware business. They’ve shifted their focus to the embedded business. VUDU set top box is getting harder to find from what I hear on the web. VUDU have also announced a number of embedded wins — Entone STB, LG (BD Blue Ray and HDTVs), Mitsubishi HDTVs.

It make sense to me. I’ve always thought it was dumb to try to break into the consumer hardware business when you don’t have the deep pocket to keep you going through the first few years. It’s better to license the designs and software.

Dec 4, 2009

It has been confirmed that VUDU is out of the hw business.  They’ve closed off their production facilities and stop selling new STB (you might be able to find places that still have stocks, but once they are out, that’s it!).

They’ve gone to the embedded model.  Also seem to be changing from P2P to streaming as the embedded devices generally frown on P2P.  On one hand, it’s good because of less infrastructure to maintain on the VUDU side, also less support (P2P has a whole host of issues with regards to firewall, home routers, etc. not to mention blocking by ISPs such as Comcast).  On the con side, this requires the clients to have higher bandwidth.

Btw, less infrastructure for VUDU because they can use CDNs such as Akamai, Limewire, etc.  It’s harder to distribute to CDN with P2P, trust me, it’s harder.  When I was there, I was looking into how to do this…. and it was not easy.

So, keep your eyes out for VUDU to be embedded in more devices.   At this point, I can’t decide if VUDU has done enough to save whatever market they have or if it’s too late.  I’ve just bought a WD TV Live (media center device) that let’s me play pretty much any video formats out there, along with music and in HD to boot.  I’ll have more to talk about regarding the WDTV in another post.