Category: Tech

Technologies

Web drop box

This is something that my users have been clamoring for, for a long time now.

It is the ability to share large files with external clients, but in a secure manner, e.g. encrypted, access control, time limited, etc.

I don’t want these files on my web server, e.g. www.mycompany.com site. And putting up FTP/SFTP servers is a pain (think, unsophisticated users).

I know there are companies out there offering these services – box.net, drop.io, mediafire, foldershare, MS Live, Google, etc. But I want something under my control.

So far I’ve found few and far between, unless I want to roll my own (always an option). The closest that I’ve come to what I need at the moment is AjaXplorer. There are limitations, but it is usable.

Anyone got other suggestions?

8/22/2010

I’ve found a few, I’ll add to this post over time (with comments if I’ve tried it).

  • http://www.mollify.org/features.php  – Mollify (AJAX interface, similar to DropBox.  Open Source, free for personal use).
  • http://www.ajaxplorer.info/wordpress/ – AjaxPlorer (AJAX interface, I’ve used this for a few clients.  It’s ok.  It’s also used in LG N4B1 NAS box).

Moving RAID 10 from one Dell R410 to another

Spent all of last week fighting fires.  We have a production server that was suffering software (application bugs) and hardware failures.  Crashing left and right, I got very little sleep responding to my pager and going online to restart the app and/or server.

The app was using too much memory (Java app) the server just can’t take more memory (we already have 32GB in it).  So we decided to throw more hw at it.  App and Postgresql was running on it (yes, I know, bad, bad, bad design — my excuse is that it was not me that set this up, I joined later).  Anyway, brought up a new, faster server (Dell R410) and moved Java app over on to it, leaving Postgresql on the old server.  The plan is that if we run into problem, it’s easy to move right back to old server.  Also easier quicker this way, no down time to take down DB, copy data over, etc.   Besides which, the DB is currenlty over 65GB, will take a while to copy over.

Well, guess what…. the new R410 started experiencing hw problem!  I have RAID 10 setup on the 4 drives.  Drives 1 & 2 (one of each RAID1 element) faulted, CRAP!  Swapped drives.  Still faulting.  I get message from the kernel (dmesg) that it kept having to rescan the SAS bus as the drives kept dropping out.  (Running CentOS 5.2 64 bit).

Talked with Dell support…. ah, what a pain in the rear they are.  They insisted that it was a firmware issue!!!!  Google for “Dell RAID controllers rejecting non-DELL drives”.  We paid for same day support and we want support now!  After a couple hours on the phone, we got them to agree to swap motherboard and RAID controller the next day.

In the mean time, we have another R410 sitting the same rack (but in use).  The apps on it can be move to another server though.  So I spent a couple hours at the data center moving the drives from failing R410 over to the other one.  I was afraid there might be problem because the current state of the RAID is degraded (2 drives in the RAID10 faulted and still syncing).  But it worked like a charm.  Shut down both systems, swapped drives (two at a time, drive 0, drive 1, drive2, drive3 so I don’t mess up).  Bring up the good R410….

It came up fine.  Saw the new RAID drives and asked if I want to import foreign config.  Said yes, and press Ctrl-R anyway so I can check and the RAID controller saw the RAID10.  It told me that the two drives are syncing.  Great, exit out and reboot.

Then I noticed that this system only have 16GB RAM…. aw CRAP!  Shut it down, pull them both off the rack, open the case, swapped DIMMs.  Put them both back in, boot up the good one…. hold my breath…..  and YES, it came up, 32GB, saw the RAID drives…

Once I got the login: prompt, login, check around, making sure everything is there.  Realize that the network is not up.  Spent a couple panic stricken minutes checking cable, switch ports, etc.  Then I remember that with RedHat (and CentOS) the ifcfg-ethN script is updated at boot and uses the MAC address.  Since I moved the drives to another server, the MAC changed and RH/CentOS noticed that the MAC address in existing ifcfg-ethN does not match current MAC, it updated those files.  Luckily it renamed the existing one to ifcfg-ethN.old.

I fired up vi and updated the old ifcfg-ethN.old file with new MAC address, rename them back to ifcfg-ethN (eth0 and eth1).  Bring them down and back up (ifdown eth0, then ifup eth0) and the network is up.

Reboot the server just to be sure that everything work, login and start up the app.  Checked from an external address (ssh to my home server, point my browser to squid at home) via a browser that the app is running and acessible from the outside world.

I’ve done this before, e.g. moving entire RAID (it was RAID1 and RAID5) from one Dell server to another identical hw Dell server.  So I know it works.  Only difference was the degraded mode of the RAID, but I am glad that it worked fine too.

Chromium OS

Like everyone else, I just had to try it out.  Since I also have a couple of EEEpc netbooks around (2 900A, and a 1000HE), plenty to test on.

Looks like Google is mainly Ubuntu on the engineering side.  I spent a little time trying to get Chromium to build under FC12 and just too much trouble for a side project.  I setup an Ubuntu 9.10 VM, followed Google’s online instruction to git d/l the source and built ChromiumOS.

So far, it’s kinda fun and I can see the use for it.  It came right up on my eeePC 900A, see all the hw, wired network, wifi network, buttons work.  Nice!

It’s just Linux underneath, and with this build, I have access to the cli via Alt-Ctrl-T key combo.

There are already plenty of people providing the live images.  If anyone is interested in my build, let me know and I’ll put a copy online.

I’ll have more when I have time to play with this.

sieve filtering

Here is some basic sieve filtering (under dovecot, should be same for other imap server though).

I am using FC11, each account can have their own sieve filter.  The default is ${HOME}/.dovecot.sieve which is a text file.

Here is a snippet of my .dovecot.sieve file.

NOTE: you need the sieve plugins (envelope, subaddress, regex, relational)


require ["reject", "fileinto", "envelope", "subaddress", "regex", "relational"];
if  anyof (
header :contains ["Return-path"] "[email protected]",
header :contains ["List-Id"] "Firewall Wizards Security Mailing List",
header :contains ["List-Post"] "mailto:[email protected]") {
fileinto "lists.firewall";
} elsif anyof (
header :contains ["X-Mailing-List"] "pgsql-general",
header :contains ["List-ID"] "pgsql-general.postgresql.org",
header :contains ["X-Mailing-List"] "pgsql-performance",
header :contains ["List-ID"] "pgsql-performance.postgresql.org") {
fileinto "lists.postgresql";
} elsif anyof (
header :contains ["X-BeenThere"] "rubyonrails-talk",
header :contains ["List-ID"] "rubyonrails-talk.googlegroups.com") {
fileinto "lists.rails";
} else {
# The rest goes into INBOX
# default is "implicit keep", we do it explicitly here
keep;
}

Converting from mbox to maildir, procmail to deliver+dovecot+sieve

I’ve been wanting to do this for a while, but too busy, too lazy and finally too worry that I might lose my mail archive…. etc…etc…etc..

Over last weekend, I bit the bullet and did it.  Partly because my DSL line was down, so there was no network activities, no mail, so might as well take advantage of the down time.  My next project is to drop AT&T DSL, I am tired of their bs.

Not meaning to get off the subject, but I am one of the earliest DSL sub in the neighborhood.  Got mine in ’96 (only 768k down/256K up).  I am about ~14-15000 ft from nearest CO, limit of DSL line.  Back then, Pacbell promised they will upgrade facility in the area so I can get faster speed.  Fast forward to 2009, 14 years later, still same old… and still crappy service.

Ok, back to converting from mbox+procmail+dovecot to maildir+deliver+dovecot+sieve.  My mail archive goes back a long way, still have emails from 1987 (UUCP/Usenet mail).  I’ve always used mbox format, as that was the default then, and since I never used qmail….

I do separate my mail into folders (just another mbox file) of course, otherwise my main mailbox would be in the gigabyte range.  I use procmail to filter incoming email into these separate boxes, to automate dealing with requests, to bounce spam, etc.  My .procmailrc is around 30K in size with lots of recipes, and also pulling in dozens of include files.

I knew that mbox is inefficient and that procmail is killing my system I/O, but sheer inertia kept me where I am.

After some research, I found mb2md.pl — a Perl script — that will do most of the grunt work in converting an mbox file into maildir.   I am using Postfix for my MTA and dovecot for IMAP, so support for maildir is already there.  Supposedly, procmail supports maildir, but…

Here is the basic HOW-TO for converting mbox to maildir.

Basics: I am using Postfix, dovecot on FC11

1. Stop postfix and dovecot.

2. Install dovecot-sieve (and dovecot-managesieve if you have users beside yourself)

3. Edit /etc/postfix/main.cf and change mailbox_command to /usr/libexec/dovecot/deliver

mailbox_command = /usr/libexec/dovecot/deliver -a “$RECIPIENT”

The -a “$RECIPIENT” part is if you want to use [email protected]

4. Edit /etc/dovecot.conf

  • look for protocol lda { section and make sure this is there, add/uncomment/edit if not
  • # Support for dynamically loadable plugins. mail_plugins is a space separated
    # list of plugins to load.
    mail_plugins = sieve
    mail_plugin_dir = /usr/lib/dovecot/lda
  • look for plugin { section and look for these lines:
  • # Sieve plugin (http://wiki.dovecot.org/LDA/Sieve) and ManageSieve service
    #
    # Location of the active script. When ManageSieve is used this is actually
    # a symlink pointing to the active script in the sieve storage directory.
    sieve=~/.dovecot.sieve
    #
    # The path to the directory where the personal Sieve scripts are stored. For
    # ManageSieve this is where the uploaded scripts are stored.
    sieve_dir=~/sieve
  • Make sure /usr/libexec/dovecot/deliver exists and is executable.

5. If your mbox is in /var/spool/mail/yourname (or in /var/mail/yourname), i.e. mine is /var/spool/mail/tin, then you can run the following command to convert that mbox into maildir.

mb2md -s /var/spool/mail/tin

mb2md will automaticaly create a directory named Maildir in your home directory (i.e. it created /home/tin/Maildir for me).  The converted maildir files will go in there (from /var/spool/mail/tin).

My filtered incoming mail (via procmail) is in the mail directory of my home directory, i.e. /home/tin/mail.  There are multiple directories and multiple mbox files in there.

There are also directories such as .imap directory, created by dovecot.  Anything there that is not a valid mbox file will be ignored by mb2md.

So now run

mb2md -s /home/tin/mail -R

That should recursively travel into your mail folder and convert all mbox files there into maildir format in your Maildir directory.

I did found some mbox files that was ignored or failed to convert.  But manually running mb2md on those file worked, e.g. mb2md -s /home/tin/mail/missed-mbox-file.

To make sure there won’t be an accident, I renamed my mail folder to mail.old.  Then start up postfix, dovecot.

I’ll add more information about the .dovecot.sieve file later.