Category: Tech

Technologies

Moving RAID 10 from one Dell R410 to another

Spent all of last week fighting fires.  We have a production server that was suffering software (application bugs) and hardware failures.  Crashing left and right, I got very little sleep responding to my pager and going online to restart the app and/or server.

The app was using too much memory (Java app) the server just can’t take more memory (we already have 32GB in it).  So we decided to throw more hw at it.  App and Postgresql was running on it (yes, I know, bad, bad, bad design — my excuse is that it was not me that set this up, I joined later).  Anyway, brought up a new, faster server (Dell R410) and moved Java app over on to it, leaving Postgresql on the old server.  The plan is that if we run into problem, it’s easy to move right back to old server.  Also easier quicker this way, no down time to take down DB, copy data over, etc.   Besides which, the DB is currenlty over 65GB, will take a while to copy over.

Well, guess what…. the new R410 started experiencing hw problem!  I have RAID 10 setup on the 4 drives.  Drives 1 & 2 (one of each RAID1 element) faulted, CRAP!  Swapped drives.  Still faulting.  I get message from the kernel (dmesg) that it kept having to rescan the SAS bus as the drives kept dropping out.  (Running CentOS 5.2 64 bit).

Talked with Dell support…. ah, what a pain in the rear they are.  They insisted that it was a firmware issue!!!!  Google for “Dell RAID controllers rejecting non-DELL drives”.  We paid for same day support and we want support now!  After a couple hours on the phone, we got them to agree to swap motherboard and RAID controller the next day.

In the mean time, we have another R410 sitting the same rack (but in use).  The apps on it can be move to another server though.  So I spent a couple hours at the data center moving the drives from failing R410 over to the other one.  I was afraid there might be problem because the current state of the RAID is degraded (2 drives in the RAID10 faulted and still syncing).  But it worked like a charm.  Shut down both systems, swapped drives (two at a time, drive 0, drive 1, drive2, drive3 so I don’t mess up).  Bring up the good R410….

It came up fine.  Saw the new RAID drives and asked if I want to import foreign config.  Said yes, and press Ctrl-R anyway so I can check and the RAID controller saw the RAID10.  It told me that the two drives are syncing.  Great, exit out and reboot.

Then I noticed that this system only have 16GB RAM…. aw CRAP!  Shut it down, pull them both off the rack, open the case, swapped DIMMs.  Put them both back in, boot up the good one…. hold my breath…..  and YES, it came up, 32GB, saw the RAID drives…

Once I got the login: prompt, login, check around, making sure everything is there.  Realize that the network is not up.  Spent a couple panic stricken minutes checking cable, switch ports, etc.  Then I remember that with RedHat (and CentOS) the ifcfg-ethN script is updated at boot and uses the MAC address.  Since I moved the drives to another server, the MAC changed and RH/CentOS noticed that the MAC address in existing ifcfg-ethN does not match current MAC, it updated those files.  Luckily it renamed the existing one to ifcfg-ethN.old.

I fired up vi and updated the old ifcfg-ethN.old file with new MAC address, rename them back to ifcfg-ethN (eth0 and eth1).  Bring them down and back up (ifdown eth0, then ifup eth0) and the network is up.

Reboot the server just to be sure that everything work, login and start up the app.  Checked from an external address (ssh to my home server, point my browser to squid at home) via a browser that the app is running and acessible from the outside world.

I’ve done this before, e.g. moving entire RAID (it was RAID1 and RAID5) from one Dell server to another identical hw Dell server.  So I know it works.  Only difference was the degraded mode of the RAID, but I am glad that it worked fine too.

Chromium OS

Like everyone else, I just had to try it out.  Since I also have a couple of EEEpc netbooks around (2 900A, and a 1000HE), plenty to test on.

Looks like Google is mainly Ubuntu on the engineering side.  I spent a little time trying to get Chromium to build under FC12 and just too much trouble for a side project.  I setup an Ubuntu 9.10 VM, followed Google’s online instruction to git d/l the source and built ChromiumOS.

So far, it’s kinda fun and I can see the use for it.  It came right up on my eeePC 900A, see all the hw, wired network, wifi network, buttons work.  Nice!

It’s just Linux underneath, and with this build, I have access to the cli via Alt-Ctrl-T key combo.

There are already plenty of people providing the live images.  If anyone is interested in my build, let me know and I’ll put a copy online.

I’ll have more when I have time to play with this.

sieve filtering

Here is some basic sieve filtering (under dovecot, should be same for other imap server though).

I am using FC11, each account can have their own sieve filter.  The default is ${HOME}/.dovecot.sieve which is a text file.

Here is a snippet of my .dovecot.sieve file.

NOTE: you need the sieve plugins (envelope, subaddress, regex, relational)


require ["reject", "fileinto", "envelope", "subaddress", "regex", "relational"];
if  anyof (
header :contains ["Return-path"] "[email protected]",
header :contains ["List-Id"] "Firewall Wizards Security Mailing List",
header :contains ["List-Post"] "mailto:[email protected]") {
fileinto "lists.firewall";
} elsif anyof (
header :contains ["X-Mailing-List"] "pgsql-general",
header :contains ["List-ID"] "pgsql-general.postgresql.org",
header :contains ["X-Mailing-List"] "pgsql-performance",
header :contains ["List-ID"] "pgsql-performance.postgresql.org") {
fileinto "lists.postgresql";
} elsif anyof (
header :contains ["X-BeenThere"] "rubyonrails-talk",
header :contains ["List-ID"] "rubyonrails-talk.googlegroups.com") {
fileinto "lists.rails";
} else {
# The rest goes into INBOX
# default is "implicit keep", we do it explicitly here
keep;
}

Converting from mbox to maildir, procmail to deliver+dovecot+sieve

I’ve been wanting to do this for a while, but too busy, too lazy and finally too worry that I might lose my mail archive…. etc…etc…etc..

Over last weekend, I bit the bullet and did it.  Partly because my DSL line was down, so there was no network activities, no mail, so might as well take advantage of the down time.  My next project is to drop AT&T DSL, I am tired of their bs.

Not meaning to get off the subject, but I am one of the earliest DSL sub in the neighborhood.  Got mine in ’96 (only 768k down/256K up).  I am about ~14-15000 ft from nearest CO, limit of DSL line.  Back then, Pacbell promised they will upgrade facility in the area so I can get faster speed.  Fast forward to 2009, 14 years later, still same old… and still crappy service.

Ok, back to converting from mbox+procmail+dovecot to maildir+deliver+dovecot+sieve.  My mail archive goes back a long way, still have emails from 1987 (UUCP/Usenet mail).  I’ve always used mbox format, as that was the default then, and since I never used qmail….

I do separate my mail into folders (just another mbox file) of course, otherwise my main mailbox would be in the gigabyte range.  I use procmail to filter incoming email into these separate boxes, to automate dealing with requests, to bounce spam, etc.  My .procmailrc is around 30K in size with lots of recipes, and also pulling in dozens of include files.

I knew that mbox is inefficient and that procmail is killing my system I/O, but sheer inertia kept me where I am.

After some research, I found mb2md.pl — a Perl script — that will do most of the grunt work in converting an mbox file into maildir.   I am using Postfix for my MTA and dovecot for IMAP, so support for maildir is already there.  Supposedly, procmail supports maildir, but…

Here is the basic HOW-TO for converting mbox to maildir.

Basics: I am using Postfix, dovecot on FC11

1. Stop postfix and dovecot.

2. Install dovecot-sieve (and dovecot-managesieve if you have users beside yourself)

3. Edit /etc/postfix/main.cf and change mailbox_command to /usr/libexec/dovecot/deliver

mailbox_command = /usr/libexec/dovecot/deliver -a “$RECIPIENT”

The -a “$RECIPIENT” part is if you want to use [email protected]

4. Edit /etc/dovecot.conf

  • look for protocol lda { section and make sure this is there, add/uncomment/edit if not
  • # Support for dynamically loadable plugins. mail_plugins is a space separated
    # list of plugins to load.
    mail_plugins = sieve
    mail_plugin_dir = /usr/lib/dovecot/lda
  • look for plugin { section and look for these lines:
  • # Sieve plugin (http://wiki.dovecot.org/LDA/Sieve) and ManageSieve service
    #
    # Location of the active script. When ManageSieve is used this is actually
    # a symlink pointing to the active script in the sieve storage directory.
    sieve=~/.dovecot.sieve
    #
    # The path to the directory where the personal Sieve scripts are stored. For
    # ManageSieve this is where the uploaded scripts are stored.
    sieve_dir=~/sieve
  • Make sure /usr/libexec/dovecot/deliver exists and is executable.

5. If your mbox is in /var/spool/mail/yourname (or in /var/mail/yourname), i.e. mine is /var/spool/mail/tin, then you can run the following command to convert that mbox into maildir.

mb2md -s /var/spool/mail/tin

mb2md will automaticaly create a directory named Maildir in your home directory (i.e. it created /home/tin/Maildir for me).  The converted maildir files will go in there (from /var/spool/mail/tin).

My filtered incoming mail (via procmail) is in the mail directory of my home directory, i.e. /home/tin/mail.  There are multiple directories and multiple mbox files in there.

There are also directories such as .imap directory, created by dovecot.  Anything there that is not a valid mbox file will be ignored by mb2md.

So now run

mb2md -s /home/tin/mail -R

That should recursively travel into your mail folder and convert all mbox files there into maildir format in your Maildir directory.

I did found some mbox files that was ignored or failed to convert.  But manually running mb2md on those file worked, e.g. mb2md -s /home/tin/mail/missed-mbox-file.

To make sure there won’t be an accident, I renamed my mail folder to mail.old.  Then start up postfix, dovecot.

I’ll add more information about the .dovecot.sieve file later.

Updates on ODV

2/26/10 Walmart to acquire VUDU

This is so funny.  All of a sudden I got a bunch of IM pings and email from ex-VUDU people asking me if I had heard about “it” yet….  What was “it”?

That turned out to be the BIG news that Walmart is buying VUDU, lock, stock and barrel!  I guess VUDU will survive with Walmart’s muscle behind it… or will it?  I just don’t think Walmart is a risk taking company, nor do I think that it is innovative.  I would love to be proven wrong though.

There are so many companies now in this space.  The space itself is getting fragmented.  We have:

  • Server based video streaming over Internet – this is divided into those requiring a box/appliance vs those streaming to your computer.  Some provide services to both (Amazon, Blockbuster, Netflix, etc.).
    • Require Appliance
      1. Amazon
      2. Blockbuster
      3. Netflix
      4. TiVo
      5. VUDU
    • Stream to computer
      1. BBC
      2. Hulu
  • Video streaming OTA (Over The Air)
  • Video streaming over cable (traditional like Comcast, Charter, Time Warner, etc.)
  • Video streaming over satelite (Dish, DirecTV, etc.)

9/15/09

I believe VUDU is getting out of the hardware business. They’ve shifted their focus to the embedded business. VUDU set top box is getting harder to find from what I hear on the web. VUDU have also announced a number of embedded wins — Entone STB, LG (BD Blue Ray and HDTVs), Mitsubishi HDTVs.

It make sense to me. I’ve always thought it was dumb to try to break into the consumer hardware business when you don’t have the deep pocket to keep you going through the first few years. It’s better to license the designs and software.

Dec 4, 2009

It has been confirmed that VUDU is out of the hw business.  They’ve closed off their production facilities and stop selling new STB (you might be able to find places that still have stocks, but once they are out, that’s it!).

They’ve gone to the embedded model.  Also seem to be changing from P2P to streaming as the embedded devices generally frown on P2P.  On one hand, it’s good because of less infrastructure to maintain on the VUDU side, also less support (P2P has a whole host of issues with regards to firewall, home routers, etc. not to mention blocking by ISPs such as Comcast).  On the con side, this requires the clients to have higher bandwidth.

Btw, less infrastructure for VUDU because they can use CDNs such as Akamai, Limewire, etc.  It’s harder to distribute to CDN with P2P, trust me, it’s harder.  When I was there, I was looking into how to do this…. and it was not easy.

So, keep your eyes out for VUDU to be embedded in more devices.   At this point, I can’t decide if VUDU has done enough to save whatever market they have or if it’s too late.  I’ve just bought a WD TV Live (media center device) that let’s me play pretty much any video formats out there, along with music and in HD to boot.  I’ll have more to talk about regarding the WDTV in another post.