tinman alley

Spent all of last week fighting fires.  We have a production server that was suffering software (application bugs) and hardware failures.  Crashing left and right, I got very little sleep responding to my pager and going online to restart the app and/or server.

The app was using too much memory (Java app) the server just can’t take more memory (we already have 32GB in it).  So we decided to throw more hw at it.  App and Postgresql was running on it (yes, I know, bad, bad, bad design — my excuse is that it was not me that set this up, I joined later).  Anyway, brought up a new, faster server (Dell R410) and moved Java app over on to it, leaving Postgresql on the old server.  The plan is that if we run into problem, it’s easy to move right back to old server.  Also easier quicker this way, no down time to take down DB, copy data over, etc.   Besides which, the DB is currenlty over 65GB, will take a while to copy over.

Well, guess what…. the new R410 started experiencing hw problem!  I have RAID 10 setup on the 4 drives.  Drives 1 & 2 (one of each RAID1 element) faulted, CRAP!  Swapped drives.  Still faulting.  I get message from the kernel (dmesg) that it kept having to rescan the SAS bus as the drives kept dropping out.  (Running CentOS 5.2 64 bit).

Talked with Dell support…. ah, what a pain in the rear they are.  They insisted that it was a firmware issue!!!!  Google for “Dell RAID controllers rejecting non-DELL drives”.  We paid for same day support and we want support now!  After a couple hours on the phone, we got them to agree to swap motherboard and RAID controller the next day.

In the mean time, we have another R410 sitting the same rack (but in use).  The apps on it can be move to another server though.  So I spent a couple hours at the data center moving the drives from failing R410 over to the other one.  I was afraid there might be problem because the current state of the RAID is degraded (2 drives in the RAID10 faulted and still syncing).  But it worked like a charm.  Shut down both systems, swapped drives (two at a time, drive 0, drive 1, drive2, drive3 so I don’t mess up).  Bring up the good R410….

It came up fine.  Saw the new RAID drives and asked if I want to import foreign config.  Said yes, and press Ctrl-R anyway so I can check and the RAID controller saw the RAID10.  It told me that the two drives are syncing.  Great, exit out and reboot.

Then I noticed that this system only have 16GB RAM…. aw CRAP!  Shut it down, pull them both off the rack, open the case, swapped DIMMs.  Put them both back in, boot up the good one…. hold my breath…..  and YES, it came up, 32GB, saw the RAID drives…

Once I got the login: prompt, login, check around, making sure everything is there.  Realize that the network is not up.  Spent a couple panic stricken minutes checking cable, switch ports, etc.  Then I remember that with RedHat (and CentOS) the ifcfg-ethN script is updated at boot and uses the MAC address.  Since I moved the drives to another server, the MAC changed and RH/CentOS noticed that the MAC address in existing ifcfg-ethN does not match current MAC, it updated those files.  Luckily it renamed the existing one to ifcfg-ethN.old.

I fired up vi and updated the old ifcfg-ethN.old file with new MAC address, rename them back to ifcfg-ethN (eth0 and eth1).  Bring them down and back up (ifdown eth0, then ifup eth0) and the network is up.

Reboot the server just to be sure that everything work, login and start up the app.  Checked from an external address (ssh to my home server, point my browser to squid at home) via a browser that the app is running and acessible from the outside world.

I’ve done this before, e.g. moving entire RAID (it was RAID1 and RAID5) from one Dell server to another identical hw Dell server.  So I know it works.  Only difference was the degraded mode of the RAID, but I am glad that it worked fine too.

Like everyone else, I just had to try it out.  Since I also have a couple of EEEpc netbooks around (2 900A, and a 1000HE), plenty to test on.

Looks like Google is mainly Ubuntu on the engineering side.  I spent a little time trying to get Chromium to build under FC12 and just too much trouble for a side project.  I setup an Ubuntu 9.10 VM, followed Google’s online instruction to git d/l the source and built ChromiumOS.

So far, it’s kinda fun and I can see the use for it.  It came right up on my eeePC 900A, see all the hw, wired network, wifi network, buttons work.  Nice!

It’s just Linux underneath, and with this build, I have access to the cli via Alt-Ctrl-T key combo.

There are already plenty of people providing the live images.  If anyone is interested in my build, let me know and I’ll put a copy online.

I’ll have more when I have time to play with this.

Here is some basic sieve filtering (under dovecot, should be same for other imap server though).

I am using FC11, each account can have their own sieve filter.  The default is ${HOME}/.dovecot.sieve which is a text file.

Here is a snippet of my .dovecot.sieve file.

NOTE: you need the sieve plugins (envelope, subaddress, regex, relational)


require ["reject", "fileinto", "envelope", "subaddress", "regex", "relational"];
if  anyof (
header :contains ["Return-path"] "firewall-wizards-bounces@listserv.icsalabs.com",
header :contains ["List-Id"] "Firewall Wizards Security Mailing List",
header :contains ["List-Post"] "mailto:firewall-wizards@listsserv..icsalabs.com") {
fileinto "lists.firewall";
} elsif anyof (
header :contains ["X-Mailing-List"] "pgsql-general",
header :contains ["List-ID"] "pgsql-general.postgresql.org",
header :contains ["X-Mailing-List"] "pgsql-performance",
header :contains ["List-ID"] "pgsql-performance.postgresql.org") {
fileinto "lists.postgresql";
} elsif anyof (
header :contains ["X-BeenThere"] "rubyonrails-talk",
header :contains ["List-ID"] "rubyonrails-talk.googlegroups.com") {
fileinto "lists.rails";
} else {
# The rest goes into INBOX
# default is "implicit keep", we do it explicitly here
keep;
}

I’ve been wanting to do this for a while, but too busy, too lazy and finally too worry that I might lose my mail archive…. etc…etc…etc..

Over last weekend, I bit the bullet and did it.  Partly because my DSL line was down, so there was no network activities, no mail, so might as well take advantage of the down time.  My next project is to drop AT&T DSL, I am tired of their bs.

Not meaning to get off the subject, but I am one of the earliest DSL sub in the neighborhood.  Got mine in ‘96 (only 768k down/256K up).  I am about ~14-15000 ft from nearest CO, limit of DSL line.  Back then, Pacbell promised they will upgrade facility in the area so I can get faster speed.  Fast forward to 2009, 14 years later, still same old… and still crappy service.

Ok, back to converting from mbox+procmail+dovecot to maildir+deliver+dovecot+sieve.  My mail archive goes back a long way, still have emails from 1987 (UUCP/Usenet mail).  I’ve always used mbox format, as that was the default then, and since I never used qmail….

I do separate my mail into folders (just another mbox file) of course, otherwise my main mailbox would be in the gigabyte range.  I use procmail to filter incoming email into these separate boxes, to automate dealing with requests, to bounce spam, etc.  My .procmailrc is around 30K in size with lots of recipes, and also pulling in dozens of include files.

I knew that mbox is inefficient and that procmail is killing my system I/O, but sheer inertia kept me where I am.

After some research, I found mb2md.pl — a Perl script — that will do most of the grunt work in converting an mbox file into maildir.   I am using Postfix for my MTA and dovecot for IMAP, so support for maildir is already there.  Supposedly, procmail supports maildir, but…

Here is the basic HOW-TO for converting mbox to maildir.

Basics: I am using Postfix, dovecot on FC11

1. Stop postfix and dovecot.

2. Install dovecot-sieve (and dovecot-managesieve if you have users beside yourself)

3. Edit /etc/postfix/main.cf and change mailbox_command to /usr/libexec/dovecot/deliver

mailbox_command = /usr/libexec/dovecot/deliver -a “$RECIPIENT”

The -a “$RECIPIENT” part is if you want to use user+foo@domain.com

4. Edit /etc/dovecot.conf

• look for protocol lda { section and make sure this is there, add/uncomment/edit if not
• # Support for dynamically loadable plugins. mail_plugins is a space separated
  # list of plugins to load.
  mail_plugins = sieve
  mail_plugin_dir = /usr/lib/dovecot/lda
• look for plugin { section and look for these lines:
• # Sieve plugin (http://wiki.dovecot.org/LDA/Sieve) and ManageSieve service
  #
  # Location of the active script. When ManageSieve is used this is actually
  # a symlink pointing to the active script in the sieve storage directory.
  sieve=~/.dovecot.sieve
  #
  # The path to the directory where the personal Sieve scripts are stored. For
  # ManageSieve this is where the uploaded scripts are stored.
  sieve_dir=~/sieve
• Make sure /usr/libexec/dovecot/deliver exists and is executable.

5. If your mbox is in /var/spool/mail/yourname (or in /var/mail/yourname), i.e. mine is /var/spool/mail/tin, then you can run the following command to convert that mbox into maildir.

mb2md -s /var/spool/mail/tin

mb2md will automaticaly create a directory named Maildir in your home directory (i.e. it created /home/tin/Maildir for me).  The converted maildir files will go in there (from /var/spool/mail/tin).

My filtered incoming mail (via procmail) is in the mail directory of my home directory, i.e. /home/tin/mail.  There are multiple directories and multiple mbox files in there.

There are also directories such as .imap directory, created by dovecot.  Anything there that is not a valid mbox file will be ignored by mb2md.

So now run

mb2md -s /home/tin/mail -R

That should recursively travel into your mail folder and convert all mbox files there into maildir format in your Maildir directory.

I did found some mbox files that was ignored or failed to convert.  But manually running mb2md on those file worked, e.g. mb2md -s /home/tin/mail/missed-mbox-file.

To make sure there won’t be an accident, I renamed my mail folder to mail.old.  Then start up postfix, dovecot.

I’ll add more information about the .dovecot.sieve file later.

2/26/10 Walmart to acquire VUDU

This is so funny.  All of a sudden I got a bunch of IM pings and email from ex-VUDU people asking me if I had heard about “it” yet….  What was “it”?

That turned out to be the BIG news that Walmart is buying VUDU, lock, stock and barrel!  I guess VUDU will survive with Walmart’s muscle behind it… or will it?  I just don’t think Walmart is a risk taking company, nor do I think that it is innovative.  I would love to be proven wrong though.

There are so many companies now in this space.  The space itself is getting fragmented.  We have:

• Server based video streaming over Internet – this is divided into those requiring a box/appliance vs those streaming to your computer.  Some provide services to both (Amazon, Blockbuster, Netflix, etc.).
  • Require Appliance
    1. Amazon
    2. Blockbuster
    3. Netflix
    4. TiVo
    5. VUDU
  • Stream to computer
    1. BBC
    2. Hulu
• Video streaming OTA (Over The Air)
• Video streaming over cable (traditional like Comcast, Charter, Time Warner, etc.)
• Video streaming over satelite (Dish, DirecTV, etc.)

9/15/09

I believe VUDU is getting out of the hardware business. They’ve shifted their focus to the embedded business. VUDU set top box is getting harder to find from what I hear on the web. VUDU have also announced a number of embedded wins — Entone STB, LG (BD Blue Ray and HDTVs), Mitsubishi HDTVs.

It make sense to me. I’ve always thought it was dumb to try to break into the consumer hardware business when you don’t have the deep pocket to keep you going through the first few years. It’s better to license the designs and software.

Dec 4, 2009

It has been confirmed that VUDU is out of the hw business.  They’ve closed off their production facilities and stop selling new STB (you might be able to find places that still have stocks, but once they are out, that’s it!).

They’ve gone to the embedded model.  Also seem to be changing from P2P to streaming as the embedded devices generally frown on P2P.  On one hand, it’s good because of less infrastructure to maintain on the VUDU side, also less support (P2P has a whole host of issues with regards to firewall, home routers, etc. not to mention blocking by ISPs such as Comcast).  On the con side, this requires the clients to have higher bandwidth.

Btw, less infrastructure for VUDU because they can use CDNs such as Akamai, Limewire, etc.  It’s harder to distribute to CDN with P2P, trust me, it’s harder.  When I was there, I was looking into how to do this…. and it was not easy.

So, keep your eyes out for VUDU to be embedded in more devices.   At this point, I can’t decide if VUDU has done enough to save whatever market they have or if it’s too late.  I’ve just bought a WD TV Live (media center device) that let’s me play pretty much any video formats out there, along with music and in HD to boot.  I’ll have more to talk about regarding the WDTV in another post.

I am collecting links to Linux based Multimedia Live CD/USB distros.

• GeeXboX
• dyne:bolic
• ArtistX
• Amarok Live
• LiMP
• Mediainlinux
• Apodio

I am a long time MySQL user, so Postgres administration is unfamiliar to me.  I did used Postgres a long time ago… when I first started my web hosting company, but back then (early 90’s), Postgres was young and not suited for our needs.  Discovered MySQL and never looked back.

Yeah, yeah, stop with the religious war already.  I believe that using the right tool for the job is more important than anything else.

Back to the problem at hand.  As a new, incoming admin, there is a lot of things I have to pick up on-the-fly.  There is a lot of historical knowledge that I do not have at new place.  Such as Postgres, which *work* uses and now I have to admin.  It seem that no one know the postgres user password, or don’t want to share ;->

I really don’t care for politics, just want to do my work!  So things break and I need access to system tables to fix it.  Can’t login to Postgres as postgres, yes, I have root and can su postgres, but still can not login via psql.

E.g. psql -h localhost -U postgress -d template1  ask me for password and I don’t know postgres user password

Searching (googling) found a lot of people asking for help, etc. but no specific way to solve the problem.  At least for a newbie Postgres admin like me.

Finally, someone pointed to pg_hba.conf…. reading the manual, backward, forward, sideways, etc…. and finally, finally, figured it out!

Here it is to save time for others.

Edit pg_hba.conf (usually in /var/lib/psql/data directory, at least on RH/CentOS/Fedora).

Add this line at top, first ACL match wins, so don’t worry about the rest there.

host all postgres 127.0.0.1/32 trust

Then at command line, run pg_ctl reload

Now you can psql -h localhost -d template1 -U postgres and login without password.  Change the password to something you know.

alter user postgres with encrypted password ‘newpassword’;

Exit and comment out the line you added to pg_hba.conf, then reload config via

pg_ctl reload

Tada!  All fixed.

It’s harder than it look, or perhaps I am just making it harder than it really is.  Anyway, I have needs to monitory performance of java based applications.

There are actually two types (to me) of Java apps.  There is the standalone apps that you run on your workstation (Eclipse, I count embedded web applets in this category), and then there is the server based types such as Tomcat/JSP/J2EE/etc.

The category that I am most interested in is the server based apps.  I need to be able to look inside the JVM they are running in, and also the container (Tomcat/J2EE/etc.).  I am not an expert in this area, so at the moment, it’s a blackbox to me.

I’ve been searching around (yes, started with Googling ) and found lots of information all over the place.  I am going to try to gather them into one spot for my benefits, and hopefully save others some time.  As always, if you have corrections, additions, please feel free to send them to me.

List of Java performance monitoring tools.

• this came from here
  1. jconsole comes with JDK 1.5 and above. It is a Java Monitoring and Management Console – JMX-compliant graphical tool for monitoring a Java virtual machine. It can monitor both local and remote JVMs.
  2. VisualVM is a visual tool that integrates several existing JDK software tools and lightweight memory and CPU profiling capabilities. This tool is designed for both production and development time use and further enhances the capability of monitoring and performance analysis for the Java SE platform.
  3. HeapAnalyzer allows the finding of a possible Java™ heap leak area through its heuristic search engine and analysis of the JavaTM heap dump in Java applications. It analyzes Java heap dumps by parsing the Java heap dump, creating directional graphs, transforming them into directional trees, and executing the heuristic search engine.
  4. PerfAnal is a GUI-based tool for analyzing the performance of applications on the Java 2 Platform. You can use PerfAnal to identify performance problems in your code and locate code that needs tuning.
  5. JAMon is a free, simple, high performance, thread safe, Java API that allows developers to easily monitor production applications.
  6. Eclipse Memory Analyzer is a fast and feature-rich Java heap analyzer that helps you find memory leaks and reduce memory consumption.
  7. GCViewer is a free open source tool to visualize data produced by the Java VM options -verbose:gc and -Xloggc:<file>. It also calculates garbage collection related performance metrics (throughput, accumulated pauses, longest pause, etc.).
  8. HPjmeter
     • Identify and diagnose performance problems in Java™ applications running on HP-UX
     • Monitor live Java™ applications and analyze profiling data
     • Capture profiling data with zero preparation when using JDK/JRE 5.0.04 or higher
     • Run the HPjmeter console on HP-UX, Linux, and Windows® systems
     • Improve garbage collection performance
  9. HPjconfig is a Java configuration tool for tuning your HP-UX 11i HP Integrity Itanium® and HP 9000 PA-RISC system kernel parameters to match the characteristics of your application. HPjconfig provides kernel parameter recommendations tailored to your HP-UX hardware platform. It offers save and restore functions for easy distribution of tailored recommendations across your customer base. When given specific Java and HP-UX versions, HPjconfig will determine if all of the latest HP-UX patches required for Java performance and functionality are installed on the system, and highlight any missing or superseded patches.
  10. Java Out-of-Box Tool is a stand-alone bundle that upon installation will install startup (RC) scripts, modify kernel parameters, rebuild the kernel, and reboot the system. During startup, the startup scripts will modify system tunables, thus providing better “Out of The Box” behavior for Java.
• eclipse TPTP and Netbeans Profiler
• YourKit
• JProfiler
• JIP
• crap4j
• JRockit

12/12/09 Updates

Yes, I will also consider commercial tools.  It’s been a few years since I develop Java full time.  My focus these days are on the Operation side – Network/System Administration, architectures, day to day operations.

I stumbled onto the Intel SS4200, which is intended as a SOHO platform for NAS.  Essentially it’s a small form factor PC, based on 1.6Ghz Celeron, 4 hot swap SATA drives, to be use as a home office NAS.

Turn out lots of other companies OEM this box (Fujitsu Siemens Scaleo is one).

Based on cursory Google search, there is a lot of people hacking this box.  It is very hackable, hw and sw wise.  One can add more memory (alas, only one slot, but you can go from 512KB to 2GB stick), upgrade the CPU (from Celeron to Core 2 Duo), etc.

Some useful links:

• Accessing linux console.
• Upgrade firmware (EMC Lifeline)

Update 01/14/10

Not that I have a lot of time, but I could not resist.  Company on eBay is selling a new SS4200-E for $135 (no drive, but 1GB RAM).  I’ve ordered one, and will start hacking once I have it.

VUDU Goes Box-free for Entone
LA Time – VUDU and Entone team up…

Heh, that mean they (both of ‘em) are still alive. That’s because they are doing something to stay above the tide of competition.

I’ve always thought that VUDU can’t survive on their own.  They need to partner with someone else, but it has to be the right partner, otherwise it’s a waste of resources (time, effort, fund…).  They had a couple of chances with some very good possibilities, but they screwed up.  Actually, IMHO, both sides screwed up….

Anyway, this is a step in the right direction, although I wish VUDU has partnered with one of the big players in the video on demand or consumer electronics field — e.g. Amazon, Samsung, LG, Sony, etc.

I think Samsung or Sony would be perfect partners for VUDU.  It’s a win-win for both side.