MongoDB and Riak

12/18/12 UPDATE

Since I am the only DevOps working on this, and there are tons of other things requiring my attention, I had to drop riak. The engineers only know mongodb anyway, and they are reluctant to learn a new nosql (riak). Crap! So this project had been killed. Too bad.

I have some python scripts that I wrote to copy mongodb collections over to riak, if I have time, I’ll open source them.

======================

I’ve been working with MongoDB at current $WORK and previous jobs. It (used-to-be) is the nice, shiny toy that everyone rushed to. I’ve run into numerous limitation in trying to scale it up. Operationally, it can be a nightmare if the architecture was not setup correctly at the beginning.

Mongo is also a PITA to scale. There are major sites that have Mongos in the thousands, but at that point it become throwing hw and money at the problem. That just seem stupid for startups.

So at current place of $WORK, they are currently testing MongoDB, but I wanted to look for an alternative solution before we become fully committed to yet another operational nightmare.

After a lot of googling, testing, experimenting, etc. I decide on trying Riak from Basho.

Googling shows a number of companies migrated from MongoDB to Riak. Their experiences was useful, but I was looking more for concrete HOWTO to move large MongoDB over to Riak.

First, of course was to get hands-on experiences with Riak. Installed, play with it, etc. Then I used the riak-python-client lib to start migrating some data over. I wrote a script to work through all collections in a Mongo DB, for each collection, create a Riak bucket and add the Mongo doc to bucket using the Mongo _id as the key.

Right away, I run into some issues with Riak. I have a 3 nodes Riak (on 3 physical CentOS 5.8 servers). The MongoDB I was copying over was large, about 2GB on disk file size and over a million records. Partway through the conversion, 2 Riak nodes crashed and died…. WTF! No matter what I do, they wouldn’t start back up (Riak log shows some kind of erlang errors, but I don’t know erlang). So I stopped the only node running, ‘rm -rf /var/lib/riak/*’, ‘killall epmd’, restarted all 3 nodes and they came back up.

I don’t have time to debug this problem, so restarted conversion with a smaller subset of Mongo data. But this crash worries me. The erl_crash.dump shows Riak run into resource issues, unable to allocate heap memory. Hmmm.

More on my adventure in evaluating Riak vs MongoDB in the future.

Toyota Rav4 recall

11/16/2012 Update: Yep, I got letter from Toyota that my 2010 RAV4 is part of the recall. Crap!

My brother recently alerted me to this. Evidently the Rav4 rear suspension arm might come loose if the nuts holding it together are not tightened correctly.

Here is link to Toyota’s announcement about it.

Check VIN here to see if your Rav4 is part of recall.

August 01, 2012
Toyota Announces Recall of Certain RAV4 and Lexus HS 250h Vehicles

TORRANCE, Calif., August 1, 2012 — Toyota Motor Sales (TMS), U.S.A., Inc., today announced that it will conduct a voluntary safety recall involving 2006 to early 2011 model year RAV4 and 2010 model year Lexus HS 250h vehicles sold in the U.S. No other Toyota or Lexus vehicles are involved.

Approximately 760,000 Toyota RAV4
Approximately 18,000 Lexus HS 250h
Toyota has determined that if the nuts on the rear suspension arm are not tightened following the proper procedure and torque specification during a rear wheel alignment service, excessive play may occur at the threaded portion of the arm, followed by rust formation. If this were to occur, the threaded portion of the rear suspension arm may wear and cause the arm to separate.

Toyota is currently developing a remedy. Once the remedy is available, we will send a notification letter by first class mail advising owners to make an appointment with an authorized Toyota or Lexus dealer to have the remedy performed at no charge.

Detailed information and answers to questions are available to customers at www.toyota.com/recall and the Toyota Customer Experience Center (1 800-331-4331) or www.lexus.com/recall and Lexus Customer Satisfaction (1 800-255-3987).

Media Contacts:
Vehicle Safety and Quality Communications
Brian R. Lyons (310) 468-2552
John Hanson (310) 468-4718
Katy Soto (310) 468-8068
Toyota Motor Sales, USA, Inc. Media Line (310) 468-5297

dynamic robots.txt file in Rails 3.x

We have a need for dynamic handling of robots.txt file as we have different requirements for production, staging, dev, test, etc.

Google-fu shows various way to do this, some for Rails 2.x, some for Rails 3.x. Here is my version.

First is to edit config/routes.rb and add this line:


match '/robots.txt' => RobotsGenerator

Then add the following to app_root/lib/classes/robots_generator.rb.

NOTE: We have an old domain, foo.com, that redirects to our newfoo.com. We don’t want foo.com to get indexed, so I have special treatment for that in production

class RobotsGenerator
  # Use the config/robots.txt in production.
  # Disallow everything for all other environments.
  def self.call(env)
    req = ActionDispatch::Request.new(env)
    headers = {}
    body = if Rails.env.production?
      if req.host.downcase =~ /foo.com$/
        headers = { 'X-Robots-Tag' => "noindex,nofollow" }
        "User-agent: *\nDisallow: /"
      else
        File.read Rails.root.join('config', 'robots.txt')
      end
    else
        "User-agent: *\nDisallow: /"
    end

    [200, headers, [body]]
  rescue Errno::ENOENT
    [404, {}, "User-agent: *\nDisallow: /"]
  end
end

Finally, you want to move public/robots.txt to config/robots.txt.

I want to give credits to the people that inspired my version.

My Latest brites on Britely site

11/08/2013

Britely was purchased by Groupon in mid-2011.  I’ve went on to work somewhere.  Their site is gone, so all the links below are broken.

 


 

Did you know that you can create your very own Brites on Britely website?

 

This should always contain the most up-to-date list of all the brites I created and published on Britely.

 

[britely=http://www.britely.com/tinleorg2/funny-signs width=”460″]

[britely=http://www.britely.com/tinleorg2/do-you-feel width=”460″]

[britely=http://www.britely.com/tinleorg2/guide-to-online-streaming-video width=”460″]

[britely=http://www.britely.com/tinleorg2/dots-life width=”460″]

[britely=http://www.britely.com/tinleorg2/yoda-force-follow-it width=”460″]