Summary of Overcoming RoR Performance Challenges Meetup on Wed 2/29/12

Overcoming RoR Performance Challenges Meetup

The talk was on best practices and some tips on looking for problems and how the panelists worked around them. There are no “magic” bullet like Ruby or RoR has 🙂

Essentials:

  1. Watch out when using ActiveRecord. It make it too easy to use DB. It make it too easy to use DB. One more time, it make it too easy to use DB.

    Essentially, ActiveRecord and DB is not always the right tool. Sometime using other tool could work better for a particular problem.

    Things mentioned:

    • Using Redis as a queueing system, to buffer writes, which later go to DB. (this is what Blitz, Bleacher Report use to increase their performance).
    • Use NoSQL (CouchBase, Mongo and Cassandara were mentioned as being used by panelists).
    • Cache results as much as possible. Don’t hit DB all the time.
    • Hand optimize queries might be needed. ActiveRecord is not the best at generating optimized DB calls.
  2. Cache as much as possible. Bleacher Reports put in caching layer everywhere, memcache, front end web cache, etc. They also have scripts that pre-warmed their cache (“goal is to never have users be the one who triggered a cache request”).

    Use the cache in newer RoR (3.2).

  3. Write code in ways that make it easy to update to latest Ruby and RoR.

    Ruby EE has flags to allow you to use more memory for internal cache. Sometime it make sense to test for and try different memory configuration there (based on 2 panelists’ experiences).RoR 3.2 has good Rack/Rails cache. Read the doc and use them.

  4. Background processes.
    • Use bg proc whenever possible.
    • Anytime you need to make calls to external website (external API), use a bg process, to not tie up your RoR web process.
    • Blitz put jobs into Redis queue, then bg server check Q for job, run it and put partial results back into Redis, Ajax call then check and format/display result to web client.
    • Bleacher Reports and Mixbooks also do similar things. They use Redis as a job queueing system, among other things (see 1 above).
  5. They all mention using other web server for production (not using webrick). The following were mentioned as being used by panelists.
    • Passenger
    • Thin
    • Unicorn
  6. Related to (ActiveRecord) above is the N+1 problem. Where you add 1 line of code and the DB calls increased manifold.
    • Advice essentially say to develop and use coding best practices and train developers to look out for them.
    • There is a possible test that can be use to automated looking out for N+1 issue.
    • Solving n+1 problem with special tests: http://en.oreilly.com/rails2009/public/schedule/detail/8615 Query testing – see PDF of slides page 75
    • Panelists all recommended RSpec for automated testing.
  7. Monitoring for issues and performance.
    • All panelists point to NewRelic as the tool they use all the time.
    • The host of the meeting Blitz also did a marketing spiel on their tool to use for performance testing (it look really good, and available as a plugin on Heroku). I am going to test it and see about using it for performance/load testing our site.
  8. For ease of scaling infrastructure, leverage AWS EC2, Heroku, Engine Yard and other cloud providers.

Errors while installing unicorn gem on OS X

I was updating my bundle for a rails 3.1 app, when I run into the following errors.

Installing unicorn (4.2.0) with native extensions
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

/opt/local/bin/ruby1.9 extconf.rb
checking for SIZEOF_OFF_T in ruby.h... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.

Provided configuration options:
--with-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/opt/local/bin/ruby1.9
/opt/local/lib/ruby1.9/1.9.1/mkmf.rb:381:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:491:in `block in try_compile'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:443:in `with_werror'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:491:in `try_compile'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:686:in `macro_defined?'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:822:in `block in have_macro'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:790:in `block in checking_for'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:284:in `block (2 levels) in postpone'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:254:in `open'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:284:in `block in postpone'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:254:in `open'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:280:in `postpone'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:789:in `checking_for'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:821:in `have_macro'
from extconf.rb:4:in `

'

Gem files will remain installed in /Users/tin/.bundler/tmp/34826/gems/unicorn-4.2.0 for inspection.
Results logged to /Users/tin/.bundler/tmp/34826/gems/unicorn-4.2.0/ext/unicorn_http/gem_make.out
An error occured while installing unicorn (4.2.0), and Bundler cannot continue.
Make sure that `gem install unicorn -v '4.2.0'` succeeds before bundling.

And the mkmf.log contain this, which was the clue I needed.

"/usr/bin/gcc-4.2 -o conftest -I/opt/local/include/ruby-1.9.1/x86_64-darwin10 -I/opt/local/include/ruby-1.9.1/ruby/backward -I/opt/local/include/ruby-1.9.1 -I. -I/opt/local/include -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -I/opt/local/include -pipe -O2 -arch x86_64 -fno-common conftest.c -L. -L/opt/local/lib -L/opt/local/lib -L. -L/opt/local/lib -arch x86_64 -L/usr/local/lib -lruby.1.9.1 -lpthread -ldl -lobjc "
i686-apple-darwin10-gcc-4.2.1: vfork: Operation timed out
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main() {return 0;}
/* end */

Notice the vfork: Operation timed out line? It turns out that I was running out of system resources (VM) and gcc can’t fork. Ah, my poor 8GB i7 laptop…. 🙂

Fix was simple, I shutdown Firefox and Chrome w/their gadzillion windows and tabs opened. Then retried “bundle update” again, and voila!

Documenting it here in case other people run into similar problem.