dynamic robots.txt file in Rails 3.x

We have a need for dynamic handling of robots.txt file as we have different requirements for production, staging, dev, test, etc.

Google-fu shows various way to do this, some for Rails 2.x, some for Rails 3.x. Here is my version.

First is to edit config/routes.rb and add this line:

match '/robots.txt' => RobotsGenerator

Then add the following to app_root/lib/classes/robots_generator.rb.

NOTE: We have an old domain, foo.com, that redirects to our newfoo.com. We don’t want foo.com to get indexed, so I have special treatment for that in production

class RobotsGenerator
  # Use the config/robots.txt in production.
  # Disallow everything for all other environments.
  def self.call(env)
    req = ActionDispatch::Request.new(env)
    headers = {}
    body = if Rails.env.production?
      if req.host.downcase =~ /foo.com$/
        headers = { 'X-Robots-Tag' => "noindex,nofollow" }
        "User-agent: *\nDisallow: /"
        File.read Rails.root.join('config', 'robots.txt')
        "User-agent: *\nDisallow: /"

    [200, headers, [body]]
  rescue Errno::ENOENT
    [404, {}, "User-agent: *\nDisallow: /"]

Finally, you want to move public/robots.txt to config/robots.txt.

I want to give credits to the people that inspired my version.

My Latest brites on Britely site


Britely was purchased by Groupon in mid-2011.  I’ve went on to work somewhere.  Their site is gone, so all the links below are broken.



Did you know that you can create your very own Brites on Britely website?


This should always contain the most up-to-date list of all the brites I created and published on Britely.


[britely=http://www.britely.com/tinleorg2/funny-signs width=”460″]

[britely=http://www.britely.com/tinleorg2/do-you-feel width=”460″]

[britely=http://www.britely.com/tinleorg2/guide-to-online-streaming-video width=”460″]

[britely=http://www.britely.com/tinleorg2/dots-life width=”460″]

[britely=http://www.britely.com/tinleorg2/yoda-force-follow-it width=”460″]

The following has been making the rounds on Google+. I had a good laugh reading it, so wanted to share here.


16 Things That it Took Me Over 50 Years to Learn (by Dave Barry)

1. Never, under any circumstances, take a sleeping pill and a laxative on the same night.
2. If you had to identify, in one word, the reason why the human race has not achieved, and never will achieve, its full potential, that word would be “meetings.”
3. There is a very fine line between “hobby” and “mental illness.”
4. People who want to share their religious views with you almost never want you to share yours with them.
5. You should not confuse your career with your life.
6. Nobody cares if you can’t dance well. Just get up and dance.
7. Never lick a steak knife.
8. The most destructive force in the universe is gossip.
9. You will never find anybody who can give you a clear and compelling reason why we observe daylight savings time.
10. You should never say anything to a woman that even remotely suggests that you think she’s pregnant unless you can see an actual baby emerging from her at that moment.
11. There comes a time when you should stop expecting other people to make a big deal about your birthday. That time is age eleven.
12. The one thing that unites all human beings, regardless of age, gender, religion, economic status or ethnic background, is that, deep down inside, we ALL believe that we are above average drivers.
13. A person, who is nice to you but rude to the waiter, is not a nice person (This is very important. Pay attention. It never fails.)
14. Your friends love you anyway.
15. Never be afraid to try something new. Remember that a lone amateur built the Ark. A large group of professionals built the Titanic.
16. Final thought for the day: Men are like fine wine. They start out as grapes, and it’s up to the women to stomp the snot out of them until they turn into something acceptable to have dinner with.

— Dave Barry