Kibana 4 with tribe node MasterNotDiscoveredException

I use tribe nodes quite a lot at $work. It’s how we federate disparate ELK clusters and able to search across them. There are many reasons to have distinct ELK clusters in each data center and/or region.

Some of these are:

1. Elasticsearch does not work well when there is network latencies, which is guaranteed when your nodes are located geographically distant places. You could spend a lot of money to get fast network connection, or you can just have only local clusters. (Me? I pick saving money and avoiding head aches :-)).

2. It can get insanely expensive to create an ES cluster that span data centers/regions. The network bandwidth requirement, the data charges, the care and feeding of such a latency sensitive cluster…. OMG!

3. I don’t really think a 3rd reason is needed.

Although tribe nodes are great for federating ES clusters, there are some quirks in setting them up and caring for them (not as bad as ES clusters that span datacenter though).

One big gotcha for many people who are setting up tribe nodes for the first time is that tribe node can not create index. Tribe can only update, modify an existing index. What this mean is that if you point Kibana at a tribe node, you must first make sure you Kibana index is already created in one of the downstream ES cluster. Otherwise, you will have to create it yourself.

Otherwise, the first time you create an index pattern and tried to save it, you will get an error similar to the subject of this post.

MasterNotDiscoveredException

The error message is wrong and misleading. It has nothing to do with Master node. It has everything to do with tribe node not able to create (PUT) a Kibana index.

Personally, I prefer to make the Kibana index that I use with tribe to have its own unique name. So I run a dedicated Kibana instance pointing to the dedicated tribe (client) node.

Here are the steps I do to get a tribe node and its associated Kibana ready for use.

1. Configure the tribe node to know all the ES clusters I want to federate data from.

tribe.elasticsearch.yml:

cluster.name: toplevel_tribe
node.name: ${HOSTNAME}
node.master: false
node.data: false
tribe:
  DC1-appservice:
     cluster.name: logging-DC1
     network.host: 0.0.0.0
     network.publish_host: ${HOSTNAME}
     discovery.zen.ping.unicast.hosts:
      - dc1-app13225.prod.example.com
      - dc1-app13226.prod.example.com
      - dc1-app13227.prod.example.com
  DC2-appservice:
     cluster.name: logging-DC2
     network.host: 0.0.0.0
     network.publish_host: ${HOSTNAME}
     discovery.zen.ping.unicast.hosts:
      - dc2-app12281.prod.example.com
      - dc2-app12282.prod.example.com
      - dc2-app12283.prod.example.com
   DC3.....etc to DCNN

  my-es-dedicated-config-cluster:
     cluster.name: es-config-CORP
     network.host: 0.0.0.0
     network.publish_host: ${HOSTNAME}
     discovery.zen.ping.unicast.hosts:
      - corp-app1234.example.com

 on_conflict: prefer_my-es-dedicated-config-cluster

2. Now pre-create the Kibana index in my-ES-dedicated-config-cluster. This is a small cluster in my admin/corp data center that is only for housing configurations, Kibana dashboards, etc.

3. A simpler and more correct way is to temporary point Kibana to the dedicated ES cluster (instead of the tribe).

Do this via this setting in your kibana.yml file:

# The Elasticsearch instance to use for all your queries.
elasticsearch.url: “http://ES-node:9200”

Start Kibana, let it create the index.  Then stop it, change the setting back to point to your tribe node.

Doing it this way ensure that your kibana is correct.

curl command for pre-creating kibana (3 and 4) index:


curl -s -XPUT "http://localhost:9200/kibana3-int/" -d '{ "settings" : { "number_of_shards" : 3, "number_of_replicas" : 2 },
"mappings" : { "temp" : { "properties" : { "dashboard" : { "type" : "string" }, "group" : { "type" : "string" }, "title" : { "type" : "string" }, "user" : { "type" : "string" } } }, "dashboard" : { "properties" : { "dashboard" : { "type" : "string" }, "group" : { "type" : "string" }, "title" : { "type" : "string" }, "user" : { "type" : "string" } } } }'


# Kibana4
curl -s -XPUT "http://localhost:9200/TRIBENAME-kibana4" -d '{ "index.mapper.dynamic" : true, "settings" : { "number_of_shards" : 1, "number_of_replicas" : 0 },"mappings" : {"search" : {"_timestamp" : { },"properties" : {"columns" : {"type" : "string"},"description" : {"type" : "string"},"hits" : {"type" : "long"},"kibanaSavedObjectMeta" : {"properties" : {"searchSourceJSON" : {"type" : "string"}}},"sort" : {"type" : "string"},"title" : {"type" : "string"},"version" : {"type" : "long"}}},"dashboard" : {"_timestamp" : { },"properties" : {"description" : {"type" : "string"},"hits" : {"type" : "long"},"kibanaSavedObjectMeta" : {"properties" : {"searchSourceJSON" : {"type" : "string"}}},"optionsJSON" : {"type" : "string"},"panelsJSON" : {"type" : "string"},"timeRestore" : {"type" : "boolean"},"title" : {"type" : "string"},"uiStateJSON" : {"type" : "string"},"version" : {"type" : "long"}}},"visualization" : {"_timestamp" : { },"properties" : {"description" : {"type" : "string"},"kibanaSavedObjectMeta" : {"properties" : {"searchSourceJSON" : {"type" : "string"}}},"savedSearchId" : {"type" : "string"},"title" : {"type" : "string"},"uiStateJSON" : {"type" : "string"},"version" : {"type" : "long"},"visState" : {"type" : "string"}}},"config" : {"_timestamp" : { },"properties" : {"buildNum" : {"type" : "long"},"defaultIndex" : {"type" : "string"}}},"index-pattern" : {"_timestamp" : { },"properties" : {"customFormats" : {"type" : "string"},"fieldFormatMap" : {"type" : "string"},"fields" : {"type" : "string"},"intervalName" : {"type" : "string"},"timeFieldName" : {"type" : "string"},"title" : {"type" : "string"}}}}}'

Toyota Rav4 recall

11/16/2012 Update: Yep, I got letter from Toyota that my 2010 RAV4 is part of the recall. Crap!

My brother recently alerted me to this. Evidently the Rav4 rear suspension arm might come loose if the nuts holding it together are not tightened correctly.

Here is link to Toyota’s announcement about it.

Check VIN here to see if your Rav4 is part of recall.

August 01, 2012
Toyota Announces Recall of Certain RAV4 and Lexus HS 250h Vehicles

TORRANCE, Calif., August 1, 2012 — Toyota Motor Sales (TMS), U.S.A., Inc., today announced that it will conduct a voluntary safety recall involving 2006 to early 2011 model year RAV4 and 2010 model year Lexus HS 250h vehicles sold in the U.S. No other Toyota or Lexus vehicles are involved.

Approximately 760,000 Toyota RAV4
Approximately 18,000 Lexus HS 250h
Toyota has determined that if the nuts on the rear suspension arm are not tightened following the proper procedure and torque specification during a rear wheel alignment service, excessive play may occur at the threaded portion of the arm, followed by rust formation. If this were to occur, the threaded portion of the rear suspension arm may wear and cause the arm to separate.

Toyota is currently developing a remedy. Once the remedy is available, we will send a notification letter by first class mail advising owners to make an appointment with an authorized Toyota or Lexus dealer to have the remedy performed at no charge.

Detailed information and answers to questions are available to customers at www.toyota.com/recall and the Toyota Customer Experience Center (1 800-331-4331) or www.lexus.com/recall and Lexus Customer Satisfaction (1 800-255-3987).

Media Contacts:
Vehicle Safety and Quality Communications
Brian R. Lyons (310) 468-2552
John Hanson (310) 468-4718
Katy Soto (310) 468-8068
Toyota Motor Sales, USA, Inc. Media Line (310) 468-5297

The following has been making the rounds on Google+. I had a good laugh reading it, so wanted to share here.

=============================================

16 Things That it Took Me Over 50 Years to Learn (by Dave Barry)

1. Never, under any circumstances, take a sleeping pill and a laxative on the same night.
2. If you had to identify, in one word, the reason why the human race has not achieved, and never will achieve, its full potential, that word would be “meetings.”
3. There is a very fine line between “hobby” and “mental illness.”
4. People who want to share their religious views with you almost never want you to share yours with them.
5. You should not confuse your career with your life.
6. Nobody cares if you can’t dance well. Just get up and dance.
7. Never lick a steak knife.
8. The most destructive force in the universe is gossip.
9. You will never find anybody who can give you a clear and compelling reason why we observe daylight savings time.
10. You should never say anything to a woman that even remotely suggests that you think she’s pregnant unless you can see an actual baby emerging from her at that moment.
11. There comes a time when you should stop expecting other people to make a big deal about your birthday. That time is age eleven.
12. The one thing that unites all human beings, regardless of age, gender, religion, economic status or ethnic background, is that, deep down inside, we ALL believe that we are above average drivers.
13. A person, who is nice to you but rude to the waiter, is not a nice person (This is very important. Pay attention. It never fails.)
14. Your friends love you anyway.
15. Never be afraid to try something new. Remember that a lone amateur built the Ark. A large group of professionals built the Titanic.
16. Final thought for the day: Men are like fine wine. They start out as grapes, and it’s up to the women to stomp the snot out of them until they turn into something acceptable to have dinner with.

— Dave Barry

WD TV Live

As you know, I’ve been playing around with media players, dvr, etc. for a while now. Started with TiVo series 2, 2wire 2040 (AT&T uVerse), VUDU… and this xmas, I picked up WD TV box.

Originally bought the WD TV (USB only) box as it was onsale for $70 at local BB. But then I found out about the Live model, which just came out. The Live model has builtin wired Ethernet (RJ-45 port), save me time, and besides which I hate USB-to-Ethernet or wireless adapters.

Side note: Most people don’t realize or understand that USB adapters are slow. Doesn’t matter if you use USB2. USB depends on your system CPU (it’s a _SERIAL PORT_ for godsake!). If you have a slow CPU, then you will never get the full bandwidth of your link.  Not only that, but it will kill your system performance.

I put the hacked WDLXTV-LIVE firmware from b-rad (http://b-rad.cc/) on my box.  Nice.  Didn’t really have time to exercise everything (family visits during the holiday season), but works as expected.

How to setup development environment for hacking WDTV Live is here:

http://wiki.wdtv.org/doku.php?id=setting_up_the_environment_for_compiling