Archive

Author Archive

ELK Operational Tips

February 22nd, 2015 No comments

I’ve been running ELK clusters for over a year now, and want to share tips and tricks that I’ve found to be useful.

Feel free to post questions and corrections. I’ll try to answer and update when possible.

Elasticsearch

  • Split brained – this is when you have more than one node in your cluster becoming master.
    • It is best to avoid ever having this happen.   Use the rule of thumb, e.g. if you have N nodes, the number of nodes that can be master is N/2 + 1.   Even better, set aside a dedicated pool of master nodes (I recommend minimum of 3 master capable nodes).
    • If split brained does happen, you want to stop one of the master node ASAP.   Depending on whether you have replicas or not, it could be easy fix, or you might end up having to re-index if your indices has gotten out of sync by having the replica promoted to primary and new index data sent to it.
  • Failed node(s) – one or more failed nodes.  There are many scenarios, from failing hardware to outages causing data corruption, etc.
  • Planned maintenance – several scenarios.
  • Indexing take too long.
  • Recovery take too long.
  • Search/query take too long.

Logstash

Kibana

 

Online debugging/tutorial tools

October 16th, 2014 No comments

Online Tools

In the course of my career, I’ve jumped from one platform, OS, programming, scripting languages, etc. to another.   I’ve found that what make it easier to transition into a new “whatever” is the quality of tutorial and debugging tools available to me.

Besides local tools, there are some awesome web sites that are setup to help with debugging various problems.   I am going to try to compile them here.

Please feel free to let me know of others that I’ve missed.

Go

Javascript

Python

 

Regular Expressions

Golang meetup group in Mountain View

September 26th, 2014 No comments

I am glad to find a Golang meetup group near me.  This one is in Mountain View, CA.

Golang Mountain View

Mountain View, CA
580 Gophers

Golang Mountain View is a community of people interested in the Go programming language(golang).(from golang.org) The Go programming language is an open source project to mak…

Check out this Meetup Group →

Categories: Golang, Tech Tags: ,

logstash-forwarder TLS handshake errors

July 3rd, 2014 3 comments

I started using logstash-forwarder to send logs from my cloud hosted servers to my ELK server for analysis.   Since it’s just a simple setup, I used the self-gen cert as described on logstash-forwarder’s github page.

Unfortunately, using the example generated a cert that is only good for 30 days.   So suddenly my kibana graph show no data for my cloud servers…. ???  After some digging, I found errors like this in the log.

 logstash-forwarder[4367]: 2014/07/01 23:24:08.559691 Failed to tls handshake with 172.25.28.52 x509: certificate has expired or is not yet valid

openssl x509 -in logstash-forwarder.crt -noout -text  show that the Validity period was only 30 days.  D’oh! 🙂

So I generated a new set, this time for 10 years.  Why not, it’s for my use and if I am still using it 10 years from now…

openssl req -x509 -batch -nodes -newkey rsa:2048 -days 3560 -keyout logstash-forwarder.key -out logstash-forwarder.crt

 

Update 2014-07-28

Tried to bring up another server with logstash-forwarder.  Except I used latest logstash-forwarder (git pull today 2014/07/25) and started getting this error when starting up LS.

Failed to tls handshake with 172.25.28.52 x509: certificate is valid for , not foo.bar.le.org

After a bit of debugging, comparing certs (exact same MD5 as the ones on working servers), I went googling and bingo!

https://github.com/elasticsearch/logstash-forwarder/issues/221

I see people blaming Go v1.3 TLS changes, but I am still using the same Go v1.2.1 that I built the currently working logstash-forwarder.   And as a matter of fact, copying logstash-forwarder from existing working servers over to the new one and it works just fine!   So I do not think that it’s Go, but something in the latest commits to logstash-forwarder that broke TLS.

 Update 2014-08-17

Turned out to be my self-gen cert ;-P   I created a new one, using properly filled out openssl.cnf and a wildcard domain.  That works fine with latest trunk and built using go v1.2.1.   I’ll update to go v1.3 soon.

 

adding CORS support to elasticsearch-head plugin

May 27th, 2014 No comments

There are two vulnerabilities in Elasticsearch that I recently patched in my installations.

One is the ‘script’ vuln, mentioned here.

Fix by adding

script.disable_dynamic: true

to your Elasticsearch.yml config file.

The other one has to do with CORS, which exposes data via REST endpoints.

Fix by adding

http.cors.allow-origin: "http://your.FQDN.domain.name"

to your Elasticsearch.yml config file.

In fixing the second one (CORS), I run into a problem where that broke my usage of elasticsearch-head plugin.  I use the plugin as a checked out git repo on my laptop and port forward to the actual ES server.   E.g. the URL I use is something like this

file:///Users/tinle/src/opensource/elasticsearch-head/index.html?base_uri=http://127.0.0.1:9200/

So I ended up having to patch elasticsearch-head to make it work with CORS.

diff --git a/dist/app.js b/dist/app.js
index 5bce2a3..7e58acb 100644
--- a/dist/app.js
+++ b/dist/app.js
@@ -1188,6 +1188,9 @@
                request: function( params ) {
                        return $.ajax( $.extend({
                                url: this.base_uri + params.path,
+      /**
+       * 2014/06/01 tinle
+       **/
                                dataType: "jsonp",
         crossDomain: true,
                                error: function(xhr, type, message) {
diff --git a/dist/vendor.js b/dist/vendor.js
index fb1a448..2b74180 100644
--- a/dist/vendor.js
+++ b/dist/vendor.js
@@ -6838,6 +6838,10 @@ jQuery.each( [ "get", "post" ], function( i, method ) {
                return jQuery.ajax({
                        type: method,
                        url: url,
+      /**
+       * HACK 2014/06/03 tinle
+       */
+      crossDomain: true,
                        data: data,
                        success: callback,
                        dataType: type
@@ -14439,4 +14443,4 @@ under the License.
                }
                throw "could not process value " + v;
        };
-})();
\ No newline at end of file
+})();

 

Updated: 6/4/2014 – I think the above patch should work.  I’ve been using it last few days and I am able to GET/PUT/POST, e.g. make changes to ES via elasticsearch-head.

 

 

Elasticsearch, Logstash and Kibana Meetup @ LinkedIn

May 23rd, 2014 No comments

We had a great ELK Meetup on Wed 5/21/2014 at LinkedIn.  The recorded video is available here.

http://www.ustream.tv/recorded/47864947

We had Kurt Hurtado, one of the logstash dev, speaking on ELK in the DevOps Environment.  Then a nice long Q&A session after, joined by Uri Boness, one of the Elasticsearch core dev.

 

Categories: NoSQL, Search, Tech Tags: , , ,

Why you should care about “Net Neutrality” and the FCC

May 16th, 2014 No comments

Linking to an article written by a colleague.  I agree with its content.

 

https://www.linkedin.com/today/post/article/20140516165054-86049424-why-you-should-care-about-the-fcc-and-net-neutrality?trk=object-title

 

We should do all we can to protect what has made the Internet successful.  Do not turn it into a pay-wall, where only those who can afford it can use it.

 

Categories: Networking, Tech Tags: ,

Elasticsearch, Logstash, Kibana (ELK) group on LinkedIn

April 29th, 2014 No comments

I started a group for ELK on LinkedIn.  The direct link is

https://www.linkedin.com/groups?home=&gid=6686144&trk=anet_ug_hm

I have also started a Meetup for the Mountain View area.

Mountain View ELK Meetup

 

Congratulations to LinkedIn for reaching 300million members!

April 20th, 2014 No comments

That’s quite an accomplishment.

http://blog.linkedin.com/2014/04/18/the-next-three-billion/

There are now sites in that stratospheric level, but still, it is not an easy accomplishment.  Congraluations LinkedIn!

 

 

Categories: Facts, Tech Tags: ,

SGI T-shirt

March 19th, 2014 No comments

A post on LinkedIn’s ex-SGIers group gotme looking for my SGI t-shirts.  One of my favorite pastime is to collect t-shirts, cups, etc. from the companies that I’ve worked for…. and I’ve worked for a lot of companies here in Silicon Valley.

I found my Jurassic T-shirt that I got when Jurassic Park, the movie, came out.  That was very big at the time, and I remember SGI rented the near by Shoreline movie theater and let the entire company watch it.  That was awesome!

Here is a picture of my T-shirts.  It’s a little bit worn out — what do you expect for a 20 years old t-shirt.  Also found another SGI t-shirt, this one is for Digital Studio  package.

 

Digital Studio T-shirt

Digital Studio

SGI Jurassic Park t-shirt

Jurassic Park

Categories: Tech Tags: , ,