I started using logstash-forwarder to send logs from my cloud hosted servers to my ELK server for analysis. Since it’s just a simple setup, I used the self-gen cert as described on logstash-forwarder’s github page.
Unfortunately, using the example generated a cert that is only good for 30 days. So suddenly my kibana graph show no data for my cloud servers…. ??? After some digging, I found errors like this in the log.
logstash-forwarder[4367]: 2014/07/01 23:24:08.559691 Failed to tls handshake with 172.25.28.52 x509: certificate has expired or is not yet valid
openssl x509 -in logstash-forwarder.crt -noout -text show that the Validity period was only 30 days. D’oh! 🙂
So I generated a new set, this time for 10 years. Why not, it’s for my use and if I am still using it 10 years from now…
openssl req -x509 -batch -nodes -newkey rsa:2048 -days 3560 -keyout logstash-forwarder.key -out logstash-forwarder.crt
Update 2014-07-28
Tried to bring up another server with logstash-forwarder. Except I used latest logstash-forwarder (git pull today 2014/07/25) and started getting this error when starting up LS.
Failed to tls handshake with 172.25.28.52 x509: certificate is valid for , not foo.bar.le.org
After a bit of debugging, comparing certs (exact same MD5 as the ones on working servers), I went googling and bingo!
https://github.com/elasticsearch/logstash-forwarder/issues/221
I see people blaming Go v1.3 TLS changes, but I am still using the same Go v1.2.1 that I built the currently working logstash-forwarder. And as a matter of fact, copying logstash-forwarder from existing working servers over to the new one and it works just fine! So I do not think that it’s Go, but something in the latest commits to logstash-forwarder that broke TLS.
Update 2014-08-17
Turned out to be my self-gen cert ;-P I created a new one, using properly filled out openssl.cnf and a wildcard domain. That works fine with latest trunk and built using go v1.2.1. I’ll update to go v1.3 soon.