HOW TO add search-guard-ssl to Elasticsearch

If you have a need to encrypt communication between your Elasticsearch nodes, but do not (yet) need the complicated ACL provided from either Shield (Elastic commercial product) or Search-Guard (open source), then you can use Search-Guard-SSL (open source).

I am going to show you how to add Search-Guard-SSL (SG-SSL for short) to Elasticsearch. There are a few requirements.

SG-SSL requires Elasticsearch version 2.0.x or newer. Make sure you are using the correct version!

First, download the correct version (zip) file from here.

Second, verify the integrity of your downloaded file.

$ curl -o search-guard-ssl-2.2.1.7.jar https://oss.sonatype.org/content/repositories/releases/com/floragunn/search-guard-ssl/2.2.1.7/search-guard-ssl-2.2.1.7.jar
$ curl -o search-guard-ssl-2.2.1.7.jar.asc https://oss.sonatype.org/content/repositories/releases/com/floragunn/search-guard-ssl/2.2.1.7/search-guard-ssl-2.2.1.7.jar.asc

$ gpg --verify search-guard-ssl-2.2.1.7.jar.asc search-guard-ssl-2.2.1.7.jar

Third, you need to use a cert — either generate your own; or one that you have purchased/generated by your Corp IT — I am not going to go into it here.

Fourth, decide where your trust store and cert are going to reside and configure elasticsearch.yml as appropriate.

Below is just the configuration specific to SG-SSL that need to be added to your elasticsearch.yml. Edit it as appropriate and add it to your Elasticsearch config.


######################################################################################
# HTTP/REST layer SSL
# NOTE: Here, I am only using transport (node to node) encryption.
# I am NOT using HTTP encryption as I want to be able to use the REST API without
# requiring HTTPS. I have HTTP (port 9200) bind to localhost only. You may need to
# turn it on depending on your security policy.
######################################################################################
searchh.guard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: PKCS12
searchguard.ssl.transport.keystore_filepath: /export/apps/my-elk-cluster/var/identity.p12
# Alias name (default: first alias which could be found)
#searchguard.ssl.transport.keystore_alias: my_alias
# passwords here are not really in use. Java has a bug where password-less keystores don't work.
searchguard.ssl.transport.keystore_password: my-keystore-password
searchguard.ssl.transport.truststore_type: JKS
searchguard.ssl.transport.truststore_filepath: /etc/pki/certs/cacerts
# Alias name (default: first alias which could be found)
#searchguard.ssl.transport.truststore_alias: my_alias
searchguard.ssl.transport.truststore_password: changeit
searchguard.ssl.transport.truststore_alias: my-alias
searchguard.ssl.transport.enforce_hostname_verification: true
searchguard.ssl.transport.resolve_hostname: true
searchguard.ssl.transport.enable_openssl_if_available: false

#####################################################################################
# Enable or disable rest layer security - https, (default: false)
searchguard.ssl.http.enabled: false
# JKS or PKCS12 (default: JKS)
#searchguard.ssl.http.keystore_type: PKCS12
# Relative path to the keystore file (this stores the server certificates), must be placed under the config/ d
ir
#searchguard.ssl.http.keystore_filepath: keystore_https_node1.jks
# Alias name (default: first alias which could be found)
#searchguard.ssl.http.keystore_alias: my_alias
# Keystore password (default: changeit)
#searchguard.ssl.http.keystore_password: changeit
# Do the clients (typically the browser or the proxy) have to authenticate themself to the http server, defaul
t is false
#searchguard.ssl.http.enforce_clientauth: false
# JKS or PKCS12 (default: JKS)
#searchguard.ssl.http.truststore_type: PKCS12
# Relative path to the truststore file (this stores the client certificates), must be placed under the config/
dir
#searchguard.ssl.http.truststore_filepath: truststore_https.jks
# Alias name (default: first alias which could be found)
#searchguard.ssl.http.truststore_alias: my_alias
# Truststore password (default: changeit)
#searchguard.ssl.http.truststore_password: changeit
# Use native Open SSL instead of JDK SSL if available (default: true)
searchguard.ssl.http.enable_openssl_if_available: false

That’s it. Now deploy to all your nodes in cluster and the nodes should be communicating over SSL. The above SG-SSL config only turn on SSL for node(transport) but leave REST (HTTP) un-encrypted. This is because I have my ES nodes bind HTTP (9200) to localhost, you have to be able to login to my ES nodes to access the REST port.

NOTE

ES 2.0 and newer has the JDK security policy manager on by default. This will prevent SG-SSL from reading your truststore and certs if it is not located in ES config directory tree.

You will need to provide your own security policy file to give ES read permission to these files.

Here is how to do that:.

First is that you must tell the JDK you want to use your security policy mapping.


export JAVA_OPTS="-Djava.security.policy=/export/apps/my-elk-instance/var/java.policy $JAVA_OPTS"

$ cat /export/apps/my-elk-instance/var/java.policy
/* this lets ES mess with a folder in a strange place. */
grant {
permission java.io.FilePermission "/export/apps/my-elk-instance/-", "read";
permission java.io.FilePermission "/etc/pki/certs/*", "read";
};

More documentation can be found here: modules-scripting-security

Next post will show you have to get Tribe ES node working with SG-SSL.

Migrating from Bamboo to Cedar (calendon stack) on Heroku

We just migrated from Bamboo (bamboo-ree-1.8.7) to Cedar (calendon rails 3.1 stack) on Heroku. Here are the steps I took to make it work. Note that I didn’t do the code migration, my dev handle that, I do the server and infrastructure part.

Basically this mean moving from Rails 2.x (and ruby 1.8.7) to Rails 3.1 (ruby 1.9.2). Some of the changes require upgrading gem packages. Things such as images, css and js has to be put into an asset bundle.

On the site itself, I left the old site running, e.g. oldsite.heroku.com, and created a new site at newsite.herokuapp.com.

We created a new git dev branch and pushed to newsite, e.g.

git push heroku dev:master

Since we use SSL, I have to make sure custom_domains addon is there. But…

heroku addons:add custom_domains:basic custom_domains:wildcard

I can’t add the ssl addon until I am ready, because Heroku requires that I defined the domains for the app first! Chicken and egg, as it mean I have to take down the currently running production site.

So, make sure everything is running on new site first. Because the next steps mean production site will be down during the changes.

0. Make sure your SSL cert is up-to-date and you have both part, domain.cert and private.key. And most importantly, your key is passphrase-less!

1. Make sure your DNS records are updated and have shortest possible TTL. You are changing them.

2. Wait till DNS changes (TTL) have settle, could be a few hours. Then update your production site CNAME from *.yourdomain.com to newsite.herokuapp.com.

3. Clear out the domains in your old Heroku site:

heroku domains:clear --app oldsite

4. Add domains to new site:

heroku domains:add '*.yourdomain.com' --app newsite
heroku domains:add 'www.yourdomain.com' --app newsite

5. Upload SSL cert to your new app:

heroku ssl:add yourdomain.crt private.key

6. Now you can add the ssl addon.

heroku addons:add ssl:hostname

7. You should get an email from Heroku that has the hostname for your SSL DNS record to point your DNS to. The name should be something like this:

appid1234567herokucom-1234567890.us-east-1.elb.amazonaws.com

Update your DNS CNAME for www.yourdomain.com to point to this name.

8. Test, test, and test again your spanking new site. Add the New Relic addon if you haven’t (life saver!) and monitor traffic to your new site.

9. Monitor the log of your new site:

heroku logs -t --app newsite

10. Check the processes:

heroku ps --app newsite

11. Wait a week, if everything look good, you can take down the oldsite.