Archive

Archive for the ‘Tech’ Category

Fair use of web content

August 11th, 2017 1 comment

This news was buried among many other news, but I felt that it deserves more people knowing about it.

It is about “fair use” of publicly available web content. What is “fair use” and when can content be restricted.

The original article is here.

A small company called hiQ is locked in a high-stakes battle over Web scraping with LinkedIn. It’s a fight that could determine whether an anti-hacking law can be used to curtail the use of scraping tools across the Web.

HiQ scrapes data about thousands of employees from public LinkedIn profiles, then packages the data for sale to employers worried about their employees quitting. LinkedIn, which was acquired by Microsoft last year, sent hiQ a cease-and-desist letter warning that this scraping violated the Computer Fraud and Abuse Act, the controversial 1986 law that makes computer hacking a crime. HiQ sued, asking courts to rule that its activities did not, in fact, violate the CFAA.

James Grimmelmann, a professor at Cornell Law School, told Ars that the stakes here go well beyond the fate of one little-known company.

I will leave it up to you to read and make up your own opinion about it.

LinkedIn Celebrate 500M members April, 2017

April 28th, 2017 No comments

LinkedIn 500M members celabratory picture.

Big celebration for LinkedIn as the company hits 500M + members.

This picture was at LinkedIn HQ in Sunnyvale.  I am the guy in the middle of that red circle.

The picture was taken by a mavic pro drone, flying above the building.  The drone belongs to one of my colleague.

 

 

Courier Fetch Error: unhandled courier request error: Authorization Exception in Chrome/Safari on Kibana 4.5.0

August 22nd, 2016 No comments

Getting this error in your Kibana?

You need to increase your max header size as default netty is only 8KB.   You can change the value in your elasticsearch.yml file.

Add the following line (or uncomment it if it is already there).

http.max_header_size: 32kb

 

Moving or copying files from one Google drive account to another

July 20th, 2016 No comments

I have seen questions on the web about how to migrate (copy/move) files from one GDrive account to another. There are many reasons, such as migrating from one Google account (such as company) to your personal account, etc.

WARNING: you may be violating your company policy by moving/copying files from your company Google account to a personal. I advise you to consult your company security officer or equivalent before doing this.

There are other reasons for wanting to copy or moving large number of files from one GDrive to another. Such as for me. I shared a folder in my GDrive with my family for putting our family photos in a central location. My family have G account, and there own GDrive. It seem that Google make it painful to copy files from one GDrive to another. Their suggestions is some form of downloading the files to your local drive first, and then uploading it to the other GDrive that you want.

This is painful!!! There are so many reasons why it’s painful…. 😉

The solution I’ve used is to install Google Drive app (supports OSX, Windows, Linux, Android and IOS).

Link Google Drive app to one Google account, and now you can treat the files in it as on your local drive and drag from there to the GDrive account you want to copy to.

Fixing ‘plugin:elasticsearch [document_already_exists_exception] [config][4.5.1]: document already exists’

June 11th, 2016 No comments

Substitute in the version ‘4.5.1’ with the version you are upgrading to. So far I’ve seen it since Kibana 4.1.x to 4.5.1.

It seem that if you upgrade Kibana, there is a timing bug in how Kibana note its current version. You will get lots of these errors in Kibana logs:

log [08:08:30.649] [error][status][plugin:elasticsearch] Status changed from green to red - [document_already_exists_exception] [config][4.5.1]: document already exists, with: {"shard":"0","index":".kibana"}

These came from me upgrading version 4.5.0 to 4.5.1. I’ve seen same thing when I went from 4.1.4 to 4.5.0.

The fix is to delete the config record in your .kibana index. Don’t worry, it gets recreated again. No loss as far as I know.

curl -XDELETE elasticsearchserver:9200/.kibana/config/4.5.1

The Kibana bug is documented here: kibana issues #5519.

If deleting record does not work, you will also need to refresh your kibana index, e.g. this will flush the data!!!!

curl -XPOST elasticsearchserver:9200/.kibana/_refresh

Categories: Elasticsearch, ELK, Tech Tags: ,

HOW TO add search-guard-ssl to Elasticsearch

March 21st, 2016 1 comment

If you have a need to encrypt communication between your Elasticsearch nodes, but do not (yet) need the complicated ACL provided from either Shield (Elastic commercial product) or Search-Guard (open source), then you can use Search-Guard-SSL (open source).

I am going to show you how to add Search-Guard-SSL (SG-SSL for short) to Elasticsearch. There are a few requirements.

SG-SSL requires Elasticsearch version 2.0.x or newer. Make sure you are using the correct version!

First, download the correct version (zip) file from here.

Second, verify the integrity of your downloaded file.

$ curl -o search-guard-ssl-2.2.1.7.jar https://oss.sonatype.org/content/repositories/releases/com/floragunn/search-guard-ssl/2.2.1.7/search-guard-ssl-2.2.1.7.jar
$ curl -o search-guard-ssl-2.2.1.7.jar.asc https://oss.sonatype.org/content/repositories/releases/com/floragunn/search-guard-ssl/2.2.1.7/search-guard-ssl-2.2.1.7.jar.asc

$ gpg --verify search-guard-ssl-2.2.1.7.jar.asc search-guard-ssl-2.2.1.7.jar

Third, you need to use a cert — either generate your own; or one that you have purchased/generated by your Corp IT — I am not going to go into it here.

Fourth, decide where your trust store and cert are going to reside and configure elasticsearch.yml as appropriate.

Below is just the configuration specific to SG-SSL that need to be added to your elasticsearch.yml. Edit it as appropriate and add it to your Elasticsearch config.


######################################################################################
# HTTP/REST layer SSL
# NOTE: Here, I am only using transport (node to node) encryption.
# I am NOT using HTTP encryption as I want to be able to use the REST API without
# requiring HTTPS. I have HTTP (port 9200) bind to localhost only. You may need to
# turn it on depending on your security policy.
######################################################################################
searchh.guard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: PKCS12
searchguard.ssl.transport.keystore_filepath: /export/apps/my-elk-cluster/var/identity.p12
# Alias name (default: first alias which could be found)
#searchguard.ssl.transport.keystore_alias: my_alias
# passwords here are not really in use. Java has a bug where password-less keystores don't work.
searchguard.ssl.transport.keystore_password: my-keystore-password
searchguard.ssl.transport.truststore_type: JKS
searchguard.ssl.transport.truststore_filepath: /etc/pki/certs/cacerts
# Alias name (default: first alias which could be found)
#searchguard.ssl.transport.truststore_alias: my_alias
searchguard.ssl.transport.truststore_password: changeit
searchguard.ssl.transport.truststore_alias: my-alias
searchguard.ssl.transport.enforce_hostname_verification: true
searchguard.ssl.transport.resolve_hostname: true
searchguard.ssl.transport.enable_openssl_if_available: false

#####################################################################################
# Enable or disable rest layer security - https, (default: false)
searchguard.ssl.http.enabled: false
# JKS or PKCS12 (default: JKS)
#searchguard.ssl.http.keystore_type: PKCS12
# Relative path to the keystore file (this stores the server certificates), must be placed under the config/ d
ir
#searchguard.ssl.http.keystore_filepath: keystore_https_node1.jks
# Alias name (default: first alias which could be found)
#searchguard.ssl.http.keystore_alias: my_alias
# Keystore password (default: changeit)
#searchguard.ssl.http.keystore_password: changeit
# Do the clients (typically the browser or the proxy) have to authenticate themself to the http server, defaul
t is false
#searchguard.ssl.http.enforce_clientauth: false
# JKS or PKCS12 (default: JKS)
#searchguard.ssl.http.truststore_type: PKCS12
# Relative path to the truststore file (this stores the client certificates), must be placed under the config/
dir
#searchguard.ssl.http.truststore_filepath: truststore_https.jks
# Alias name (default: first alias which could be found)
#searchguard.ssl.http.truststore_alias: my_alias
# Truststore password (default: changeit)
#searchguard.ssl.http.truststore_password: changeit
# Use native Open SSL instead of JDK SSL if available (default: true)
searchguard.ssl.http.enable_openssl_if_available: false

That’s it. Now deploy to all your nodes in cluster and the nodes should be communicating over SSL. The above SG-SSL config only turn on SSL for node(transport) but leave REST (HTTP) un-encrypted. This is because I have my ES nodes bind HTTP (9200) to localhost, you have to be able to login to my ES nodes to access the REST port.

NOTE

ES 2.0 and newer has the JDK security policy manager on by default. This will prevent SG-SSL from reading your truststore and certs if it is not located in ES config directory tree.

You will need to provide your own security policy file to give ES read permission to these files.

Here is how to do that:.

First is that you must tell the JDK you want to use your security policy mapping.


export JAVA_OPTS="-Djava.security.policy=/export/apps/my-elk-instance/var/java.policy $JAVA_OPTS"

$ cat /export/apps/my-elk-instance/var/java.policy
/* this lets ES mess with a folder in a strange place. */
grant {
permission java.io.FilePermission "/export/apps/my-elk-instance/-", "read";
permission java.io.FilePermission "/etc/pki/certs/*", "read";
};

More documentation can be found here: modules-scripting-security

Next post will show you have to get Tribe ES node working with SG-SSL.

Categories: Elasticsearch, ELK, Tech Tags: , ,

Kibana 4 with tribe node MasterNotDiscoveredException

December 19th, 2015 No comments

I use tribe nodes quite a lot at $work. It’s how we federate disparate ELK clusters and able to search across them. There are many reasons to have distinct ELK clusters in each data center and/or region.

Some of these are:

1. Elasticsearch does not work well when there is network latencies, which is guaranteed when your nodes are located geographically distant places. You could spend a lot of money to get fast network connection, or you can just have only local clusters. (Me? I pick saving money and avoiding head aches :-)).

2. It can get insanely expensive to create an ES cluster that span data centers/regions. The network bandwidth requirement, the data charges, the care and feeding of such a latency sensitive cluster…. OMG!

3. I don’t really think a 3rd reason is needed.

Although tribe nodes are great for federating ES clusters, there are some quirks in setting them up and caring for them (not as bad as ES clusters that span datacenter though).

One big gotcha for many people who are setting up tribe nodes for the first time is that tribe node can not create index. Tribe can only update, modify an existing index. What this mean is that if you point Kibana at a tribe node, you must first make sure you Kibana index is already created in one of the downstream ES cluster. Otherwise, you will have to create it yourself.

Otherwise, the first time you create an index pattern and tried to save it, you will get an error similar to the subject of this post.

MasterNotDiscoveredException

The error message is wrong and misleading. It has nothing to do with Master node. It has everything to do with tribe node not able to create (PUT) a Kibana index.

Personally, I prefer to make the Kibana index that I use with tribe to have its own unique name. So I run a dedicated Kibana instance pointing to the dedicated tribe (client) node.

Here are the steps I do to get a tribe node and its associated Kibana ready for use.

1. Configure the tribe node to know all the ES clusters I want to federate data from.

tribe.elasticsearch.yml:

cluster.name: toplevel_tribe
node.name: ${HOSTNAME}
node.master: false
node.data: false
tribe:
  DC1-appservice:
     cluster.name: logging-DC1
     network.host: 0.0.0.0
     network.publish_host: ${HOSTNAME}
     discovery.zen.ping.unicast.hosts:
      - dc1-app13225.prod.example.com
      - dc1-app13226.prod.example.com
      - dc1-app13227.prod.example.com
  DC2-appservice:
     cluster.name: logging-DC2
     network.host: 0.0.0.0
     network.publish_host: ${HOSTNAME}
     discovery.zen.ping.unicast.hosts:
      - dc2-app12281.prod.example.com
      - dc2-app12282.prod.example.com
      - dc2-app12283.prod.example.com
   DC3.....etc to DCNN

  my-es-dedicated-config-cluster:
     cluster.name: es-config-CORP
     network.host: 0.0.0.0
     network.publish_host: ${HOSTNAME}
     discovery.zen.ping.unicast.hosts:
      - corp-app1234.example.com

 on_conflict: prefer_my-es-dedicated-config-cluster

2. Now pre-create the Kibana index in my-ES-dedicated-config-cluster. This is a small cluster in my admin/corp data center that is only for housing configurations, Kibana dashboards, etc.

3. A simpler and more correct way is to temporary point Kibana to the dedicated ES cluster (instead of the tribe).

Do this via this setting in your kibana.yml file:

# The Elasticsearch instance to use for all your queries.
elasticsearch.url: “http://ES-node:9200”

Start Kibana, let it create the index.  Then stop it, change the setting back to point to your tribe node.

Doing it this way ensure that your kibana is correct.

curl command for pre-creating kibana (3 and 4) index:


curl -s -XPUT "http://localhost:9200/kibana3-int/" -d '{ "settings" : { "number_of_shards" : 3, "number_of_replicas" : 2 },
"mappings" : { "temp" : { "properties" : { "dashboard" : { "type" : "string" }, "group" : { "type" : "string" }, "title" : { "type" : "string" }, "user" : { "type" : "string" } } }, "dashboard" : { "properties" : { "dashboard" : { "type" : "string" }, "group" : { "type" : "string" }, "title" : { "type" : "string" }, "user" : { "type" : "string" } } } }'


# Kibana4
curl -s -XPUT "http://localhost:9200/TRIBENAME-kibana4" -d '{ "index.mapper.dynamic" : true, "settings" : { "number_of_shards" : 1, "number_of_replicas" : 0 },"mappings" : {"search" : {"_timestamp" : { },"properties" : {"columns" : {"type" : "string"},"description" : {"type" : "string"},"hits" : {"type" : "long"},"kibanaSavedObjectMeta" : {"properties" : {"searchSourceJSON" : {"type" : "string"}}},"sort" : {"type" : "string"},"title" : {"type" : "string"},"version" : {"type" : "long"}}},"dashboard" : {"_timestamp" : { },"properties" : {"description" : {"type" : "string"},"hits" : {"type" : "long"},"kibanaSavedObjectMeta" : {"properties" : {"searchSourceJSON" : {"type" : "string"}}},"optionsJSON" : {"type" : "string"},"panelsJSON" : {"type" : "string"},"timeRestore" : {"type" : "boolean"},"title" : {"type" : "string"},"uiStateJSON" : {"type" : "string"},"version" : {"type" : "long"}}},"visualization" : {"_timestamp" : { },"properties" : {"description" : {"type" : "string"},"kibanaSavedObjectMeta" : {"properties" : {"searchSourceJSON" : {"type" : "string"}}},"savedSearchId" : {"type" : "string"},"title" : {"type" : "string"},"uiStateJSON" : {"type" : "string"},"version" : {"type" : "long"},"visState" : {"type" : "string"}}},"config" : {"_timestamp" : { },"properties" : {"buildNum" : {"type" : "long"},"defaultIndex" : {"type" : "string"}}},"index-pattern" : {"_timestamp" : { },"properties" : {"customFormats" : {"type" : "string"},"fieldFormatMap" : {"type" : "string"},"fields" : {"type" : "string"},"intervalName" : {"type" : "string"},"timeFieldName" : {"type" : "string"},"title" : {"type" : "string"}}}}}'

%d bloggers like this: