tinman alley

A question that I’ve seen asked many time on the web is people asking how to shorten their Grafana legend/label.

E.g. using hostname as a legend will usually return the full FQDN, which can be too long if you have many hosts and make a mess of your panel.

Lots of searching show people having similar questions and a number of request for enhancements. In my case, there is a simple solution that work. Here is how.

Use the function label_replace(). So

rate(nginx_http_requests_total{instance=~"$instance", host="$host"}[5m])

turn into

label_replace(rate(nginx_http_requests_total{instance=~"$instance", host="$host"}[5m]), "hname", "$1", "instance", "(.*).foo.bar.local")

 

And the legend format changed from

{{instance}}-{{status}}  to {{hname}}-{{status}}

https://gluesolution.xyz/devops/2018/05/22/What-Is-Difference-Between-An-API-Gateway-And-A-Service-Mesh.html

Intuitively, I knew they are different, but could not explain it as clearly as the above post.

 

If you are using sendgrid as a service for your outbound email, you would want to monitor and be able to answer questions such as:

• how much email are you sending
• status of sent email – success, bounced, delayed, etc.
• trends
• etc.

We get questions all the time from $WORK customer support folks on whether an email sent to a customer got there (customer claimed they never got it).   There could be any number of reasons why customer do not see email sent from us.

• our email is filtered into customer spam folder
• email is reject/bounced by customer mail service
• any number of network/server/services related errors between us and customer mail service
• the email address customer provided is invalid (and email bounced)

If we have access to event logs from sendgrid, we would be able to quickly answer these types of questions.

Luckily sendgrid offers Event Webhook.

Verbatim quote from above link.

SendGrid’s Event Webhook will notify a URL of your choice via HTTP POST with information about events that occur as SendGrid processes your email. Common uses of this data are to remove unsubscribes, react to spam reports, determine unengaged recipients, identify bounced email addresses, or create advanced analytics of your email program. With Unique Arguments and Category parameters, you can insert dynamic data that will help build a sharp, clear image of your mailings.

Login to your sendgrid account and click on Mail Settings.

Then click on Event Notification

 

In HTTP Post URL, enter the DNS name of the service endpoint you are going to setup next.

For example, mine is (not a valid endpoint, but close enough): https://sendlog.mydomain.com/logger

Since I do not believe in re-inventing the wheel, Adly Abdullah has already written a simple sendgrid event listener (Note: this is my forked version, which works with ES 6.x).   This is a nodejs service.  You can install it via npm.

$ sudo npm install -g sendgrid-event-logger pm2

You want to install pm2 (nodejs Process Manager version 2).  Very nice nodejs process manager.

Next is to edit and configure sendgrid-event-logger (SEL for short).   If the default config works for you, then no need to do anything.  Check and make sure it is pointing to where your ES host is located (mine is running on the same instance, hence localhost).   I also left SEL listening on port 8080 as that is available on this instance.

$ cat /etc/sendgrid-event-logger.json
{
    "elasticsearch_host": "localhost:9200",
    "port": 8080,
    "use_basicauth": true,
    "basicauth": {
    "user": "sendgridlogger",
    "password": "KLJSDG(#@%@!gBigSecret"
},
"use_https": false,
    "https": {
        "key_file": "",
        "cert_file": ""
    },
    "days_to_retain_log": 365
}

NOTE: I have use_https set to false because my nginx front-end is already using https.

Since SEL is listening on port 8080, you can run it as yourself.

$ pm2 start sendgrid-event-logger -i 0 --name "sendgrid-event-logger"

Verify that SEL is running.

$ pm2 show 0

Describing process with id 0 - name sendgrid-event-logger
┌───────────────────┬──────────────────────────────────────────────────────┐
│ status            │ online                                               │
│ name              │ sendgrid-event-logger                                │
│ restarts          │ 0                                                    │
│ uptime            │ 11m                                                  │
│ script path       │ /usr/bin/sendgrid-event-logger                       │
│ script args       │ N/A                                                  │
│ error log path    │ $HOME/.pm2/logs/sendgrid-event-logger-error-0.log    │
│ out log path      │ $HOME/.pm2/logs/sendgrid-event-logger-out-0.log      │
│ pid path          │ $HOME/.pm2/pids/sendgrid-event-logger-0.pid          │
│ interpreter       │ node                                                 │
│ interpreter args  │ N/A                                                  │
│ script id         │ 0                                                    │
│ exec cwd          │ $HOME                                                │
│ exec mode         │ fork_mode                                            │
│ node.js version   │ 8.11.1                                               │
│ watch & reload .  │ ✘                                                    │
│ unstable restarts │ 0                                                    │
│ created at        │ 2018-02-14T23:36:06.705Z                             │
└───────────────────┴──────────────────────────────────────────────────────┘
Code metrics value
┌─────────────────┬────────┐
│ Loop delay .    │ 0.68ms │
│ Active requests │ 0      │
│ Active handles  │ 4      │
└─────────────────┴────────┘

I use nginx and here is my nginx config for SEL.

/etc/nginx/sites-available $ cat sendgrid-logger
upstream sendgrid_logger {
  server 127.0.0.1:8080;
}

server {
  server_name slog.mysite.org slog;
  listen 443 ssl ;

  include snippets/ssl.conf;
  access_log /var/log/nginx/slog/access.log;
  error_log /var/log/nginx/slog/error.log;
  proxy_connect_timeout 5m;
  proxy_send_timeout 5m;
  proxy_read_timeout 5m;

  location / {
    proxy_pass http://sendgrid_logger;
  }
}
$ sudo ln -s /etc/nginx/sites-available/sendgrid-logger /etc/nginx/sites-enabled/
$ sudo systemctl reload nginx

Make sure Sendgrid Event webhook is turned on and you should be seeing events coming in.   Check your Elasticsearch cluster for new indices.

$ curl -s localhost:9200/_cat/indices|grep mail
green open mail-2018.03.31 -g6Tw9b9RfqZnBVYLdrF-g 1 0 2967 0 1.4mb 1.4mb
green open mail-2018.03.28 GxTRx2PgR4yT5kiH0RKXrg 1 0 8673 0 4.2mb 4.2mb
green open mail-2018.04.06 2PO9YV1eS7eevZ1dfFrMGw 1 0 10216 0 4.9mb 4.9mb
green open mail-2018.04.11 _ZINqVPTSwW7b8wSgkTtTA 1 0 8774 0 4.3mb 4.3mb
etc.

Go to Kibana, setup index pattern.  In my case, it’s mail-*.  Go to Discover, select mail-* index pattern and play around.

Here is my simple report.  I see around 9am, something happened to cause a huge spike in mail events.

 

Next step is for you to create dashboards to fit your needs.

Enjoy!

 

I created an RSS for cryptocurrency prices. It list the 10 that I am interested in, but I can certainly add more. Let me know if you want to add one to the list.

https://blog.tinle.org/blockchain/

This is an awesome article from Alexy Ivanov on tuning your web servers.

https://blogs.dropbox.com/tech/2017/09/optimizing-web-servers-for-high-throughput-and-low-latency/

So sad… but it’s inevitable, Oracle killing Solaris and Sun.

Oracle Finally Killed Sun

This news was buried among many other news, but I felt that it deserves more people knowing about it.

It is about “fair use” of publicly available web content. What is “fair use” and when can content be restricted.

The original article is here.

A small company called hiQ is locked in a high-stakes battle over Web scraping with LinkedIn. It’s a fight that could determine whether an anti-hacking law can be used to curtail the use of scraping tools across the Web.

HiQ scrapes data about thousands of employees from public LinkedIn profiles, then packages the data for sale to employers worried about their employees quitting. LinkedIn, which was acquired by Microsoft last year, sent hiQ a cease-and-desist letter warning that this scraping violated the Computer Fraud and Abuse Act, the controversial 1986 law that makes computer hacking a crime. HiQ sued, asking courts to rule that its activities did not, in fact, violate the CFAA.

James Grimmelmann, a professor at Cornell Law School, told Ars that the stakes here go well beyond the fate of one little-known company.

I will leave it up to you to read and make up your own opinion about it.

Big celebration for LinkedIn as the company hits 500M + members.

This picture was at LinkedIn HQ in Sunnyvale.  I am the guy in the middle of that red circle.

The picture was taken by a mavic pro drone, flying above the building.  The drone belongs to one of my colleague.

 

 

Getting this error in your Kibana?

You need to increase your max header size as default netty is only 8KB.   You can change the value in your elasticsearch.yml file.

Add the following line (or uncomment it if it is already there).

http.max_header_size: 32kb

 

I have seen questions on the web about how to migrate (copy/move) files from one GDrive account to another. There are many reasons, such as migrating from one Google account (such as company) to your personal account, etc.

WARNING: you may be violating your company policy by moving/copying files from your company Google account to a personal. I advise you to consult your company security officer or equivalent before doing this.

There are other reasons for wanting to copy or moving large number of files from one GDrive to another. Such as for me. I shared a folder in my GDrive with my family for putting our family photos in a central location. My family have G account, and there own GDrive. It seem that Google make it painful to copy files from one GDrive to another. Their suggestions is some form of downloading the files to your local drive first, and then uploading it to the other GDrive that you want.

This is painful!!! There are so many reasons why it’s painful…. 😉

The solution I’ve used is to install Google Drive app (supports OSX, Windows, Linux, Android and IOS).

Link Google Drive app to one Google account, and now you can treat the files in it as on your local drive and drag from there to the GDrive account you want to copy to.