tinman alley

Enjoyed reading this post

https://gluesolution.xyz/devops/2018/05/22/What-Is-Difference-Between-An-API-Gateway-And-A-Service-Mesh.html

Intuitively, I knew they are different, but could not explain it as clearly as the above post.

If you are using sendgrid as a service for your outbound email, you would want to monitor and be able to answer questions such as:

• how much email are you sending
• status of sent email – success, bounced, delayed, etc.
• trends
• etc.

We get questions all the time from $WORK customer support folks on whether an email sent to a customer got there (customer claimed they never got it).   There could be any number of reasons why customer do not see email sent from us.

• our email is filtered into customer spam folder
• email is reject/bounced by customer mail service
• any number of network/server/services related errors between us and customer mail service
• the email address customer provided is invalid (and email bounced)

If we have access to event logs from sendgrid, we would be able to quickly answer these types of questions.

Luckily sendgrid offers Event Webhook.

Verbatim quote from above link.

SendGrid’s Event Webhook will notify a URL of your choice via HTTP POST with information about events that occur as SendGrid processes your email. Common uses of this data are to remove unsubscribes, react to spam reports, determine unengaged recipients, identify bounced email addresses, or create advanced analytics of your email program. With Unique Arguments and Category parameters, you can insert dynamic data that will help build a sharp, clear image of your mailings.

Login to your sendgrid account and click on Mail Settings.

Then click on Event Notification

In HTTP Post URL, enter the DNS name of the service endpoint you are going to setup next.

For example, mine is (not a valid endpoint, but close enough): https://sendlog.mydomain.com/logger

Since I do not believe in re-inventing the wheel, Adly Abdullah has already written a simple sendgrid event listener (Note: this is my forked version, which works with ES 6.x).   This is a nodejs service.  You can install it via npm.

$ sudo npm install -g sendgrid-event-logger pm2

You want to install pm2 (nodejs Process Manager version 2).  Very nice nodejs process manager.

Next is to edit and configure sendgrid-event-logger (SEL for short).   If the default config works for you, then no need to do anything.  Check and make sure it is pointing to where your ES host is located (mine is running on the same instance, hence localhost).   I also left SEL listening on port 8080 as that is available on this instance.

$ cat /etc/sendgrid-event-logger.json
{
    "elasticsearch_host": "localhost:9200",
    "port": 8080,
    "use_basicauth": true,
    "basicauth": {
    "user": "sendgridlogger",
    "password": "KLJSDG(#@%@!gBigSecret"
},
"use_https": false,
    "https": {
        "key_file": "",
        "cert_file": ""
    },
    "days_to_retain_log": 365
}

NOTE: I have use_https set to false because my nginx front-end is already using https.

Since SEL is listening on port 8080, you can run it as yourself.

$ pm2 start sendgrid-event-logger -i 0 --name "sendgrid-event-logger"

Verify that SEL is running.

$ pm2 show 0

Describing process with id 0 - name sendgrid-event-logger
┌───────────────────┬──────────────────────────────────────────────────────┐
│ status            │ online                                               │
│ name              │ sendgrid-event-logger                                │
│ restarts          │ 0                                                    │
│ uptime            │ 11m                                                  │
│ script path       │ /usr/bin/sendgrid-event-logger                       │
│ script args       │ N/A                                                  │
│ error log path    │ $HOME/.pm2/logs/sendgrid-event-logger-error-0.log    │
│ out log path      │ $HOME/.pm2/logs/sendgrid-event-logger-out-0.log      │
│ pid path          │ $HOME/.pm2/pids/sendgrid-event-logger-0.pid          │
│ interpreter       │ node                                                 │
│ interpreter args  │ N/A                                                  │
│ script id         │ 0                                                    │
│ exec cwd          │ $HOME                                                │
│ exec mode         │ fork_mode                                            │
│ node.js version   │ 8.11.1                                               │
│ watch & reload .  │ ✘                                                    │
│ unstable restarts │ 0                                                    │
│ created at        │ 2018-02-14T23:36:06.705Z                             │
└───────────────────┴──────────────────────────────────────────────────────┘
Code metrics value
┌─────────────────┬────────┐
│ Loop delay .    │ 0.68ms │
│ Active requests │ 0      │
│ Active handles  │ 4      │
└─────────────────┴────────┘

I use nginx and here is my nginx config for SEL.

/etc/nginx/sites-available $ cat sendgrid-logger
upstream sendgrid_logger {
  server 127.0.0.1:8080;
}

server {
  server_name slog.mysite.org slog;
  listen 443 ssl ;

  include snippets/ssl.conf;
  access_log /var/log/nginx/slog/access.log;
  error_log /var/log/nginx/slog/error.log;
  proxy_connect_timeout 5m;
  proxy_send_timeout 5m;
  proxy_read_timeout 5m;

  location / {
    proxy_pass http://sendgrid_logger;
  }
}
$ sudo ln -s /etc/nginx/sites-available/sendgrid-logger /etc/nginx/sites-enabled/
$ sudo systemctl reload nginx

Make sure Sendgrid Event webhook is turned on and you should be seeing events coming in.   Check your Elasticsearch cluster for new indices.

$ curl -s localhost:9200/_cat/indices|grep mail
green open mail-2018.03.31 -g6Tw9b9RfqZnBVYLdrF-g 1 0 2967 0 1.4mb 1.4mb
green open mail-2018.03.28 GxTRx2PgR4yT5kiH0RKXrg 1 0 8673 0 4.2mb 4.2mb
green open mail-2018.04.06 2PO9YV1eS7eevZ1dfFrMGw 1 0 10216 0 4.9mb 4.9mb
green open mail-2018.04.11 _ZINqVPTSwW7b8wSgkTtTA 1 0 8774 0 4.3mb 4.3mb
etc.

Go to Kibana, setup index pattern.  In my case, it’s mail-*.  Go to Discover, select mail-* index pattern and play around.

Here is my simple report.  I see around 9am, something happened to cause a huge spike in mail events.

Next step is for you to create dashboards to fit your needs.

Enjoy!

USB Ethernet Adapters for TiVo

plaza-prize

Here is a collected list of USB adapters I got from http://www.tivoco
mmunity.com/tivo-vb/showthread.php?s=&threadid=54620&pagenumber=3

I bought a cheap one (Farallon USB1.1 to ethernet) for $13 from Computer
Geek, and it worked great. Just plug-n-play 🙂

09/11/2005Got word from Antonio Carlos that a Linksys USB200M
works great.

06/17/2004 I’ve received feedback from Rob Clark
that a D-Link DSB-H3ETX (USB to enet adapter) also work.
He bought his locally for $15 and the link he sent is http://support.dlink.com/products/v
iew.asp?productid=DSB%2DH3ETX.

Basically any USB-enet adapters that uses the Pegasus chipset should
work with Tivo as Linux has driver support for that chip.

3COM
3Com USB Ethernet 3C460B

ABOCOM
USB 10/100 Fast Ethernet
USB HPNA/Ethernet

ACCTON
Accton USB 10/100 Ethernet Adapter
SpeedStream USB 10/100 Ethernet

ADMTEK
ADMtek ADM8511 Pegasus II USB Ethernet
ADMtek AN986 Pegasus USB Ethernet (eval. board)

ALLIEDTEL
Allied Telesyn Int. AT-USB100

BELKIN
Belkin F5D5050 USB Ethernet

BILLIONTON
Billionton USB-100
Billionton USBE-100
Billionton USBEL-100
Billionton USBLP-100

COMPAQ
iPAQ Networking 10/100 USB

COREGA
Corega FEter USB-TX

DLINK
D-Link DSB-650
D-Link DSB-650TX
D-Link DSB-650TX(PNA)

ELSA
Elsa Micolink USB2Ethernet

HAWKING
Hawking UF100 10/100 Ethernet

IODATA
IO DATA USB ET/TX
IO DATA USB ET/TX-S

KINGSTON
Kingston KNU101TX Ethernet

LANEED
LANEED USB Ethernet LD-USB/T
LANEED USB Ethernet LD-USB/TX

LINKSYS
Linksys USB100TX
Linksys USB10TX
Linksys USB Ethernet Adapter
Linksys USB USB10TX
Linksys USB100M
Linksys USB200M

MELCO
MELCO/BUFFALO LUA2-TX
MELCO/BUFFALO LUA-TX

SIEMENS
SpeedStream USB 10/100 Ethernet

SMARTBRIDGES
smartNIC 2 PnP Adapter

SMC
SMC 202 USB Ethernet

SOHOWARE
SOHOware NUB100 Ethernet

Tin’s Home…

Last Updated: $Date: 2003/08/19 04:32:49 $

I created an RSS for cryptocurrency prices. It list the 10 that I am interested in, but I can certainly add more. Let me know if you want to add one to the list.

https://blog.tinle.org/blockchain/

You may have notice that this blog is mostly unavailable or showing 5xx lately. It’s because I am on AWS and the recent Intel vulns has all the cloud vendors patching and rebooting their hypervisors. It’s causing various issues with my infrastructure.

I don’t blame the vendors, they are doing what they are supposed to be doing :-). I am waiting for my turn…. when the clouds are done with their patching, then I have to patch my instances and reboot them too. Ugh, joy….

This is an awesome article from Alexy Ivanov on tuning your web servers.

https://blogs.dropbox.com/tech/2017/09/optimizing-web-servers-for-high-throughput-and-low-latency/

So sad… but it’s inevitable, Oracle killing Solaris and Sun.

Oracle Finally Killed Sun

This news was buried among many other news, but I felt that it deserves more people knowing about it.

It is about “fair use” of publicly available web content. What is “fair use” and when can content be restricted.

The original article is here.

A small company called hiQ is locked in a high-stakes battle over Web scraping with LinkedIn. It’s a fight that could determine whether an anti-hacking law can be used to curtail the use of scraping tools across the Web.

HiQ scrapes data about thousands of employees from public LinkedIn profiles, then packages the data for sale to employers worried about their employees quitting. LinkedIn, which was acquired by Microsoft last year, sent hiQ a cease-and-desist letter warning that this scraping violated the Computer Fraud and Abuse Act, the controversial 1986 law that makes computer hacking a crime. HiQ sued, asking courts to rule that its activities did not, in fact, violate the CFAA.

James Grimmelmann, a professor at Cornell Law School, told Ars that the stakes here go well beyond the fate of one little-known company.

I will leave it up to you to read and make up your own opinion about it.

Anyone having issue getting warranty service for their solar panel converter? Enphase claimed 15 years warranty. My system was installed in mid 2010, and one of the converter had failed. It’s the only one reporting low/no voltage for the past 4 weeks. The rest of my 20+ panels and converters are fine.

It look like Enphase is not geared to support home end users. They kept re-directing me to my “installer”. Unfortunately, my installer had gone out of business a few years ago. Yes, live and learn for me. Next time I’ll use a more reputable company.

In any case, Enphase is giving me the run around. Sounds like time to complain to Consumer Protection Agency and local state agency.

Amazon customer service reps reaction after answering a call.

I feel like I am caught in an episode of bad customer service TV show, except this is real and it’s from Amazon.

Hello customer. I’ll be glad to help you, but first, please verify you are a customer.

But, but, that’s the reason I am calling….

I am sorry, I can’t help you if you can not prove you are a customer.

As a long time Amazon Prime customer, I have 2FA turns on. It’s the usual send a code to my phone, with the fall back is to run an authenticator app on the phone and enter code from that app. All well and good, works well for some time now. Except that phone happens to be my work phone, and now that I am no longer working there, I do not have the phone anymore. Oops!

In my defense, I did remember to update all other online accounts to use my personal phone. The only one I forgot was my Amazon account.

Last night, Friday, after coming home from my last day at $WORK. I tried updating my Amazon account and immediately run into problems. I can’t login since Amazon want to send code to my work phone, which I do not have any longer. I can not use code from authenticator app (Google Authenticator), since it’s also tied to that phone. I contacted Amazon customer service via the phone number that popped up when I was having problem login.

Cue dramatic music. The customer service rep was courteous and nice, but could not help me because I could not tell him the code that he sent to my phone…. LOL. I tried explaining that I no longer have that phone. This person did not know what to do, put me on hold for 10 minutes and come back online to tell me he can’t help me. I asked to escalate to his manager, his answer was because he could not verify that I am who I said I am, he can not escalate and can not help me…. I hang up and sent an email to Amazon support asking for help, explaining my problem and ask them to call my house phone number to verify (my house number is in my account settings).

I got an email reply from Amazon support telling me that they can not call me, but they gave me a number to call for help. I called the number and run into the same problem. The service rep can not verify me by sending a code to “my phone”. After explaining again the issue, the rep said that he will check with someone. I was put on hold for more than 10 minutes, and got disconnected while waiting.

By this point, I was not happy, so I clicked on the support email feedback which takes me to an Amazon web page. After giving the lowest rating and clicking submit, right away I got a popup that ask for my current phone number (so Amazon can call you back). After entering my home number, I immediately got a call from Amazon support. We went through the same process as before, where we got to the verification point and the customer service rep realized I do not have phone to receive verification code and I can not run authenticator app on that phone. This rep also asked to put me on hold so he can consult with someone. After a long 15 minutes or so wait, he came back and said someone will email me with instruction on how to resolve this.

That conversation was around noon today (Saturday 4/29/2017). It is now almost 6pm, I still have not seen any email from Amazon support.

It’s hard to believe that I am the first Amazon customer to run into this issue. This over reliance on using a phone as the proof of identity is single point of failure. What about all the other information? Such as my home phone? Obviously they can and do call my home phone as my earlier support call from them show. They could and did ask about other information in my account to verify, why is that not enough?

I have shot off another email to Amazon support asking for escalation. Funny thing, while trying to send this email to support, Amazon wants me to login to my account…. arrrgggg!

I’ll just post this experience here as a tale of how not to design your 2FA without adequate fall back. Problems happen and you need to have another method to reset login that does not depends on the very device that is used for 2FA. Most importantly, you need to give your customer service personnel ways to deal with unexpected circumstances, beyond just reading from scripts. My experiences with Amazon customer service was terrible! Refusing to help me because they can’t verify me? Will not escalate until I can prove I am a customer? Seriously?

Update 2017/04/30 Sun – Amazon service responded to my last email asking for help with the response that they are going to reset my password. That’s not going to help. I know my password, I don’t have my phone so I can’t login. This reminds me of another person’s trouble trying to get help from Amazon customer service.

Bad Amazon customer service

Update 2017/04/30 Sun – once more, I clicked on the feedback button in the support email, and gave a 1 star review. Got a popup to enter my phone number so Amazon can call me. This time, I got someone calling from state of Washington (last 3 times were from non-US support centers). Fourth time was the charm. This service rep had run into another customer with similar problem as mine before and she knew what was needed. She had to pass me over to the 2FA customer service team (hmmm). They sent me an email with link to: Amazon 2FA recovery web page.

Essentially, I have to verify my identity by uploading a picture (scan or photo) of government-issued identity document. It will take 1-2 days after that for Amazon to do what they need and remove the 2FA from my account.

Took a picture of my driver license, blacked out sensitive data, leaving only my name and home address and submitted it to the recovery page. I find it amusing that Amazon think this is more secure. With today’s graphic editors, I could have easily created a fake photo ID claiming to be me.

Update 2017/05/01 Mon – I received an email from Amazon support saying that they have disabled my 2FA. I tested it and was able to login to my account. I went to reset 2FA to my new personal phone and tested it again. Finally! Everything is working again.

Conclusion

Terrible customer experience. Bad security theater. I understand the need to verify users and protect their data, but the methods for doing so and the training Amazon provide to their customer service personnel is very lacking.

Customer Service training

Amazon need to train service reps on how to deal with the unexpected, beyond their scripted responses. They should enable their reps to escalate to higher level of support as needed. The big fail was refusing to help because a user can not provide proof of their identity. Imagine you just got robbed and now the police refused to help you because you can not prove your identity.

Bad security

Requiring additional verification when the primary method failed or not available is fine. But making users jump through hoop based on perceived security is not. Requiring users to send in photos or scans of government issued IDs is security theater. With modern graphic editing tools, and so many scanned pictures already available on the internet, it is easy to create spoofed IDs and submitting them. Especially when the only requirement was the name and home address on the ID has to match what Amazon has about user.

Since Amazon already have shared information about the users, why not query the user on that as proof of identity? If there are concern about access to personal data, then have dedicated support team that is only for this identity verification task. This team can only access a particular user’s data when that user need to be verified. The access is logged and documented.