Archive for the ‘AWS’ Category

Troubleshooting network connection

May 24, 2018

How to troubleshoot network connection between AWS Lambda, AWS EC2 through Application Load Balancer and outside world.

First try

curl https://thehost.com

returns empty.

Second try

curl --dump-header - https://thehost.com

returns something more meaningful but a bit misleading:

HTTP/1.1 302 Found
 Date: Thu, 24 May 2018 01:15:46 GMT
 Content-Type: text/plain; charset=utf-8
 Content-Length: 0
 Connection: keep-alive
 Location: login_page

telnet connects though:

telnet thehost 443
Trying 55.66.222.111...
Connected to thehost.com.
Escape character is '^]'.
^]
telnet> status
Connected to thehost.com.
Operating in obsolete linemode
Local character echo
Escape character is '^]'.
Connection closed by foreign host

Third try

curl -svo /dev/null https://thehost.com

gives us a hint with TLS info and the root of a problem below:

* Rebuilt URL to: https://thehost.com/
 * Trying 54.66.241.172...
 * Connected to thehost.com (55.66.222.111) port 443 (#0)
 * found 148 certificates in /etc/ssl/certs/ca-certificates.crt
 * found 592 certificates in /etc/ssl/certs
 * ALPN, offering http/1.1
 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
 * server certificate verification OK
 * server certificate status verification SKIPPED
 * common name: *.thehost.com (matched)
 * server certificate expiration date OK
 * server certificate activation date OK
 * certificate public key: RSA
 * certificate version: #3
 * subject: CN=*.thehost.com
 * start date: Thu, 22 Mar 2018 00:00:00 GMT
 * expire date: Mon, 22 Apr 2019 12:00:00 GMT
 * issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
 * compression: NULL
 * ALPN, server accepted to use http/1.1
 > GET / HTTP/1.1
 > Host: thehost.com
 > User-Agent: curl/7.47.0
 > Accept: */*
 >
 < HTTP/1.1 502 Bad Gateway
 < Server: awselb/2.0
 < Date: Thu, 24 May 2018 01:25:00 GMT
 < Content-Type: text/html
 < Content-Length: 138
 < Connection: keep-alive

Referring to the doco, and here is the answer:

ALB does support HTTP/2, but only for HTTPS listeners. You will not be able to send plaintext HTTP/2 requests.

So, no traffic will go through HTTP/2 AWS ALB unless it’s on 443 port. Damn!

References:

  1. https://forums.aws.amazon.com/thread.jspa?threadID=122901
  2. https://forums.aws.amazon.com/thread.jspa?threadID=238804
  3. https://forums.aws.amazon.com/thread.jspa?threadID=263770
  4. https://forums.aws.amazon.com/thread.jspa?threadID=94483

P.S.

Tricks to try next time:

curl -L http://example.com/
wget -S -O /dev/null http://www.example.com
wget -s -O /dev/null --header="Host: foo.bar" http://www.example.com
openssl s_client -connect thehost.com:443
curl -v --http2 https://www.example.com

Another possible reason:

There were no “Content-Length” and “Transfer-Encoding” headers in the response, and the backend used keep-alive and didn’t close connection as ELB was expecting it to do. Placing Apache in between backend and ELB should solve this problem (I haven’t tested it yet).

Amazon Linux useful stuff

August 24, 2017

The easiest way to get ec2 instance remediation via Auto Scaling is to implement health check API and put it behind Load Balancer. But still it can take 5-10 mins till ELB detects ec2 failure and tells ASG to tear it down.

EC2 System and Instance Status Checks monitor CPU, memory, os, file system, network and hardware of the instance (like loss of network sudo ifdown eth0). But they don’t give a sh*t about failure of custom software running on the instance – unless you tell them to.

This is how ec2 instance can explicitly tell Auto Scaling Group to replace it:

AWS_ACCESS_KEY_ID=AMYIDMYIDMYQ AWS_SECRET_ACCESS_KEY=0MYKEYMYKEYMYKEYMYKEYK aws autoscaling set-instance-health --instance-id "$(curl http://169.254.169.254/latest/meta-data/instance-id)" --health-status Unhealthy --region ap-southeast-2

It should have AWS credentials of a user with proper IAM permissions though.