Everyday Ubuntu commands

Host IP:

hostname -I

Disks:

df

Tasks:

top

Processes:

ps -ef | grep whatever

Find the largest files and dirs:

sudo du -a / | sort -n -r | head -n 20

to be continued…

SSH tricks

Check SSH configuration
sshd -t

Show active SSH connections
lsof -i -n | egrep '\<ssh\>'

Script to keep alive SSH connection with varying IP:

#!/bin/bash

if [ -z "$(ps ax | awk '{print $9}' | grep 2200:svarm:2200)" ]
then
ssh -N -f -R 2200:svarm:2200 -p 2222 mynetwork.net
fi

Establish SSH tunnel to DB on background, save its PID and kill it afterwards:

#!/bin/bash

ssh  -N -L 3333:db.myserver.com:3306 jump-host &
pid=$!
do something
kill -9 $pid

 

 

 

 

cron tricks

Edit current user’s cron:

crontab -e

Execute a script once your computer boots up:

@reboot ~/script.sh

Check cron log:

grep cron /var/log/syslog

 

 

Enabling Jenkins on Ubuntu to ssh to Bitbucket

On your workstation use

$ ssh-keygen

to generate a private-public keypair.

In your Bitbucket account Settings -> GENERAL -> Access keys add a key for Jenkins and copy content of a public one in it.

On your Jenkins machine, sign in as jenkins user and do:

$ nano ~/.ssh/id_jenkins_rsa

and copy content of a private key in there. Then:

$ nano .bash_profile

and copy there following:

[ -z "$SSH_AUTH_SOCK" ] && eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_jenkins_rsa

Save and exit. Make sure you have right permissions:

$ chmod 755 .bash_profile

Logout and log back in. Check if it works:

$ ssh -T git@bitbucket.org

Should authenticate you via a deploy key. Then jenkins can use it like:

$ git clone git@bitbucket.org:youraccount/yourrepo.git

10 Cool Linux Shell Tricks

An extract from this article: https://support.blue.net.au/support/34/

1. Send Output and Errors
command &> file

2. Parallelize Your Loops
for HOST in $(< ListOfHosts); do ssh $HOST ’sudo apt-get update’ & done

3. Catch Memory Leaks By Using Top via Cron
crontab – <<< ‘*/15 * * * * top -n 1 -b’

4. Stdin directly from the command line
<<<

5. Set a Random Initial Password, That Must be Changed
umask u=rw,go=
openssl rand -base64 6 | tee -a PasswordFile | passwd –stdin joe
chage -d 0 joe

6. Add Your Public Key to Remote Machines
ssh-copy-id -i .ssh/id_rsa.pub hostname

7. Extract an RPM without any additional software
rpm -ivh –root /tmp/deleteme –nodeps –noscripts package.rpm

8. See How a File Has Changed from Factory Defaults
dpkg -S /etc/foo/foo.conf
rpm -qf /etc/foo/foo.conf
diff /etc/foo/foo.conf /tmp/deleteme/etc/foo/foo.conf

9. Undo Your Network Screwups After You’ve Lost the Connection
at now + 5 minutes <<< ‘cp /etc/ssh/sshd_config.old /etc/ssh/sshd_config; service sshd restart’

10. Check if SSH Port is Open
nc -w 3 server 22 ssh <<< ''

Multi hop SSH tunnel

There is a private API endpoint listening to port 9999 accessible from EC2 in a private subnet on AWS. Private subnet is accessible from a bastion host which is in a Public subnet and has a Public IP. Both hosts have different keipairs.

The task is to enable developers call API endpoint from their workstations.

Open local port forwarding:

ssh -vtA -i ec2.pem  -L9999:apiendpointIP:9999 ubuntu@ec2IP \
 -o 'ProxyCommand = ssh -vtA -i bastionhost.pem -L9999:localhost:9999 \
 ubuntu@bastionhostIP -W ec2IP:22'

Now developers can access the API endpoint:

localhost:9999

The next step is to create a limited user and chroot it to jail.

Generate random UUID token that starts with what you want

I wanted to personalize my environments by assigning them unique tokens which will be used by consumer APIs. To generate those tokens, I wrote a simple program on Java:

import java.util.UUID;

public class RandomStringUUID {
    public static void main(String[] args) {

        while(true) {
            // Creating a random UUID (Universally unique identifier).
            UUID uuid = UUID.randomUUID();
            String randomUUIDString = uuid.toString();

            if (randomUUIDString.startsWith("c001")) {

                System.out.println("UUID token = " + randomUUIDString);
                System.out.println("UUID version       = " + uuid.version());
                System.out.println("UUID variant       = " + uuid.variant());

                break;
            }
        }
    }
}

Output:

UUID token = c001ab8b-4224-4530-bb93-0fab6ad3f83b
UUID version = 4
UUID variant = 2

Process finished with exit code 0

Now, looking at the token I can tell to which environment it belongs. And my log analyzer too 🙂

Troubleshooting network connection

How to troubleshoot network connection between AWS Lambda, AWS EC2 through Application Load Balancer and outside world.

First try

curl https://thehost.com

returns empty.

Second try

curl --dump-header - https://thehost.com

returns something more meaningful but a bit misleading:

HTTP/1.1 302 Found
 Date: Thu, 24 May 2018 01:15:46 GMT
 Content-Type: text/plain; charset=utf-8
 Content-Length: 0
 Connection: keep-alive
 Location: login_page

telnet connects though:

telnet thehost 443
Trying 55.66.222.111...
Connected to thehost.com.
Escape character is '^]'.
^]
telnet> status
Connected to thehost.com.
Operating in obsolete linemode
Local character echo
Escape character is '^]'.
Connection closed by foreign host

Third try

curl -svo /dev/null https://thehost.com

gives us a hint with TLS info and the root of a problem below:

* Rebuilt URL to: https://thehost.com/
 * Trying 54.66.241.172...
 * Connected to thehost.com (55.66.222.111) port 443 (#0)
 * found 148 certificates in /etc/ssl/certs/ca-certificates.crt
 * found 592 certificates in /etc/ssl/certs
 * ALPN, offering http/1.1
 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
 * server certificate verification OK
 * server certificate status verification SKIPPED
 * common name: *.thehost.com (matched)
 * server certificate expiration date OK
 * server certificate activation date OK
 * certificate public key: RSA
 * certificate version: #3
 * subject: CN=*.thehost.com
 * start date: Thu, 22 Mar 2018 00:00:00 GMT
 * expire date: Mon, 22 Apr 2019 12:00:00 GMT
 * issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
 * compression: NULL
 * ALPN, server accepted to use http/1.1
 > GET / HTTP/1.1
 > Host: thehost.com
 > User-Agent: curl/7.47.0
 > Accept: */*
 >
 < HTTP/1.1 502 Bad Gateway
 < Server: awselb/2.0
 < Date: Thu, 24 May 2018 01:25:00 GMT
 < Content-Type: text/html
 < Content-Length: 138
 < Connection: keep-alive

Referring to the doco, and here is the answer:

ALB does support HTTP/2, but only for HTTPS listeners. You will not be able to send plaintext HTTP/2 requests.

So, no traffic will go through HTTP/2 AWS ALB unless it’s on 443 port. Damn!

References:

1. https://forums.aws.amazon.com/thread.jspa?threadID=122901
2. https://forums.aws.amazon.com/thread.jspa?threadID=238804
3. https://forums.aws.amazon.com/thread.jspa?threadID=263770
4. https://forums.aws.amazon.com/thread.jspa?threadID=94483

P.S.

Tricks to try next time:

curl -L http://example.com/
wget -S -O /dev/null http://www.example.com
wget -s -O /dev/null --header="Host: foo.bar" http://www.example.com
openssl s_client -connect thehost.com:443
curl -v --http2 https://www.example.com

Another possible reason:

There were no “Content-Length” and “Transfer-Encoding” headers in the response, and the backend used keep-alive and didn’t close connection as ELB was expecting it to do. Placing Apache in between backend and ELB should solve this problem (I haven’t tested it yet).

Copying a file from AWS ec2 into a private subnet over ssh tunnel

In one terminal session establish ssh tunnel to deployment server through jumphost:

ssh -L 1234:10.16X.server.privateIP:22 -p 22 ec2-user@54.25X.jumphost.publicIP

it logged me into jumphost, ec2-user is the user on the jumphost. 1234 is the port on localhost where the tunnel is now established.

In other terminal session on localhost, do the command:

scp -P 1234 ubuntu@localhost:/var/lib/jenkins/branches/master/path/to/deployment/package .

ubuntu is the user on deployment server which in in private subnet. Both servers are listening to ssh on port 22.

Voila!

Now you can close the tunnel in first terminal session:

[ec2-user@ip-10-161-jumphost-privateIP ~]$ exit

Install mono, dotnet, octo.exe on Amazon Linux

# mono
yum install yum-utils
rpm --import "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF"
yum-config-manager --enable http://download.mono-project.com/repo/centos/
wget http://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/l/libpng15-1.5.30-1.fc28.x86_64.rpm
yum install -y libpng15-1.5.30-1.fc28.x86_64.rpm
yum install -y mono
#yum install -y mono-devel
#yum install -y mono-complete

# dotnet
mkdir /opt/dotnet
cd /opt/dotnet/
sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
sudo sh -c 'echo -e "[packages-microsoft-com-prod]\nname=packages-microsoft-com-prod \nbaseurl=https://packages.microsoft.com/yumrepos/microsoft-rhel7.3-prod\nenabled=1\ngpgcheck=1\ngpgkey=https://packages.microsoft.com/keys/microsoft.asc" > /etc/yum.repos.d/dotnetdev.repo'
sudo yum -y update
sudo yum -y install libunwind libicu
#sudo yum -y install dotnet-sdk-2.0.0
wget ftp://rpmfind.net/linux/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/o/openssl-libs-1.1.0f-9.fc27.x86_64.rpm
rpm -i --replacefiles openssl-libs-1.1.0f-9.fc27.x86_64.rpm
sudo yum install dotnet-sdk-2.0.0

# octo.exe
mkdir /opt/octopus
cd /opt/octopus
wget https://download.octopusdeploy.com/octopus-tools/4.24.2/OctopusTools.4.24.2.rhel.7-x64.tar.gz
tar -xvzf OctopusTools.4.24.2.rhel.7-x64.tar.gz
ln -s /opt/octopus/Octo /usr/bin/octo.exe