Archive for the ‘DevOps’ Category

Everyday Ubuntu commands

February 4, 2020

Host IP:

hostname -I

Disks:

df

Tasks:

top

Processes:

ps -ef | grep whatever

Find the largest files and dirs:

sudo du -a / | sort -n -r | head -n 20

to be continued…

cron tricks

April 1, 2019

Edit current user’s cron:

crontab -e

Execute a script once your computer boots up:

@reboot ~/script.sh

Check cron log:

grep cron /var/log/syslog

 

 

Enabling Jenkins on Ubuntu to ssh to Bitbucket

November 27, 2018

On your workstation use

$ ssh-keygen

to generate a private-public keypair.

In your Bitbucket account Settings -> GENERAL -> Access keys add a key for Jenkins and copy content of a public one in it.

On your Jenkins machine, sign in as jenkins user and do:

$ nano ~/.ssh/id_jenkins_rsa

and copy content of a private key in there. Then:

$ nano .bash_profile

and copy there following:

[ -z "$SSH_AUTH_SOCK" ] && eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_jenkins_rsa

Save and exit. Make sure you have right permissions:

$ chmod 755 .bash_profile

Logout and log back in. Check if it works:

$ ssh -T git@bitbucket.org

Should authenticate you via a deploy key. Then jenkins can use it like:

$ git clone git@bitbucket.org:youraccount/yourrepo.git

10 Cool Linux Shell Tricks

August 9, 2018

An extract from this article: https://support.blue.net.au/support/34/

1. Send Output and Errors
command &> file

2. Parallelize Your Loops
for HOST in $(< ListOfHosts); do ssh $HOST ’sudo apt-get update’ & done

3. Catch Memory Leaks By Using Top via Cron
crontab – <<< ‘*/15 * * * * top -n 1 -b’

4. Stdin directly from the command line
<<<

5. Set a Random Initial Password, That Must be Changed
umask u=rw,go=
openssl rand -base64 6 | tee -a PasswordFile | passwd –stdin joe
chage -d 0 joe

6. Add Your Public Key to Remote Machines
ssh-copy-id -i .ssh/id_rsa.pub hostname

7. Extract an RPM without any additional software
rpm -ivh –root /tmp/deleteme –nodeps –noscripts package.rpm

8. See How a File Has Changed from Factory Defaults
dpkg -S /etc/foo/foo.conf
rpm -qf /etc/foo/foo.conf
diff /etc/foo/foo.conf /tmp/deleteme/etc/foo/foo.conf

9. Undo Your Network Screwups After You’ve Lost the Connection
at now + 5 minutes <<< ‘cp /etc/ssh/sshd_config.old /etc/ssh/sshd_config; service sshd restart’

10. Check if SSH Port is Open
nc -w 3 server 22 ssh <<< ''

Multi hop SSH tunnel

August 6, 2018

There is a private API endpoint listening to port 9999 accessible from EC2 in a private subnet on AWS. Private subnet is accessible from a bastion host which is in a Public subnet and has a Public IP. Both hosts have different keipairs.

The task is to enable developers call API endpoint from their workstations.

Open local port forwarding:

ssh -vtA -i ec2.pem  -L9999:apiendpointIP:9999 ubuntu@ec2IP \
 -o 'ProxyCommand = ssh -vtA -i bastionhost.pem -L9999:localhost:9999 \
 ubuntu@bastionhostIP -W ec2IP:22'

Now developers can access the API endpoint:

localhost:9999

The next step is to create a limited user and chroot it to jail.

Generate random UUID token that starts with what you want

May 31, 2018

I wanted to personalize my environments by assigning them unique tokens which will be used by consumer APIs. To generate those tokens, I wrote a simple program on Java:

import java.util.UUID;

public class RandomStringUUID {
    public static void main(String[] args) {

        while(true) {
            // Creating a random UUID (Universally unique identifier).
            UUID uuid = UUID.randomUUID();
            String randomUUIDString = uuid.toString();

            if (randomUUIDString.startsWith("c001")) {

                System.out.println("UUID token = " + randomUUIDString);
                System.out.println("UUID version       = " + uuid.version());
                System.out.println("UUID variant       = " + uuid.variant());

                break;
            }
        }
    }
}

Output:

UUID token = c001ab8b-4224-4530-bb93-0fab6ad3f83b
UUID version = 4
UUID variant = 2

Process finished with exit code 0

Now, looking at the token I can tell to which environment it belongs. And my log analyzer too 🙂

Troubleshooting network connection

May 24, 2018

How to troubleshoot network connection between AWS Lambda, AWS EC2 through Application Load Balancer and outside world.

First try

curl https://thehost.com

returns empty.

Second try

curl --dump-header - https://thehost.com

returns something more meaningful but a bit misleading:

HTTP/1.1 302 Found
 Date: Thu, 24 May 2018 01:15:46 GMT
 Content-Type: text/plain; charset=utf-8
 Content-Length: 0
 Connection: keep-alive
 Location: login_page

telnet connects though:

telnet thehost 443
Trying 55.66.222.111...
Connected to thehost.com.
Escape character is '^]'.
^]
telnet> status
Connected to thehost.com.
Operating in obsolete linemode
Local character echo
Escape character is '^]'.
Connection closed by foreign host

Third try

curl -svo /dev/null https://thehost.com

gives us a hint with TLS info and the root of a problem below:

* Rebuilt URL to: https://thehost.com/
 * Trying 54.66.241.172...
 * Connected to thehost.com (55.66.222.111) port 443 (#0)
 * found 148 certificates in /etc/ssl/certs/ca-certificates.crt
 * found 592 certificates in /etc/ssl/certs
 * ALPN, offering http/1.1
 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
 * server certificate verification OK
 * server certificate status verification SKIPPED
 * common name: *.thehost.com (matched)
 * server certificate expiration date OK
 * server certificate activation date OK
 * certificate public key: RSA
 * certificate version: #3
 * subject: CN=*.thehost.com
 * start date: Thu, 22 Mar 2018 00:00:00 GMT
 * expire date: Mon, 22 Apr 2019 12:00:00 GMT
 * issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
 * compression: NULL
 * ALPN, server accepted to use http/1.1
 > GET / HTTP/1.1
 > Host: thehost.com
 > User-Agent: curl/7.47.0
 > Accept: */*
 >
 < HTTP/1.1 502 Bad Gateway
 < Server: awselb/2.0
 < Date: Thu, 24 May 2018 01:25:00 GMT
 < Content-Type: text/html
 < Content-Length: 138
 < Connection: keep-alive

Referring to the doco, and here is the answer:

ALB does support HTTP/2, but only for HTTPS listeners. You will not be able to send plaintext HTTP/2 requests.

So, no traffic will go through HTTP/2 AWS ALB unless it’s on 443 port. Damn!

References:

  1. https://forums.aws.amazon.com/thread.jspa?threadID=122901
  2. https://forums.aws.amazon.com/thread.jspa?threadID=238804
  3. https://forums.aws.amazon.com/thread.jspa?threadID=263770
  4. https://forums.aws.amazon.com/thread.jspa?threadID=94483

P.S.

Tricks to try next time:

curl -L http://example.com/
wget -S -O /dev/null http://www.example.com
wget -s -O /dev/null --header="Host: foo.bar" http://www.example.com
openssl s_client -connect thehost.com:443
curl -v --http2 https://www.example.com

Another possible reason:

There were no “Content-Length” and “Transfer-Encoding” headers in the response, and the backend used keep-alive and didn’t close connection as ELB was expecting it to do. Placing Apache in between backend and ELB should solve this problem (I haven’t tested it yet).

Copying a file from AWS ec2 into a private subnet over ssh tunnel

April 6, 2018

In one terminal session establish ssh tunnel to deployment server through jumphost:

ssh -L 1234:10.16X.server.privateIP:22 -p 22 ec2-user@54.25X.jumphost.publicIP

it logged me into jumphost, ec2-user is the user on the jumphost. 1234 is the port on localhost where the tunnel is now established.

In other terminal session on localhost, do the command:

scp -P 1234 ubuntu@localhost:/var/lib/jenkins/branches/master/path/to/deployment/package .

ubuntu is the user on deployment server which in in private subnet. Both servers are listening to ssh on port 22.

Voila!

Now you can close the tunnel in first terminal session:

[ec2-user@ip-10-161-jumphost-privateIP ~]$ exit

Ansible role to install maven with yum

September 8, 2017

It should work on CentOS, Amazon Linux, etc.

– name: “Download maven”
get_url:
url: http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo
dest: /etc/yum.repos.d/epel-apache-maven.repo
validate_certs: no
timeout: 60
mode: 0777
register: apache-maven

– replace:
path: /etc/yum.repos.d/epel-apache-maven.repo
regexp: ‘\$releasever/’
replace: ‘6/’
backup: yes

– name: “Install dependencies”
yum:
name: “{{ item }}”
state: present
update_cache: yes
with_items:
– apache-maven