Host IP:
hostname -I
Disks:
df
Tasks:
top
Processes:
ps -ef | grep whatever
Find the largest files and dirs:
sudo du -a / | sort -n -r | head -n 20
to be continued…
Host IP:
hostname -I
Disks:
df
Tasks:
top
Processes:
ps -ef | grep whatever
Find the largest files and dirs:
sudo du -a / | sort -n -r | head -n 20
to be continued…
Check SSH configuration
sshd -t
Show active SSH connections
lsof -i -n | egrep '\<ssh\>'
Script to keep alive SSH connection with varying IP:
#!/bin/bash if [ -z "$(ps ax | awk '{print $9}' | grep 2200:svarm:2200)" ] then ssh -N -f -R 2200:svarm:2200 -p 2222 mynetwork.net fi
Establish SSH tunnel to DB on background, save its PID and kill it afterwards:
#!/bin/bash ssh -N -L 3333:db.myserver.com:3306 jump-host & pid=$! do something kill -9 $pid
Edit current user’s cron:
crontab -e
Execute a script once your computer boots up:
@reboot ~/script.sh
Check cron log:
grep cron /var/log/syslog
On your workstation use
$ ssh-keygen
to generate a private-public keypair.
In your Bitbucket account Settings -> GENERAL -> Access keys add a key for Jenkins and copy content of a public one in it.
On your Jenkins machine, sign in as jenkins user and do:
$ nano ~/.ssh/id_jenkins_rsa
and copy content of a private key in there. Then:
$ nano .bash_profile
and copy there following:
[ -z "$SSH_AUTH_SOCK" ] && eval "$(ssh-agent -s)" ssh-add ~/.ssh/id_jenkins_rsa
Save and exit. Make sure you have right permissions:
$ chmod 755 .bash_profile
Logout and log back in. Check if it works:
$ ssh -T git@bitbucket.org
Should authenticate you via a deploy key. Then jenkins can use it like:
$ git clone git@bitbucket.org:youraccount/yourrepo.git
An extract from this article: https://support.blue.net.au/support/34/
1. Send Output and Errors
command &> file
2. Parallelize Your Loops
for HOST in $(< ListOfHosts); do ssh $HOST ’sudo apt-get update’ & done
3. Catch Memory Leaks By Using Top via Cron
crontab – <<< ‘*/15 * * * * top -n 1 -b’
4. Stdin directly from the command line
<<<
5. Set a Random Initial Password, That Must be Changed
umask u=rw,go=
openssl rand -base64 6 | tee -a PasswordFile | passwd –stdin joe
chage -d 0 joe
6. Add Your Public Key to Remote Machines
ssh-copy-id -i .ssh/id_rsa.pub hostname
7. Extract an RPM without any additional software
rpm -ivh –root /tmp/deleteme –nodeps –noscripts package.rpm
8. See How a File Has Changed from Factory Defaults
dpkg -S /etc/foo/foo.conf
rpm -qf /etc/foo/foo.conf
diff /etc/foo/foo.conf /tmp/deleteme/etc/foo/foo.conf
9. Undo Your Network Screwups After You’ve Lost the Connection
at now + 5 minutes <<< ‘cp /etc/ssh/sshd_config.old /etc/ssh/sshd_config; service sshd restart’
10. Check if SSH Port is Open
nc -w 3 server 22 ssh <<< ''
There is a private API endpoint listening to port 9999 accessible from EC2 in a private subnet on AWS. Private subnet is accessible from a bastion host which is in a Public subnet and has a Public IP. Both hosts have different keipairs.
The task is to enable developers call API endpoint from their workstations.
Open local port forwarding:
ssh -vtA -i ec2.pem -L9999:apiendpointIP:9999 ubuntu@ec2IP \ -o 'ProxyCommand = ssh -vtA -i bastionhost.pem -L9999:localhost:9999 \ ubuntu@bastionhostIP -W ec2IP:22'
Now developers can access the API endpoint:
localhost:9999
The next step is to create a limited user and chroot it to jail.
I wanted to personalize my environments by assigning them unique tokens which will be used by consumer APIs. To generate those tokens, I wrote a simple program on Java:
import java.util.UUID; public class RandomStringUUID { public static void main(String[] args) { while(true) { // Creating a random UUID (Universally unique identifier). UUID uuid = UUID.randomUUID(); String randomUUIDString = uuid.toString(); if (randomUUIDString.startsWith("c001")) { System.out.println("UUID token = " + randomUUIDString); System.out.println("UUID version = " + uuid.version()); System.out.println("UUID variant = " + uuid.variant()); break; } } } }
Output:
UUID token = c001ab8b-4224-4530-bb93-0fab6ad3f83b UUID version = 4 UUID variant = 2 Process finished with exit code 0
Now, looking at the token I can tell to which environment it belongs. And my log analyzer too 🙂
How to troubleshoot network connection between AWS Lambda, AWS EC2 through Application Load Balancer and outside world.
curl https://thehost.com
returns empty.
curl --dump-header - https://thehost.com
returns something more meaningful but a bit misleading:
HTTP/1.1 302 Found Date: Thu, 24 May 2018 01:15:46 GMT Content-Type: text/plain; charset=utf-8 Content-Length: 0 Connection: keep-alive Location: login_page
telnet connects though:
telnet thehost 443 Trying 55.66.222.111... Connected to thehost.com. Escape character is '^]'. ^] telnet> status Connected to thehost.com. Operating in obsolete linemode Local character echo Escape character is '^]'. Connection closed by foreign host
curl -svo /dev/null https://thehost.com
gives us a hint with TLS info and the root of a problem below:
* Rebuilt URL to: https://thehost.com/ * Trying 54.66.241.172... * Connected to thehost.com (55.66.222.111) port 443 (#0) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 592 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.thehost.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: RSA * certificate version: #3 * subject: CN=*.thehost.com * start date: Thu, 22 Mar 2018 00:00:00 GMT * expire date: Mon, 22 Apr 2019 12:00:00 GMT * issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon * compression: NULL * ALPN, server accepted to use http/1.1 > GET / HTTP/1.1 > Host: thehost.com > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 502 Bad Gateway < Server: awselb/2.0 < Date: Thu, 24 May 2018 01:25:00 GMT < Content-Type: text/html < Content-Length: 138 < Connection: keep-alive
Referring to the doco, and here is the answer:
ALB does support HTTP/2, but only for HTTPS listeners. You will not be able to send plaintext HTTP/2 requests.
So, no traffic will go through HTTP/2 AWS ALB unless it’s on 443 port. Damn!
Tricks to try next time:
curl -L http://example.com/ wget -S -O /dev/null http://www.example.com wget -s -O /dev/null --header="Host: foo.bar" http://www.example.com openssl s_client -connect thehost.com:443 curl -v --http2 https://www.example.com
Another possible reason:
There were no “Content-Length” and “Transfer-Encoding” headers in the response, and the backend used keep-alive and didn’t close connection as ELB was expecting it to do. Placing Apache in between backend and ELB should solve this problem (I haven’t tested it yet).
In one terminal session establish ssh tunnel to deployment server through jumphost:
ssh -L 1234:10.16X.server.privateIP:22 -p 22 ec2-user@54.25X.jumphost.publicIP
it logged me into jumphost, ec2-user is the user on the jumphost. 1234 is the port on localhost where the tunnel is now established.
In other terminal session on localhost, do the command:
scp -P 1234 ubuntu@localhost:/var/lib/jenkins/branches/master/path/to/deployment/package .
ubuntu is the user on deployment server which in in private subnet. Both servers are listening to ssh on port 22.
Voila!
Now you can close the tunnel in first terminal session:
[ec2-user@ip-10-161-jumphost-privateIP ~]$ exit
# mono
yum install yum-utils
rpm --import "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF"
yum-config-manager --enable http://download.mono-project.com/repo/centos/
wget http://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/l/libpng15-1.5.30-1.fc28.x86_64.rpm
yum install -y libpng15-1.5.30-1.fc28.x86_64.rpm
yum install -y mono
#yum install -y mono-devel
#yum install -y mono-complete
# dotnet
mkdir /opt/dotnet
cd /opt/dotnet/
sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
sudo sh -c 'echo -e "[packages-microsoft-com-prod]\nname=packages-microsoft-com-prod \nbaseurl=https://packages.microsoft.com/yumrepos/microsoft-rhel7.3-prod\nenabled=1\ngpgcheck=1\ngpgkey=https://packages.microsoft.com/keys/microsoft.asc" > /etc/yum.repos.d/dotnetdev.repo'
sudo yum -y update
sudo yum -y install libunwind libicu
#sudo yum -y install dotnet-sdk-2.0.0
wget ftp://rpmfind.net/linux/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/o/openssl-libs-1.1.0f-9.fc27.x86_64.rpm
rpm -i --replacefiles openssl-libs-1.1.0f-9.fc27.x86_64.rpm
sudo yum install dotnet-sdk-2.0.0
# octo.exe
mkdir /opt/octopus
cd /opt/octopus
wget https://download.octopusdeploy.com/octopus-tools/4.24.2/OctopusTools.4.24.2.rhel.7-x64.tar.gz
tar -xvzf OctopusTools.4.24.2.rhel.7-x64.tar.gz
ln -s /opt/octopus/Octo /usr/bin/octo.exe