Bash things I always need to look up.
Snippets
If/then/else
if test ! $(which jq)
then
echo "Looks like you don't have jq installed. Installing jq..."
brew install jq
echo "...jq installed!"
else
echo "You've already installed jq. Cool!"
fi
Code language: PHP (php)
For loop
for site in 1202859 1202862 1201051 1207663 1228567 1213504 1224728 1229567 1239206
do
echo "iteration for $site"
done
Code language: PHP (php)
- When using a
$variable
, you have to use “, not ‘ ! - Variable assignment from the results of another command:
name=$(command)
- Array assignment from the results of another command: `array_name=($(command))
Tools
curl
- -o for saving to a file
jq
jq – command line JSON processor
- use
-r
to remove quotes around the data. raw. - Use
| @sh
for transforming arrays into space separated strings
# Select all children of the data object and extract the value of the id keys
| jq '.data[].id'
# Select the first element of the data object and return the value of the filesystemBackupId key, raw/no quotes
| jq -r '.data[0].filesystemBackupId'
Code language: PHP (php)
tr
Transformations! Replacements, changing case, https://linuxhint.com/bash_tr_command/
Use | tr '\n' ' ')
to replace new lines with spaces.
awk
Awk is a command line text-processing tool.
Use awk to replace a text with an empty string:
curl -s $url | pup -p 'li#user-company text{}' | awk '{sub(/Employer:/,"")} 1'
Code language: PHP (php)
pup
pup is a command line tool for processing HTML, inspired by jq.
https://github.com/ericchiang/pup
Get the Employer section from a WordPress.org profile:
<code>curl -s <a rel="noreferrer noopener" href="https://profiles.wordpress.org/cagrimmett/" target="_blank">https://profiles.wordpress.org/cagrimmett/</a> | pup -p 'li#user-company text{}'</code>
Code language: HTML, XML (xml)
Clean that up to remove the “Employer:”, new lines, and tabs with awk
and tr
:
curl -s $url | pup -p 'li#user-company text{}' | awk '{sub(/Employer:/,"")} 1' | tr -d '\n' | tr -d '\t'
Code language: PHP (php)
Use in a script:
// Assumes an input file named 5-1.txt with a list of profile URLs
// dependent on pup
for url in $(head -n800 5-1.txt); do
employer="$(curl -s $url | pup -p 'li#user-company text{}' | awk '{sub(/Employer:/,"")} 1' | tr -d '\n' | tr -d '\t')"
contributions="$(curl -s $url | pup -p 'div.item-meta-contribution text{}' | tr -d '\n' | tr -d '\t')"
echo "$url | $employer | $contributions" >> 5-1_contributors.txt
done
Code language: JavaScript (javascript)
nohup
nohup (No Hang Up) is a command in Linux systems that runs the process even after logging out from the shell/terminal.
nohup command [command-argument ...]
Code language: CSS (css)
wget
wget can download recursively!
wget -r
-1 0 https://site.com/….
Set the maximum number of subdirectories that Wget will recurse into to depth. In order to prevent one from accidentally downloading very large websites when using recursion this is limited to a depth of 5 by default, i.e., it will traverse at most 5 directories deep starting from the provided URL. Set ‘-l 0’ or ‘-l inf’ for infinite recursion depth.
https://www.gnu.org/software/wget/manual/wget.html#Recursive-Retrieval-Options
history with timestamp
using zsh: history -f
concat commands with &&
cd ~/Projects/team51-metrics && java -jar metabase.jar
Write out a file to standard output
Helpful for short files.
cat ~/.ssh/id_rsa.pub
Copy contents to your clipboard
pbcopy < ~/.ssh/id_rsa.pub
SSH
Adding your SSH key to the ssh-agent
open ~/.ssh/config
- If it doesn’t exist, create it:
touch ~/.ssh/config
- If it doesn’t exist, create it:
- Move the new keys to your ~/.ssh folder
- Edit the ~/.ssh/config file:
Host *
AddKeysToAgent yes
UseKeychain yes
IdentityFile ~/.ssh/YOUR SSH PRIVATE KEY FILE
Code language: JavaScript (javascript)
ssh-add -K ~/.ssh/YOUR KEY
The above snippet works for all hosts globally. If you need to specify, you can do something like:
Host ssh.atomicsites.net
AddKeysToAgent yes
UseKeychain yes
IdentityFile ~/.ssh/id_rsa
Code language: JavaScript (javascript)
Removing keys from the ssh-agent
ssh-add -D
Proxying SSH
Two ways: In the command line or in a config file
Command line: use the -o option with ProxyCommand
ssh -o ProxyCommand="nc -x 127.0.0.1:8080 %h %p" -i ~/.ssh/id_rsa [username]@[serveraddress.net]
Code language: JavaScript (javascript)
Config file: Add ProxyCommand under the host:
Host ssh.atomicsites.net client-ssh.atomicsites.net
ProxyCommand nc -x 127.0.0.1:8080 %h %p
AddKeysToAgent yes
UseKeychain yes
IdentityFile ~/.ssh/id_rsa
Code language: JavaScript (javascript)