Bash learnings

Hierarchy:

Categories

Updated 2022-10-02

things I always need to look up.

Snippets

If/then/else

if  ! $(which jq)
then
    echo "Looks like you don't have jq installed. Installing jq..."
    brew install jq
    echo "...jq installed!"
else 
    echo "You've already installed jq. Cool!"
fi

For loop

for site in  1202859 1202862 1201051 1207663 1228567 1213504 1224728 1229567 1239206
do
    echo "iteration for $site"
done
  • When using a $variable, you have to use “, not ‘ !
  • Variable assignment from the results of another command: name=$(command)
  • Array assignment from the results of another command: `array_name=($(command))

curl

  • -o for saving to a file

jq

jq – command line JSON processor

  • use -r to remove quotes around the data. raw.
  • Use | @sh for transforming arrays into space separated strings
# Select all children of the data object and extract the value of the id keys
| jq '.data[].id'

# Select the first element of the data object and return the value of the filesystemBackupId key, raw/no quotes
| jq -r '.data[0].filesystemBackupId'

tr

Transformations! Replacements, changing case, https://linuxhint.com/bash_tr_command/

Use | tr '\n' ' ') to replace new lines with spaces.

awk

Awk is a text- tool.

Use awk to replace a text with an empty string:

curl -s $url | pup -p 'li#user-company text{}' | awk '{sub(/Employer:/,"")} 1'

pup

pup is a command line tool for processing HTML, inspired by jq.

https://github.com/ericchiang/pup

Get the Employer section from a .org profile:

curl -s https://profiles.wordpress.org/cagrimmett/ | pup -p 'li#user-company text{}'

Clean that up to remove the “Employer:”, new lines, and tabs with awk and tr:

curl -s $url | pup -p 'li#user-company text{}' | awk '{sub(/Employer:/,"")} 1' | tr -d '\n' | tr -d '\t'

Use in a script:

// Assumes an input file named 5-1.txt with a list of profile URLs
// dependent on pup
for url in $(head -n800 5-1.txt); do
    employer="$(curl -s $url | pup -p 'li#user-company text{}' | awk '{sub(/Employer:/,"")} 1' | tr -d '\n' | tr -d '\t')"
    contributions="$(curl -s $url | pup -p 'div.item-meta-contribution text{}' | tr -d '\n' | tr -d '\t')"
    echo "$url | $employer | $contributions" >> 5-1_contributors.txt
done

nohup

nohup (No Hang Up) is a command in Linux systems that runs the process even after logging out from the shell/terminal. 

nohup command [command-argument ...]

wget

wget can download recursively!

wget -r -1 0 https://site.com/….

Set the maximum number of subdirectories that Wget will recurse into to depth. In order to prevent one from accidentally downloading very large websites when using recursion this is limited to a depth of 5 by default, i.e., it will traverse at most 5 directories deep starting from the provided URL. Set ‘-l 0’ or ‘-l inf’ for infinite recursion depth.

https://www.gnu.org/software/wget/manual/wget.html#Recursive-Retrieval-Options

history with timestamp

using zsh: history -f

concat commands with &&

cd ~/Projects/team51-metrics && java -jar metabase.jar

Write out a file to standard output

Helpful for short files.

cat ~/.ssh/id_rsa.pub

Copy contents to your clipboard

pbcopy

SSH

Adding your SSH key to the ssh-agent

  1. open ~/.ssh/config
    1. If it doesn’t exist, create it: touch ~/.ssh/config
  2. Move the new keys to your ~/.ssh folder
  3. Edit the ~/.ssh/config file:
Host *
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/YOUR SSH PRIVATE KEY FILE
  1. ssh-add -K ~/.ssh/YOUR KEY

The above snippet works for all hosts globally. If you need to specify, you can do something like:

Host ssh.atomicsites.net
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_rsa

Removing keys from the ssh-agent

ssh-add -D

Proxying SSH

Two ways: In the command line or in a config file

Command line: use the -o option with ProxyCommand

ssh -o ProxyCommand="nc -x 127.0.0.1:8080 %h %p" -i ~/.ssh/id_rsa [username]@[serveraddress.net]

Config file: Add ProxyCommand under the host:

Host ssh.atomicsites.net client-ssh.atomicsites.net
	ProxyCommand nc -x 127.0.0.1:8080 %h %p
	AddKeysToAgent yes
	UseKeychain yes
	IdentityFile ~/.ssh/id_rsa

Related Posts