Hippocamplus My Second Memory

Shell

This page focuses on configuration of the terminal, one-liners and scripting. Command-line tools I used (e.g. git, snakemake, docker) are described in the Tools page.

GNU screen

I had the following problems in some HPCs:

  • screen not clearing out after commands like less or emacs.
  • Ctl-l not clearing out the screen.
  • Emacs rendering badly with annoying visual glitches. I had to constantly Ctl-L to clean them.

The solution was to add in ~/.screenrc:

# clear out screen after less etc
altscreen on
# fixes ctr-l and emacs visual glitches
term screen-256color

.bashrc aliases

  • alias terml='gnome-terminal --tab-with-profile="Sol Light"' to open a new terminal with light theme.
  • alias termd='gnome-terminal --tab-with-profile="Sol Dark"' same with dark theme.
  • alias rf='ls -rt | tail -1' to quickly get the most recent file. Then for example evince `rf` to open the latest file.
  • alias evincel='ls -rt *.pdf | tail -1 | xargs evince' opens the last PDF document
  • alias R='R --no-save --no-restore' to avoid R asking about saving the history.

Misc

  • Redirect stderr to stdout: 2>&1.
  • Show special characters (end of line, tabs) with cat -A.

Colors in less

Use the -r or -R option. Sometimes the previous command needs a parameter make sure the output includes color codes.

tree -C | less -R
ls --color | less -R
grep --color=always | less -R

Avoid killing ssh jobs

nohup function (apparently).

One-liners

Bash & Pipes

Add headers with cat

cat file.txt | cat headers.txt -

Remove empty directories

find . -type d | xargs rmdir

Git add all untracked files

git st -s | grep '??' | cut -f2 -d ' ' | xargs git add

Open recent PDFs

find pdf -mtime -1 | xargs evince

AWK

  • Capture group in string regexp using match($0, /xxx(.*)yyy/, a); print a[1]'

Perl one-liners

Inspired from these: Catonmat, Ksplice

  • -p run command on each line and print line.
  • -n run command on each line but don’t print line.
  • -i operates on the file in-place, i.e. updating the input file.
  • -a split the lines by white spaces into a @F array (-F to choose the separator)
  • -l removes the trailing new line, and adds it back if -p is used.
  • $_ contains the line.
  • $. contains the line number.

  • Replace text line-by-line in a file: perl -pe 's/you/me/g'
  • Replace text if line contains foo: perl -pe 's/you/me/g if /foo/
  • Replace text line-by-line with regexp: perl -pe 's/ID(\d+)/$1/g'
  • Operations on a CSV file: perl -F, -ane 'print $F[3]+$F[8]
  • Operations on a CSV file and print at the end: perl -F, -ane $t+=$F[3]+$F[8]; END{print $t} %
  • Print matches from regular expression: perl -ne 'print "$1\n" if /foo=([^;]*)/'

Shell scripting

Start a script with one of the shebang

#!/bin/sh

Variables

To remove a specific prefix/suffix from a variable name:

foo=${foop#$prefix}
foo=${foos%$suffix}

If Then Else

A simple example:

if [ $VAL == "YEP" ]; then
    echo "It's a yes!"
else
    echo "No no no :("
fi

Or with multiple conditions:

if [ $VAL == "YEP" ] && [ COND ]; then
    echo "It's a yes!"
else
    echo "No no no :("
fi

The spacing is quite important, and the conditions can be built with:

  • -eq equal to.
  • -ne not equal to.
  • -lt less than.
  • -le less or equal than.
  • -gt great than.
  • -ge great or equal than.
  • -s file exists and not empty.
  • -f file exists and not directory.
  • -d directory exists.
  • -x file executable.
  • -w file writable.
  • -r file readable.

Modules

For example installing Emacs as a module on a HPC. After installing Emacs locally in a ~/softwares/emacs/emacs-24.4 I create a file ~/myModules/jmonlong/emacs/24.4 with:

#%Module1.0
proc ModulesHelp { } {
  puts stderr "\tMUMmer "
}
module-whatis "mummer"

set             root                /home/jmonlong/softwares/emacs/emacs-24.4
prepend-path    PATH                $root/bin
prepend-path    LIBRARY_PATH        $root/lib/
prepend-path    LD_LIBRARY_PATH     $root/lib/

Then to use the module:

module use ~/myModules
module load jmonlong/emacs

Ubuntu

Sound problems

I touched something my headphones sometime bug (white noise and bad sound quality). This fixes it (in my ~/.bashrc):

alias soundfix='amixer -c PCH cset "name=Headphone Mic Boost Volume" 1'

OSX

Mount server

I created a directory sftp (I don’t know why I choose this name…anyway) and mount the root of the different servers there. Eventually I created a symbolic link at the root of my computer to point there so that paths like /gs/projects/... work directly, as if in the cluster.

To mount a server I use the following command:

sshfs jmonlong@guillimin.hpc.mcgill.ca:/ /Users/jeanmonlong/sftp/guillimin -ovolname=NAME

Setup WD Elements external hard drive

By default the disk is formatted in NTFS, which OSX could read only. To write, the solution that worked for me was to add this line to /etc/fstab:

LABEL=Elements none ntfs rw,auto,nobrowse

Note: If there are spaces in the label, replace them by \040.

The disk can then be accessed through the Volumes folder (/Volumes).