Lesson 2: Basic command line skills

(c) 2018 Justin Bois and Axel Müller. With the exception of pasted graphics, where the source is noted, this work is licensed under a Creative Commons Attribution License CC-BY 4.0. All code contained herein is licensed under an MIT license.

This document was prepared at Caltech with financial support from the Donna and Benjamin M. Rosen Bioengineering Center.

This lesson was generated from an Jupyter notebook. You can download the notebook here.

Perhaps the first step toward really empowering you to command your computer to do whatever you will is to learn how to use the command line. This lesson provides a brief introduction to command line skills.

What is a shell?

A shell is a program that takes commands from files or entered interactively through the keyboard and passes them on to the operating system to be executed. A shell is accessed through a terminal or terminal emulator.

We will use JupyterLab's terminal in this tutorial and throughout most of the bootcamp. However, you need not limit yourself to using this. If you are using macOS, you can use the Terminal application. For a Mac, you can fire up the Terminal application. It is typically in the /Applications/Utilities folder. Otherwise, hold down Command ⌘-space bar and type terminal in the search box, and select the Terminal Application. For Windows, you can use PowerShell, which you can launch through the Start menu. If you are using Linux, it's a good bet you already know how to navigate a terminal, so we will not give specific instructions for Linux.

A very brief historical overview

Ken Thompson of Bell Labs developed the predecessor to the modern shell for the first release of UNIX in 1971. In 1977, Stephen Bourne introduced the Bourne shell (sh) which added the ability to invoke scripts (small reusable programs) from within the shell. The Bourne shell remains relevant. In some cases it is still the default root shell. Shortly afterwards, the C shell (csh) was developed which made use of a C-like scripting language. tcsh is built on csh and is still very common. bash, the Bourne again shell was developed by Brian Fox to replace the Bourne shell. It adds many useful features to sh and is the default shell for macOS and several Linux distributions.

Windows 10 enables you to also use bash, but you need to activate the Windows Subsystem for Linux. We will not use bash for Windows users in the bootcamp (unless you want to), but will instead use PowerShell, which is also the default shell in JupyterLab's terminal for Windows users. For the simple command line operations we will do in the bootcamp, PowerShell is almost always sufficient and the syntax is the same. This cheatsheet is a useful reference for comparing bash and PowerShell commands.

Getting started with the command line

Now, as you did in lesson 0, launch a terminal in JupyterLab.

As we go through this tutorial, any indented, fixed-width text contains commands you should enter at the command line.

pwd and ls

Let's start out by using the pwd command to figure out what directory we're in.

pwd

pwd tells you the path of your current directory. A path for a directory or file is the list of all its parent directories, separated by slashes (/), up to the root directory signified by the initial /. You are probably in your home directory.

To list all files and folders in the current directory, we employ the ls command.

ls

cd, change directory:

Let's make sure we are in the home directory:

cd

Use pwd to check where you are now. Invoking the cd command without specifying a target directory defaults to the home directory. Another way to specify your home directory is by its shortcut, ~/. In general, the tilde-slash means "home directory."

mkdir and rmdir

To make a directory, the command is mkdir, hence the name make directory. followed by the name of the directory you want to create. For example, to make a directory called bootcamp:

mkdir bootcamp

You now have an empty directory called bootcamp. You can see it if you list the contents of the directory.

ls

We do not need (nor want) this directory, which we will be using the bootcamp directory under version control with Git, so let's delete it. To delete an empty directory, the command is rmdir.

rmdir bootcamp

git

We will be using git a lot starting with the second day of the bootcamp, and will in fact use it right now to set things up for you for this tutorial. Put briefly, git is a version control system that allows multiple programmers to work together and allows individuals to keep track of their work. You are going to use it in just a moment to get all of the code your need for the bootcamp on your machine organized exactly as I have it organized on my machine.

In my work, I like to have a directory in my home directory called git that has all of the code from git repositories that I work on. We'll have you do the same, though you are welcome to change how you organize things after the bootcamp. In Lesson 0, you make a git directory to house all of your repositories.

Let's venture into that directory.

cd ~/git/bootcamp

I may have updated some things in that repository between the time where you cloned the repository and now. You can update the directory.

git pull upstream master

This pulls in all of the changes from the upstream repository (which is mine, the one you forked from).

Word to the wise: NO SPACES

Look at what is in the directory using ls.

ls

You will notice a few files and some directories. The directory command_line_tutorial/ has some files that will help us through this lesson. Note that there are no spaces in the directory name. In general, you avoid spaces in directory and file names, even though your operating system often has them in there. Trust me on this, they can make things a total mess, especially on the command line, since a space also separates commands. Really. NO SPACES.

mv: renaming files

Since we're currently doing a command line tutorial, let's go into that directory and see what is there.

cd command_line_tutorial
ls

We see that we have a directory called sequences, as well as a FASTA file named some sequence.fasta. This file name has the annoying space in it. We would like to rename it something without a space, say some_sequence.fasta. To do this, we us the mv command, short for "move." We enter mv, followed by the name of the file we want to rename, and then its new name.

mv some sequence.fasta some_sequence.fasta

Uh-oh! That gave us some strange output, talking about the usage of mv. This is because the space in the file some sequence.fasta was interpreted as a gap between arguments of the mv command. To specify that the space is part of the file name, we need to use an escape character, \. The space following the escape character is not considered as an argument separator. This works (but don't do it just yet):

mv some\ sequence.fasta some_sequence.fasta   # Don't do this

Because these files are under version control, you should precede the mv command with git. That way, Git will keep track of the naming changes you made. So, do this:

git mv some\ sequence.fasta some_sequence.fasta

Now, we probably want this file in the sequences directory. We can also move files into directories (without changing their file names) using the mv command.

git mv some_sequence.fasta sequences/

The trailing slash is not necessary, but I always include it out of habit to remind myself that I am moving a file to a directory.

Now let's go into the sequences directory and see what we have.

cd sequences
ls

We see that some_sequence.fasta is there, along with other FASTA files.

Exploring file content

We would like to see what is in the sequence files. Bash offers various ways to display the content of files. We'll look at the genome of the dengue virus in the file dengue.fasta. There are lots of ways to do it. We'll start with less. It got its name because it is more feature-rich than more, which was used to look at files before less came to be. ("less is more," get it?) It allows using the arrow up and arrow down keys traverse up or down by line. It also allows scrolling by touchpad or mouse. Since it doesn't require the whole file to be read before displaying the top content, it's ideal for larger files. It also supports searching initiated by "/" followed by the query; shift+g will go to the end of the file; gg to the beginning; and you can specify a line number by ":" followed by the line number.

less dengue.fasta

We'll now look at several other ways to look at files. Just substitute them for less in the above command.

cat

cat prints the entire file to the standard output (terminal). This is especially useful if the files are very small.

head

head just prints the top lines of the file to the standard output. The default can be changed:

head -5 

This will print the first 5 lines to the standard output

tail

Like head, but for the last lines of the file.

Copying files and directories: cp

If you want to retain a copy of the folder/file in the original folder you can use the copy command cp. It works straightforwardly with files. Applied to directories it requires a flag: cp -r, meaning "recursive." A flag typically begins with a hyphen (-) and gives the command some extra directions on how you want to do things. In this case, we are telling cp to work recursively.

Let's have a look at the cp command in action.

cp dengue.fasta copy_of_dengue.fasta

Maybe we want a copy of the entire sequences directory. To do that, we will cd one directory up to the command_line_tutorial directory.

cd ../

We went up one directory using ../. This is an example of a relative path. The current directory is "./", "../../" is two directories up, "../../../" is three directories up, and so on. This is very very useful when navigating directory structures. Now let's try copying an entire directory with the -r flag.

cp -r sequences copy_of_sequences

We can also rename directories with the mv command. Let's rename copy_of_sequences to sequences_copy. This is silly, but illustrates how things work.

mv copy_of_sequences sequences_copy

Removing files and directories with rm

Yes, some of the things we just did are silly. We have no need for having a copy of a given sequence or a copy of the whole sequences directory. We can clean things up by deleting them. First, let's get rid of our copy of the dengue sequence. Let's cd into the sequences directory and make sure it's there.

cd sequences
ls

Now let's remove the file and verify it is gone.

rm copy_of_dengue.fasta
ls

And poof!, its gone! And I mean gone. It is pretty much irrecoverable. Warning: rm is a wrecking ball. It will destroy any files you have that do not have restrictive permissions. This is so important, I will say it again.

rm is unforgiving

Therefore, I always like to use the -i flag, which means that rm will ask me if I'm sure before deletion.

rm -i some_sequence.fasta

You will get a prompt. Answer "n" if you do not want to delete it.

Now, let's use rm to remove an entire directory. To do this, we need to use the -r flag.

rm -r sequences_copy

Aliases (PowerShell users, skip this section)

Yes, rm is a wrecking ball, but we can temper it using the -i flag. For safety, we would like rm to always ask us about deletion. We can instruct bash to do this for us by creating an alias.

alias rm="rm -i"

After executing this, any time we use rm, bash will instead execute rm -i, thereby keeping us out of trouble.

One of my favorite aliases is to make ls list things more prettily.

alias ls="ls -FG"

The -F flag makes ls put a slash at the end of directories. This helps us tell the difference between files and directories. The -G flag enables coloring of the output, also useful for differentiating file types.

Updating your bootcamp directory

You have not updated the name of the file some_sequence.fasta. Git kept track of that, so you should commit and push your change. We will talk more about Git later in the bootcamp. For now, do the following commands to commit your change and then push the change to your master branch, which is your fork.

git commit -m "Changed file name of some_sequence.fasta."
git push origin master

You are now empowered

You are now already to manage files and navigate your way around the command line! My computer runs Mac OS X. I very rarely use Finder to copy, move, or even read files. I do it all on the command line. Once you get the hang of it, you will find the command line very efficient.