Lesson 2: Basic command line skills¶
Perhaps the first step toward really empowering you to command your computer to do whatever you will is to learn how to use the command line. This lesson provides a brief introduction to command line skills.
What is a shell?¶
A shell is a program that takes commands from files or entered interactively through the keyboard and passes them on to the operating system to be executed. A shell is accessed through a terminal or terminal emulator.
We will use JupyterLab’s terminal in this tutorial and throughout most of the bootcamp. However, you need not limit yourself to using this. If you are using macOS, you can use the Terminal application. For a Mac, you can fire up the Terminal application. It is typically in the
/Applications/Utilities folder. Otherwise, hit ⌘-space bar and type
terminal in the search box, and select the Terminal Application. For Windows, you can use PowerShell, which you can launch through the
menu. If you are using Linux, it’s a good bet you already know how to navigate a terminal, so we will not give specific instructions for Linux.
A very brief historical overview¶
Ken Thompson of Bell Labs developed the predecessor to the modern shell for the first release of UNIX in 1971. In 1977, Stephen Bourne introduced the Bourne shell (
sh) which added the ability to invoke scripts (small reusable programs) from within the shell. The Bourne shell remains relevant. In some cases it is still the default root shell. Shortly afterwards, the C shell (
csh) was developed which made use of a C-like scripting language.
tcsh is built on
csh and is still very
common. Bash, the Bourne again shell, was developed by Brian Fox to replace the Bourne shell. It adds many useful features to sh and is the default shell for macOS and several Linux distributions. In 1990, Paul Falsted developed Zsh (pronounced Z-shell) as a further improvement of Bash. Starting in October of 2019, Zsh is the default shell for macOS.
Windows vs. macOS/Linux¶
Windows 10 enables you to also use Bash, but you need to activate the Windows Subsystem for Linux. We will not use
bash for Windows users in the bootcamp (unless you want to), but will instead use PowerShell, which is also the default shell in JupyterLab’s terminal for Windows users. For the simple command line operations we will do in the bootcamp, PowerShell is almost always sufficient
and the syntax is the same. This cheatsheet is a useful reference for comparing Bash and PowerShell commands.
In what follows in this lesson, we will show the Zsh commands and sometimes provide commentary for Windows users. Here is a brief table comparing Bash and Zsh to PowerShell commands.
Does not exist
Getting started with the command line¶
Now, as you did in Lesson 0, launch a terminal in JupyterLab.
As we go through this tutorial, any text in green boxes (or indented fixed-width text in the Jupyter notebook version of this tutorial) contains commands you should enter at the command line.
pwd and ls¶
Let’s start out by using the
pwd command to figure out what directory we’re in.
pwd tells you the path of your current directory. A path for a directory or file is the list of all its parent directories, separated by slashes (
/), up to the root directory signified by the initial
/. You are probably in your home directory.
To list all files and folders in the current directory, we employ the
cd, change directory:¶
Let’s make sure we are in the home directory:
pwd to check where you are now. Invoking the
cd command without specifying a target directory defaults to the home directory. Another way to specify your home directory is by its shortcut,
~/. In general, the tilde-slash means “home directory.”
mkdir and rmdir¶
To make a directory, the command is
mkdir, hence the name make directory. followed by the name of the directory you want to create. For example, to make a directory called
You now have an empty directory called bootcamp. You can see it if you list the contents of the directory.
We do not need (nor want) this directory, since we will be using the bootcamp directory under version control with Git, so let’s delete it. To delete an empty directory, the command is
We will be using
git a lot starting with the second day of the bootcamp, and will in fact use it right now to set things up for you for this tutorial. Put briefly,
git is a version control system that allows multiple programmers to work together and allows individuals to keep track of their work. You are going to use it in just a moment to get all of the code your need for the bootcamp on your machine organized exactly as I have it organized on my machine.
In my work, I like to have a directory in my home directory called
git that has all of the code from
git repositories that I work on. We’ll have you do the same, though you are welcome to change how you organize things after the bootcamp. In Lesson 0, you made a
git directory to house all of your repositories.
Let’s venture into that directory.
I may have updated some things in that repository between the time where you cloned the repository and now. You can update the directory.
git pull upstream master
This pulls in all of the changes from the upstream repository (which is mine, the one you forked from).
Word to the wise: NO SPACES¶
Look at what is in the directory using
You will notice a few files and some directories. The directory
command_line_tutorial/ has some files that will help us through this lesson. Note that there are no spaces in the directory name. In general, you avoid spaces in directory and file names, even though your operating system often has them in there. Trust me on this, they can make things a total mess, especially on the command line, since a space also separates commands. Really. NO SPACES.
mv: renaming files¶
Since we’re currently doing a command line tutorial, let’s go into that directory and see what is there.
We see that we have a directory called
sequences, as well as a FASTA file named
some sequence.fasta. This file name has the annoying space in it. We would like to rename it something without a space, say
some_sequence.fasta. To do this, we us the ``mv`` command, short for “move.” We enter
mv, followed by the name of the file we want to rename, and then its new name.
mv some sequence.fasta some_sequence.fasta
Uh-oh! That gave us some strange output, talking about the usage of
mv. This is because the space in the file
some sequence.fasta was interpreted as a gap between arguments of the
mv command. To specify that the space is part of the file name, we need to use an escape character. The escape character for macOS or Linux is
\. With Windows, you can use a caret
^ as an escape character or you can enclose the file name with a space in single quotes. The space following the
escape character is not considered as an argument separator. This works (but don’t do it just yet):
macOS or Linux:
mv some\ sequence.fasta some_sequence.fasta # Don't do this
mv 'some sequence.fasta' some_sequence.fasta # Don't do this
Because these files are under version control, you should precede the
mv command with
git. That way, Git will keep track of the naming changes you made. So, do this:
macOS or Linux:
git mv some\ sequence.fasta some_sequence.fasta
git mv 'some sequence.fasta' some_sequence.fasta
Now, we probably want this file in the
sequences directory. We can also move files into directories (without changing their file names) using the
git mv some_sequence.fasta sequences/
The trailing slash is not necessary, but I always include it out of habit to remind myself that I am moving a file to a directory.
Now let’s go into the
sequences directory and see what we have.
We see that
some_sequence.fasta is there, along with other FASTA files.
Exploring file content¶
We would like to see what is in the sequence files. Bash offers various ways to display the content of files. We’ll look at the genome of the dengue virus in the file
dengue.fasta. There are lots of ways to do it. We’ll start with
less. It got its name because it is more feature-rich than
more, which was used to look at files before
less came to be. (“
more,” get it?) It allows using the arrow up and arrow down keys traverse up or down by line. It also allows
scrolling by touchpad or mouse. Since it doesn’t require the whole file to be read before displaying the top content, it’s ideal for larger files. It also supports searching initiated by “/” followed by the query;
shift+g will go to the end of the file;
gg to the beginning; and you can specify a line number by “
:” followed by the line number.
macOS or Linux:
We’ll now look at several other ways to look at files. Just substitute them for
less in the above command.
cat prints the entire file to the standard output (terminal). This is especially useful if the files are very small. Windows users, use
!type instead of
head just prints the top lines of the file to the standard output. The default can be changed:
This will print the first 5 lines to the standard output. Windows users, note alternative command.
Copying files and directories: cp¶
If you want to retain a copy of the folder/file in the original folder you can use the copy command
cp. It works straightforwardly with files. Applied to directories it requires a flag:
cp -r, meaning “recursive.” A flag typically begins with a hyphen (
-) and gives the command some extra directions on how you want to do things. In this case, we are telling
cp to work recursively.
Let’s have a look at the
cp command in action.
cp dengue.fasta copy_of_dengue.fasta
Maybe we want a copy of the entire
sequences directory. To do that, we will
cd one directory up to the
We went up one directory using
../. This is an example of a relative path. The current directory is “
../../” is two directories up, “
../../../” is three directories up, and so on. This is very very useful when navigating directory structures. Now let’s try copying an entire directory with the
cp -r sequences copy_of_sequences
We can also rename directories with the
mv command. Let’s rename
sequences_copy. This is silly, but illustrates how things work.
mv copy_of_sequences sequences_copy
Removing files and directories with rm¶
Yes, some of the things we just did are silly. We have no need for having a copy of a given sequence or a copy of the whole sequences directory. We can clean things up by deleting them. First, let’s get rid of our copy of the dengue sequence. Let’s
cd into the sequences directory and make sure it’s there.
Now let’s remove the file and verify it is gone.
And poof!, its gone! And I mean gone. It is pretty much irrecoverable. Warning:
rm is a wrecking ball. It will destroy any files you have that do not have restrictive permissions. This is so important, I will say it again.
rm is unforgiving.
Therefore, I always like to use the
-i flag, which means that
rm will ask me if I’m sure before deletion.
rm -i some_sequence.fasta
You will get a prompt. Answer
n if you do not want to delete it.
Now, let’s use
rm to remove an entire directory. To do this, we need to use the
rm -r sequences_copy
Aliases (PowerShell users, skip this section)¶
rm is a wrecking ball, but we can temper it using the
-i flag. For safety, we would like
rm to always ask us about deletion. We can instruct
bash to do this for us by creating an alias.
alias rm="rm -i"
After executing this, any time we use
bash will instead execute
rm -i, thereby keeping us out of trouble.
One of my favorite aliases is to make
ls list things more prettily.
alias ls="ls -FG"
-F flag makes
ls put a slash at the end of directories. This helps us tell the difference between files and directories. The
-G flag enables coloring of the output, also useful for differentiating file types.
Updating your bootcamp directory¶
You have now updated the name of the file
some_sequence.fasta. Git kept track of that, so you should commit and push your change. We will talk more about Git later in the bootcamp. For now, do the following commands to commit your change and then push the change to your master branch, which is your fork.
git commit -m "Changed file name of some_sequence.fasta."
git push origin master
You are now empowered¶
You are now already to manage files and navigate your way around the command line! My computer runs macOS. I very rarely use Finder to copy, move, or even read files. I do it all on the command line. Once you get the hang of it, you will find the command line very efficient.
%watermark -v -p jupyterlab