Lesson 11: File I/O


[1]:
import os
import glob

Reading data in from files and then writing your results out again is one of the most common practices in scientific computing. In this tutorial, we will learn about some of Python’s File I/O capabilities. We will use a PDB file as an example. The PDB file contains the crystal structure for the tetramerization domain of p53.It is stored in the file ~/git/bootcamp/data/1OLG.pdb. (Make sure you launch your notebook from the ~/git/bootcamp/ directory.) Note that 1OLG is its unique Protein Databank identifier.

File objects

To open a file, we use the built-in open() function. When opening files, we should do this using context management. I will demonstrate how to open a file and then describe the syntax.

[2]:
with open('data/1OLG.pdb', 'r') as f:
    print(type(f))
<class '_io.TextIOWrapper'>

Python has a wonderful keyword, with. This keyword enables context management. Upon entry into a with block, variables have certain meaning. In this case, the variable f has the meaning of an open file, an instance of the _io.TextIOWrapper class. Upon exit, certain operations take place. For file objects created by opening them, the file is automatically closed upon exit, even if there is an error. This is important. If your program raises an exception before you have a chance to close the file, it won’t get closed and you could be in trouble. If you use context management, the file will still get closed. So here is an important tip:

Use context management using with when working with files.

Let’s focus for a moment on the variable f in the above code cell. It is a Python file object, which has methods and attributes, just like any other object. We’ll explore those in a moment, but first, let’s look at how we opened the file. The first argument to open() is a string that has the name of the file, with the full path if necessary. The second argument is a string that says what we will be doing with the file. I.e., are we reading or writing to the file? The possible strings for this second argument are

string

meaning

'r'

open a text file for reading

'w'

create and open a text file for writing

'a'

append an existing text file

'r+'

open a text file for reading and writing

append 'b' to any of the above

same as above, except for binary files

We will mostly be working with text files in the bootcamp, so the first three are the most useful. A big warning, though….

Trying to open an existing file with ‘w’ will wipe it out and create a new file.

Reading data out of the file with file object methods

We will focus on the methods f.read() and f.readlines(). What do they do?

method

task

f.read()

Read the entire contents of the file into a string

f.readlines()

Read the entire file into a list with each item being a string representing a line

First, we’ll try using the first method to get a single string with the entire contents of the file.

[3]:
# Read file into string
with open('data/1OLG.pdb', 'r') as f:
    f_str = f.read()

# Let's look at the first 1000 characters
f_str[:1000]
[3]:
'HEADER    ANTI-ONCOGENE                           13-JUN-94   1OLG              \nTITLE     HIGH-RESOLUTION SOLUTION STRUCTURE OF THE OLIGOMERIZATION             \nTITLE    2 DOMAIN OF P53 BY MULTI-DIMENSIONAL NMR                               \nCOMPND    MOL_ID: 1;                                                            \nCOMPND   2 MOLECULE: TUMOR SUPPRESSOR P53 (OLIGOMERIZATION DOMAIN);             \nCOMPND   3 CHAIN: A, B, C, D;                                                   \nCOMPND   4 ENGINEERED: YES                                                      \nSOURCE    MOL_ID: 1;                                                            \nSOURCE   2 ORGANISM_SCIENTIFIC: HOMO SAPIENS;                                   \nSOURCE   3 ORGANISM_COMMON: HUMAN;                                              \nSOURCE   4 ORGANISM_TAXID: 9606                                                 \nKEYWDS    ANTI-ONCOGENE                                                         \nEXPDTA    SOLUTION NMR      '

We see lots of \n, which signifies a new line. The backslash is known as an escape character, meaning that the n after it does not signify the letter n, but that \n together means a new line.

Now, let’s try reading it in as a list.

[4]:
# Read contents of the file in as a list
with open('data/1OLG.pdb', 'r') as f:
    f_list = f.readlines()

# Look at the list (first ten entries)
f_list[:10]
[4]:
['HEADER    ANTI-ONCOGENE                           13-JUN-94   1OLG              \n',
 'TITLE     HIGH-RESOLUTION SOLUTION STRUCTURE OF THE OLIGOMERIZATION             \n',
 'TITLE    2 DOMAIN OF P53 BY MULTI-DIMENSIONAL NMR                               \n',
 'COMPND    MOL_ID: 1;                                                            \n',
 'COMPND   2 MOLECULE: TUMOR SUPPRESSOR P53 (OLIGOMERIZATION DOMAIN);             \n',
 'COMPND   3 CHAIN: A, B, C, D;                                                   \n',
 'COMPND   4 ENGINEERED: YES                                                      \n',
 'SOURCE    MOL_ID: 1;                                                            \n',
 'SOURCE   2 ORGANISM_SCIENTIFIC: HOMO SAPIENS;                                   \n',
 'SOURCE   3 ORGANISM_COMMON: HUMAN;                                              \n']

We see that each entry is a line, including the newline character. To look at lines in files, the rstrip() method for strings can come it handy. It strips all whitespace, including newlines, from the end of a string.

[5]:
f_list[0].rstrip()
[5]:
'HEADER    ANTI-ONCOGENE                           13-JUN-94   1OLG'

Reading line-by-line

What if we do not want to read the entire file into a list? For example, if a file is several gigabytes, we do not want to spend all of our RAM storing a list. Instead, we can read it line-by-line. Conveniently, the file object can be used as an iterator.

[6]:
# Print the first ten lines of the file
with open('data/1OLG.pdb', 'r') as f:
    for i, line in enumerate(f):
        print(line.rstrip())
        if i >= 10:
            break
HEADER    ANTI-ONCOGENE                           13-JUN-94   1OLG
TITLE     HIGH-RESOLUTION SOLUTION STRUCTURE OF THE OLIGOMERIZATION
TITLE    2 DOMAIN OF P53 BY MULTI-DIMENSIONAL NMR
COMPND    MOL_ID: 1;
COMPND   2 MOLECULE: TUMOR SUPPRESSOR P53 (OLIGOMERIZATION DOMAIN);
COMPND   3 CHAIN: A, B, C, D;
COMPND   4 ENGINEERED: YES
SOURCE    MOL_ID: 1;
SOURCE   2 ORGANISM_SCIENTIFIC: HOMO SAPIENS;
SOURCE   3 ORGANISM_COMMON: HUMAN;
SOURCE   4 ORGANISM_TAXID: 9606

Alternatively, we can use the method f.readline() to read a single line in the file and return it as a string.

[7]:
# Print the first ten lines of the file
with open('data/1OLG.pdb', 'r') as f:
    i = 0
    while i < 10:
        print(f.readline().rstrip())
        i += 1
HEADER    ANTI-ONCOGENE                           13-JUN-94   1OLG
TITLE     HIGH-RESOLUTION SOLUTION STRUCTURE OF THE OLIGOMERIZATION
TITLE    2 DOMAIN OF P53 BY MULTI-DIMENSIONAL NMR
COMPND    MOL_ID: 1;
COMPND   2 MOLECULE: TUMOR SUPPRESSOR P53 (OLIGOMERIZATION DOMAIN);
COMPND   3 CHAIN: A, B, C, D;
COMPND   4 ENGINEERED: YES
SOURCE    MOL_ID: 1;
SOURCE   2 ORGANISM_SCIENTIFIC: HOMO SAPIENS;
SOURCE   3 ORGANISM_COMMON: HUMAN;

Each subsequent call to f.readline() reads in the next line of the file. (As we read through a file, we keep moving forward in the bytes of the file and we have to use f.seek() to rewind.)

Writing to a file

Writing to a file has similar syntax. We already saw how to open a file for writing. Again, context management is useful. However, before trying to open a file, we should check to make sure a file of the same name does not exist before opening it. The os.path module is useful. The function os.path.isfile() function checks to see if a file exists.

[8]:
os.path.isfile('data/1OLG.pdb')
[8]:
True

Now that we know how to check existence of a file so we do not overwrite it, we can open and write a file.

[9]:
if os.path.isfile('mastery.txt'):
    raise RuntimeError('File mastery.txt already exists.')

with open('mastery.txt', 'w') as f:
    f.write('This is my file.')
    f.write('There are many like it, but this one is mine.')
    f.write('I must master my file like I must master my life.')

Note that we can use the f.write() method to write strings to a file. Let’s look at the file contents.

[10]:
!cat mastery.txt
This is my file.There are many like it, but this one is mine.I must master my file like I must master my life.

Ah! There are no newlines! When writing to a file, unlike when you use the print() function, you must include the newline characters. Let’s try again, intentionally obliterating our first attempt.

[11]:
with open('mastery.txt', 'w') as f:
    f.write('This is my file.\n')
    f.write('There are many like it, but this one is mine.\n')
    f.write('I must master my file like I must master my life.\n')

!cat mastery.txt
This is my file.
There are many like it, but this one is mine.
I must master my file like I must master my life.

That’s better. Note also that f.write() only takes strings as arguments. You cannot pass numbers. They must be converted to strings first.

[12]:
# This will result in an exception
with open('gimme_phi.txt', 'w') as f:
    f.write('The golden ratio is φ = ')
    f.write(1.61803398875)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-9265e8a96772> in <module>
      2 with open('gimme_phi.txt', 'w') as f:
      3     f.write('The golden ratio is φ = ')
----> 4     f.write(1.61803398875)

TypeError: write() argument must be str, not float

Yup. It must be a string. Let’s try again.

[13]:
with open('gimme_phi.txt', 'w') as f:
    f.write('The golden ratio is φ = ')
    f.write('{phi:.8f}'.format(phi=1.61803398875))

!cat gimme_phi.txt
The golden ratio is φ = 1.61803399

That works!

An exercise: extract atomic coordinates for first chain in tetramer

As an example on how to do file I/O, we will take the PDB file and extract only the ATOM records for the first chain of the tetramer and write only those entries to a new file.

It is useful to know that according to the PDB format specification, column 21 in the ATOM entry gives the ID of the chain.

We also conveniently use the fact that we can have multiple files open in our with block, separating them with commas.

[14]:
with open('data/1OLG.pdb', 'r') as f, open('atoms_chain_A.txt', 'w') as f_out:
    # Put the ATOM lines from chain A in new file
    for line in f:
        if len(line) > 21 and line[:4] == 'ATOM' and line[21] == 'A':
            f_out.write(line)

Let’s see how we did!

[15]:
!head -10 atoms_chain_A.txt
ATOM      1  N   LYS A 319      18.634  25.437  10.685  1.00  4.81           N
ATOM      2  CA  LYS A 319      17.984  25.295   9.354  1.00  4.32           C
ATOM      3  C   LYS A 319      18.160  23.876   8.818  1.00  3.74           C
ATOM      4  O   LYS A 319      19.259  23.441   8.537  1.00  3.67           O
ATOM      5  CB  LYS A 319      18.609  26.282   8.371  1.00  4.67           C
ATOM      6  CG  LYS A 319      18.003  26.056   6.986  1.00  5.15           C
ATOM      7  CD  LYS A 319      16.476  26.057   7.091  1.00  5.90           C
ATOM      8  CE  LYS A 319      16.014  27.341   7.784  1.00  6.51           C
ATOM      9  NZ  LYS A 319      16.388  28.518   6.952  1.00  7.33           N
ATOM     10  H1  LYS A 319      18.414  24.606  11.281  1.00  5.09           H
[16]:
!tail -10 atoms_chain_A.txt
ATOM    689  HD2 PRO A 359       0.183  25.663  13.542  1.00  4.71           H
ATOM    690  HD3 PRO A 359       0.246  23.956  13.062  1.00  4.53           H
ATOM    691  N   GLY A 360      -3.984  26.791  10.832  1.00  5.45           N
ATOM    692  CA  GLY A 360      -4.489  28.138  10.445  1.00  5.95           C
ATOM    693  C   GLY A 360      -5.981  28.236  10.765  1.00  6.77           C
ATOM    694  O   GLY A 360      -6.401  27.621  11.732  1.00  7.24           O
ATOM    695  OXT GLY A 360      -6.679  28.924  10.039  1.00  7.15           O
ATOM    696  H   GLY A 360      -4.589  26.020  10.828  1.00  5.72           H
ATOM    697  HA2 GLY A 360      -3.950  28.896  10.995  1.00  5.99           H
ATOM    698  HA3 GLY A 360      -4.341  28.288   9.386  1.00  6.05           H

Nice!

Finding files and with glob

In the above snippet of code, we extracted all atom records from a PDB file. We might want to do this (or some other operation) for many files. For example, the directory ~/git/data/ has four PDB files in it. For the present discussion, let’s say we want to pull the sequence of chain A out of each PDB file.

The glob module from the standard library enables us to get a list of all files that match a pattern. In our case, we want all files matching data/*.pdb, where * is a wild card character, meaning that any matches of characters where * appears are allowed. Let’s see what glob.glob() gives us.

[17]:
file_list = glob.glob('data/*.pdb')

file_list
[17]:
['data/1OLG.pdb', 'data/1J6Z.pdb', 'data/1FAG.pdb', 'data/2ERK.pdb']

We have the four PDB files. We can now loop over them and pull out the sequences.

[18]:
# Dictionary to hold sequences
seqs = {}

# Loop through all matching files
for file_name in file_list:
    # Extract PDB ID
    pdb_id = file_name[file_name.find('/')+1:file_name.rfind('.')]

    # Initialize sequence string, which we build as we go along
    seq = ''
    with open(file_name, 'r') as f:
        for line in f:
            if len(line) > 11 and line[:6] == 'SEQRES' and line[11] == 'A':
                seq += line[19:].rstrip() + ' '

    # Build sequence with dash-joined three letter codes
    seq = '-'.join(seq.split())

    # Store in the dictionary
    seqs[pdb_id] = seq

Let’s take a look at what we got. We’ll look at actin.

[19]:
seqs['1J6Z']
[19]:
'ASP-GLU-ASP-GLU-THR-THR-ALA-LEU-VAL-CYS-ASP-ASN-GLY-SER-GLY-LEU-VAL-LYS-ALA-GLY-PHE-ALA-GLY-ASP-ASP-ALA-PRO-ARG-ALA-VAL-PHE-PRO-SER-ILE-VAL-GLY-ARG-PRO-ARG-HIS-GLN-GLY-VAL-MET-VAL-GLY-MET-GLY-GLN-LYS-ASP-SER-TYR-VAL-GLY-ASP-GLU-ALA-GLN-SER-LYS-ARG-GLY-ILE-LEU-THR-LEU-LYS-TYR-PRO-ILE-GLU-HIC-GLY-ILE-ILE-THR-ASN-TRP-ASP-ASP-MET-GLU-LYS-ILE-TRP-HIS-HIS-THR-PHE-TYR-ASN-GLU-LEU-ARG-VAL-ALA-PRO-GLU-GLU-HIS-PRO-THR-LEU-LEU-THR-GLU-ALA-PRO-LEU-ASN-PRO-LYS-ALA-ASN-ARG-GLU-LYS-MET-THR-GLN-ILE-MET-PHE-GLU-THR-PHE-ASN-VAL-PRO-ALA-MET-TYR-VAL-ALA-ILE-GLN-ALA-VAL-LEU-SER-LEU-TYR-ALA-SER-GLY-ARG-THR-THR-GLY-ILE-VAL-LEU-ASP-SER-GLY-ASP-GLY-VAL-THR-HIS-ASN-VAL-PRO-ILE-TYR-GLU-GLY-TYR-ALA-LEU-PRO-HIS-ALA-ILE-MET-ARG-LEU-ASP-LEU-ALA-GLY-ARG-ASP-LEU-THR-ASP-TYR-LEU-MET-LYS-ILE-LEU-THR-GLU-ARG-GLY-TYR-SER-PHE-VAL-THR-THR-ALA-GLU-ARG-GLU-ILE-VAL-ARG-ASP-ILE-LYS-GLU-LYS-LEU-CYS-TYR-VAL-ALA-LEU-ASP-PHE-GLU-ASN-GLU-MET-ALA-THR-ALA-ALA-SER-SER-SER-SER-LEU-GLU-LYS-SER-TYR-GLU-LEU-PRO-ASP-GLY-GLN-VAL-ILE-THR-ILE-GLY-ASN-GLU-ARG-PHE-ARG-CYS-PRO-GLU-THR-LEU-PHE-GLN-PRO-SER-PHE-ILE-GLY-MET-GLU-SER-ALA-GLY-ILE-HIS-GLU-THR-THR-TYR-ASN-SER-ILE-MET-LYS-CYS-ASP-ILE-ASP-ILE-ARG-LYS-ASP-LEU-TYR-ALA-ASN-ASN-VAL-MET-SER-GLY-GLY-THR-THR-MET-TYR-PRO-GLY-ILE-ALA-ASP-ARG-MET-GLN-LYS-GLU-ILE-THR-ALA-LEU-ALA-PRO-SER-THR-MET-LYS-ILE-LYS-ILE-ILE-ALA-PRO-PRO-GLU-ARG-LYS-TYR-SER-VAL-TRP-ILE-GLY-GLY-SER-ILE-LEU-ALA-SER-LEU-SER-THR-PHE-GLN-GLN-MET-TRP-ILE-THR-LYS-GLN-GLU-TYR-ASP-GLU-ALA-GLY-PRO-SER-ILE-VAL-HIS-ARG-LYS-CYS-PHE'

Computing environment

[20]:
%load_ext watermark
%watermark -v -p jupyterlab
CPython 3.7.7
IPython 7.13.0

jupyterlab 1.2.6