Lesson 15: Testing and test-driven development¶
(c) 2018 Justin Bois and Davi Ortega. With the exception of pasted graphics, where the source is noted, this work is licensed under a Creative Commons Attribution License CC-BY 4.0. All code contained herein is licensed under an MIT license.
This document was prepared at Caltech with financial support from the Donna and Benjamin M. Rosen Bioengineering Center.
This lesson was generated from a Jupyter notebook. You can download the notebook here.
Test-driven development, or TDD, is a paradigm for developing software. The idea is that a programmer thinks about a design specification for a bit of code, usually a function. I.e., she lays out what the input and output should be. She then writes a test (that will fail) for the bit of code. She then writes or updates the code to pass the test. She does this incrementally as she builds her code. Let's try this by example.
An example of TDD¶
We will write a function that computes the number of negatively charged residues in a protein. In other words, we count up the number of glutamate (E) and aspartate (D) residues.
We'll call the function number_negatives(), and will just make an empty function for now as a placeholder.
def number_negatives(seq):
"""Number of negative residues a protein sequence"""
# Do nothing for now
pass
Now, we'll write a trivial test. It is just a conditional expression stating the obvious: the number of negative charges in a sequence with a single glutamate should be 1.
number_negatives('E') == 1
It should have been 1, but our function did not calculate that correctly, thus the False output. We failed the test! But before we focus on the test failure, let's think about what we just did.
We defined the prototype for the function. We know we want it to take in a sequence (a string) and return an integer. So, in building the test, we have designed the interface for the function.
Back to the test failure. We will now revisit the function to write it so that it will pass the test.
def number_negatives(seq):
"""Number of negative residues a protein sequence"""
# Count E's and D's, since these are the negative residues
return seq.count('E') + seq.count('D')
We'll try out test again.
number_negatives('E') == 1
Hurray! We passed our first test. Now, lets write some more test based on what we expect from this function.
print(number_negatives('E') == 1)
print(number_negatives('D') == 1)
print(number_negatives('') == 0)
print(number_negatives('ACKLWTTAE') == 1)
print(number_negatives('DDDDEEEE') == 8)
Our function appears to be working well. But let's think carefully about how it could break. What if we input lowercase letters? I.e., what would we want
number_negatives('acklwttae')
to return? Should we allow lowercase inputs?
This is an example where coming up with tests is how we define the interface of the function, or in other words, how the function should behave giving a range of inputs. Note that we weren't done designing it on the first pass!
Moving on, let's say we want to allow lowercase symbols. But, before we mess with our function, let's write a test that defines the expected behavior.
number_negatives('acklwttae') == 1
We failed, as expected. Now, back to the function. We will add a line to convert the input sequence to uppercase.
def number_negatives(seq):
"""Number of negative residues a protein sequence"""
# Convert sequence to upper case
seq = seq.upper()
# Count E's and D's, since these are the negative residues
return seq.count('E') + seq.count('D')
Let's try the test again
number_negatives('acklwttae') == 1
Now that this passes, we need to make sure all the old tests also pass. We have to make sure everything passes.
print(number_negatives('E') == 1)
print(number_negatives('D') == 1)
print(number_negatives('') == 0)
print(number_negatives('ACKLWTTAE') == 1)
print(number_negatives('DDDDEEEE') == 8)
print(number_negatives('acklwttae') == 1)
Great! This works now.
You can see how the cycle proceeds. Right now we might be happy with our function, but use cases we have not thought of might creep up as we use the function in different contexts. For every unexpected behavior or bug you find, write another test that covers it. Importantly, any time you update your code, you need to run all of your tests to make sure the function is still performing in all cases after the update!
The assert statement¶
In our example, we used a bunch of print statements to check our tests. Conveniently, Python has a built-in way to do your tests using the assert keyword. For example, our first test using assert is as follows.
assert number_negatives('E') == 1
This ran without issue. Now, let's try asserting something we know will fail.
assert number_negatives('E') == 2
We get an AssertionError, indicating that our assertion failed. We can even append the assert statement with a comment describing the error.
assert number_negatives('E') == 2, 'Failed on sequence of length 1'
So, we see the basic syntax of assert statements. After assert, we have a conditional expression that evaluates to True or False. If it evaluates False, an AssertionError is raised, meaning that the test was failed. Optionally, the conditional expression can be followed with a comma and a string that describes how it failed. So, we could write all of our tests together as a series of assertions. Actually, it would be best to write a function that does all the testing.
def test_number_negatives():
"""Perform unit tests on number_negatives."""
assert number_negatives('E') == 1
assert number_negatives('D') == 1
assert number_negatives('') == 0
assert number_negatives('ACKLWTTAE') == 1
assert number_negatives('DDDDEEEE') == 8
assert number_negatives('acklwttae') == 1
# Run all the tests
test_number_negatives()
Excellent! Everything passed!
It might be a little underwhelming that Python exits silently when all our tests pass. Fortunately, someone else felt that way too and implemented a testing tool that is more into positive reinforcement.
Introducing pytest¶
The pytest (a.k.a. py.test) module comes with a standard Anaconda installation and is useful tool for automating our testing. It gives detailed feedback on tests and you can read its documentation here.
The unittest module from the standard library and nose are two other major testing packages for Python. All three are in common usage. We use pytest here because I think it is the easiest to use and understand.
Pytest is not only a module but also a command line application that searches for tests in your code, runs them and let you know if they fail, and if they pass; finally some positive reinforcement.
Using pytest¶
To take the most advantage of pytest, we should take a step back and write the functions we have been working with in this lesson to a .py file. Using the Jupyter Lab text editor, we'll write seq_features_and_tests.py. Copy the following in the text editor.
def number_negatives(seq):
"""Number of negative residues a protein sequence"""
# Convert sequence to upper case
seq = seq.upper()
# Count E's and D's, since these are the negative residues
return seq.count('E') + seq.count('D')
def test_number_negatives():
"""Perform unit tests on n_neg."""
assert number_negatives('E') == 1
assert number_negatives('D') == 1
assert number_negatives('') == 0
assert number_negatives('ACKLWTTAE') == 1
assert number_negatives('DDDDEEEE') == 8
assert number_negatives('acklwttae') == 1
Now, pytest makes it easy to verify if all these tests pass or not:
!pytest seq_features_and_tests.py
We can try the option -v for even more sugar.
!pytest -v seq_features_and_tests.py
Separating tests in functional units¶
In more complicated set of tests, it is a good idea to separate the tests by meaningful functional units, so when something breaks, we can easily find the problem and fix it.
pytest allows us to build multiple test functions with different names that indicates what they are testing. For example, let's change the content of the file seq_feature_and_tests.py to this:
def number_negatives(seq):
"""Number of negative residues a protein sequence"""
# Convert sequence to upper case
seq = seq.upper()
# Count E's and D's, since these are the negative residues
return seq.count('E') + seq.count('D')
def test_number_negatives_for_single_AA():
"""Perform unit tests on number_negative for single AA"""
assert number_negatives('E') == 1
assert number_negatives('D') == 1
def test_number_negatives_for_empty():
"""Perform unit tests on number_negative for empty entry"""
assert number_negatives('') == 0
def test_number_negatives_for_short_sequence():
"""Perform unit tests on number_negative for short sequence"""
assert number_negatives('ACKLWTTAE') == 1
assert number_negatives('DDDDEEEE') == 8
def test_number_negatives_for_lowercase():
"""Perform unit tests on number_negative for lowercase"""
assert number_negatives('acklwttae') == 1
and let's run it again without -v and with.
!pytest seq_features_and_tests.py
Look! Four dots instead of 1. And when we run it with the -v flag, it lists all four tests.
!pytest -v seq_features_and_tests.py
pytest is smart¶
Pytest is such a smart application that you don't even need to tell it explicitly which file it should look at. By default, pytest will search for files starting with test_ and ending with .py in the whole directory tree.
Also, for the sake of clarity, let's separate our functions from our code. So let's make another file, named test_seq_features.py and place just the functions with the assert statements. You can delete them now from the seq_features_and_tests.py and rename that file to seq_features.py.
The directory should have now these two files:
seq_features.py
def number_negatives(seq):
"""Number of negative residues a protein sequence"""
# Convert sequence to upper case
seq = seq.upper()
# Count E's and D's, since these are the negative residues
return seq.count('E') + seq.count('D')
and
test_seq_features.py
import seq_features
def test_number_negatives_single_E_or_D():
"""Perform unit tests on number_negative for single AA"""
assert seq_features.number_negatives('E') == 1
assert seq_features.number_negatives('D') == 1
def test_number_negatives_for_empty():
"""Perform unit tests on number_negative for empty entry"""
assert seq_features.number_negatives('') == 0
def test_number_negatives_for_short_sequences():
"""Perform unit tests on number_negative for short sequence"""
assert seq_features.number_negatives('ACKLWTTAE') == 1
assert seq_features.number_negatives('DDDDEEEE') == 8
def test_number_negatives_for_lowercase():
"""Perform unit tests on number_negative for lowercase"""
assert seq_features.number_negatives('acklwttae') == 1
Note that because the number_negatives() function is in a different file than the tests, we must import the seq_features module in the file with tests.
Now you can run the test as:
!pytest -v
The obvious thing to do next is to test some other cases. Think: what else could go wrong? What if there is an invalid residue in the sequence? How we expect our code to behave?
These and other semi-existential questions will be addressed in the next lesson.