(c) 2017 Justin Bois. This work is licensed under a Creative Commons Attribution License CC-BY 4.0. All code contained herein is licensed under an MIT license.
This tutorial was generated from a Jupyter notebook. You can download the notebook here.
import numpy as np
import pandas as pd
Pandas can be a bit frustrating during your first experiences with it. In this lesson, we will practice using Pandas. The more and more you use it, the more distant the memory of life without it will become.
We will again use the frog tongue adhesion data set. Your goal here is to extract certain entries out of the DataFrame. If it is not in your namespace, load in the DataFrame using pd.read_csv().
df = pd.read_csv('data/frog_tongue_adhesion.csv', comment='#')
a) Extract the impact time of all impacts that had an adhesive strength of magnitude greater than 2000 Pa.
b) Extract the impact force and adhesive force for all of Frog II's strikes.
c) Extract the adhesive force and the time the frog pulls on the target for juvenile frogs (Frogs III and IV).
# a) impact times for frogs with |adh. strenght| < 2000.
df.loc[np.abs(df['adhesive strength (Pa)'])<2000, 'impact time (ms)']
# b) Impact force and adhesive force for Frog II
df.loc[df['ID']=='II', ['impact force (mN)', 'adhesive force (mN)']]
# c) Adhesive force and time frog pulls for frogs III and IV
df.loc[df['ID'].isin(['III', 'IV']),
['adhesive force (mN)', 'time frog pulls on target (ms)']]
groupby()¶In addition to his tidy data paper, Hadley Wickham also wrote an influential paper on the split-apply-combine idea. To quote him, "You see the split-apply-combine strategy whenever you break up a big problem into manageable pieces, operate on each piece independently and then put all the pieces back together." An example of a split-apply-combine operation on our present data set is this:
You can imagine that split-apply-combine operations happy very often in analyzing biological data.
The groupby() method of Pandas enables convenient split-apply-combine. It works by splitting up a DataFrame based on some criterion. Once that happens, we can then apply a function to these split up DataFrame. Upon application of the function, we get a recombined DataFrame with the result.
This is best shown by example. We'll compute mean impact force of each frog. First, though, let's do it the "long way."
- Extract all of Frog I's impact forces and compute the mean.
- Do the same for the other three frogs.
- Write a
forloop to do this and return a NumPy array with the four mean impact forces.
Well, we can actually do it with a list comprehension in one line, but you can imagine such operations can get clunky in a hurry.
[(frog_id, np.mean(df.loc[df['ID']==frog_id, 'impact force (mN)']))
for frog_id in df['ID'].unique()]
Now, unfortunately, you don't get a DataFrame out of this. You only get a list of typles with the frog IDs and mean impact forces. It would require more code to get these back into a convenient DataFrame.
Using groupby(), eliminates this hassle. Let's see by example.
# Make a GroupBy object
gb = df.groupby('ID')
# Apply the np.mean function to the grouped object
df_mean = gb.apply(np.mean)
# Look at the new DataFrame
df_mean
Sweet! Look at that! We actually calculated the mean of every variable for each frog. We have a DataFrame with the results. We can pull the mean impact force for a frog of interest using loc.
df_mean.loc['III', 'impact force (mN)']
Now, what if we want more information, like both the mean and the median? We can apply multiple functions to a GroupBy object using the agg() method. The argument of this method is a list of functions you want to apply.
df_mean_median = gb.agg([np.mean, np.median])
df_mean_median
We now have a MultiIndex for the column headers. We can index with tuples.
df_mean_median.loc[:, ('impact force (mN)', 'median')]
Now, let's practice with groupby().
a) Compute standard deviation of the impact forces for each frog.
b) Write a function, coeff_of_var(data), which computes the coefficient of variation of a data set. This is the standard deviation divided by the absolute value of the mean.
c) Compute the coefficient of variation of the impact forces and adhesive forces for each frog.
d) And now, finally.... Compute a DataFrame that has the mean, median, standard deviation, and coefficient of variation of the impact forces and adhesive forces for each frog.
# a) standard deviation of impact forces
gb.apply(np.std)['impact force (mN)']
# b and c: coeff. of variation for impact and adhesive force
def coeff_of_var(data):
"""Coefficient of variation."""
return np.std(data) / np.abs(np.mean(data))
# Make GroupBy object with two columns of interest in DataFrame for convenience
gb = df[['ID', 'impact force (mN)', 'adhesive force (mN)']].groupby('ID')
# Applot the coeff_of_var_function
gb.apply(coeff_of_var)
# d) Apply all of the great stats functions!
df_result = gb.agg([np.mean, np.median, np.std, coeff_of_var])
df_result
We can index these things using the MultiIndex of the columns, but we much prefer tidy DataFrames, which we can generate again use pd.melt().
# First make the index (frog ID) column of DataFrame
df_result['ID'] = df_result.index
# Melt the DataFrame to make it tidy
pd.melt(df_result, var_name=['quantity', 'statistic'], id_vars='ID')