Python for data analysis: Instructor Notes

Challenge solutions

Install the required workshop packages on Mac OS with Homebrew

Homebrew is one of the popular package managers used by many Mac users.

To install Homebrew, you need have xcode command line tools installed,

from the terminal, type:

xcode-select --install

then

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Run the following command to ensure Homebrew is installed properly:

brew doctor

install Python 3:

brew install python3

All the rest of workshop packages can be installed by pip3 which comes together with Python 3:

pip3 install numpy pandas matplotlib jupyter
pip3 install bokeh ggplot

pip3 can automatically figure out all the dependencies that required by the above packages.

Launch a Python command prompt, type:

python3

Checking installations.

In the include/scripts directory, you will find a script called check_env.py This checks the functionality of the Anaconda install.

By default, Data Carpentry does not have people pull the whole repository with all the scripts and addenda. Therefore, you, as the instructor, get to decide how you’d like to provide this script to learners, if at all. To use this, students can navigate into includes/scripts terminal, and execute the following:

python check_env.py

If learners receive an AssertionError, it will inform you how to help them correct this installation. Otherwise, it will tell you that the system is good to go and ready for Data Carpentry!

00-short-introduction-to-Python

Tuples Challenges

Dictionaries Challenges

Make sure it is also clear that access to ‘the second value’ is actually just about the key name. Add for example rev[10] = "ten" to clarify it is not about the position.

rev[2] = "apple-sauce"

{1: 'one', 2: 'apple-sauce', 3: 'three'}

01-starting-with-data

###Bug Note:

Pandas < .18.1 has a bug where surveys_df[‘weight’].describe() may return a runtime error.

Dataframe Challenges

Calculating Statistics Challenges

Both do result in the same output, making it alternative ways of getting the unique values. nunique combines the count and unique value extraction.

Grouping Challenges

surveys_df.groupby(['site_id','sex']).agg({"year": 'min',
                                           "hindfoot_length": 'median',
                                           "weight": 'mean'})`

surveys_df.groupby(['site_id'])['weight'].describe()

Plotting Challenges

surveys_df.groupby('site_id').mean()["weight"].plot(kind='bar')

average weight across all species for each site

surveys_df.groupby('sex').count()["record_id"].plot(kind='bar')

total males versus total females for the entire dataset

02-index-slice-subset

Tip: use .head() method throughout this lesson to keep your display neater for students. Encourage students to try with and without .head() to reinforce this useful tool and then to use it or not at their preference. For example, if a student worries about keeping up in pace with typing, let them know they can skip the .head(), but that you’ll use it to keep more lines of previous steps visible.

Indexing Challenges

Selection Challenges

Advanced Selection Challenges

surveys_df[~surveys_df["sex"].isin(['M', 'F'])]

Masking Challenges

new = surveys_df[~surveys_df['sex'].isin(['M', 'F'])].copy()
new['sex']='x'
print(len(new))

Can verify the number of Nan values with sum(surveys_df['sex'].isnull()), which is equal to the number of none female/male records.

# selection of the data with isin
stack_selection = surveys_df[(surveys_df['sex'].isin(['M', 'F'])) &
							surveys_df["weight"] > 0.][["sex", "weight", "site_id"]]
# calculate the mean weight for each site id and sex combination:
stack_selection = stack_selection.groupby(["site_id", "sex"]).mean().unstack()
# and we can make a stacked bar plot from this:
stack_selection.plot(kind='bar', stacked=True)

Suggestion: As we now the other values are all Nan values, we could also select all not null values (just preview, more on this in next lesson):

stack_selection = surveys_df[(surveys_df['sex'].notnull()) &
					surveys_df["weight"] > 0.][["sex", "weight", "site_id"]]

average weight for each site per sex

However, due to the unstack command, the legend header contains two levels. In order to remove this, the column naming needs to be simplified :

stack_selection.columns = stack_selection.columns.droplevel()

average weight for each site per sex

03-data-types-and-format

Challenge - Changing Types

Pandas cannot convert types from float to int if the column contains NaN values.

Challenge - Counting

surveys_df.isnull()

If the students have trouble generating the output, or anything happens with that, there is a file called “sample output” that contains the data file they should generate.

04-merging-data

# read the files:
survey2001 = pd.read_csv("data/survey2001.csv")
survey2002 = pd.read_csv("data/survey2002.csv")
# concatenate
survey_all = pd.concat([survey2001, survey2002], axis=0)
# get the weight for each year, grouped by sex:
weight_year = survey_all.groupby(['year', 'sex']).mean()["wgt"].unstack()
# plot:
weight_year.plot(kind="bar")
plt.tight_layout()  # tip(!)

average weight for each year, grouped by sex

# writing to file:
weight_year.to_csv("weight_for_year.csv")
# reading it back in:
pd.read_csv("weight_for_year.csv", index_col=0)

merged_left = pd.merge(left=surveys_df,right=species_df, how='left', on="species_id")

Then calculate and plot the distribution of:

1. taxa per site (number of species of each taxa per site):

Species distribution (number of taxa for each site) can be derived as follows:

merged_left.groupby(["site_id"])["taxa"].nunique().plot(kind='bar')

taxa per site

Suggestion: It is also possible to plot the number of individuals for each taxa in each site (stacked bar chart):

merged_left.groupby(["site_id", "taxa"]).count()["record_id"].unstack().plot(kind='bar', stacked=True)
plt.legend(loc='upper center', ncol=3, bbox_to_anchor=(0.5, 1.05))

(the legend otherwise overlaps the bar plot)

taxa per site

2. taxa by sex by site: Providing the Nan values with the M|F values (can also already be changed to ‘x’):

merged_left.loc[merged_left["sex"].isnull(), "sex"] = 'M|F'

Number of taxa for each site/sex combination:

ntaxa_sex_site= merged_left.groupby(["site_id", "sex"])["taxa"].nunique().reset_index(level=1)
ntaxa_sex_site = ntaxa_sex_plot.pivot_table(values="taxa", columns="sex", index=ntaxa_sex_plot.index)
ntaxa_sex_site.plot(kind="bar", legend=False)
plt.legend(loc='upper center', ncol=3, bbox_to_anchor=(0.5, 1.08),
           fontsize='small', frameon=False)

taxa per site per sex

Suggestion (for discussion only):

The number of individuals for each taxa in each site per sex can be derived as well.

sex_taxa_site  = merged_left.groupby(["site_id", "taxa", "sex"]).count()['record_id']
sex_taxa_site.unstack(level=[1, 2]).plot(kind='bar', logy=True)
plt.legend(loc='upper center', ncol=3, bbox_to_anchor=(0.5, 1.15),
           fontsize='small', frameon=False)

taxa per site per sex

This is not really the best plot choice: not readable,… A first option to make this better, is to make facets. However, pandas/matplotlib do not provide this by default. Just as a pure matplotlib example (M|F if for not-defined sex records):

fig, axs = plt.subplots(3, 1)
for sex, ax in zip(["M", "F", "M|F"], axs):
    sex_taxa_site[sex_taxa_plot["sex"] == sex].plot(kind='bar', ax=ax, legend=False)
    ax.set_ylabel(sex)
    if not ax.is_last_row():
        ax.set_xticks([])
        ax.set_xlabel("")
axs[0].legend(loc='upper center', ncol=5, bbox_to_anchor=(0.5, 1.3),
              fontsize='small', frameon=False)

taxa per site per sex

However, it would be better to link to Seaborn and Altair for its kind of multivariate visualisations.

site_info = pd.read_csv("data/sites.csv")
site_info.groupby("site_type").count()

merged_site_type = pd.merge(merged_left, site_info, on='site_id')
# For each site, get the number of species for each site
nspecies_site = merged_site_type.groupby(["site_id"])["species"].nunique().rename("nspecies")
# For each site, get the number of individuals
nindividuals_site = merged_site_type.groupby(["site_id"]).count()['record_id'].rename("nindiv")
# combine the two series
diversity_index = pd.concat([nspecies_site, nindividuals_site], axis=1)
# calculate the diversity index
diversity_index['diversity'] = diversity_index['nspecies']/diversity_index['nindiv']

Making a bar chart:

diversity_index['diversity'].plot(kind="barh")
plt.xlabel("Diversity index")

taxa per site per sex

05-loops-and-functions

Basic Loop Challenges

for creature in animals:
    print(creature+',', end='')

This loop also adds a comma after the last animal. A better, loop-free solution would be: ','.join(animals)

Looping Over Dataframe Challenges

surveys_year = surveys_df[surveys_df.year == year].dropna()

An empty file with only the headers

You could just make a list manually, however, why not check the first and last year making use of the code itself?

n_year = 5  # better overview by making variable from it
first_year = surveys_df['year'].min()
last_year = surveys_df['year'].max()

for year in range(first_year, last_year, n_year):
    print(year)

    # Select data for the year
    surveys_year = surveys_df[surveys_df.year == year].dropna()

Similar to previous example, but use the species_id column. surveys_df['species_id'].unique(). However, the species names would improve interpretation of the file naming. A join with the species: merged_left = pd.merge(left=surveys,right=species, how='left', on="species_id") and using the species column.

Functions Challenges

Show these in a debugging environment to make this more clear!

Additional Functions Challenges

def one_year_csv_writer(this_year, all_data, folder_to_save, root_name):
    """
    Writes a csv file for data from a given year.

    Parameters
    ---------
    this_year : int
        year for which data is extracted
    all_data: pd.DataFrame
        DataFrame with multi-year data
    folder_to_save : str
        folder to save the data files
    root_name: str
        root of the filenames to save the data
    """

    # Select data for the year
    surveys_year = all_data[all_data.year == this_year]

    # Write the new DataFrame to a csv file
    filename = os.path.join(folder_to_save, ''.join([root_name, str(this_year), '.csv']))
    surveys_year.to_csv(filename)

Also adapt function yearly_data_csv_writer with the additional inputs.

Additional Functions Challenges

Adapt the input arguments, e.g. 1978, 1979.

Output Management Challenges

Implementation inside the function:

filenames = []
for year in range(start_year, end_year+1):
    filenames.append(one_year_csv_writer(year, all_data, folder_to_save, root_name))
return filenames

NoneType

yearly_data_arg_test(surveys_df, end_year=2001)

def one_year_csv_writer(this_year, all_data, folder_to_save='./', root_name='survey'):
    """
    Writes a csv file for data from a given year.

    Parameters
    ---------
    this_year : int
        year for which data is extracted
    all_data: pd.DataFrame
        DataFrame with multi-year data
    folder_to_save : str
        folder to save the data files
    root_name: str
        root of the filenames to save the data
    """

    # Select data for the year
    surveys_year = all_data[all_data.year == this_year]

    # Write the new DataFrame to a csv file
    filename = os.path.join(folder_to_save, ''.join([root_name, str(this_year), '.csv']))
    surveys_year.to_csv(filename)

    # Write the new DataFrame to a csv file
    if len(surveys_year) > 0:
        filename = os.path.join(folder_to_save, ''.join([root_name, str(this_year), '.csv']))
        surveys_year.to_csv(filename)
    else:
        print("No data for year " + str(this_year))

def yearly_data_csv_writer(all_data, yearcolumn="year",
                           folder_to_save='./', root_name='survey'):
    """
    Writes separate csv files for each year of data.

    all_data --- DataFrame with multi-year data
    yearcolumn --- column name containing the year of the data
    folder_to_save --- folder name to store files
    root_name --- start of the file names stored
    """
    years = all_data["year"].unique()

    # "end_year" is the last year of data we want to pull, so we loop to end_year+1
    filenames = []
    for year in years:
        filenames.append(one_year_csv_writer(year, all_data, folder_to_save, root_name))
    return filenames

06-plotting-with-ggplot

If the students have trouble generating the output, or anything happens with that, there is a file called “sample output” that contains the data file they should have generated in lesson 3.

iPython notebooks for plotting can be viewed in the _extras folder

07-putting-it-all-together

Scientists often operate on mathematical equations. Being able to use them in their graphics has a lot of added value. Luckily, Matplotlib provides powerful tools for text control. One of them is the ability to use LaTeX mathematical notation, whenever text is used (you can learn more about LaTeX math notation here: https://en.wikibooks.org/wiki/LaTeX/Mathematics). To use mathematical notation, surround your text using the dollar sign (“$”). LaTeX uses the backslash character (“") a lot. Since backslash has a special meaning in the Python strings, you should replace all the LaTeX-related backslashes with two backslashes.

plt.plot(t, t, 'r--', label='$y=x$')
plt.plot(t, t**2 , 'bs-', label='$y=x^2$')
plt.plot(t, (t - 5)**2 + 5 * t - 0.5, 'g^:', label='$y=(x - 5)^2 + 5  x - \\frac{1}{2}$') # note the double backslash

plt.legend(loc='upper left', shadow=True, fontsize='x-large')

# Note the double backslashes in the line below.
plt.xlabel('This is the x axis. It can also contain math such as $\\bar{x}=\\frac{\\sum_{i=1}^{n} {x}} {N}$')
plt.ylabel('This is the y axis')
plt.title('This is the figure title')

plt.show()

This page contains more information.