Climate data analysis @ WHOI: a tutorial

The purpose of this tutorial is to provide practical guidance on how to analyze gridded climate data stored on WHOI's servers using python. The tutorial is designed to take place over ~6 hour-long sessions, and is designed to cover four topics: (1) data pre-processing, (2) model validation, (3) using pre-industrial control runs to detect a climate change signal, and (4) model intercomparison. In addition to the tutorials we've provided several additional examples which illustrate the use of large ensembles and paleo-proxies, for example. Below, you will find recent announcements, a description of the individual tutorial sessions and other examples, setup instructions, an overview of the project structure, and other potentially useful links. Before going into these details, here's a preview:

Pre-processing with xarray

from xarray_reference.ipynb

Model validation

2m-temperature bias CESM2 (relative to ERA5), from model_validation.ipynb

Gulf Stream bias in CESM2 (relative to ORAS5), from T2m_and_gulfstream_validation.ipynb

Climate change detection

Woods Hole 2m-temperature compared to PI control (CESM2), from climate_change_detection.ipynb

Intermodel comparison and large ensembles

Woods Hole 2m-temperature projections (1% year^-1 CO₂ scenario), from (intermodel_comparison.ipynb)

1,000-member toy model ensemble (fig. from (stochastic_large_ensemble.ipynb)

7/19/24 updates

• The project structure (including filenames) has changed since the last tutorial! The outline below contains links to notebooks that we went through together in class.
• Rendered results from the tutorial notebooks can be found in the results folder. To see the results for a given notebook, click through to the folder with the notebook's name, then open the markdown (".md") file inside.
• See below for links to (i) the CMIP6 overview paper and (ii) CMIP naming conventions

Outline for summer 2024 tutorial

Date	Topic	Notebook
7/9/24	Connecting to the CMIP5* and CMIP6 data servers	N/A (see instructions below)
7/10/24	Pre-processing using xarray	xarray_reference.ipynb
7/11/24	Defining a climate index	woodshole_climate_index.ipynb
7/16/24	Climate model validation	model_validation.ipynb
7/17/24	Detecting climate change using models	climate_change_detection.ipynb
7/18/24	Model intercomparison using WHOI's servers	model_intercomparison.ipynb

Note: see the task list for detailed instructions on how to run the tutorial notebooks from 7/11 - 7/18

^*CMIP = Coupled Model Intercomparison Project

Description of other examples

Topic	Notebook
Validating 2m-temperature and Gulf Stream position in CESM2	T2m_and_gulfstream_validation.ipynb
Detecting climate change in a stochastic model	stochastic_large_ensemble.ipynb
Reproducing results from a recent Nature Geoscience paper1	azores.ipynb

¹Cresswell-Clay, N. et al. "Twentieth-century Azores High expansion unprecedented in the past 1,200 years". Nat. Geosci. 15, 548–553 (2022).

Setup

Getting the code

• Option 1: if you're comfortable with Github, fork the repository (see this page for more on forking), then clone it to your PC.
• Option 2: Download the repository as a .zip file: go to the repository home page, then click "Code -> Download ZIP".

Virtual environment

1. Set up mamba or conda (if not already). To set up, download and install miniforge following the instructions here: https://github.com/conda-forge/miniforge.
2. Navigate to the project home folder (e.g., with cd ~/whoi-climate-data-tutorial)
3. Create a conda/mamba environment for the project with: mamba create -p ./envs and activate the environment with conda activate ./envs
4. Next, install necessary packages in the environment with:^3,4
   • (Mac/Linux) mamba env update -p ./envs --file environment.yml
   • (Windows) mamba env update -p ./envs --file environment_no_cdo.yml
5. Install custom module (src) in the environment with pip install -e .

³The CDO package, used for regridding data in this tutorial, is not available for Windows through conda (thanks to Haakon Pihlaja for catching this). This may cause the mamba env update command to "hang" when used with environment.yml, the full list of packages (which includes CDO). While it's possible to use CDO on Windows, it's probably not worth setting this up just for the tutorial. Instead, use the package list without CDO, environment_no_cdo.yml.

⁴If you're using conda and the conda install ... / conda env update ... commands are taking a long time, you could try updating the solver to "libmamba". If this doesn't work, you could also try setting the channel priority to flexible, with conda config --set channel_priority flexible (thanks to Lilli Enders for suggesting this).

Running the code (locally)

1. Navigate to project folder
2. Activate virtual environment (conda activate ./envs)
3. Start jupyter lab with by typing jupyter lab in terminal / command window

Accessing the climate data servers

Note: to access the data, you must be on the WHOI network (i.e., on the WHOI wifi or connected by VPN).

Option 1 (preferred): mounting the network file system

• Windows and (non-Mac) Linux users: follow online instructions for CMIP5 or for CMIP6.
• Mac users: for CMIP6, open Finder, then select "Go" from the top menu bar and click "Connect to Server". Then, enter smb://vast.whoi.edu/proj/cmip6. If prompted, enter your WHOI username (without "@whoi.edu") and password. Note the default mount location for the server is /Volumes/data. For CMIP5, use the same process but with the following server address: smb://cmip5.whoi.edu.

Option 2: downloading the data and running locally

For reanalysis and model data used in the tutorial:

• Download data from the shared Google Drive folder

For other (CMIP6) model output:

• Go to http://cmip6.whoi.edu/search.
• Click on the the "HTTP" link in the "Files" column for a dataset you'd like to download.
• On the next page, select individual files to download.
• (optional) Save these files to a folder called data in this project.

Option 3: running in the cloud

• Go to https://colab.research.google.com/
• In Google Colab, click "File -> Open notebook -> Upload" and select one of the tutorial notebooks.
• Go to the shared Google Drive folder
• Click the three dots next to the "climate-data" folder and click "Organize -> Add shortcut", then select "My Drive"
• Note: if using Google Colab, the regridding components of the tutorial will not work, owing to package compatibility issues (cannot import the xesmf package in Colab, possibly related to this issue).

Description of high-level folders & files

Folder/file	Description
scripts/tutorials	contains jupyter notebooks used in tutorial
scripts/examples	contains other examples of topics covered in the tutorial
results	contains .md files with rendered output from tutorials and examples
docs	contains FAQ and detailed instructions for completing tutorials
src	custom module containing functions used in examples
setup.py	file needed to import src module
environment*.yml	files containing list of packages needed for tutorial
.gitignore	list of files and extensions git should ignore

Other potentially useful links

• The Good Research Code Handbook (A guide for how to organize research code for non-computer scientists)
• Description of CMIP naming conventions, including variant ID
• CMIP6 overview paper