# S2S Machine Learning Forecast Competition - Draft Idea

*Please note: the text below and the code example are meant to provide an example of what the competition could look like. It is provided as a rough initial draft idea, for further discussion.*

## Introduction

Each team's submission could consist of a Renku Project - which is  a git repository with some bells and whistles. A Juptyer Notebook written in Python could be provided as a starter project computatational interface, facilitating access to the S2S database and observational data, ML model building and training on past (hindcast) data, and the creation of a set of forecasts for the target (test) year. 

Renku also provides a computatational platform at SDSC which could be used for participants/teams with less demanding ML algorithms, while the git repository can be easily cloned to run on other machines for more demanding cases. These could be the team's own machines or possibly cloud computing resources.

Renku provides a pipeline that takes the forecasts for the test year(s) created and submitted by each team's Renku Project. These could be processed centrally to calculate skill scores from each team's submission, and summary scores piped to a real-time leaderboard.

Teams would be free to design their own Juptyer Notebook in a language of their choice (run at SDSC or elsewhere), with the competition requiring that the code be submitted in the form of a Juptyer Notebook at the close of the project. This would enable to check for adherence to protocol (no data is allowed to be used after the forecast start date), as well as for code sharing after the close of the competition. 

The Renku framework differs from the one used by Forecast Rodeo, which provided no software or hardware resources(?), simply requiring that the forecasts for the test period be submitted as NetCDF files. Topcoder was used to score them and provide the leaderboard.
https://www.usbr.gov/research/challenges/forecastrodeo.html

## Example Starter Project

The starter project here is a Juptyer Notebook written in Python; this could be used as a wrapper to call R libraries if desired.

### Main elements of Starter Project Juptyer Notebook:
* Read pre-processed S2S and obs precip and 2mT data via IRI Data Library (IRIDL)
* Train and select ML model (this demo uses logistic regression) using S2S hindcasts and observation data
* Run Test (Apply ML model to S2S forecasts, say for 2020)

### Results & Leaderboard:
* Pipe test results to RENKU/SDSC auto test & diagnostic suite. This could use the NCAR library "climpred" for verification of weather and climate forecasts.
* Pipe result to competition leaderboard

## This Demo: s2sforecast.ipynb

This Jupyter Notebook uses Extended Logistic Regression as a simple example of a ML method. For the competition, the ML method would be supplied via a Python function or wrapper. West Africa Jun-Aug case demo using GEFS SubX hindcasts (1999-2015) and forecasts (2017-2020).

1. Download GEFS (SubX) reforecasts and forecasts of week 3-4 precip, made every Wednesday over W Africa domain, from IRIDL. Pre-processing including computing biweekly averages over the W Africa domain, and anomaly calculation is done in IRIDL.
1. Download matching CHIRPS obs precip anomaly data.
1. Train "ML" model on 1999-2015 Jun-Aug hindcast period.
1. Compute in-sample hindcast RPSS for model selection.
1. Generate calibrated forecasts for 2017-2020 test period.
1. Verify against observations using RPSS.
1. Compare against "counting" benchmark.
For the competition, the last two steps would be omitted from the Jupyter Notebook; indstead the forecast from each team would be saved for scoring by SDSC.

## Discussion points

* Contractor needs to be an experienced python coder, fluent in git, Juptyer, etc.
* Interface with IRIDL also needs some support, although the basic elements exist for S2S precip and 2mT. 

***

## Working with the project

The simplest way to start your project is right from the Renku
platform - just click on the `Environments` tab and start a new session.
This will start an interactive environment right in your browser.

To work with the project anywhere outside the Renku platform,
click the `Settings` tab where you will find the
git repo URLs - use `git` to clone the project on whichever machine you want.

### Changing interactive environment dependencies

Initially we install a very minimal set of packages to keep the images small.
However, you can add python and conda packages in `requirements.txt` and
`environment.yml` to your heart's content. If you need more fine-grained
control over your environment, please see [the documentation](https://renku.readthedocs.io/en/latest/user/advanced_interfaces.html#dockerfile-modifications).

## Project configuration

Project options can be found in `.renku/renku.ini`. In this
project there is currently only one option, which specifies
the default type of environment to open, in this case `/lab` for
JupyterLab. You may also choose `/tree` to get to the "classic" Jupyter
interface.

## Moving forward

Once you feel at home with your project, we recommend that you replace
this README file with your own project documentation! Happy data wrangling!