final changes to scorer for RPSS from averaged RPS
  • After last minute community feedback, we decided to modify the way we average the RPS for RPSS. Following Weigel et al. 2007 s2s-ai-challenge#50, !23 implements RPSS=1-<RPS_ML>/<RPS_clim>, whereas before we had RPSS=<1-RPS_ML/RPS_clim>, where angle brackets <> denote the average of the scores over a given number of forecast–observation pairs. Furthermore, we now penalize NaNs where numerical values where expected by RPS=2.
  • We restart the s2saichallengescorer, which automatically fetches the new scores of previous submissions, so your old submissions are re-evaluated. If you want to evaluate your score yourself, please use the updated skill_by_year.