Commit dc38ff62 authored by Aaron Spring's avatar Aaron Spring 🚼
Browse files

Merge branch 'SubX_magic' into 'master'

IRIDL server-side processing

Server-side processing:
- [x] select and aggregate `lead_time` `L` and `forecast_time` `S`
- [x] regrid `X` and `Y` to 1.5 deg grid
- [x] SubX intake catalog biweekly 
- [x] requires unreleased climetlab_s2s_ai_challenge plugin
- [ ] init frequency forecast: keep as todo
- [x] IRIDL cat S2S 
- [x] IRIDL obs regrid
- [x] check proper lead ints or floats to match my int lead_time

closes #9

See merge request aaron.spring/s2s-ai-challenge-template!10
parents ae40fed3 fc6223c7
Pipeline #209132 passed with stage
in 19 seconds
...@@ -20,8 +20,10 @@ ...@@ -20,8 +20,10 @@
- `curl` & `wget`, see [wget_curl.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/wget_curl.ipynb) - `curl` & `wget`, see [wget_curl.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/wget_curl.ipynb)
- `intake`, see [intake.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/intake.ipynb) - `intake`, see [intake.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/intake.ipynb)
- `IRIDL` including overview, see [IRIDL.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/IRIDL.ipynb) - `IRIDL` including overview, see [IRIDL.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/IRIDL.ipynb)
- `intake` catalogs for `IRIDL` [`SubX`](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/SubX_catalog.yml) and [`S2S`](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/S2S_catalog.yml) see [IRIDL.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/IRIDL.ipynb) (#9 !10, [Aaron Spring](https://renkulab.io/gitlab/aaron.spring)
- SubX-Project models: `IRIDL` including overview, see [IRIDL.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/IRIDL.ipynb) - SubX-Project models: `IRIDL` including overview, see [IRIDL.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/IRIDL.ipynb)
- How to access password-protected S2S-Project output from IRIDL with xarray? see [IRIDL.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/IRIDL.ipynb) - How to access password-protected S2S-Project output from IRIDL with xarray? see [IRIDL.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/IRIDL.ipynb)
- Access NOAA CPC observations via opendap from `IRIDL`, see [IRIDL.ipynb](https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge-template/-/blob/master/notebooks/data_access/IRIDL.ipynb) (#9 !10, [Aaron Spring](https://renkulab.io/gitlab/aaron.spring)
- fix `netcdf4` version to `1.5.4` for `opendap` to work lazily with `xarray` (!2, !7, [Aaron Spring](https://renkulab.io/gitlab/aaron.spring)) - fix `netcdf4` version to `1.5.4` for `opendap` to work lazily with `xarray` (!2, !7, [Aaron Spring](https://renkulab.io/gitlab/aaron.spring))
......
...@@ -24,8 +24,8 @@ dependencies: ...@@ -24,8 +24,8 @@ dependencies:
- netcdf4 - netcdf4
- pip - pip
- pip: - pip:
- climetlab >= 0.7.0 - climetlab >= 0.8.0
- climetlab_s2s_ai_challenge >= 0.6.3 - climetlab_s2s_ai_challenge >= 0.7.1
- configargparse # for weatherbench - configargparse # for weatherbench
- netcdf4==1.5.4 - netcdf4==1.5.4
prefix: "/opt/conda" prefix: "/opt/conda"
This diff is collapsed.
plugins:
source:
- module: intake_xarray
sources:
training-input:
description: S2S hindcasts from IRIDL regridded to 1.5 deg grid and aggregated by mean over lead, https://iridl.ldeo.columbia.edu/SOURCES/.ECMWF/.S2S/overview.html
driver: opendap
parameters:
center:
description: name of the center issuing the hindcast
type: str
default: ECMF
allowed: [BOM, CNRM, ECCC, ECMF, HMCR, ISAC, JMA, KMA, NCEP, UKMO]
grid:
description: regrid to this global resolution
type: float
default: 1.5
lead_name:
description: name of the lead_time dimension
type: str
default: LA
allowed: [LA, L]
lead_start:
description: aggregation start lead passed to RANGEEDGES
type: int
default: 14
lead_end:
description: aggregation end lead passed to RANGEEDGES
type: int
default: 27
experiment_type:
description: type of experiment
type: str
default: perturbed
allowed: [control, perturbed, RMMS]
group:
description: group of variables
type: str
default: 2m_above_ground
#allowed: [2m_above_ground, ...] see https://iridl.ldeo.columbia.edu/SOURCES/.ECMWF/.S2S/.ECMF/.reforecast/.perturbed/
param:
description: variable name
type: str
default: 2t
#allowed: [2t] see https://iridl.ldeo.columbia.edu/SOURCES/.ECMWF/.S2S/.ECMF/.reforecast/.perturbed/
xarray_kwargs:
engine: netcdf4
args:
urlpath: http://iridl.ldeo.columbia.edu/SOURCES/.ECMWF/.S2S/.{{center}}/.reforecast/.{{experiment_type}}/.{{group}}/{{param}}/{{lead_name}}/({{lead_start}})/({{lead_end}})/RANGEEDGES/[{{lead_name}}]average/X/0/{{grid}}/358.5/GRID/Y/90/{{grid}}/-90/GRID/dods
test-input:
description: S2S forecasts from IRIDL regridded to 1.5 deg grid and aggregated by mean over lead, https://iridl.ldeo.columbia.edu/SOURCES/.ECMWF/.S2S/overview.html
driver: opendap
parameters:
center:
description: name of the center issuing the hindcast
type: str
default: ECMF
allowed: ['BOM','CNRM','ECCC','ECMF','HMCR','ISAC','JMA','KMA','NCEP','UKMO']
grid:
description: regrid to this global resolution
type: float
default: 1.5
lead_name:
description: name of the lead_time dimension
type: str
default: LA
allowed: [LA, L, L1]
lead_start:
description: aggregation start lead passed to RANGEEDGES
type: int
default: 14
lead_end:
description: aggregation end lead passed to RANGEEDGES
type: int
default: 27
experiment_type:
description: type of experiment
type: str
default: perturbed
allowed: [control, perturbed, RMMS]
group:
description: group of variables
type: str
default: 2m_above_ground
#allowed: see https://iridl.ldeo.columbia.edu/SOURCES/.ECMWF/.S2S/.ECMF/.reforecast/.perturbed/
param:
description: variable name
type: str
default: 2t
#allowed: [2t] see https://iridl.ldeo.columbia.edu/SOURCES/.ECMWF/.S2S/.ECMF/.reforecast/.perturbed/
xarray_kwargs:
engine: netcdf4
args:
urlpath: http://iridl.ldeo.columbia.edu/SOURCES/.ECMWF/.S2S/.{{center}}/.forecast/.{{experiment_type}}/.{{group}}/{{param}}/S/(0000%201%20Jan%202020)/(0000%2031%20Dec%202020)/RANGEEDGES/{{lead_name}}/({{lead_start}})/({{lead_end}})/RANGEEDGES/[{{lead_name}}]average/X/0/{{grid}}/358.5/GRID/Y/90/{{grid}}/-90/GRID/dods
plugins:
source:
- module: intake_xarray
sources:
training-input:
description: SubX hindcasts from IRIDL regridded to 1.5 deg grid and aggregated by mean over lead, http://iridl.ldeo.columbia.edu/SOURCES/.Models/.SubX/outline.html
driver: opendap
parameters:
center:
description: name of the center issuing the hindcast
type: str
default: EMC
allowed: [CESM, ECCC, EMC, ESRL, GMAO, NCEP, NRL, RSMAS]
model:
description: name of the model
type: str
default: GEFS
allowed: [30LCESM1, 46LCESM1, GEM, GEPS6, GEPS5, GEFS, GEFSv12, FIMr1p1, GEOS_V2p1, CFSv2, NESM, CCSM4]
grid:
description: regrid to this global resolution
type: float
default: 1.5
lead_start:
description: aggregation start lead passed to RANGEEDGES
type: int
default: 14
lead_end:
description: aggregation end lead passed to RANGEEDGES
type: int
default: 27
param:
description: variable name
type: str
default: pr
#allowed: [pr]
xarray_kwargs:
engine: netcdf4
args:
urlpath: http://iridl.ldeo.columbia.edu/SOURCES/.Models/.SubX/.{{center}}/.{{model}}/.hindcast/.{{param}}/L/({{lead_start}})/({{lead_end}})/RANGEEDGES/[L]average/X/0/{{grid}}/358.5/GRID/Y/90/{{grid}}/-90/GRID/dods
test-input:
description: SubX forecasts from IRIDL regridded to 1.5 deg grid and aggregated by mean over lead, http://iridl.ldeo.columbia.edu/SOURCES/.Models/.SubX/outline.html
driver: opendap
parameters:
center:
description: name of the center issuing the forecast
type: str
default: EMC
allowed: [CESM, ECCC, EMC, ESRL, GMAO, NCEP, NRL, RSMAS]
model:
description: name of the model
type: str
default: GEFS
allowed: [30LCESM1, 46LCESM1, GEM, GEPS6, GEPS5, GEFS, GEFSv12, FIMr1p1, GEOS_V2p1, CFSv2, NESM, CCSM4]
grid:
description: regrid to this global resolution
type: float
default: 1.5
lead_start:
description: aggregation start lead passed to RANGEEDGES
type: int
default: 14
lead_end:
description: aggregation end lead passed to RANGEEDGES
type: int
default: 27
param:
description: variable name, see http://iridl.ldeo.columbia.edu/SOURCES/.Models/.SubX/outline.html
type: str
default: pr
#allowed: [pr]
xarray_kwargs:
engine: netcdf4
args:
urlpath: http://iridl.ldeo.columbia.edu/SOURCES/.Models/.SubX/.{{center}}/.{{model}}/.forecast/.{{param}}/S/(0000%201%20Jan%202020)/(0000%2031%20Dec%202020)/RANGEEDGES/L/({{lead_start}})/({{lead_end}})/RANGEEDGES/[L]average/X/0/{{grid}}/358.5/GRID/Y/90/{{grid}}/-90/GRID/dods
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment