Enhanced Intake-ESM Catalog Demo¶
This Jupyter Notebook compares the original Intake-ESM Catalog with an enhanced catalog that includes additional attributes. Both catalogs are an inventory of the NCAR Community Earth System Model (CESM) Large Ensemble (LENS) data hosted on AWS S3 (doi:10.26024/wt24-5j82).
Set up environment¶
[1]:
import intake
import pandas as pd
import pprint
# Not used here:
###import numpy as np
###import xarray as xr
# Allow multiple lines per cell to be displayed without print (default is just last line)
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
# Enable more explicit control of DataFrame display (e.g., to omit annoying line numbers)
from IPython.display import HTML
Inspect original intake-esm catalog¶
[2]:
# Open original collection description file
cat_url_orig = 'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.json'
coll_orig = intake.open_esm_datastore(cat_url_orig)
[3]:
print(coll_orig.esmcol_data['description']) #Description of collection
print("Catalog file:", coll_orig.esmcol_data['catalog_file'])
print(coll_orig) # Summary of collection structure
This is an ESM collection for CESM1 Large Ensemble Zarr dataset publicly available on Amazon S3 (us-west-2 region)
Catalog file: https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.csv
<aws-cesm1-le catalog with 56 dataset(s) from 409 asset(s)>
[4]:
# Show expanded version of collection structure with details
uniques_orig = coll_orig.unique(columns=["component", "frequency", "experiment", "variable"])
pprint.pprint(uniques_orig, compact=True, indent=1, width=80)
{'component': {'count': 5, 'values': ['atm', 'ice_nh', 'ice_sh', 'lnd', 'ocn']},
'experiment': {'count': 4, 'values': ['20C', 'CTRL', 'HIST', 'RCP85']},
'frequency': {'count': 6,
'values': ['daily', 'hourly6-1990-2005', 'hourly6-2026-2035',
'hourly6-2071-2080', 'monthly', 'static']},
'variable': {'count': 65,
'values': ['DIC', 'DOC', 'FLNS', 'FLNSC', 'FLUT', 'FSNO', 'FSNS',
'FSNSC', 'FSNTOA', 'FW', 'H2OSNO', 'HMXL', 'ICEFRAC',
'LHFLX', 'O2', 'PD', 'PRECC', 'PRECL', 'PRECSC',
'PRECSL', 'PSL', 'Q', 'QFLUX', 'QRUNOFF', 'QSW_HBL',
'QSW_HTP', 'RAIN', 'RESID_S', 'SALT', 'SFWF',
'SFWF_WRST', 'SHF', 'SHFLX', 'SHF_QSW', 'SNOW',
'SOILLIQ', 'SOILWATER_10CM', 'SSH', 'SST', 'T', 'TAUX',
'TAUX2', 'TAUY', 'TAUY2', 'TEMP', 'TMQ', 'TREFHT',
'TREFHTMN', 'TREFHTMX', 'TS', 'U', 'UES', 'UET',
'UVEL', 'VNS', 'VNT', 'VVEL', 'WTS', 'WTT', 'WVEL',
'Z3', 'aice', 'aice_d', 'hi', 'hi_d']}}
[5]:
# Show the first few lines of the catalog. There are as many lines as there are paths.
# The order is the same as that of the CSV catalog file listed in the JSON description file.
print("Catalog file:", coll_orig.esmcol_data['catalog_file'])
df = coll_orig.df
HTML(df.head(10).to_html(index=False))
Catalog file: https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.csv
[5]:
| component | frequency | experiment | variable | path | variable_long_name | dim_per_tstep | start | end |
|---|---|---|---|---|---|---|---|---|
| atm | daily | 20C | FLNS | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNS.zarr | net longwave flux at surface | 2.0 | 1920-01-01 12:00:00 | 2005-12-30 12:00:00 |
| atm | daily | 20C | FLNSC | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNSC.zarr | clearsky net longwave flux at surface | 2.0 | 1920-01-01 12:00:00 | 2005-12-30 12:00:00 |
| atm | daily | 20C | FLUT | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLUT.zarr | upwelling longwave flux at top of model | 2.0 | 1920-01-01 12:00:00 | 2005-12-30 12:00:00 |
| atm | daily | 20C | FSNS | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FSNS.zarr | net solar flux at surface | 2.0 | 1920-01-01 12:00:00 | 2005-12-30 12:00:00 |
| atm | daily | 20C | FSNSC | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FSNSC.zarr | clearsky net solar flux at surface | 2.0 | 1920-01-01 12:00:00 | 2005-12-30 12:00:00 |
| atm | daily | 20C | FSNTOA | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FSNTOA.zarr | net solar flux at top of atmosphere | 2.0 | 1920-01-01 12:00:00 | 2005-12-30 12:00:00 |
| atm | daily | 20C | ICEFRAC | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-ICEFRAC.zarr | fraction of sfc area covered by sea-ice | 2.0 | 1920-01-01 12:00:00 | 2005-12-30 12:00:00 |
| atm | daily | 20C | LHFLX | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-LHFLX.zarr | surface latent heat flux | 2.0 | 1920-01-01 12:00:00 | 2005-12-30 12:00:00 |
| atm | daily | 20C | PRECL | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-PRECL.zarr | large-scale (stable) precipitation rate (liq + ice) | 2.0 | 1920-01-01 12:00:00 | 2005-12-30 12:00:00 |
| atm | daily | 20C | PRECSC | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-PRECSC.zarr | convective snow rate (water equivalent) | 2.0 | 1920-01-01 12:00:00 | 2005-12-30 12:00:00 |
Table: First few lines of the original Intake-ESM Catalog showing the model component, the temporal frequency, the experiment, the abbreviated variable name, and the AWS S3 path for each Zarr store.
Finding Data¶
If you happen to know the meaning of the variable names, you can find what data are available for that variable. For example:
[6]:
# Filter the catalog to find available data for one variable
df = coll_orig.search(variable='FLNS').df
HTML(df.to_html(index=False))
[6]:
| component | frequency | experiment | variable | path | variable_long_name | dim_per_tstep | start | end |
|---|---|---|---|---|---|---|---|---|
| atm | daily | 20C | FLNS | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNS.zarr | net longwave flux at surface | 2.0 | 1920-01-01 12:00:00 | 2005-12-30 12:00:00 |
| atm | daily | CTRL | FLNS | s3://ncar-cesm-lens/atm/daily/cesmLE-CTRL-FLNS.zarr | net longwave flux at surface | 2.0 | 0402-01-01 12:00:00 | 2200-12-31 12:00:00 |
| atm | daily | HIST | FLNS | s3://ncar-cesm-lens/atm/daily/cesmLE-HIST-FLNS.zarr | net longwave flux at surface | 1.0 | 1850-01-01 00:00:00 | 1919-12-31 12:00:00 |
| atm | daily | RCP85 | FLNS | s3://ncar-cesm-lens/atm/daily/cesmLE-RCP85-FLNS.zarr | net longwave flux at surface | 2.0 | 2006-01-01 12:00:00 | 2100-12-31 12:00:00 |
| atm | hourly6-1990-2005 | 20C | FLNS | s3://ncar-cesm-lens/atm/hourly6-1990-2005/cesmLE-20C-FLNS.zarr | net longwave flux at surface | 2.0 | 1990-01-01 00:00:00 | 2006-01-01 00:00:00 |
| atm | hourly6-2026-2035 | RCP85 | FLNS | s3://ncar-cesm-lens/atm/hourly6-2026-2035/cesmLE-RCP85-FLNS.zarr | net longwave flux at surface | 2.0 | 2026-01-01 00:00:00 | 2036-01-01 00:00:00 |
| atm | hourly6-2071-2080 | RCP85 | FLNS | s3://ncar-cesm-lens/atm/hourly6-2071-2080/cesmLE-RCP85-FLNS.zarr | net longwave flux at surface | 2.0 | 2071-01-01 00:00:00 | 2081-01-01 00:00:00 |
| atm | monthly | 20C | FLNS | s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-FLNS.zarr | net longwave flux at surface | 2.0 | 1920-01-16 12:00:00 | 2005-12-16 12:00:00 |
| atm | monthly | CTRL | FLNS | s3://ncar-cesm-lens/atm/monthly/cesmLE-CTRL-FLNS.zarr | net longwave flux at surface | 2.0 | 0400-01-16 12:00:00 | 2200-12-16 12:00:00 |
| atm | monthly | HIST | FLNS | s3://ncar-cesm-lens/atm/monthly/cesmLE-HIST-FLNS.zarr | net longwave flux at surface | 1.0 | 1850-01-16 12:00:00 | 1919-12-16 12:00:00 |
| atm | monthly | RCP85 | FLNS | s3://ncar-cesm-lens/atm/monthly/cesmLE-RCP85-FLNS.zarr | net longwave flux at surface | 2.0 | 2006-01-16 12:00:00 | 2100-12-16 12:00:00 |
Table: All available Zarr stores for the “FLNS” data.
[7]:
# Narrow the filter to specific frequency and expiriment
df = coll_orig.search(variable='FLNS', frequency='daily', experiment='RCP85').df
HTML(df.to_html(index=False))
[7]:
| component | frequency | experiment | variable | path | variable_long_name | dim_per_tstep | start | end |
|---|---|---|---|---|---|---|---|---|
| atm | daily | RCP85 | FLNS | s3://ncar-cesm-lens/atm/daily/cesmLE-RCP85-FLNS.zarr | net longwave flux at surface | 2.0 | 2006-01-01 12:00:00 | 2100-12-31 12:00:00 |
Table: The single Zarr store for daily “FLNS” data from “RCP85” experiment.
The Problem¶
Do all potential users know that “FLNS” is a CESM-specific abbreviation for “Net longwave flux at surface”? How would a novice user find out, other than by finding separate documentation, or by opening a Zarr store in the hopes that the long name might be recorded there? How do we address the fact that every climate model code seems to have a different, non-standard name for all the variables, thus making multi-source research needlessly difficult?
Enhanced Intake-ESM Catalog¶
By adding additional columns to the Intake-ESM catalog, we should be able to improve semantic interoperability and provide potentially useful information to the users.
[8]:
# Open enhanced collection description file
cat_url = 'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le-enhanced.json'
coll = intake.open_esm_datastore(cat_url)
coll
aws-cesm1-le catalog with 27 dataset(s) from 365 asset(s):
| unique | |
|---|---|
| component | 5 |
| dim | 2 |
| frequency | 5 |
| experiment | 6 |
| start | 10 |
| end | 11 |
| variable | 75 |
| long_name | 75 |
| path | 365 |
[9]:
print(coll.esmcol_data['description']) # Description of collection
print("Catalog file:", coll.esmcol_data['catalog_file'])
print(coll) # Summary of collection structure
This is an inventory of the Community Earth System Model (CESM) Large Ensemble (LENS) dataset in Zarr format publicly available on Amazon S3 (https://doi.org/10.26024/wt24-5j82)
Catalog file: https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le-enhanced.csv
<aws-cesm1-le catalog with 27 dataset(s) from 365 asset(s)>
Long names¶
In the summary above, note the addition of additional elements: long_name, start, end, and dim.
[10]:
# Show the first few lines of the enhanced catalog.
print("Catalog file:", coll.esmcol_data['catalog_file'])
HTML(coll.df.head(10).to_html(index=False))
Catalog file: https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le-enhanced.csv
[10]:
| component | dim | frequency | experiment | start | end | variable | long_name | path |
|---|---|---|---|---|---|---|---|---|
| atm | 2D | monthly | HIST | 1850-01 | 1919-12 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/monthly/cesmLE-HIST-FLNS.zarr |
| atm | 2D | monthly | 20C | 1920-01 | 2005-12 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-FLNS.zarr |
| atm | 2D | daily | 20C | 1920-01-01 | 2005-12-31 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNS.zarr |
| atm | 2D | monthly | RCP85 | 2006-01 | 2100-12 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/monthly/cesmLE-RCP85-FLNS.zarr |
| atm | 2D | daily | RCP85 | 2006-01-01 | 2100-12-31 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/daily/cesmLE-RCP85-FLNS.zarr |
| atm | 2D | monthly | CTRL | 0400-01 | 2200-12 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/monthly/cesmLE-CTRL-FLNS.zarr |
| atm | 2D | monthly | CTRL_AMIP | 0001-01 | 2600-12 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/monthly/cesmLE-CTRL_AMIP-FLNS.zarr |
| atm | 2D | monthly | CTRL_SLAB_OCN | 0001-01 | 1000-12 | FLNS | Net longwave flux at surface | s3://ncar-cesm-lens/atm/monthly/cesmLE-CTRL_SLAB_OCN-FLNS.zarr |
| atm | 2D | monthly | HIST | 1850-01 | 1919-12 | FLNSC | Clearsky net longwave flux at surface | s3://ncar-cesm-lens/atm/monthly/cesmLE-HIST-FLNSC.zarr |
| atm | 2D | monthly | 20C | 1920-01 | 2005-12 | FLNSC | Clearsky net longwave flux at surface | s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-FLNSC.zarr |
Table: First few lines of the enhanced catalog, listing of the same information as the original catalog as well as the long name of each variable and an indication of whether each variable is 2D or 3D.
ISSUE: The long names arenotCF Standard Names, but rather are those documented at http://www.cgd.ucar.edu/ccr/strandwg/CESM-CAM5-BGC_LENS_fields.html. For interoperability, the long_name column should be replaced by a cf_name column and possibly an attribute column to disambiguate if needed.
[11]:
# List all available variables by Long Name, sorted alphabetically
uniques = coll.unique(columns=['long_name'])
nameList = sorted(uniques['long_name']['values'])
print(*nameList, sep='\n') #note *list to unpack each item for print function
Clearsky net longwave flux at surface
Clearsky net solar flux at surface
Convective precipitation rate (liq + ice)
Convective snow rate (water equivalent)
Dissolved Inorganic Carbon
Dissolved Organic Carbon
Dissolved Oxygen
Fraction of sfc area covered by sea-ice
Free-Surface Residual Heat Flux
Free-Surface Residual Salt Flux
Freshwater Flux
Geopotential Height (above sea level)
Geopotential Z at 500 mbar pressure surface
Heat Flux across top face
Heat Flux in grid-x direction
Heat Flux in grid-y direction
Horizontal total wind speed average at the surface
Internal Ocean Heat Flux Due to Ice Formation
Large-scale (stable) precipitation rate (liq + ice)
Large-scale (stable) snow rate (water equivalent)
Lowest model level zonal wind
Maximum (convective and large-scale) precipitation rate (liq+ice)
Maximum reference height temperature over output period
Meridional wind
Minimum reference height temperature over output period
Mixed-Layer Depth
Net longwave flux at surface
Net solar flux at surface
Net solar flux at top of atmosphere
Potential Density Ref to Surface
Potential Temperature
Reference height humidity
Reference height temperature
Salinity
Salt Flux across top face
Salt Flux in grid-x direction
Salt Flux in grid-y direction
Sea Surface Height
Sea level pressure
Solar Short-Wave Heat Flux
Solar Short-Wave Heat Flux in boundary layer
Solar Short-Wave Heat Flux in top layer
Specific Humidity at 850 mbar pressure surface
Specific humidity
Surface latent heat flux
Surface pressure
Surface sensible heat flux
Surface temperature (radiative)
Temperature
Total (convective and large-scale) precipitation rate (liq + ice)
Total (vertically integrated) precipitable water
Total Surface Heat Flux including short-wave
Upwelling longwave flux at top of model
Velocity in grid-x direction
Velocity in grid-y direction
Vertical Velocity
Virtual Salt Flux due to weak restoring
Virtual Salt Flux in FW Flux formulation
Wind stress (squared) in grid-x direction
Wind stress (squared) in grid-y direction
Wind stress in grid-x direction
Wind stress in grid-y direction
Zonal wind
atmospheric rain
atmospheric snow
fraction of ground covered by snow
grid cellmean ice thickness
grid cellmean ice thickness (daily)
grid cellmean ice thickness (monthly)
ice area, aggregate (daily)
ice area, aggregate (monthly)
snow depth (liquid water)
soil liquid water (vegetated landunits only)
soil liquid water + ice in top 10cm of soil (veg landunits only)
total liquid runoff (does not include QSNWCPICE)
[12]:
# Show all available data for a specific variable based on long name
myName = 'Salinity'
HTML(coll.search(long_name=myName).df.to_html(index=False))
[12]:
| component | dim | frequency | experiment | start | end | variable | long_name | path |
|---|---|---|---|---|---|---|---|---|
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-SALT.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-SALT.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-SALT.zarr |
Table: All available data in this catalog for selected variable
Substring matches¶
ISSUE: The current version of intake-esm search() function requires an exact full-string case-sensitive match of the long_name. (This has been reported as an issue at https://github.com/NCAR/cesm-lens-aws/issues/48)
Demonstrate a work-around: find all variables with a particular substring in the long name
[13]:
myTerm = 'Wind'
myTerm = myTerm.lower() #search regardless of case
partials = [name for name in nameList if myTerm in name.lower()]
print(f"All datasets with name containing {myTerm}:")
print(*partials, sep='\n')
All datasets with name containing wind:
Horizontal total wind speed average at the surface
Lowest model level zonal wind
Meridional wind
Wind stress (squared) in grid-x direction
Wind stress (squared) in grid-y direction
Wind stress in grid-x direction
Wind stress in grid-y direction
Zonal wind
[14]:
# Display full table for each match (could be lengthy if many matches)
for name in partials:
df = coll.search(long_name=name).df[['component', 'dim', 'experiment', 'variable', 'long_name']]
HTML(df.to_html(index=False))
###df.head(1) #show only first entry in each group for compactness
# Note: It is also possible to hide column(s) instead of specifying desired columns
###coll.search(long_name=name).df.drop(columns=['path'])
[14]:
| component | dim | experiment | variable | long_name |
|---|---|---|---|---|
| atm | 2D | 20C | WSPDSRFAV | Horizontal total wind speed average at the surface |
| atm | 2D | RCP85 | WSPDSRFAV | Horizontal total wind speed average at the surface |
[14]:
| component | dim | experiment | variable | long_name |
|---|---|---|---|---|
| atm | 2D | 20C | UBOT | Lowest model level zonal wind |
| atm | 2D | RCP85 | UBOT | Lowest model level zonal wind |
[14]:
| component | dim | experiment | variable | long_name |
|---|---|---|---|---|
| atm | 3D | HIST | V | Meridional wind |
| atm | 3D | 20C | V | Meridional wind |
| atm | 3D | 20C | V | Meridional wind |
| atm | 3D | RCP85 | V | Meridional wind |
| atm | 3D | RCP85 | V | Meridional wind |
| atm | 3D | RCP85 | V | Meridional wind |
| atm | 3D | CTRL | V | Meridional wind |
| atm | 3D | CTRL_AMIP | V | Meridional wind |
| atm | 3D | CTRL_SLAB_OCN | V | Meridional wind |
[14]:
| component | dim | experiment | variable | long_name |
|---|---|---|---|---|
| ocn | 2D | 20C | TAUX2 | Wind stress (squared) in grid-x direction |
| ocn | 2D | RCP85 | TAUX2 | Wind stress (squared) in grid-x direction |
| ocn | 2D | CTRL | TAUX2 | Wind stress (squared) in grid-x direction |
[14]:
| component | dim | experiment | variable | long_name |
|---|---|---|---|---|
| ocn | 2D | 20C | TAUY2 | Wind stress (squared) in grid-y direction |
| ocn | 2D | RCP85 | TAUY2 | Wind stress (squared) in grid-y direction |
| ocn | 2D | CTRL | TAUY2 | Wind stress (squared) in grid-y direction |
[14]:
| component | dim | experiment | variable | long_name |
|---|---|---|---|---|
| ocn | 2D | 20C | TAUX | Wind stress in grid-x direction |
| ocn | 2D | RCP85 | TAUX | Wind stress in grid-x direction |
| ocn | 2D | CTRL | TAUX | Wind stress in grid-x direction |
[14]:
| component | dim | experiment | variable | long_name |
|---|---|---|---|---|
| ocn | 2D | 20C | TAUY | Wind stress in grid-y direction |
| ocn | 2D | RCP85 | TAUY | Wind stress in grid-y direction |
| ocn | 2D | CTRL | TAUY | Wind stress in grid-y direction |
[14]:
| component | dim | experiment | variable | long_name |
|---|---|---|---|---|
| atm | 3D | HIST | U | Zonal wind |
| atm | 3D | 20C | U | Zonal wind |
| atm | 3D | 20C | U | Zonal wind |
| atm | 3D | RCP85 | U | Zonal wind |
| atm | 3D | RCP85 | U | Zonal wind |
| atm | 3D | RCP85 | U | Zonal wind |
| atm | 3D | CTRL | U | Zonal wind |
| atm | 3D | CTRL_AMIP | U | Zonal wind |
| atm | 3D | CTRL_SLAB_OCN | U | Zonal wind |
Table(s): Information about all matching datasets
ISSUE: The case-insensitive substring matching is not integrated into intake-esm, so it is not clear whether resulting search results can be passed directly to Xarray to read data.
Other attributes¶
Other columns in the enhanced catalog may be useful. For example, the dimensionality column enables us to list all data from the ocean component that is 3D.
[15]:
df = coll.search(dim="3D",component="ocn").df
HTML(df.to_html(index=False))
[15]:
| component | dim | frequency | experiment | start | end | variable | long_name | path |
|---|---|---|---|---|---|---|---|---|
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | DIC | Dissolved Inorganic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-DIC.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | DIC | Dissolved Inorganic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-DIC.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | DIC | Dissolved Inorganic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-DIC.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | DOC | Dissolved Organic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-DOC.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | DOC | Dissolved Organic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-DOC.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | DOC | Dissolved Organic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-DOC.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | O2 | Dissolved Oxygen | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-O2.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | O2 | Dissolved Oxygen | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-O2.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | O2 | Dissolved Oxygen | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-O2.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | PD | Potential Density Ref to Surface | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-PD.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | PD | Potential Density Ref to Surface | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-PD.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | PD | Potential Density Ref to Surface | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-PD.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-SALT.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-SALT.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-SALT.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | TEMP | Potential Temperature | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-TEMP.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | TEMP | Potential Temperature | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-TEMP.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | TEMP | Potential Temperature | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-TEMP.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | UES | Salt Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-UES.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UES | Salt Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UES.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | UES | Salt Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-UES.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | UET | Heat Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-UET.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UET | Heat Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UET.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | UET | Heat Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-UET.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | UVEL | Velocity in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-UVEL.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UVEL | Velocity in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UVEL.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | UVEL | Velocity in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-UVEL.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | VNS | Salt Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-VNS.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VNS | Salt Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VNS.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | VNS | Salt Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-VNS.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | VNT | Heat Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-VNT.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VNT | Heat Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VNT.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | VNT | Heat Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-VNT.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | VVEL | Velocity in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-VVEL.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VVEL | Velocity in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VVEL.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | VVEL | Velocity in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-VVEL.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | WTS | Salt Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-WTS.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WTS | Salt Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WTS.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | WTS | Salt Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-WTS.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | WTT | Heat Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-WTT.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WTT | Heat Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WTT.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | WTT | Heat Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-WTT.zarr |
| ocn | 3D | monthly | 20C | 1920-01 | 2005-12 | WVEL | Vertical Velocity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-WVEL.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WVEL | Vertical Velocity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WVEL.zarr |
| ocn | 3D | monthly | CTRL | 0400-01 | 2200-12 | WVEL | Vertical Velocity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-WVEL.zarr |
Table: List of all 3D ocean datasets
Spatiotemporal filtering¶
If there were both regional and global data available (e.g., LENS and NA-CORDEX data for the same variable, both listed in same catalog), some type of coverage indicator (or columns for bounding box edges) could be listed.
Temporal extent in LENS is conveyed by the experiment (HIST, 20C, etc) but this is imprecise and requires external documentation. We have added start/end columns to the catalog, but intake-esm currently does not have built-in functionality to filter based on time.
[16]:
# We can do a simple search that exactly matches a temporal value
df = coll.search(dim="3D",component="ocn", end='2100-12').df
HTML(df.to_html(index=False))
[16]:
| component | dim | frequency | experiment | start | end | variable | long_name | path |
|---|---|---|---|---|---|---|---|---|
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | DIC | Dissolved Inorganic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-DIC.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | DOC | Dissolved Organic Carbon | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-DOC.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | O2 | Dissolved Oxygen | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-O2.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | PD | Potential Density Ref to Surface | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-PD.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | SALT | Salinity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-SALT.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | TEMP | Potential Temperature | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-TEMP.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UES | Salt Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UES.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UET | Heat Flux in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UET.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | UVEL | Velocity in grid-x direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UVEL.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VNS | Salt Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VNS.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VNT | Heat Flux in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VNT.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | VVEL | Velocity in grid-y direction | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VVEL.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WTS | Salt Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WTS.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WTT | Heat Flux across top face | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WTT.zarr |
| ocn | 3D | monthly | RCP85 | 2006-01 | 2100-12 | WVEL | Vertical Velocity | s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WVEL.zarr |
Table: *List of all 3D ocean datasets with an end month of 2100 December. Note that because intake-esm currently requires an exact match this search would not find daily data with an end date of 2100 Dec 31.
TO DO¶
Add substring search capability to intake-esm.
Add simple temporal search capability to intake-esm.
Consider adding spatial attributes (and appropriate intake-esm support) once we have some regional data available.
Have some tables in this demo only list only variable and long name rather than every single frequency and experiment.