Enhanced Intake-ESM Catalog Demo

This Jupyter Notebook compares the original Intake-ESM Catalog with an enhanced catalog that includes additional attributes. Both catalogs are an inventory of the NCAR Community Earth System Model (CESM) Large Ensemble (LENS) data hosted on AWS S3 (doi:10.26024/wt24-5j82).

Intake-esm Documentation

Set up environment

[1]:
import intake
import pandas as pd
import pprint
# Not used here:
###import numpy as np
###import xarray as xr
# Allow multiple lines per cell to be displayed without print (default is just last line)
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
# Enable more explicit control of DataFrame display (e.g., to omit annoying line numbers)
from IPython.display import HTML

Inspect original intake-esm catalog

[2]:
# Open original collection description file
cat_url_orig = 'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.json'
coll_orig = intake.open_esm_datastore(cat_url_orig)
[3]:
print(coll_orig.esmcol_data['description']) #Description of collection
print("Catalog file:", coll_orig.esmcol_data['catalog_file'])
print(coll_orig) # Summary of collection structure
This is an ESM collection for CESM1 Large Ensemble Zarr dataset publicly available on Amazon S3 (us-west-2 region)
Catalog file: https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.csv
<aws-cesm1-le catalog with 56 dataset(s) from 409 asset(s)>
[4]:
# Show expanded version of collection structure with details
uniques_orig = coll_orig.unique(columns=["component", "frequency", "experiment", "variable"])
pprint.pprint(uniques_orig, compact=True, indent=1, width=80)
{'component': {'count': 5, 'values': ['atm', 'ice_nh', 'ice_sh', 'lnd', 'ocn']},
 'experiment': {'count': 4, 'values': ['20C', 'CTRL', 'HIST', 'RCP85']},
 'frequency': {'count': 6,
               'values': ['daily', 'hourly6-1990-2005', 'hourly6-2026-2035',
                          'hourly6-2071-2080', 'monthly', 'static']},
 'variable': {'count': 65,
              'values': ['DIC', 'DOC', 'FLNS', 'FLNSC', 'FLUT', 'FSNO', 'FSNS',
                         'FSNSC', 'FSNTOA', 'FW', 'H2OSNO', 'HMXL', 'ICEFRAC',
                         'LHFLX', 'O2', 'PD', 'PRECC', 'PRECL', 'PRECSC',
                         'PRECSL', 'PSL', 'Q', 'QFLUX', 'QRUNOFF', 'QSW_HBL',
                         'QSW_HTP', 'RAIN', 'RESID_S', 'SALT', 'SFWF',
                         'SFWF_WRST', 'SHF', 'SHFLX', 'SHF_QSW', 'SNOW',
                         'SOILLIQ', 'SOILWATER_10CM', 'SSH', 'SST', 'T', 'TAUX',
                         'TAUX2', 'TAUY', 'TAUY2', 'TEMP', 'TMQ', 'TREFHT',
                         'TREFHTMN', 'TREFHTMX', 'TS', 'U', 'UES', 'UET',
                         'UVEL', 'VNS', 'VNT', 'VVEL', 'WTS', 'WTT', 'WVEL',
                         'Z3', 'aice', 'aice_d', 'hi', 'hi_d']}}
[5]:
# Show the first few lines of the catalog. There are as many lines as there are paths.
# The order is the same as that of the CSV catalog file listed in the JSON description file.
print("Catalog file:", coll_orig.esmcol_data['catalog_file'])
df = coll_orig.df
HTML(df.head(10).to_html(index=False))
Catalog file: https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.csv
[5]:
component frequency experiment variable path variable_long_name dim_per_tstep start end
atm daily 20C FLNS s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNS.zarr net longwave flux at surface 2.0 1920-01-01 12:00:00 2005-12-30 12:00:00
atm daily 20C FLNSC s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNSC.zarr clearsky net longwave flux at surface 2.0 1920-01-01 12:00:00 2005-12-30 12:00:00
atm daily 20C FLUT s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLUT.zarr upwelling longwave flux at top of model 2.0 1920-01-01 12:00:00 2005-12-30 12:00:00
atm daily 20C FSNS s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FSNS.zarr net solar flux at surface 2.0 1920-01-01 12:00:00 2005-12-30 12:00:00
atm daily 20C FSNSC s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FSNSC.zarr clearsky net solar flux at surface 2.0 1920-01-01 12:00:00 2005-12-30 12:00:00
atm daily 20C FSNTOA s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FSNTOA.zarr net solar flux at top of atmosphere 2.0 1920-01-01 12:00:00 2005-12-30 12:00:00
atm daily 20C ICEFRAC s3://ncar-cesm-lens/atm/daily/cesmLE-20C-ICEFRAC.zarr fraction of sfc area covered by sea-ice 2.0 1920-01-01 12:00:00 2005-12-30 12:00:00
atm daily 20C LHFLX s3://ncar-cesm-lens/atm/daily/cesmLE-20C-LHFLX.zarr surface latent heat flux 2.0 1920-01-01 12:00:00 2005-12-30 12:00:00
atm daily 20C PRECL s3://ncar-cesm-lens/atm/daily/cesmLE-20C-PRECL.zarr large-scale (stable) precipitation rate (liq + ice) 2.0 1920-01-01 12:00:00 2005-12-30 12:00:00
atm daily 20C PRECSC s3://ncar-cesm-lens/atm/daily/cesmLE-20C-PRECSC.zarr convective snow rate (water equivalent) 2.0 1920-01-01 12:00:00 2005-12-30 12:00:00

Table: First few lines of the original Intake-ESM Catalog showing the model component, the temporal frequency, the experiment, the abbreviated variable name, and the AWS S3 path for each Zarr store.

Finding Data

If you happen to know the meaning of the variable names, you can find what data are available for that variable. For example:

[6]:
# Filter the catalog to find available data for one variable
df = coll_orig.search(variable='FLNS').df
HTML(df.to_html(index=False))
[6]:
component frequency experiment variable path variable_long_name dim_per_tstep start end
atm daily 20C FLNS s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNS.zarr net longwave flux at surface 2.0 1920-01-01 12:00:00 2005-12-30 12:00:00
atm daily CTRL FLNS s3://ncar-cesm-lens/atm/daily/cesmLE-CTRL-FLNS.zarr net longwave flux at surface 2.0 0402-01-01 12:00:00 2200-12-31 12:00:00
atm daily HIST FLNS s3://ncar-cesm-lens/atm/daily/cesmLE-HIST-FLNS.zarr net longwave flux at surface 1.0 1850-01-01 00:00:00 1919-12-31 12:00:00
atm daily RCP85 FLNS s3://ncar-cesm-lens/atm/daily/cesmLE-RCP85-FLNS.zarr net longwave flux at surface 2.0 2006-01-01 12:00:00 2100-12-31 12:00:00
atm hourly6-1990-2005 20C FLNS s3://ncar-cesm-lens/atm/hourly6-1990-2005/cesmLE-20C-FLNS.zarr net longwave flux at surface 2.0 1990-01-01 00:00:00 2006-01-01 00:00:00
atm hourly6-2026-2035 RCP85 FLNS s3://ncar-cesm-lens/atm/hourly6-2026-2035/cesmLE-RCP85-FLNS.zarr net longwave flux at surface 2.0 2026-01-01 00:00:00 2036-01-01 00:00:00
atm hourly6-2071-2080 RCP85 FLNS s3://ncar-cesm-lens/atm/hourly6-2071-2080/cesmLE-RCP85-FLNS.zarr net longwave flux at surface 2.0 2071-01-01 00:00:00 2081-01-01 00:00:00
atm monthly 20C FLNS s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-FLNS.zarr net longwave flux at surface 2.0 1920-01-16 12:00:00 2005-12-16 12:00:00
atm monthly CTRL FLNS s3://ncar-cesm-lens/atm/monthly/cesmLE-CTRL-FLNS.zarr net longwave flux at surface 2.0 0400-01-16 12:00:00 2200-12-16 12:00:00
atm monthly HIST FLNS s3://ncar-cesm-lens/atm/monthly/cesmLE-HIST-FLNS.zarr net longwave flux at surface 1.0 1850-01-16 12:00:00 1919-12-16 12:00:00
atm monthly RCP85 FLNS s3://ncar-cesm-lens/atm/monthly/cesmLE-RCP85-FLNS.zarr net longwave flux at surface 2.0 2006-01-16 12:00:00 2100-12-16 12:00:00

Table: All available Zarr stores for the “FLNS” data.

[7]:
# Narrow the filter to specific frequency and expiriment
df = coll_orig.search(variable='FLNS', frequency='daily', experiment='RCP85').df
HTML(df.to_html(index=False))
[7]:
component frequency experiment variable path variable_long_name dim_per_tstep start end
atm daily RCP85 FLNS s3://ncar-cesm-lens/atm/daily/cesmLE-RCP85-FLNS.zarr net longwave flux at surface 2.0 2006-01-01 12:00:00 2100-12-31 12:00:00

Table: The single Zarr store for daily “FLNS” data from “RCP85” experiment.

The Problem

Do all potential users know that “FLNS” is a CESM-specific abbreviation for “Net longwave flux at surface”? How would a novice user find out, other than by finding separate documentation, or by opening a Zarr store in the hopes that the long name might be recorded there? How do we address the fact that every climate model code seems to have a different, non-standard name for all the variables, thus making multi-source research needlessly difficult?

Enhanced Intake-ESM Catalog

By adding additional columns to the Intake-ESM catalog, we should be able to improve semantic interoperability and provide potentially useful information to the users.

[8]:
# Open enhanced collection description file
cat_url = 'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le-enhanced.json'
coll = intake.open_esm_datastore(cat_url)
coll

aws-cesm1-le catalog with 27 dataset(s) from 365 asset(s):

unique
component 5
dim 2
frequency 5
experiment 6
start 10
end 11
variable 75
long_name 75
path 365
[9]:
print(coll.esmcol_data['description']) # Description of collection
print("Catalog file:", coll.esmcol_data['catalog_file'])
print(coll) # Summary of collection structure
This is an inventory of the Community Earth System Model (CESM) Large Ensemble (LENS) dataset in Zarr format publicly available on Amazon S3 (https://doi.org/10.26024/wt24-5j82)
Catalog file: https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le-enhanced.csv
<aws-cesm1-le catalog with 27 dataset(s) from 365 asset(s)>

Long names

In the summary above, note the addition of additional elements: long_name, start, end, and dim.

[10]:
# Show the first few lines of the enhanced catalog.
print("Catalog file:", coll.esmcol_data['catalog_file'])
HTML(coll.df.head(10).to_html(index=False))
Catalog file: https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le-enhanced.csv
[10]:
component dim frequency experiment start end variable long_name path
atm 2D monthly HIST 1850-01 1919-12 FLNS Net longwave flux at surface s3://ncar-cesm-lens/atm/monthly/cesmLE-HIST-FLNS.zarr
atm 2D monthly 20C 1920-01 2005-12 FLNS Net longwave flux at surface s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-FLNS.zarr
atm 2D daily 20C 1920-01-01 2005-12-31 FLNS Net longwave flux at surface s3://ncar-cesm-lens/atm/daily/cesmLE-20C-FLNS.zarr
atm 2D monthly RCP85 2006-01 2100-12 FLNS Net longwave flux at surface s3://ncar-cesm-lens/atm/monthly/cesmLE-RCP85-FLNS.zarr
atm 2D daily RCP85 2006-01-01 2100-12-31 FLNS Net longwave flux at surface s3://ncar-cesm-lens/atm/daily/cesmLE-RCP85-FLNS.zarr
atm 2D monthly CTRL 0400-01 2200-12 FLNS Net longwave flux at surface s3://ncar-cesm-lens/atm/monthly/cesmLE-CTRL-FLNS.zarr
atm 2D monthly CTRL_AMIP 0001-01 2600-12 FLNS Net longwave flux at surface s3://ncar-cesm-lens/atm/monthly/cesmLE-CTRL_AMIP-FLNS.zarr
atm 2D monthly CTRL_SLAB_OCN 0001-01 1000-12 FLNS Net longwave flux at surface s3://ncar-cesm-lens/atm/monthly/cesmLE-CTRL_SLAB_OCN-FLNS.zarr
atm 2D monthly HIST 1850-01 1919-12 FLNSC Clearsky net longwave flux at surface s3://ncar-cesm-lens/atm/monthly/cesmLE-HIST-FLNSC.zarr
atm 2D monthly 20C 1920-01 2005-12 FLNSC Clearsky net longwave flux at surface s3://ncar-cesm-lens/atm/monthly/cesmLE-20C-FLNSC.zarr

Table: First few lines of the enhanced catalog, listing of the same information as the original catalog as well as the long name of each variable and an indication of whether each variable is 2D or 3D.

ISSUE: The long names are **not* CF Standard Names, but rather are those documented at http://www.cgd.ucar.edu/ccr/strandwg/CESM-CAM5-BGC_LENS_fields.html. For interoperability, the long_name column should be replaced by a cf_name column and possibly an attribute column to disambiguate if needed.*

[11]:
# List all available variables by Long Name, sorted alphabetically
uniques = coll.unique(columns=['long_name'])
nameList = sorted(uniques['long_name']['values'])
print(*nameList, sep='\n') #note *list to unpack each item for print function
Clearsky net longwave flux at surface
Clearsky net solar flux at surface
Convective precipitation rate (liq + ice)
Convective snow rate (water equivalent)
Dissolved Inorganic Carbon
Dissolved Organic Carbon
Dissolved Oxygen
Fraction of sfc area covered by sea-ice
Free-Surface Residual Heat Flux
Free-Surface Residual Salt Flux
Freshwater Flux
Geopotential Height (above sea level)
Geopotential Z at 500 mbar pressure surface
Heat Flux across top face
Heat Flux in grid-x direction
Heat Flux in grid-y direction
Horizontal total wind speed average at the surface
Internal Ocean Heat Flux Due to Ice Formation
Large-scale (stable) precipitation rate (liq + ice)
Large-scale (stable) snow rate (water equivalent)
Lowest model level zonal wind
Maximum (convective and large-scale) precipitation rate (liq+ice)
Maximum reference height temperature over output period
Meridional wind
Minimum reference height temperature over output period
Mixed-Layer Depth
Net longwave flux at surface
Net solar flux at surface
Net solar flux at top of atmosphere
Potential Density Ref to Surface
Potential Temperature
Reference height humidity
Reference height temperature
Salinity
Salt Flux across top face
Salt Flux in grid-x direction
Salt Flux in grid-y direction
Sea Surface Height
Sea level pressure
Solar Short-Wave Heat Flux
Solar Short-Wave Heat Flux in boundary layer
Solar Short-Wave Heat Flux in top layer
Specific Humidity at 850 mbar pressure surface
Specific humidity
Surface latent heat flux
Surface pressure
Surface sensible heat flux
Surface temperature (radiative)
Temperature
Total (convective and large-scale) precipitation rate (liq + ice)
Total (vertically integrated) precipitable water
Total Surface Heat Flux including short-wave
Upwelling longwave flux at top of model
Velocity in grid-x direction
Velocity in grid-y direction
Vertical Velocity
Virtual Salt Flux due to weak restoring
Virtual Salt Flux in FW Flux formulation
Wind stress (squared) in grid-x direction
Wind stress (squared) in grid-y direction
Wind stress in grid-x direction
Wind stress in grid-y direction
Zonal wind
atmospheric rain
atmospheric snow
fraction of ground covered by snow
grid cellmean ice thickness
grid cellmean ice thickness (daily)
grid cellmean ice thickness (monthly)
ice area, aggregate (daily)
ice area, aggregate (monthly)
snow depth (liquid water)
soil liquid water (vegetated landunits only)
soil liquid water + ice in top 10cm of soil (veg landunits only)
total liquid runoff (does not include QSNWCPICE)
[12]:
# Show all available data for a specific variable based on long name
myName = 'Salinity'
HTML(coll.search(long_name=myName).df.to_html(index=False))
[12]:
component dim frequency experiment start end variable long_name path
ocn 3D monthly 20C 1920-01 2005-12 SALT Salinity s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-SALT.zarr
ocn 3D monthly RCP85 2006-01 2100-12 SALT Salinity s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-SALT.zarr
ocn 3D monthly CTRL 0400-01 2200-12 SALT Salinity s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-SALT.zarr

Table: All available data in this catalog for selected variable

Substring matches

ISSUE: The current version of intake-esm search() function requires an exact full-string case-sensitive match of the long_name. (This has been reported as an issue at https://github.com/NCAR/cesm-lens-aws/issues/48)

Demonstrate a work-around: find all variables with a particular substring in the long name

[13]:
myTerm = 'Wind'
myTerm = myTerm.lower() #search regardless of case
partials = [name for name in nameList if myTerm in name.lower()]
print(f"All datasets with name containing {myTerm}:")
print(*partials, sep='\n')
All datasets with name containing wind:
Horizontal total wind speed average at the surface
Lowest model level zonal wind
Meridional wind
Wind stress (squared) in grid-x direction
Wind stress (squared) in grid-y direction
Wind stress in grid-x direction
Wind stress in grid-y direction
Zonal wind
[14]:
# Display full table for each match (could be lengthy if many matches)
for name in partials:
    df = coll.search(long_name=name).df[['component', 'dim', 'experiment', 'variable', 'long_name']]
    HTML(df.to_html(index=False))
    ###df.head(1) #show only first entry in each group for compactness
    # Note: It is also possible to hide column(s) instead of specifying desired columns
    ###coll.search(long_name=name).df.drop(columns=['path'])
[14]:
component dim experiment variable long_name
atm 2D 20C WSPDSRFAV Horizontal total wind speed average at the surface
atm 2D RCP85 WSPDSRFAV Horizontal total wind speed average at the surface
[14]:
component dim experiment variable long_name
atm 2D 20C UBOT Lowest model level zonal wind
atm 2D RCP85 UBOT Lowest model level zonal wind
[14]:
component dim experiment variable long_name
atm 3D HIST V Meridional wind
atm 3D 20C V Meridional wind
atm 3D 20C V Meridional wind
atm 3D RCP85 V Meridional wind
atm 3D RCP85 V Meridional wind
atm 3D RCP85 V Meridional wind
atm 3D CTRL V Meridional wind
atm 3D CTRL_AMIP V Meridional wind
atm 3D CTRL_SLAB_OCN V Meridional wind
[14]:
component dim experiment variable long_name
ocn 2D 20C TAUX2 Wind stress (squared) in grid-x direction
ocn 2D RCP85 TAUX2 Wind stress (squared) in grid-x direction
ocn 2D CTRL TAUX2 Wind stress (squared) in grid-x direction
[14]:
component dim experiment variable long_name
ocn 2D 20C TAUY2 Wind stress (squared) in grid-y direction
ocn 2D RCP85 TAUY2 Wind stress (squared) in grid-y direction
ocn 2D CTRL TAUY2 Wind stress (squared) in grid-y direction
[14]:
component dim experiment variable long_name
ocn 2D 20C TAUX Wind stress in grid-x direction
ocn 2D RCP85 TAUX Wind stress in grid-x direction
ocn 2D CTRL TAUX Wind stress in grid-x direction
[14]:
component dim experiment variable long_name
ocn 2D 20C TAUY Wind stress in grid-y direction
ocn 2D RCP85 TAUY Wind stress in grid-y direction
ocn 2D CTRL TAUY Wind stress in grid-y direction
[14]:
component dim experiment variable long_name
atm 3D HIST U Zonal wind
atm 3D 20C U Zonal wind
atm 3D 20C U Zonal wind
atm 3D RCP85 U Zonal wind
atm 3D RCP85 U Zonal wind
atm 3D RCP85 U Zonal wind
atm 3D CTRL U Zonal wind
atm 3D CTRL_AMIP U Zonal wind
atm 3D CTRL_SLAB_OCN U Zonal wind

Table(s): Information about all matching datasets

ISSUE: The case-insensitive substring matching is not integrated into intake-esm, so it is not clear whether resulting search results can be passed directly to Xarray to read data.

Other attributes

Other columns in the enhanced catalog may be useful. For example, the dimensionality column enables us to list all data from the ocean component that is 3D.

[15]:
df = coll.search(dim="3D",component="ocn").df
HTML(df.to_html(index=False))
[15]:
component dim frequency experiment start end variable long_name path
ocn 3D monthly 20C 1920-01 2005-12 DIC Dissolved Inorganic Carbon s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-DIC.zarr
ocn 3D monthly RCP85 2006-01 2100-12 DIC Dissolved Inorganic Carbon s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-DIC.zarr
ocn 3D monthly CTRL 0400-01 2200-12 DIC Dissolved Inorganic Carbon s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-DIC.zarr
ocn 3D monthly 20C 1920-01 2005-12 DOC Dissolved Organic Carbon s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-DOC.zarr
ocn 3D monthly RCP85 2006-01 2100-12 DOC Dissolved Organic Carbon s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-DOC.zarr
ocn 3D monthly CTRL 0400-01 2200-12 DOC Dissolved Organic Carbon s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-DOC.zarr
ocn 3D monthly 20C 1920-01 2005-12 O2 Dissolved Oxygen s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-O2.zarr
ocn 3D monthly RCP85 2006-01 2100-12 O2 Dissolved Oxygen s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-O2.zarr
ocn 3D monthly CTRL 0400-01 2200-12 O2 Dissolved Oxygen s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-O2.zarr
ocn 3D monthly 20C 1920-01 2005-12 PD Potential Density Ref to Surface s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-PD.zarr
ocn 3D monthly RCP85 2006-01 2100-12 PD Potential Density Ref to Surface s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-PD.zarr
ocn 3D monthly CTRL 0400-01 2200-12 PD Potential Density Ref to Surface s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-PD.zarr
ocn 3D monthly 20C 1920-01 2005-12 SALT Salinity s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-SALT.zarr
ocn 3D monthly RCP85 2006-01 2100-12 SALT Salinity s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-SALT.zarr
ocn 3D monthly CTRL 0400-01 2200-12 SALT Salinity s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-SALT.zarr
ocn 3D monthly 20C 1920-01 2005-12 TEMP Potential Temperature s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-TEMP.zarr
ocn 3D monthly RCP85 2006-01 2100-12 TEMP Potential Temperature s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-TEMP.zarr
ocn 3D monthly CTRL 0400-01 2200-12 TEMP Potential Temperature s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-TEMP.zarr
ocn 3D monthly 20C 1920-01 2005-12 UES Salt Flux in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-UES.zarr
ocn 3D monthly RCP85 2006-01 2100-12 UES Salt Flux in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UES.zarr
ocn 3D monthly CTRL 0400-01 2200-12 UES Salt Flux in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-UES.zarr
ocn 3D monthly 20C 1920-01 2005-12 UET Heat Flux in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-UET.zarr
ocn 3D monthly RCP85 2006-01 2100-12 UET Heat Flux in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UET.zarr
ocn 3D monthly CTRL 0400-01 2200-12 UET Heat Flux in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-UET.zarr
ocn 3D monthly 20C 1920-01 2005-12 UVEL Velocity in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-UVEL.zarr
ocn 3D monthly RCP85 2006-01 2100-12 UVEL Velocity in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UVEL.zarr
ocn 3D monthly CTRL 0400-01 2200-12 UVEL Velocity in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-UVEL.zarr
ocn 3D monthly 20C 1920-01 2005-12 VNS Salt Flux in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-VNS.zarr
ocn 3D monthly RCP85 2006-01 2100-12 VNS Salt Flux in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VNS.zarr
ocn 3D monthly CTRL 0400-01 2200-12 VNS Salt Flux in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-VNS.zarr
ocn 3D monthly 20C 1920-01 2005-12 VNT Heat Flux in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-VNT.zarr
ocn 3D monthly RCP85 2006-01 2100-12 VNT Heat Flux in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VNT.zarr
ocn 3D monthly CTRL 0400-01 2200-12 VNT Heat Flux in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-VNT.zarr
ocn 3D monthly 20C 1920-01 2005-12 VVEL Velocity in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-VVEL.zarr
ocn 3D monthly RCP85 2006-01 2100-12 VVEL Velocity in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VVEL.zarr
ocn 3D monthly CTRL 0400-01 2200-12 VVEL Velocity in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-VVEL.zarr
ocn 3D monthly 20C 1920-01 2005-12 WTS Salt Flux across top face s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-WTS.zarr
ocn 3D monthly RCP85 2006-01 2100-12 WTS Salt Flux across top face s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WTS.zarr
ocn 3D monthly CTRL 0400-01 2200-12 WTS Salt Flux across top face s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-WTS.zarr
ocn 3D monthly 20C 1920-01 2005-12 WTT Heat Flux across top face s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-WTT.zarr
ocn 3D monthly RCP85 2006-01 2100-12 WTT Heat Flux across top face s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WTT.zarr
ocn 3D monthly CTRL 0400-01 2200-12 WTT Heat Flux across top face s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-WTT.zarr
ocn 3D monthly 20C 1920-01 2005-12 WVEL Vertical Velocity s3://ncar-cesm-lens/ocn/monthly/cesmLE-20C-WVEL.zarr
ocn 3D monthly RCP85 2006-01 2100-12 WVEL Vertical Velocity s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WVEL.zarr
ocn 3D monthly CTRL 0400-01 2200-12 WVEL Vertical Velocity s3://ncar-cesm-lens/ocn/monthly/cesmLE-CTRL-WVEL.zarr

Table: List of all 3D ocean datasets

Spatiotemporal filtering

  • If there were both regional and global data available (e.g., LENS and NA-CORDEX data for the same variable, both listed in same catalog), some type of coverage indicator (or columns for bounding box edges) could be listed.

  • Temporal extent in LENS is conveyed by the experiment (HIST, 20C, etc) but this is imprecise and requires external documentation. We have added start/end columns to the catalog, but intake-esm currently does not have built-in functionality to filter based on time.

[16]:
# We can do a simple search that exactly matches a temporal value
df = coll.search(dim="3D",component="ocn", end='2100-12').df
HTML(df.to_html(index=False))
[16]:
component dim frequency experiment start end variable long_name path
ocn 3D monthly RCP85 2006-01 2100-12 DIC Dissolved Inorganic Carbon s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-DIC.zarr
ocn 3D monthly RCP85 2006-01 2100-12 DOC Dissolved Organic Carbon s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-DOC.zarr
ocn 3D monthly RCP85 2006-01 2100-12 O2 Dissolved Oxygen s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-O2.zarr
ocn 3D monthly RCP85 2006-01 2100-12 PD Potential Density Ref to Surface s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-PD.zarr
ocn 3D monthly RCP85 2006-01 2100-12 SALT Salinity s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-SALT.zarr
ocn 3D monthly RCP85 2006-01 2100-12 TEMP Potential Temperature s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-TEMP.zarr
ocn 3D monthly RCP85 2006-01 2100-12 UES Salt Flux in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UES.zarr
ocn 3D monthly RCP85 2006-01 2100-12 UET Heat Flux in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UET.zarr
ocn 3D monthly RCP85 2006-01 2100-12 UVEL Velocity in grid-x direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-UVEL.zarr
ocn 3D monthly RCP85 2006-01 2100-12 VNS Salt Flux in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VNS.zarr
ocn 3D monthly RCP85 2006-01 2100-12 VNT Heat Flux in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VNT.zarr
ocn 3D monthly RCP85 2006-01 2100-12 VVEL Velocity in grid-y direction s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-VVEL.zarr
ocn 3D monthly RCP85 2006-01 2100-12 WTS Salt Flux across top face s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WTS.zarr
ocn 3D monthly RCP85 2006-01 2100-12 WTT Heat Flux across top face s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WTT.zarr
ocn 3D monthly RCP85 2006-01 2100-12 WVEL Vertical Velocity s3://ncar-cesm-lens/ocn/monthly/cesmLE-RCP85-WVEL.zarr

Table: *List of all 3D ocean datasets with an end month of 2100 December. Note that because intake-esm currently requires an exact match this search would not find daily data with an end date of 2100 Dec 31.

TO DO

  • Add substring search capability to intake-esm.

  • Add simple temporal search capability to intake-esm.

  • Consider adding spatial attributes (and appropriate intake-esm support) once we have some regional data available.

  • Have some tables in this demo only list only variable and long name rather than every single frequency and experiment.