ERDDAP Access

Find OOI and IOOS salinity data in a specified time interval and bounding box using the ERDDAP advanced search and data access RESTful APIs

Note: This Jupyter notebook originated froman ERDDAPY notebook from the IOOS gallery

ERDDAP is a data server that gives you a simple, consistent way to download data in the format and the spatial and temporal coverage that you want. ERDDAP is a web application with an interface for people to use. It is also a RESTful web service that allows data access directly from any computer program (e.g. Matlab, R, or webpages).”

This notebook uses the python client erddapy to help construct the RESTful URLs and translate the responses into Pandas and Xarray objects.

A typical ERDDAP RESTful URL looks like:

https://data.ioos.us/gliders/erddap/tabledap/whoi_406-20160902T1700.mat?depth,latitude,longitude,salinity,temperature,time&time>=2016-07-10T00:00:00Z&time<=2017-02-10T00:00:00Z &latitude>=38.0&latitude<=41.0&longitude>=-72.0&longitude<=-69.0

Let’s break it down to smaller parts:

  • server: https://data.ioos.us/gliders/erddap/

  • protocol: tabledap

  • dataset_id: whoi_406-20160902T1700

  • response: .mat

  • variables: depth,latitude,longitude,temperature,time

  • constraints:

    • time>=2016-07-10T00:00:00Z

    • time<=2017-02-10T00:00:00Z

    • latitude>=38.0

    • latitude<=41.0

    • longitude>=-72.0

    • longitude<=-69.0

[1]:
import pandas as pd
from erddapy import ERDDAP
from erddapy.utilities import urlopen
import hvplot.xarray
import hvplot.pandas
import holoviews as hv
import numpy as np

1. Search ERDDAP “catalog”

[2]:
# ERDDAP for OOI
server = 'http://erddap.dataexplorer.oceanobservatories.org/erddap'
protocol = 'tabledap'
[3]:
ooi = ERDDAP(server=server, protocol=protocol)

A search for everything looks like this. The only effective filtering parameters being passed are protocol=tabledap and response=csv.

[4]:
df = pd.read_csv(urlopen(ooi.get_search_url(response='csv', search_for='all')))
len(df)
[4]:
540

Now we’ll refine our search by adding temporal, bounding box and variable constraints.

[5]:
min_time = '2018-07-01T00:00:00Z'
max_time = '2018-07-15T00:00:00Z'
min_lon, max_lon = -127, -122
min_lat, max_lat = 44, 48
standard_name = 'sea_water_practical_salinity'
cdm_data_type = 'timeseries'

kw = {
    'standard_name': standard_name,
    'min_lon': min_lon,'max_lon': max_lon,'min_lat': min_lat,'max_lat': max_lat,
    'min_time': min_time,'max_time': max_time, 'cdm_data_type':cdm_data_type
}
[6]:
search_url = ooi.get_search_url(response='csv', **kw)
search_df = pd.read_csv(urlopen(search_url))
search_df = search_df[['Institution', 'Dataset ID','tabledap']]
search_df
[6]:
Institution Dataset ID tabledap
0 Ocean Observatories Initiative (OOI) ooi-ce01issm-rid16-02-flortd000 http://erddap.dataexplorer.oceanobservatories....
1 Ocean Observatories Initiative (OOI) ooi-ce01issm-rid16-03-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
2 Ocean Observatories Initiative (OOI) ooi-ce01issm-rid16-03-dostad000 http://erddap.dataexplorer.oceanobservatories....
3 Ocean Observatories Initiative (OOI) ooi-ce01issm-rid16-07-nutnrb000 http://erddap.dataexplorer.oceanobservatories....
4 Ocean Observatories Initiative (OOI) ooi-ce01issm-rid16-06-phsend000 http://erddap.dataexplorer.oceanobservatories....
... ... ... ...
58 Ocean Observatories Initiative (OOI) ooi-ce07shsm-sbd11-06-metbka000 http://erddap.dataexplorer.oceanobservatories....
59 Ocean Observatories Initiative (OOI) ooi-rs01slbs-lj01a-12-ctdpfb101 http://erddap.dataexplorer.oceanobservatories....
60 Ocean Observatories Initiative (OOI) ooi-rs01sbps-pc01a-4c-flordd103 http://erddap.dataexplorer.oceanobservatories....
61 Ocean Observatories Initiative (OOI) ooi-rs01sbps-pc01a-4a-ctdpfa103 http://erddap.dataexplorer.oceanobservatories....
62 Ocean Observatories Initiative (OOI) ooi-rs01sbps-pc01a-4b-phsena102 http://erddap.dataexplorer.oceanobservatories....

63 rows × 3 columns

2. Read data from one dataset, manually

Let’s inspect a specific dataset_id.

[7]:
dataset_id = 'ooi-ce01issm-sbd17-06-ctdbpc000'

Construct the ERDDAP URL to get the data

[8]:
ooi.dataset_id = dataset_id
ooi.constraints = {'time>=': min_time,'time<=': max_time}
ooi.response='csv'
ooi.variables = [ 'time', ooi.get_var_by_attr(dataset_id=dataset_id, standard_name=standard_name)[0]]
print(ooi.get_download_url())
http://erddap.dataexplorer.oceanobservatories.org/erddap/tabledap/ooi-ce01issm-sbd17-06-ctdbpc000.csv?time,sea_water_practical_salinity&time>=1530403200.0&time<=1531612800.0

Read the data into Xarray

[9]:
ds = ooi.to_xarray(decode_times=True)
#ds = ds.swap_dims({'row':'time'})
#[ds[var].plot() for var in ds.data_vars];
[10]:
ds.sea_water_practical_salinity.hvplot(grid=True)
[10]:

3. Read data from all datasets, automatically

Let’s narrow this down by only taking the “CTDBP” data

[11]:
ctdbp = search_df[search_df['Dataset ID'].str.contains("ctdbp")].reset_index()
print(len(ctdbp))
ctdbp
14
[11]:
index Institution Dataset ID tabledap
0 1 Ocean Observatories Initiative (OOI) ooi-ce01issm-rid16-03-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
1 5 Ocean Observatories Initiative (OOI) ooi-ce01issm-mfd37-03-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
2 9 Ocean Observatories Initiative (OOI) ooi-ce01issm-sbd17-06-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
3 10 Ocean Observatories Initiative (OOI) ooi-ce04osbp-lj01c-06-ctdbpo108 http://erddap.dataexplorer.oceanobservatories....
4 16 Ocean Observatories Initiative (OOI) ooi-ce04ossm-rid27-03-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
5 22 Ocean Observatories Initiative (OOI) ooi-ce02shbp-lj01d-06-ctdbpn106 http://erddap.dataexplorer.oceanobservatories....
6 25 Ocean Observatories Initiative (OOI) ooi-ce02shsm-rid27-03-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
7 32 Ocean Observatories Initiative (OOI) ooi-ce06issm-rid16-03-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
8 36 Ocean Observatories Initiative (OOI) ooi-ce06issm-mfd37-03-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
9 40 Ocean Observatories Initiative (OOI) ooi-ce06issm-sbd17-06-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
10 42 Ocean Observatories Initiative (OOI) ooi-ce09ossm-rid27-03-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
11 46 Ocean Observatories Initiative (OOI) ooi-ce09ossm-mfd37-03-ctdbpe000 http://erddap.dataexplorer.oceanobservatories....
12 51 Ocean Observatories Initiative (OOI) ooi-ce07shsm-rid27-03-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
13 55 Ocean Observatories Initiative (OOI) ooi-ce07shsm-mfd37-03-ctdbpc000 http://erddap.dataexplorer.oceanobservatories....
[12]:
def alllonlat(e, cdm_data_type, min_time, max_time):
    url='{}/tabledap/allDatasets.csv?datasetID%2CminLongitude%2CminLatitude&cdm_data_type=%22{}%22&minTime%3C={}&maxTime%3E={}'.format(e.server,
                        cdm_data_type,max_time,min_time)
    print(url)
    df = pd.read_csv(urlopen(url), skiprows=[1])
    return df
[13]:
dfll = alllonlat(ooi, 'TimeSeries', min_time, max_time)
# extract lon,lat values of matching datasets from allDatasets dataframe
dfr = dfll[dfll['datasetID'].isin(search_df['Dataset ID'])]
http://erddap.dataexplorer.oceanobservatories.org/erddap/tabledap/allDatasets.csv?datasetID%2CminLongitude%2CminLatitude&cdm_data_type=%22TimeSeries%22&minTime%3C=2018-07-15T00:00:00Z&maxTime%3E=2018-07-01T00:00:00Z
[14]:
dfr
[14]:
datasetID minLongitude minLatitude
0 ooi-ce01issm-rid16-02-flortd000 -124.09543 44.65983
1 ooi-ce01issm-rid16-03-ctdbpc000 -124.09543 44.65983
2 ooi-ce01issm-rid16-03-dostad000 -124.09543 44.65983
3 ooi-ce01issm-rid16-07-nutnrb000 -124.09543 44.65983
5 ooi-ce01issm-rid16-06-phsend000 -124.09543 44.65983
... ... ... ...
99 ooi-ce07shsm-sbd11-06-metbka000 -124.56631 46.98408
324 ooi-rs01slbs-lj01a-12-ctdpfb101 -125.38981 44.51524
328 ooi-rs01sbps-pc01a-4c-flordd103 -125.38966 44.52897
329 ooi-rs01sbps-pc01a-4a-ctdpfa103 -125.38966 44.52897
330 ooi-rs01sbps-pc01a-4b-phsena102 -125.38966 44.52897

63 rows × 3 columns

[15]:
dfr.hvplot.points(x='minLongitude', y='minLatitude', geo=True,
                  tiles='OSM', color='red', alpha=0.2, hover_cols=['datasetID'],
                  xlim=(min_lon, max_lon), padding=20, title='OOI Stations with Salinity')
[15]:

Loop through datasets extracting data

[16]:
df_list = []
hv_list = []
for dataset_id in ctdbp['Dataset ID'].values:
    ooi.dataset_id = dataset_id
    ooi.variables = [
        'time',
        ooi.get_var_by_attr(dataset_id=dataset_id,  standard_name=standard_name)[0]
    ]
    try:
        ds = ooi.to_xarray(decode_times=True)
        df_list.append(ds)
        print(dataset_id)
        hv_list.append(ds[ooi.variables[-1]].hvplot(label=dataset_id))
    except:
        pass
ooi-ce01issm-rid16-03-ctdbpc000
ooi-ce01issm-mfd37-03-ctdbpc000
ooi-ce01issm-sbd17-06-ctdbpc000
ooi-ce04osbp-lj01c-06-ctdbpo108
ooi-ce04ossm-rid27-03-ctdbpc000
ooi-ce02shbp-lj01d-06-ctdbpn106
ooi-ce02shsm-rid27-03-ctdbpc000
ooi-ce06issm-rid16-03-ctdbpc000
ooi-ce06issm-mfd37-03-ctdbpc000
ooi-ce06issm-sbd17-06-ctdbpc000
ooi-ce07shsm-rid27-03-ctdbpc000
ooi-ce07shsm-mfd37-03-ctdbpc000
[17]:
hv.Overlay(hv_list).opts(legend_position='right', width=900, legend_offset=(0,0))
[17]:

Find all the IOOS salinity data

Let’s do the same query for IOOS

[18]:
ioos = ERDDAP(server="http://erddap.sensors.ioos.us/erddap", protocol=protocol)
[19]:
search_url = ioos.get_search_url(response='csv', **kw)
search_df = pd.read_csv(urlopen(search_url))
search_df = search_df[['Institution', 'Dataset ID','tabledap']]
search_df
[19]:
Institution Dataset ID tabledap
0 Center for Coastal Margin Observation & Predic... org_cmop_46096 http://erddap.sensors.ioos.us/erddap/tabledap/...
1 Observing System Monitoring Center (OSMC) osmc_46097 http://erddap.sensors.ioos.us/erddap/tabledap/...
2 Observing System Monitoring Center (OSMC) osmc_46098 http://erddap.sensors.ioos.us/erddap/tabledap/...
3 Observing System Monitoring Center (OSMC) osmc_46099 http://erddap.sensors.ioos.us/erddap/tabledap/...
4 Observing System Monitoring Center (OSMC) osmc_46100 http://erddap.sensors.ioos.us/erddap/tabledap/...
5 NOAA National Data Buoy Center (NDBC) wmo_46116 http://erddap.sensors.ioos.us/erddap/tabledap/...
6 University of Washington edu_washington_46120 http://erddap.sensors.ioos.us/erddap/tabledap/...
7 University of Washington edu_washington_46121 http://erddap.sensors.ioos.us/erddap/tabledap/...
8 University of Washington edu_washington_46123 http://erddap.sensors.ioos.us/erddap/tabledap/...
9 University of Washington edu_washington_46124 http://erddap.sensors.ioos.us/erddap/tabledap/...
10 University of Washington edu_washington_46125 http://erddap.sensors.ioos.us/erddap/tabledap/...
11 Northwest Association of Networked Ocean Obser... nanoos_apl_chaba http://erddap.sensors.ioos.us/erddap/tabledap/...
12 Northwest Association of Networked Ocean Obser... nanoos_apl_nemo http://erddap.sensors.ioos.us/erddap/tabledap/...
13 USGS National Water Information System (NWIS) gov_usgs_waterdata_12113415 http://erddap.sensors.ioos.us/erddap/tabledap/...
14 NOAA Center for Operational Oceanographic Prod... noaa_nos_co_ops_9437540 http://erddap.sensors.ioos.us/erddap/tabledap/...
15 Northwest Association of Networked Ocean Obser... nanoos_kc_nsaj02 http://erddap.sensors.ioos.us/erddap/tabledap/...
16 Northwest Association of Networked Ocean Obser... nanoos_kc_ptwilliams http://erddap.sensors.ioos.us/erddap/tabledap/...
17 Northwest Association of Networked Ocean Obser... nanoos_kc_ycqmh01 http://erddap.sensors.ioos.us/erddap/tabledap/...
18 Northwest Association of Networked Ocean Obser... nanoos_psi_baycenter http://erddap.sensors.ioos.us/erddap/tabledap/...
19 Northwest Association of Networked Ocean Obser... nanoos_psi_nahcotta http://erddap.sensors.ioos.us/erddap/tabledap/...
20 Northwest Association of Networked Ocean Obser... nanoos_kc_seaqysi http://erddap.sensors.ioos.us/erddap/tabledap/...
21 Center for Coastal Margin Observation & Predic... org_cmop_sefo3 http://erddap.sensors.ioos.us/erddap/tabledap/...
22 Center for Coastal Margin Observation & Predic... org_cmop_seto3 http://erddap.sensors.ioos.us/erddap/tabledap/...
23 Northwest Association of Networked Ocean Obser... nanoos_taf_dabobbay http://erddap.sensors.ioos.us/erddap/tabledap/...
24 Northwest Association of Networked Ocean Obser... nanoos_wadoh_hoodcanal1 http://erddap.sensors.ioos.us/erddap/tabledap/...
25 Northwest Association of Networked Ocean Obser... nanoos_wcsh_whiskey1 http://erddap.sensors.ioos.us/erddap/tabledap/...
[20]:
ioos.constraints = {'time>=': min_time,'time<=': max_time}
ioos.response='csv'
[21]:
df_list = []
hv_list = []

for dataset_id in search_df['Dataset ID'].values:
    ioos.dataset_id = dataset_id
    ioos.variables = [
        'time',
        ioos.get_var_by_attr(dataset_id=dataset_id,  standard_name=standard_name)[0]
    ]
    try:
        ds = ioos.to_xarray(decode_times=True)
        df_list.append(ds)
        print(dataset_id)
        hv_list.append(ds[ioos.variables[-1]].hvplot(label=dataset_id))
    except:
        pass
org_cmop_46096
osmc_46097
osmc_46098
osmc_46099
osmc_46100
wmo_46116
edu_washington_46123
edu_washington_46124
edu_washington_46125
nanoos_apl_chaba
nanoos_apl_nemo
gov_usgs_waterdata_12113415
noaa_nos_co_ops_9437540
nanoos_kc_nsaj02
nanoos_kc_ptwilliams
nanoos_kc_ycqmh01
nanoos_psi_baycenter
nanoos_psi_nahcotta
nanoos_kc_seaqysi
org_cmop_sefo3
org_cmop_seto3
nanoos_taf_dabobbay
nanoos_wadoh_hoodcanal1
[22]:
hv.Overlay(hv_list).opts(legend_position='right', width=900, legend_offset=(0,0))
[22]:
[23]:
dfll = alllonlat(ioos, 'TimeSeries', min_time, max_time)
# extract lon,lat values of matching datasets from allDatasets dataframe
dfr = dfll[dfll['datasetID'].isin(search_df['Dataset ID'])]
http://erddap.sensors.ioos.us/erddap/tabledap/allDatasets.csv?datasetID%2CminLongitude%2CminLatitude&cdm_data_type=%22TimeSeries%22&minTime%3C=2018-07-15T00:00:00Z&maxTime%3E=2018-07-01T00:00:00Z
[24]:
dfr.hvplot.points(x='minLongitude', y='minLatitude', geo=True,
                  tiles='OSM', color='red', alpha=0.4,
                  xlim=(min_lon, max_lon), padding=20, title='IOOS Stations with Salinity')
[24]: