ERDDAP Access¶
Find OOI and IOOS salinity data in a specified time interval and bounding box using the ERDDAP advanced search and data access RESTful APIs
Note: This Jupyter notebook originated froman ERDDAPY notebook from the IOOS gallery
ERDDAP is a data server that gives you a simple, consistent way to download data in the format and the spatial and temporal coverage that you want. ERDDAP is a web application with an interface for people to use. It is also a RESTful web service that allows data access directly from any computer program (e.g. Matlab, R, or webpages).”
This notebook uses the python client erddapy to help construct the RESTful URLs and translate the responses into Pandas and Xarray objects.
A typical ERDDAP RESTful URL looks like:
Let’s break it down to smaller parts:
protocol: tabledap
dataset_id: whoi_406-20160902T1700
response: .mat
variables: depth,latitude,longitude,temperature,time
constraints:
time>=2016-07-10T00:00:00Z
time<=2017-02-10T00:00:00Z
latitude>=38.0
latitude<=41.0
longitude>=-72.0
longitude<=-69.0
[1]:
import pandas as pd
from erddapy import ERDDAP
from erddapy.utilities import urlopen
import hvplot.xarray
import hvplot.pandas
import holoviews as hv
import numpy as np
1. Search ERDDAP “catalog”¶
[2]:
# ERDDAP for OOI
server = 'http://erddap.dataexplorer.oceanobservatories.org/erddap'
protocol = 'tabledap'
[3]:
ooi = ERDDAP(server=server, protocol=protocol)
A search for everything looks like this. The only effective filtering parameters being passed are protocol=tabledap and response=csv.
[4]:
df = pd.read_csv(urlopen(ooi.get_search_url(response='csv', search_for='all')))
len(df)
[4]:
540
Now we’ll refine our search by adding temporal, bounding box and variable constraints.
[5]:
min_time = '2018-07-01T00:00:00Z'
max_time = '2018-07-15T00:00:00Z'
min_lon, max_lon = -127, -122
min_lat, max_lat = 44, 48
standard_name = 'sea_water_practical_salinity'
cdm_data_type = 'timeseries'
kw = {
'standard_name': standard_name,
'min_lon': min_lon,'max_lon': max_lon,'min_lat': min_lat,'max_lat': max_lat,
'min_time': min_time,'max_time': max_time, 'cdm_data_type':cdm_data_type
}
[6]:
search_url = ooi.get_search_url(response='csv', **kw)
search_df = pd.read_csv(urlopen(search_url))
search_df = search_df[['Institution', 'Dataset ID','tabledap']]
search_df
[6]:
| Institution | Dataset ID | tabledap | |
|---|---|---|---|
| 0 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-02-flortd000 | http://erddap.dataexplorer.oceanobservatories.... |
| 1 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 2 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-03-dostad000 | http://erddap.dataexplorer.oceanobservatories.... |
| 3 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-07-nutnrb000 | http://erddap.dataexplorer.oceanobservatories.... |
| 4 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-06-phsend000 | http://erddap.dataexplorer.oceanobservatories.... |
| ... | ... | ... | ... |
| 58 | Ocean Observatories Initiative (OOI) | ooi-ce07shsm-sbd11-06-metbka000 | http://erddap.dataexplorer.oceanobservatories.... |
| 59 | Ocean Observatories Initiative (OOI) | ooi-rs01slbs-lj01a-12-ctdpfb101 | http://erddap.dataexplorer.oceanobservatories.... |
| 60 | Ocean Observatories Initiative (OOI) | ooi-rs01sbps-pc01a-4c-flordd103 | http://erddap.dataexplorer.oceanobservatories.... |
| 61 | Ocean Observatories Initiative (OOI) | ooi-rs01sbps-pc01a-4a-ctdpfa103 | http://erddap.dataexplorer.oceanobservatories.... |
| 62 | Ocean Observatories Initiative (OOI) | ooi-rs01sbps-pc01a-4b-phsena102 | http://erddap.dataexplorer.oceanobservatories.... |
63 rows × 3 columns
2. Read data from one dataset, manually¶
Let’s inspect a specific dataset_id.
[7]:
dataset_id = 'ooi-ce01issm-sbd17-06-ctdbpc000'
Construct the ERDDAP URL to get the data
[8]:
ooi.dataset_id = dataset_id
ooi.constraints = {'time>=': min_time,'time<=': max_time}
ooi.response='csv'
ooi.variables = [ 'time', ooi.get_var_by_attr(dataset_id=dataset_id, standard_name=standard_name)[0]]
print(ooi.get_download_url())
http://erddap.dataexplorer.oceanobservatories.org/erddap/tabledap/ooi-ce01issm-sbd17-06-ctdbpc000.csv?time,sea_water_practical_salinity&time>=1530403200.0&time<=1531612800.0
Read the data into Xarray
[9]:
ds = ooi.to_xarray(decode_times=True)
#ds = ds.swap_dims({'row':'time'})
#[ds[var].plot() for var in ds.data_vars];
[10]:
ds.sea_water_practical_salinity.hvplot(grid=True)
[10]:
3. Read data from all datasets, automatically¶
Let’s narrow this down by only taking the “CTDBP” data
[11]:
ctdbp = search_df[search_df['Dataset ID'].str.contains("ctdbp")].reset_index()
print(len(ctdbp))
ctdbp
14
[11]:
| index | Institution | Dataset ID | tabledap | |
|---|---|---|---|---|
| 0 | 1 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-rid16-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 1 | 5 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-mfd37-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 2 | 9 | Ocean Observatories Initiative (OOI) | ooi-ce01issm-sbd17-06-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 3 | 10 | Ocean Observatories Initiative (OOI) | ooi-ce04osbp-lj01c-06-ctdbpo108 | http://erddap.dataexplorer.oceanobservatories.... |
| 4 | 16 | Ocean Observatories Initiative (OOI) | ooi-ce04ossm-rid27-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 5 | 22 | Ocean Observatories Initiative (OOI) | ooi-ce02shbp-lj01d-06-ctdbpn106 | http://erddap.dataexplorer.oceanobservatories.... |
| 6 | 25 | Ocean Observatories Initiative (OOI) | ooi-ce02shsm-rid27-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 7 | 32 | Ocean Observatories Initiative (OOI) | ooi-ce06issm-rid16-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 8 | 36 | Ocean Observatories Initiative (OOI) | ooi-ce06issm-mfd37-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 9 | 40 | Ocean Observatories Initiative (OOI) | ooi-ce06issm-sbd17-06-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 10 | 42 | Ocean Observatories Initiative (OOI) | ooi-ce09ossm-rid27-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 11 | 46 | Ocean Observatories Initiative (OOI) | ooi-ce09ossm-mfd37-03-ctdbpe000 | http://erddap.dataexplorer.oceanobservatories.... |
| 12 | 51 | Ocean Observatories Initiative (OOI) | ooi-ce07shsm-rid27-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
| 13 | 55 | Ocean Observatories Initiative (OOI) | ooi-ce07shsm-mfd37-03-ctdbpc000 | http://erddap.dataexplorer.oceanobservatories.... |
[12]:
def alllonlat(e, cdm_data_type, min_time, max_time):
url='{}/tabledap/allDatasets.csv?datasetID%2CminLongitude%2CminLatitude&cdm_data_type=%22{}%22&minTime%3C={}&maxTime%3E={}'.format(e.server,
cdm_data_type,max_time,min_time)
print(url)
df = pd.read_csv(urlopen(url), skiprows=[1])
return df
[13]:
dfll = alllonlat(ooi, 'TimeSeries', min_time, max_time)
# extract lon,lat values of matching datasets from allDatasets dataframe
dfr = dfll[dfll['datasetID'].isin(search_df['Dataset ID'])]
http://erddap.dataexplorer.oceanobservatories.org/erddap/tabledap/allDatasets.csv?datasetID%2CminLongitude%2CminLatitude&cdm_data_type=%22TimeSeries%22&minTime%3C=2018-07-15T00:00:00Z&maxTime%3E=2018-07-01T00:00:00Z
[14]:
dfr
[14]:
| datasetID | minLongitude | minLatitude | |
|---|---|---|---|
| 0 | ooi-ce01issm-rid16-02-flortd000 | -124.09543 | 44.65983 |
| 1 | ooi-ce01issm-rid16-03-ctdbpc000 | -124.09543 | 44.65983 |
| 2 | ooi-ce01issm-rid16-03-dostad000 | -124.09543 | 44.65983 |
| 3 | ooi-ce01issm-rid16-07-nutnrb000 | -124.09543 | 44.65983 |
| 5 | ooi-ce01issm-rid16-06-phsend000 | -124.09543 | 44.65983 |
| ... | ... | ... | ... |
| 99 | ooi-ce07shsm-sbd11-06-metbka000 | -124.56631 | 46.98408 |
| 324 | ooi-rs01slbs-lj01a-12-ctdpfb101 | -125.38981 | 44.51524 |
| 328 | ooi-rs01sbps-pc01a-4c-flordd103 | -125.38966 | 44.52897 |
| 329 | ooi-rs01sbps-pc01a-4a-ctdpfa103 | -125.38966 | 44.52897 |
| 330 | ooi-rs01sbps-pc01a-4b-phsena102 | -125.38966 | 44.52897 |
63 rows × 3 columns
[15]:
dfr.hvplot.points(x='minLongitude', y='minLatitude', geo=True,
tiles='OSM', color='red', alpha=0.2, hover_cols=['datasetID'],
xlim=(min_lon, max_lon), padding=20, title='OOI Stations with Salinity')
[15]:
Loop through datasets extracting data
[16]:
df_list = []
hv_list = []
for dataset_id in ctdbp['Dataset ID'].values:
ooi.dataset_id = dataset_id
ooi.variables = [
'time',
ooi.get_var_by_attr(dataset_id=dataset_id, standard_name=standard_name)[0]
]
try:
ds = ooi.to_xarray(decode_times=True)
df_list.append(ds)
print(dataset_id)
hv_list.append(ds[ooi.variables[-1]].hvplot(label=dataset_id))
except:
pass
ooi-ce01issm-rid16-03-ctdbpc000
ooi-ce01issm-mfd37-03-ctdbpc000
ooi-ce01issm-sbd17-06-ctdbpc000
ooi-ce04osbp-lj01c-06-ctdbpo108
ooi-ce04ossm-rid27-03-ctdbpc000
ooi-ce02shbp-lj01d-06-ctdbpn106
ooi-ce02shsm-rid27-03-ctdbpc000
ooi-ce06issm-rid16-03-ctdbpc000
ooi-ce06issm-mfd37-03-ctdbpc000
ooi-ce06issm-sbd17-06-ctdbpc000
ooi-ce07shsm-rid27-03-ctdbpc000
ooi-ce07shsm-mfd37-03-ctdbpc000
[17]:
hv.Overlay(hv_list).opts(legend_position='right', width=900, legend_offset=(0,0))
[17]:
Find all the IOOS salinity data¶
Let’s do the same query for IOOS
[18]:
ioos = ERDDAP(server="http://erddap.sensors.ioos.us/erddap", protocol=protocol)
[19]:
search_url = ioos.get_search_url(response='csv', **kw)
search_df = pd.read_csv(urlopen(search_url))
search_df = search_df[['Institution', 'Dataset ID','tabledap']]
search_df
[19]:
| Institution | Dataset ID | tabledap | |
|---|---|---|---|
| 0 | Center for Coastal Margin Observation & Predic... | org_cmop_46096 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 1 | Observing System Monitoring Center (OSMC) | osmc_46097 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 2 | Observing System Monitoring Center (OSMC) | osmc_46098 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 3 | Observing System Monitoring Center (OSMC) | osmc_46099 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 4 | Observing System Monitoring Center (OSMC) | osmc_46100 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 5 | NOAA National Data Buoy Center (NDBC) | wmo_46116 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 6 | University of Washington | edu_washington_46120 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 7 | University of Washington | edu_washington_46121 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 8 | University of Washington | edu_washington_46123 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 9 | University of Washington | edu_washington_46124 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 10 | University of Washington | edu_washington_46125 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 11 | Northwest Association of Networked Ocean Obser... | nanoos_apl_chaba | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 12 | Northwest Association of Networked Ocean Obser... | nanoos_apl_nemo | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 13 | USGS National Water Information System (NWIS) | gov_usgs_waterdata_12113415 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 14 | NOAA Center for Operational Oceanographic Prod... | noaa_nos_co_ops_9437540 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 15 | Northwest Association of Networked Ocean Obser... | nanoos_kc_nsaj02 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 16 | Northwest Association of Networked Ocean Obser... | nanoos_kc_ptwilliams | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 17 | Northwest Association of Networked Ocean Obser... | nanoos_kc_ycqmh01 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 18 | Northwest Association of Networked Ocean Obser... | nanoos_psi_baycenter | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 19 | Northwest Association of Networked Ocean Obser... | nanoos_psi_nahcotta | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 20 | Northwest Association of Networked Ocean Obser... | nanoos_kc_seaqysi | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 21 | Center for Coastal Margin Observation & Predic... | org_cmop_sefo3 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 22 | Center for Coastal Margin Observation & Predic... | org_cmop_seto3 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 23 | Northwest Association of Networked Ocean Obser... | nanoos_taf_dabobbay | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 24 | Northwest Association of Networked Ocean Obser... | nanoos_wadoh_hoodcanal1 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
| 25 | Northwest Association of Networked Ocean Obser... | nanoos_wcsh_whiskey1 | http://erddap.sensors.ioos.us/erddap/tabledap/... |
[20]:
ioos.constraints = {'time>=': min_time,'time<=': max_time}
ioos.response='csv'
[21]:
df_list = []
hv_list = []
for dataset_id in search_df['Dataset ID'].values:
ioos.dataset_id = dataset_id
ioos.variables = [
'time',
ioos.get_var_by_attr(dataset_id=dataset_id, standard_name=standard_name)[0]
]
try:
ds = ioos.to_xarray(decode_times=True)
df_list.append(ds)
print(dataset_id)
hv_list.append(ds[ioos.variables[-1]].hvplot(label=dataset_id))
except:
pass
org_cmop_46096
osmc_46097
osmc_46098
osmc_46099
osmc_46100
wmo_46116
edu_washington_46123
edu_washington_46124
edu_washington_46125
nanoos_apl_chaba
nanoos_apl_nemo
gov_usgs_waterdata_12113415
noaa_nos_co_ops_9437540
nanoos_kc_nsaj02
nanoos_kc_ptwilliams
nanoos_kc_ycqmh01
nanoos_psi_baycenter
nanoos_psi_nahcotta
nanoos_kc_seaqysi
org_cmop_sefo3
org_cmop_seto3
nanoos_taf_dabobbay
nanoos_wadoh_hoodcanal1
[22]:
hv.Overlay(hv_list).opts(legend_position='right', width=900, legend_offset=(0,0))
[22]:
[23]:
dfll = alllonlat(ioos, 'TimeSeries', min_time, max_time)
# extract lon,lat values of matching datasets from allDatasets dataframe
dfr = dfll[dfll['datasetID'].isin(search_df['Dataset ID'])]
http://erddap.sensors.ioos.us/erddap/tabledap/allDatasets.csv?datasetID%2CminLongitude%2CminLatitude&cdm_data_type=%22TimeSeries%22&minTime%3C=2018-07-15T00:00:00Z&maxTime%3E=2018-07-01T00:00:00Z
[24]:
dfr.hvplot.points(x='minLongitude', y='minLatitude', geo=True,
tiles='OSM', color='red', alpha=0.4,
xlim=(min_lon, max_lon), padding=20, title='IOOS Stations with Salinity')
[24]: