Downloading and finding data from WIS2
Learning outcomes!
By the end of this practical session, you will be able to:
- use pywis-pubsub to subscribe to a Global Broker and download data to your local system
- use pywiscat to discover datasets from the Global Discovery Catalogue
Introduction
In this session you will learn how to discover data from the WIS2 Global Discovery Catalogue (GDC) and download data from a WIS2 Global Broker (GB).
Preparation
Note
Before starting please login to your student VM.
Downloading data with pywis-pubsub
The first practical session used MQTT Explorer to connect to the Météo-France Global Broker.
Let's use the pywis-pubsub to subscribe using a command line tool.
pywis-pubsub subscribe --help
Note
pywis-pubsub is pre-installed on the local training environment, but can be installed from anywhere with pip3 install pywis-pubsub
Update the sample configuration (see the sections marked TBD) to connect to the Météo-France Global Broker:
vi ~/exercise-materials/pywis-pubsub-exercises/config.yml
Update the following values in the configuration:
- broker:
mqtts://everyone:everyone@globalbroker.meteo.fr:8883
- subscribe_topics: fill in one to many topics
cache/#
(on separate lines) - storage.option.path: add a directory to where data should be downloaded
Run the pywis-pubsub
command:
pywis-pubsub subscribe -c ~/exercise-materials/pywis-pubsub-exercises/config.yml --verbosity DEBUG
At this point you should see a number of ongoing messages on your screen. Hit Crtl-C
to stop messages from arriving/download to help analyze the content.
Note
The above command will download data to your local system for demo purposes. For operational environments you will need to consider and manage diskspace requirements as part of your workflow.
Question
What is the format of the data notifications that are displayed on the screen?
Click to reveal answer
The format is GeoJSON
Question
Is there data being downloaded? How can we run the pywis-pubsub
command to be able to download the data (hint: review the options when running the pywis-pubsub subscribe --help
command)?
Click to reveal answer
Add the -d
or --download
flag to the pywis-pubsub
command
Stop the pywis-pubsub
command (CTRL-C) and update the configuration to be able to download the data
to /tmp/wis2-data
.
Try spatial filtering with a bounding box:
pywis-pubsub subscribe -c ~/exercise-materials/pywis-pubsub-exercises/config.yml --verbosity INFO -d -b -142,42,-52,84
Note
Try using your own bounding box (format is west,south,east,north
, in decimal degrees).
Finding data with pywiscat
Let's use pywiscat to query the GDC
At the moment, the available GDCs are as follows:
- Environment and Climate Change Canada, Meteorological Service of Canada: https://api.weather.gc.ca/collections/wis2-discovery-metadata
- China Meteorological Administration: https://api.weather.gc.ca/api/collections/wis2-discovery-metadata
pywiscat search --help
Note
pywiscat is pre-installed on the local training environment, but can be installed from anywhere with pip3 install pywiscat
Note
by default, pywiscat connects to Canada's Global Discovery Catalogue. To set to a different catalogue, set the PYWISCAT_GDC_URL
environment variable
pywiscat search
Question
How many records are returned from the search?
Click to reveal answer
There should be 40 records returned
Querying WIS2 GDC 🗃️ ...
Results: 40 records
...
Let's try querying the GDC with a keyword:
pywiscat search -q observations
Question
What is the data policy of the results?
Click to reveal answer
All data returned should specify "core" data
+---------------------------------------------------------------------------+------------------------------+-------------------------------------------------------+-------------+
| id | centre | title | data policy |
+---------------------------------------------------------------------------+------------------------------+-------------------------------------------------------+-------------+
| urn:x-wmo:md:dma:dominica_met_wis2node:surface-weather-observations | dominica_met_wis2node | Surface weather observations from Dominica Meteoro... | core |
| urn:x-wmo:md:gin:conakry_met_centre:surface-weather-observations | conakry_met_centre | Surface weather observations from gin.conakry_met_... | core |
| urn:x-wmo:md:atg:antigua_met_wis2node:surface-weather-observations | antigua_met_wis2node | Surface weather observations from Antigua and Barb... | core |
| urn:x-wmo:md:bfa:ouagadougou_met_centre:surface-weather-observations | ouagadougou_met_centre | Surface weather observations from bfa.ouagadougou_... | core |
...
Try additional queries with -q
Tip
The -q
flag allows for the following syntax:
-q synop
: find all records with the word "synop"-q temp
: find all records with the word "temp"-q "observations AND malawi"
: find all records with the words "observations" and "malawi"-q "observations NOT malawi"
: find all records that contain the word "observations" but not the word "malawi"-q "synop OR temp"
: find all records with both "synop" or "temp"-q "obs~"
: fuzzy search
When searching for terms with spaces, enclose in double quotes.
Let's get more details on a specific search result that we are interested in:
pywiscat get <id>
Tip
Use the id
value from the previous search.
Conclusion
Congratulations!
In this practical session, you learned how to:
- use pywis-pubsub to subscribe to a Global Broker and download data to your local system
- use pywiscat to discover datasets from the Global Discovery Catalogue