Skip to content

Downloading and finding data from WIS2

Learning outcomes!

By the end of this practical session, you will be able to:

  • use pywis-pubsub to subscribe to a Global Broker and download data to your local system
  • use pywiscat to discover datasets from the Global Discovery Catalogue

Introduction

In this session you will learn how to discover data from the WIS2 Global Discovery Catalogue (GDC) and download data from a WIS2 Global Broker (GB).

Preparation

Note

Before starting please login to your student VM.

Downloading data with pywis-pubsub

The first practical session used MQTT Explorer to connect to the Météo-France Global Broker.

Let's use the pywis-pubsub to subscribe using a command line tool.

pywis-pubsub subscribe --help

Note

pywis-pubsub is pre-installed on the local training environment, but can be installed from anywhere with pip3 install pywis-pubsub

Update the sample configuration (see the sections marked TBD) to connect to the Météo-France Global Broker:

vi ~/exercise-materials/pywis-pubsub-exercises/config.yml

Update the following values in the configuration:

  • broker: mqtts://everyone:everyone@globalbroker.meteo.fr:8883
  • subscribe_topics: fill in one to many topics cache/# (on separate lines)
  • storage.option.path: add a directory to where data should be downloaded

Run the pywis-pubsub command:

pywis-pubsub subscribe -c ~/exercise-materials/pywis-pubsub-exercises/config.yml --verbosity DEBUG

At this point you should see a number of ongoing messages on your screen. Hit Crtl-C to stop messages from arriving/download to help analyze the content.

Note

The above command will download data to your local system for demo purposes. For operational environments you will need to consider and manage diskspace requirements as part of your workflow.

Question

What is the format of the data notifications that are displayed on the screen?

Click to reveal answer

The format is GeoJSON

Question

Is there data being downloaded? How can we run the pywis-pubsub command to be able to download the data (hint: review the options when running the pywis-pubsub subscribe --help command)?

Click to reveal answer

Add the -d or --download flag to the pywis-pubsub command

Stop the pywis-pubsub command (CTRL-C) and update the configuration to be able to download the data to /tmp/wis2-data.

Try spatial filtering with a bounding box:

pywis-pubsub subscribe -c ~/exercise-materials/pywis-pubsub-exercises/config.yml --verbosity INFO -d -b -142,42,-52,84

Note

Try using your own bounding box (format is west,south,east,north, in decimal degrees).

Finding data with pywiscat

Let's use pywiscat to query the GDC

At the moment, the available GDCs are as follows:

  • Environment and Climate Change Canada, Meteorological Service of Canada: https://api.weather.gc.ca/collections/wis2-discovery-metadata
  • China Meteorological Administration: https://api.weather.gc.ca/api/collections/wis2-discovery-metadata
pywiscat search --help

Note

pywiscat is pre-installed on the local training environment, but can be installed from anywhere with pip3 install pywiscat

Note

by default, pywiscat connects to Canada's Global Discovery Catalogue. To set to a different catalogue, set the PYWISCAT_GDC_URL environment variable

pywiscat search

Question

How many records are returned from the search?

Click to reveal answer

There should be 40 records returned

Querying WIS2 GDC 🗃️ ...

Results: 40 records
...

Let's try querying the GDC with a keyword:

pywiscat search -q observations

Question

What is the data policy of the results?

Click to reveal answer

All data returned should specify "core" data

+---------------------------------------------------------------------------+------------------------------+-------------------------------------------------------+-------------+
| id                                                                        | centre                       | title                                                 | data policy |
+---------------------------------------------------------------------------+------------------------------+-------------------------------------------------------+-------------+
| urn:x-wmo:md:dma:dominica_met_wis2node:surface-weather-observations       | dominica_met_wis2node        | Surface weather observations from Dominica Meteoro... | core        |
| urn:x-wmo:md:gin:conakry_met_centre:surface-weather-observations          | conakry_met_centre           | Surface weather observations from gin.conakry_met_... | core        |
| urn:x-wmo:md:atg:antigua_met_wis2node:surface-weather-observations        | antigua_met_wis2node         | Surface weather observations from Antigua and Barb... | core        |
| urn:x-wmo:md:bfa:ouagadougou_met_centre:surface-weather-observations      | ouagadougou_met_centre       | Surface weather observations from bfa.ouagadougou_... | core        |
...

Try additional queries with -q

Tip

The -q flag allows for the following syntax:

  • -q synop: find all records with the word "synop"
  • -q temp: find all records with the word "temp"
  • -q "observations AND malawi": find all records with the words "observations" and "malawi"
  • -q "observations NOT malawi": find all records that contain the word "observations" but not the word "malawi"
  • -q "synop OR temp": find all records with both "synop" or "temp"
  • -q "obs~": fuzzy search

When searching for terms with spaces, enclose in double quotes.

Let's get more details on a specific search result that we are interested in:

pywiscat get <id>

Tip

Use the id value from the previous search.

Conclusion

Congratulations!

In this practical session, you learned how to:

  • use pywis-pubsub to subscribe to a Global Broker and download data to your local system
  • use pywiscat to discover datasets from the Global Discovery Catalogue