Guide to the WMO Information System Volume II (DRAFT) - WMO Information System 2.0

WIS2 leverages open standards to take advantage of the ecosystem of technologies available on the market, thereby avoiding the need to build bespoke solutions that can force National Meteorological and Hydrological Services (NMHSs) to procure costly systems and equipment. In today’s standards development ecosystem, standards bodies work together closely to minimize overlap and build on one another’s areas of expertise. For example, the World Wide Web Consortium provides the framework of web standards, which the Open Geospatial Consortium (OGC) and other standards bodies leverage. WIS2 leverages open standards with industry adoption and wider, stable and robust implementations, thus extending the reach of WMO data sharing and lowering the barrier to access by Members.

WIS2 prioritizes public telecommunication networks, rather than private networks, for GTS links. The Internet is therefore the best choice for a local connection, as it uses commonly available and well-understood technology.

WIS2 aims to improve the discovery, access and utilization of weather, climate and water data by adopting web technologies proven to provide a truly collaborative platform for a more participatory approach. Data exchange using the Web also facilitates easy access mechanisms. Browsers and search engines allow web users to discover data without the need for specialized software. The Web also enables additional data access platforms, such as desktop geographical information systems (GIS), mobile applications and forecaster workstations. The Web provides access control and security mechanisms that can be utilized to freely share core data as defined by Resolution 1 (Cg-Ext(2021)) and to protect the data with more restrictive licensing constraints. Web technologies also allow for authentication and authorization to enable the provider to retain control of who can access published resources and to request users to accept a license specifying the terms and conditions for using the data as a condition of being granted access.

WIS2 uses a "publish-subscribe" pattern by which users subscribe to a topic to receive new data in real time. The mechanism is similar to WhatsApp and other messaging applications. It is a reliable and straightforward way to allow users to choose their data of interest and to receive them reliably.

The cloud provides reliable platforms for data sharing and processing. It reduces the need for expensive local IT infrastructure, which constitutes a barrier to developing effective and reliable data processing workflows for some WMO Members. WIS2 encourages WMO centres to adopt cloud technologies, where appropriate, to meet users' needs. While WMO technical regulations will not mandate cloud services, WIS2 will promote the gradual adoption of cloud technologies that provide the most effective solution.

The cloud-based infrastructure allows for the easy portability of technical solutions, ensuring that a system implemented by a specific country or territory can be packaged and deployed easily in other countries/territories with similar needs. In addition, using cloud technologies allows WIS2 to deploy infrastructure and systems efficiently, while requiring minimal effort from the NMHSs, by shipping ready-made services and implementing consistent data processing and exchange techniques.

It is important to note that hosting data and services on the cloud does not affect data ownership. Even in a cloud environment, organizations retain ownership of their data, software, configuration and change management as if they were hosting their infrastructure. As a result, data authority and provenance stay with the organization, and the cloud is simply a technical means to publish the data.

WMO enables the international exchange of observations and model data for all Earth-system disciplines.

Resolution 1 (Cg-Ext(2021)) describes the Earth system data that are necessary for efforts to monitor, understand and predict the weather and climate – including the hydrological cycle, the atmospheric environment and space weather.

WIS2 is the mechanism by which these Earth system data are exchanged.

A common practice when working with data is to group them into datasets. All the data in a dataset share some common characteristics. The Data Catalog Vocabulary (DCAT) defines a dataset as a "collection of data, published or curated by a single agent, and available for access or download in one or more representations".^[3]

Why is this important? The "single agent" (such as an organization) responsible for managing the collection ensures consistency among the data. For example, in a dataset:

This consistency means that it is possible to predict the contents of a dataset, at least regarding the common characteristics, making it easier to write applications to process the data.

A dataset may be published as an immutable resource (such as data collected from a research programme), or it may be routinely updated (for example, every minute, as new observations are collected from weather stations).

A dataset may be represented as a single, structured file or object (for example, a CSV file in which each row represents a data record) or as thousands of consistent files (for example, output from a reanalysis model encoded as many thousands of General Regularly-distributed Information in Binary form (GRIB) files). Determining the best way to represent a dataset is beyond the scope of this Guide – there are many factors to consider. The key point here is that the dataset is considered a single, identifiable resource irrespective of how it is represented.

Because data are grouped into a single, conceptual resource (that is, the dataset) it is possible to:

The dataset concept is central to WIS2:

It is up to data publishers to decide how their data are grouped into datasets – effectively, to decide what datasets they publish to WIS2. That said, it is recommended that, subject to the consistency rules above, data publishers should organize their data into as few datasets as possible.

For a data publisher, this means fewer discover metadata records to maintain. For a data consumer, this means fewer topics to subscribe to and fewer places to access the data.

There are some things that are fixed requirements for datasets:

Some examples of datasets include:

In summary, datasets are important because they are how data are managed in WIS2.

The first step to using data published via WIS2 is to determine which dataset or datasets contain the data that are needed. To do this, a data consumer may browse discovery metadata provided by the Global Discovery Catalogue. Discovery metadata follow a standard scheme (see Manual on WIS, Volume II – Appendix F. WMO Core Metadata Profile (Version 2)). A data consumer may discover a dataset using keywords, a geographic area of interest, temporal information, or free text. Matching search results from the Global Discovery Catalogue provide high-level information (title, description, keywords, spatiotemporal extents, data policy, licensing, contact information), from which data consumers can assess and evaluate their interest in accessing/downloading data associated with the dataset record.

A key component of dataset records in the Global Discovery Catalogue is "actionable" links. A dataset record provides one or more links, each clearly identifying its nature and purpose (informational, direct download, application programming interface (API), subscription) so that the data consumer can interact with the data accordingly. For example, a dataset record may include a link to subscribe to notifications about the data(see 1.2.2 How to subscribe to notifications about the availability of new data), or an API, or an offline archive retrieval service.

The Global Discovery Catalogue is accessible via an API and provides a low-barrier mechanism (see 2.4.4 Global Discovery Catalogue). Internet search engines are able to index the discovery metadata in the Global Discovery Catalogue, thereby providing data consumers with an alternative means to search for WIS2 data.

WIS2 provides notifications about updates to datasets, for example, a notification may indicate that a new observation record from an automatic weather station has been added to a dataset of surface observations. These notifications are published on Message Brokers. Where data consumers need to use data rapidly once they have been published (for example, as inputs to a weather prediction model), they should subscribe to one or more Global Brokers to get notification messages using Message Queuing Telemetry Transport (MQTT) protocol.footnote[Subscribing to notifications about newly available data ensures that the data consumers do not need to continually to poll the data server to check for updates.]

In WIS2, notifications are republished by Global Brokers to ensure resilient distribution. Consequently, there will be multiple places where one can subscribe. Data consumers requiring real-time notifications must subscribe to Global Brokers. Data consumers should subscribe to more than one Global Broker to ensure that notifications continue to be received if a Global Broker instance fails.

A dataset in WIS2 is associated with a unique topic. Notifications about updates to a dataset are published to the associated topic. Topics are organized according to a standard scheme (see the Manual on WIS, Volume II - Appendix D. WIS2 Topic Hierarchy).

A data consumer can find the appropriate topic to subscribe to either by searching the Global Discovery Catalogue, by using an Internet search engine,^[5], or by browsing the topic hierarchy on a Message Broker.

WIS2 uses Global Caches to distribute core data, as defined in the WMO Unified Data Policy (Resolution 1 (Cg-Ext (2021))). Each Global Cache republishes core data on its own highly available data server and publishes a new notification message advertising the availability of those data from the Global Cache location.

Notifications from WIS2 Nodes and Global Caches are published on different topics: The root topic used by WIS2 Nodes is origin, while the root topic used by Global Caches is cache. Other than the root, the topic hierarchy is identical. For example, for synoptic weather observations published by Environment Canada:

As per clause 3.2.13 of the Manual on WIS, Volume II, data consumers should access core data from the Global Caches. In order to access these data, they must subscribe to the cache topic hierarchy. They will then receive the relevant notifications from the Global Caches, each of which will contain a link (URL) enabling them to download the relevant data from the data server of the corresponding Global Cache.

On receipt of a notification message, a data consumer needs to decide whether to download the newly available data. The content of the notification message provides the information needed to make this decision (see the Manual on WIS, Volume II - Appendix E. WIS2 Notification Message).

In many cases, data consumers will use a software application to determine whether or not to download the data. The present section explains this process.

When subscribing to multiple Global Brokers, data consumers will receive multiple copies of a notification message. Each notification message has a unique identifier, defined using the id property. Duplicate messages should be discarded.

Core data are available from both a WIS2 Node and the Global Caches, each of which will publish a different notification message advertising an alternative location from which the data may be downloaded. Because these are different messages, they will have different identifiers. However, each of these messages refers to the same data object, which is uniquely identified in the notification message using the data_id property. Notification messages from different sources can easily be compared to determine whether they refer to the same data. By subscribing to the cache root topic, data consumers will only receive notifications about data available from the Global Caches. The origin root topic should be used when subscribing to notifications about recommended data. Data consumers should not subscribe to the origin root topic for notifications about core data because the notification messages provided on these topics will refer to data published directly on the WIS2 Nodes (referred to as the "origin").

Data consumers need to consider their strategy for managing these duplicate messages. From a data perspective, it does not matter which Global Cache instance is used – they will all provide an identical copy of the data object published by the originating WIS2 Node. The simplest strategy is to accept the first notification message and download the data from the Global Cache instance that the message refers to by using a URL for the data object at that Global Cache instance. Alternatively, data consumers may have a preferred Global Cache instance, for example, one that is located in their region. Whichever Global Cache instance is chosen, data consumers will need to implement logic to discard duplicate notification messages based on id and duplicate data objects based on data_id.

A notification message also provides a small amount of metadata about the data object it references, such as location or time. Data consumers can use these metadata to decide whether the data object referenced in the message should be downloaded. This is known as client-side filtering.

The notification message should also include the metadata identifier for the dataset to which the data object belongs. A data consumer can use the metadata identifier to search the Global Discovery Catalogue and discover more about the data - in particular, whether there are any conditions on the use of those data.

Links to where data can be accessed are made available through dataset discovery metadata (via the Global Discovery Catalogue) and/or data notification messages (via Global Brokers). Links can be used to directly download the data (according to the network protocol and content description provided in the link) using a mechanism appropriate to the workflow of the data consumer. Such mechanisms could include web and/or desktop applications, custom tools and so forth.

A discovery metadata record or notification message may provide more than one download link. The preferred link will be identified as "canonical" (link relation: "rel": "canonical" ^[6]).

Where data are provided through an interactive web service, a canonical link containing a URL from which data consumers can directly download a data object may be complemented with an additional link providing the URL for the root of the web service from which data consumers can interact with or query the entire dataset.

If a download link implements access control (for example, the data consumer needs to take some additional action(s) to download the data object), it will contain a security object that provides the pertinent information (such as the access control mechanism used and where/how a data consumer would need to register to request access).

Data are shared on WIS2 in accordance with the WMO Unified Data Policy (Resolution 1 (Cg-Ext (2021))). This data policy describes two categories of data: core and recommended.

The WMO Unified Data Policy (Resolution 1 (Cg-Ext (2021))) encourages attribution of the source of the data in all cases. This ensures that, credit is given to those who have expended effort and resources in collecting, curating, generating, or processing the data. Attribution provides visibility into who is using the data, which, for many organizations, serves as crucial evidence to justify the continued provision and updating of the data.

Details of the applicable WMO data policy and any rights or licenses associated with the data are provided in the discovery metadata accompanying the data. Discovery metadata records are available from the Global Discovery Catalogue.

The Manual on WIS, Volume II – Appendix F. WMO Core Metadata Profile (Version 2), 1.18 Properties / WMO Data Policy provides details on how the WMO Data Policy, rights and/or licenses are described in the discovery metadata.

When using data from WIS2, data consumers:

Data consumers wanting to use data published via WIS2 should, at a minimum, read the following sections:

The following specifications in the Manual on WIS, Volume II also provide useful information:

The first step is to consider the data, how they can be conceptually grouped into one or more datasets (see 1.1.4 Why are datasets so important?), and whether they are core or recommended data, as per the WMO Unified Data Policy (Resolution 1 (Cg-Ext(2021))) .

Next, it is important to consider where the data are published. If the data relate to a specific country or territory, they should be published through a National Centre (NC). If they relate to a region, programme, or other specialized function within WMO, they should be published through a Data Collection or Production Centre (DCPC). The functional requirements for NCs and DCPCs are described in the Manual on WIS, Volume II - Part III Functions of WIS.

All NCs and DCPCs are affiliated with a Global Information System Centre (GISC), which is responsible for helping to establish efficient and effective data sharing on WIS. The affiliated GISC can assist in getting the data onto WIS2.

It may be possible to identify an existing NC or DCPC that can publish the data. Alternatively, it may be necessary to establish a new NC or DCPC. The main distinction between these two centres is that an NC is designated by a Member, whereas a DCPC is designated by a WMO or related international programme and/or a regional association.

Both NCs and DCPCs require the operation of a WIS2 Node (see 2.4.1 WIS2 Node). The procedure for registering a new WIS2 Node is provided in 2.6.1.1 Registration and decommissioning of a WIS2 Node.

Once the scope of the datasets has been determined, the applicable data policy has been identified, and a WIS2 Node is ready for data publication, the process can proceed to the next step: providing discovery metadata.

Discovery metadata is the mechanism by which data publishers tell potential consumers about their data, how it may be accessed, and any conditions they may place on the use of those data.

Each dataset that is published must have an associated discovery metadata record. This record is encoded as GeoJSON (See RFC 7946^[7]) and must conform to the specification given in the Manual on WIS, Volume II - Appendix F. WMO Core Metadata Profile (Version 2).

Copies of all discovery metadata records from WIS2 are held in the Global Discovery Catalogues, where data consumers can search and browse to find data that is of interest to them.

Depending on local arrangements, your GISC may be able to assist in transferring discovery metadata record(s) to the Global Discovery Catalogues. If this is not the case, data publishers will need to publish the discovery metadata record(s) themselves^[8] using one of two methods:

In both cases, a notification message needs to be published on a Message Broker that tells WIS2 that there is a new discovery metadata record to upload and that it can be accessed at the specified URL.^[10] Notification messages shall conform to the specification given in the Manual on WIS, Volume II - Appendix E. WIS2 Notification Message. They must also be published on a topic that conforms to the specification given in the Manual on WIS, Volume II - Appendix D. WIS2 Topic Hierarchy. For example, metadata published by Deutscher Wetterdienst would use the following topic: origin/a/wis2/de-dwd/metadata/core.

These discovery metadata records are then propagated through the Global Service components into the Global Discovery Catalogue, where data consumers can search and browse for datasets of interest.

Upon receipt of a new discovery metadata record, a Global Discovery Catalogue (see 2.4.4 Global Discovery Catalogue) will validate, assess, ingest, and publish the record. Validation ensures compliance with the specification, while the assessment evaluates the discovery record against good practices. The Global Discovery Catalogue will notify the data publisher if the discovery metadata record fails validation and provide recommendations for improvements.

Discovery metadata must be published in the Global Discovery Catalogues before the data are published.

WIS2 is based on the web architecture.^[11] As such it is resource oriented. Datasets are resources; the "granules" of data grouped in a dataset are resources; and the discovery metadata records that describe datasets are resources. In web architecture, every resource has a unique identifier (such as a URI^[12]), which can be used to resolve the identified resource and interact with it (for example, to download a representation of the resource over an open-standard protocol such as HTTP).

In simple terms, data (and metadata) are provided to WIS2 by assigning them a unique identifier, in this case a URL^[13], and making them available via a data server - most typically a web server using HTTP protocol.^[14] It is up to the data server to decide what to provide when resolving the identifier. For example, the URL of a data granule may resolve as a representation encoded in a given data format, whereas the URL of a dataset may resolve as a description of the dataset (that is, metadata) that includes links to access the data from which the dataset is comprised - either individual files (that is, the data granules) or an interactive API that enables users to request only the parts of the dataset they need by specifying query parameters.

The following sections cover specific considerations relating to publishing data to WIS2.

Data publishers planning to operate WIS2 Nodes, at a minimum, should read the following sections:

The following sections are recommended for further reading:

Note that PART IV. Security and PART V. Competencies reference content originally published for WIS1. These sections remain largely applicable and will be updated in subsequent releases of this Guide.

Data publishers publishing aviation weather data via WIS2 for onward transmission through the International Civil Aviation Organization (ICAO) System Wide Information Management (SWIM), should also read 2.8.1.1 Publishing meteorological data through WIS2 into ICAO SWIM.

Finally, data publishers should also review the specifications in the Manual on WIS, Volume II:

When providing a WIS2 Node, Members may use whichever software components they consider most appropriate to comply with the WIS2 technical regulations.

To assist Members, a free and open-source reference implementation called “WIS2 in a box” (wis2box) is available. wis2box implements the requirements for a WIS2 Node and contains additional enhancements. wis2box is built free and open-source software components that are mature, robust and widely adopted for operational use.

wis2box provides the functionality required for both data publisher and data consumer roles, as well as the following technical functions:

The project documentation can be found at https://docs.wis2box.wis.wmo.int.

wis2box is managed as a free and open source project. The source code, issue tracking and discussions are hosted openly on GitHub: https://github.com/wmo-im/wis2box.

The successful operation of WIS2 depends on a set of Global Services running well-managed IT environments with a very high level of reliability so that all WIS2 users and WIS2 Nodes are able to access and provide the data they need for their duties.

Depending on the nature of the Global Service, the following are the minimum capabilities needed to ensure that the level of service as a whole reaches 100% (or very close):

In addition to the above, WIS architecture can accommodate adding (or removing) Global Services. Candidate WIS centres should inform their WIS NFP and contact the WMO Secretariat to discuss their offer to provide a Global Service.

Running a Global Service is a significant commitment for a WIS centre. To maintain a very high level of service, each Global Service has a key role to play.

On receipt of an offer from a Member to operate a Global Service, the WMO Secretariat will suggest which Global Service the Member may provide to improve WIS2. This suggestion will be based on the current situation of WIS2 (such as the number of existing Global Brokers, whether an additional Global Cache is needed, and so forth).

The Manual on WIS, Volume II, the present Guide, and other available materials will help WIS centres decide how to proceed.

When a decision on how to proceed has been made, the WIS NFP will inform the WMO Secretariat of its preference. Depending on the type of Global Service, the WMO Secretariat will provide a checklist to the WIS centre so that the future Global Service can be included in WIS operations.

A WIS centre must commit to running the Global Service for a minimum of four years.

The WMO Secretariat and other Global Services will make the required changes to include the new Global Service in WIS operations.

In WIS2, Global Caches provide access to WMO core data for data consumers. This allows data providers to restrict access to their systems to Global Services, and it reduces the need for them to provide high bandwidth and low latency access to their data. Global Caches operate in a way that is transparent to end users in that they resend notification messages from data providers. These messages are updated to point to copies of the original data held in the Global Cache data store. Global Caches also resend notification messages from data providers for core data that are not stored in the Global Cache, such as when the originator specifies in the notification message that a certain dataset should not be cached. In these cases, the notification messages remain unchanged and point to the original source. Data consumers should subscribe to the notification messages from Global Caches instead of the notification messages from data providers for WMO core data. When data consumers receive a notification message, they should follow the URLs from that message, which either point to a Global Cache which has a copy of the data, or – in case of uncached content – point to the original source.

Driven by international standards for data discovery, access, and visualization, the WIS2 principles help lower the barrier for WMO Members to weather, climate, and water data. An additional benefit of adopting these standards is that Members are able to provide the same data and access mechanisms to external systems at no extra cost for implementation.

WIS2 standards are based on industry standards (OGC, W3C, IETF) and allow for broad interoperability. This means that non-traditional users can also use data from WIS2 data in the same manner, without the requirement for specialized software, tools, or applications.

The efficient and effective provision of services relying on meteorological, climatological, hydrological and oceanographic information depends on a reliable information infrastructure. This infrastructure should be guided by community best practices and standards, including recommendations and requirements on sourcing, securing, managing, archiving, exchanging and providing easy access to information. This part of the Guide aims to provide high-level guidance on these activities, which can be grouped under the term “information management”. This is done by identifying and describing the fundamental principles of good information management and by highlighting the different stages of the information management life cycle.

Note: The term "information" is used in a general sense and includes data and products.

High-level guidance on information management practices that apply in the context of information related to the Earth system is provided in this part of the Guide. Detailed technical information, such as the specification of data formats or quality control and assurance methods, is provided in other parts of the Guide and in other WMO publications. These are referenced where applicable.

The principles of information management are described below. Section 3.3 The information management life cycle describes five focus areas.

The effective management of information is essential for WMO centres to deliver operational services and information that is authoritative, seamless, secure and timely. The principles below underpin and provide a framework for information management across the full information life cycle. These principles are independent of the information type and are largely independent of technology; they are therefore expected to remain stable over time.