Draft guidance on technical specifications of WIS2

Whether providing data as files or through interactive APIs you need to decide which encodings (aka. data formats) to use. WMO Technical Regulations may require that data is encoded in particular formats. For example, synoptic observations must be encoded in BUFR. The Manual on Codes (WMO-No. 306) provides details of data formats formally approved for use in WMO. However, Technical Regulations don’t cover all data sharing requirements. In such cases, you should select data formats that are open, non-proprietary, and widely adopted and understood in their target user community. In this context, ‘open’ means that anyone can use the format without needing a license to do so – either to encode data in that format or write software that understands the format.

The simplest way to publish data through WIS2 is to persist your data as files and publish those files on a Web sserver. All these files need to be organised somehow – perhaps a flat structure or grouped into collections that resemble folders or directory structures.

To make your data usable, your users need to be able to find the specific file (or files) they need.

Naming conventions for files and/or directories are useful - but only when the scheme is understood. Assuming that your users understand your naming convention will be a barrier to widespread re-use; many users will simply treat the filename as an opaque string. Where communities commonly use file-naming conventions (e.g., with embedded metadata), these should only be used when adequate documentation is provided to users.

WIS2 does not require the use of specific naming conventions.

Another mechanism to consider is complementing the collections (i.e., directories or folders in which files are grouped) with information that describes their content. Then users,both humans and software agents, can browse the structure and find what they need. Examples of this approach include:

When publishing collections of data it is tempting to package content into zip or SIP^[17] resources - perhaps even packaging the entire collection, complete with folders, into a single resource. Similarly, WMO formats such as GRIB and BUFR allow multiple data objects (e.g., fields or observations) to be packed into a single file. Only having to download a single resource is convenient for many users, but the downside is that the user must download the entire resource and then unpack/decompress it. The convenience of downloading fewer resources needs to be balanced against the cost of forcing users to download data they may not need. Whatever your choice, you should be guided by common practice in your domain - i.e., only zip, SIP, or pack if your users expect it.

Interactive data access aims to support efficient data workflows by enabling client applications to request only the data that they need. The advantage with interactive data access is that it provides more flexibility. Data publishers can offer an API structured around how users want to work with the data rather than force them to work with the structure that is convenient for you as a data publisher.

But it is more complex to implement. You need a server running software that can:

Importantly, when considering use of interactive APIs to serve your data you need to plan for costs: every request to an interactive API requires computational resources to process.

Based on the experience of data publishers who have been using Web APIs to serve their communities, this Guide makes the following recommendations about interactive APIs:

Finally, you should consider versioning your API to avoid breaking changes when adding new features. A common approach is add a version number prefix into the API path; e.g., /v1/service/{rest-of-path} or /service/v1/{rest-of-path}.

More guidance on use of interactive APIs in WIS2 is anticipated in future versions of this Guide.

WIS2 is designed to support the data sharing needs of all WMO programmes. Among these, the World Weather Watch ^[23] drives specific needs for the rapid exchange of data to support weather forecasting.

To enable real-time data sharing^[24], WIS2 uses notification messages to advertise the availability of a new resource - data or discovery metadata - and how to access that resource. Notification messages are published to a queue on a message broker in your WIS2 Node^[25] using the MQTT protocol and immediately delivered to everyone subscribing to that queue. A queue is associated with a specific topic, such as dataset.

For example, when a new temperature profile from a radio sonde deployment is added to a dataset of upper-air data measurements, a notification message would be published that includes the URL used to access the new temperature profile data. Everyone subscribing to notification messages about the upper-air measurement dataset would receive the notification message, identify the URL and download the new temperature profile data.

Optionally, data may be embedded in a notification message using a content object in addition to publishing via the data server. Inline data must be encoded as UTF-8, Base64, or gzip, and must not exceed 4096 bytes in length once encoded.

Notification messages are encoded as GeoJSON (RFC 7946) and must conform to the Manual on WIS (WMO-No. 1060), Volume II, Appendix E: WIS2 Notification Message.

The URL used in the notification message should refer only to the newly added data object rather (e.g., the new temperature profile) than the entire dataset. However, the WIS2 Notification Message specification allows for multiple URLs to be provided. If you are providing your data through an interactive API, you might provide a "canonical" link (designated with link relation: "rel": "canonical"^[26]), and an additional link providing the URL for the root of the Web service from where one can interact with or query the entire Dataset.

You should include the dataset identifier in the notification message (metadata_id property). This allows data consumers receiving the notification to cross reference with information provided in the discovery metadata for the dataset, such as the conditions of use specified in the data policy, rights, or license.

Furthermore, if you have implemented controlled access to your data (e.g., the use of an API key), you should include a security object in the download link that provides the pertinent information (e.g., the access control mechanism used, and where/how a Data Consumer would need to register to request access).

To ensure that data consumers can easily find the topics they want to subscribe to, data publishers must publish to an authorized topic, as specified in the Manual on WIS (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy.

If your data seems to relate to more than one topic, select the most appropriate one. The topic-hierarchy is not a knowledge organisation system - it is only used to ensure uniqueness of topics for publishing notification messages. Discovery metadata is used to describe a dataset and its relevance to additional disciplines; each dataset is mapped to one, and only one, topic.

If the WIS2 Topic Hierarchy does not include a topic appropriate for your data, your should publish on an experimental topic. This allows for data exchange to be established while the formalities are considered^[27]. Experimental topics are provided for each Earth-system discipline at level 8 in the topic hierarchy (e.g., origin/a/wis2/{centre-id}/data/{earth-system-discipline}/experimental/). Data publishers can can extend the experimental branch with sub-topics as they deem appropriate. Experimental topics are subject to change and will be removed once they are no longer needed. For more information, see Manual on WIS (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy, section 7.1.2 Publishing guidelines.

Whatever topic you choose, the discovery metadata you provided to the Global Discovery Catalogue must include subscription links using that topic^[28]. The Global Broker will only republish notification messages on topics specified in your discovery metadata records.

Core data, as specified in the WMO Unified Data Policy (Resolution 1 (Cg-Ext 2021)) is considered essential for provision of services for the protection of life and property and for the well-being of all nations. Core data is provided on a free and unrestricted basis, without charge and with no conditions on use.

WIS2 ensures highly available, rapid access to most Core data via a collection of Global Caches (see Global Cache). Global Caches subscribe to notification messages about the availability of new Core data published at WIS2 Nodes, download a copy of that data and re-publish it on a high-performance data server, discarding it after the retention period expires - normally 24-hours^[29]. Global Caches do not provide any sophisticated APIs - they publish notification messages advertising the availability of data on their cache and allow users to download data via HTTPS using the URL in the notification message.

The URL included in a notification message that is used to access Core data from a WIS2 Node, or the "canonical" URL if multiple URLs are provided, must:

A Global Cache will download and cache the data object accessed via this URL.

The Global Caches are designed to support Members efficiently share real-time and near real-time data; they take on the task of making sure that Core data is available to all and cover the costs of delivering data to a global community.

Unfortunately, Global Caches cannot republish all Core data: there is a limit to how much data they can afford to serve. Currently, a Global Cache expected to cache about 100GB of data each day.

If frequent updates to your dataset are very large (e.g., weather prediction models or remote sensing observations) you will need to share the burden of distributing your data with the Global Cache operators. You should work with your GISC to determine the highest priority elements of your Core datasets that will be republished by the Global Caches.

For Core data that is not to be cached, you must set the cache property in the notification message to false^[30].

You must ensure that Core data that is not cached is publicly accessible from your WIS2 Node; i.e., with no access control mechanisms in place.

A Global Cache operator may choose to disregard your cache preference - for example, if they feel that the content you are providing is large enough to impede provision of caching services for other Members^[31]. In such cases, the Global Cache operator will log this behaviour. In collaboration with the Global Cache operators, your GISC will work with you to resolve concerns.

Finally, please note that Global Caches are under no obligation to cache data published on experimental topics. For such data, the cache property should be set to false.

Recommended data, as defined in the WMO Unified Data Policy (Resolution 1 (Cg-Ext 2021)), is exchanged on WIS2 in support of Earth system monitoring and prediction efforts and may by provided with conditions on use. This means that you may control access to Recommended data.

Access control should use only the "security schemes" for authentication and authorization specified in OpenAPI^[32].

Where access control is implemented, you should include a security object in download links provided in discovery metadata and notification messages that provide the user with pertinent information about the access control mechanism used and where/how they might register to request access.

Recommended data is never cached by the Global Caches.

Use of Core data must always be free and unrestricted. However, you may need to leverage existing systems with built-in access control when implementing the download service for your WIS2 Node.

Example 1: API key. Your data server requires a valid API key to be included in download requests. The URLs used notification messages should include a valid API key.^[33][34]

Example 2: Pre-signed URLs. Your data server uses a cloud-based object store that requires credentials to be provided when downloading data. The URLs used in notification message should be pre-signed with the data publisher’s credentials and valid for the cache retention period (e.g., 24-hours).^[35]

In both cases, the URL provided in a notification message can be directly resolved without a user, or Global Cache, needing to take additional action such as providing credentials or authenticating.

Finally, note that if you are only publishing Core data, you may be able to entirely rely on the Global Caches to distribute your data. In such cases, your WIS2 Node may use IP-filtering to allow access only from Global Services. For more details, see section 2.6 Implementation and operation of a WIS2 Node.

There is no requirement for a WIS2 Node to publish notification messages about newly available data - the mechanism is available if needed (e.g., for real-time data exchange). Data archives published through WIS2 do not need to provide notification messages for data unless the user community have expressed a need to be rapidly notified about changes (e.g., the addition of new records into a climate observation archive).

However, notification messages must still be used to share discovery metadata with WIS2. Given that provision of metadata and subsequent updates is likely to be infrequent, it may be sufficient to "hand-craft" a notification message and publish it locally on an MQTT broker^[36] or with help from a GISC. See above for more details on publishing discovery metadata to WIS2.

Note that some data archives are categorised as Core data; for example, Essential Climate Variables. Core data may be distributed via the Global Caches. However, given that these provide only short-term hosting of data (e.g., 24-hours), Global Caches are not an appropriate mechanism to provide access to archives of Core data. The archive must be accessed directly via the WIS2 Node.

Registration and decomissioning of WIS2 Nodes must be approved by the Permanent Representative to WMO (PR) for the country or territory in which the WIS Centre resides. Where the WIS2 Node is part of a Data Collection or Production Centre (DCPC), the sponsoring WMO Programme or Regional Association shall be consulted.

WMO Secretariat will operate a WIS2 register as an authoritative list of WIS2 Nodes and Global Services.

The registration of a WIS2 Node involves the following steps:

A diagram of the process of registering a WIS2 Node is presented below.

Once a WIS2 Node has been registered and connected with the Global Services, it can procede to register the datasets it will publish via WIS2. To register a dataset, the authorized WIS2 Node publishes discovery metadata about the new dataset. Validation of the discovery metadata is completed by the Global Discovery Catalogues and Global Brokers automatically subscribes to the topics provided in the discovery metadata record. For more information, see How to provide discovery metadata to WIS2.

Once the dataset has successfully been registered, the WIS2 Node can procede to exchange data - see [_how_to_provide_data_in_wis2].

When decommissioning a WIS2 Node operators must ensure that obligations relating to data sharing within WIS continue to be met after the WIS2 Node is decommissioned, for example, by migrating these data sharing obligations to another WIS2 Node. In the case of a DCPC, this may mean the responsibilities are transferred to another Member.

The Centre Identifier (centre-id) is used in WIS2 to uniquely identify a participating WIS2 Node. The Centre Identifier must conform to the specification given in the Manual on WIS (WMO-No. 1060), Volume II, Appendix D: WIS2 Topic Hierarchy, section 7.1.6 Centre identification.

The Centre Identifier comprises two dash-separated tokens.

Token 1 is a Top Level Domain (TLD) defined by IANA^[39].

This is usually a simple choice for a Member. However, overseas territories require some thought. The recommended approach depends on the governance of the overseas territory. Take some French examples. Réunion is a French Department – it’s considered part of France, it uses the Euro. Here, we would use the “fr” TLD. New Caledonia is a French overseas territory with top-level-domain of “nc”. It has separate, devolved governance. The recommendation is to use “nc”. All that said, it’s a national decision which TLD to use.

Token 2 is a descriptive name for the centre and this may contain dashes (but not other special characters).

The descriptive name should be something recognisable - not only by our community, but by other users too. Basing things on the Web domain name is likely to ensure that centre identifiers remain unique within a particular country/territory. A UK example this time: the UK’s National Meteorological Service is the Met Office (http://www.metoffice.gov.uk), so “metoffice” is better than “ukmo”^[40]. Using the 4-letter GTS centre identifiers (CCCC) is not recommended because people unfamiliar with the GTS do not understand them.

The Centre Identifier specification says that larger organisations operating multiple centres may wish to register separate centre-ids for each centre. This is good practice. Keeping with the UK example, Met Office operates a National Meteorological Centre (NMC), 9 DCPCs (e.g., a Volcanic Ash Advisory Centre) and a WIS2 Global Service, so it’s important to split them out. For example:

Using a system name in the centre-id is not a good idea because these may change over time. Functional designations are long-term durable. Appending `-test may be used to designate test WIS Nodes.

Appending “-test” may be used to designate test WIS Nodes.

When configuring your WIS2 Node you need to consider how it will accessed by Global Services and Data Consumers.

Global Brokers must authenticate when they connect to the MQTT message broker in your WIS2 Node. Username and password credentials are used^[41]. When registering your WIS2 Node with the WMO Secretariat, you will need to provide these credentials. The WMO Secretariat will share these credentials with the Global Service operators and store them in the WIS register. You should not consider these credentials as confidential or secret.

Given that Global Brokers will re-publish notification messages provided by your WIS2 Node, you may decide to restrict access to the MQTT message broker. Global Brokers operate using a fixed IP address which allows you to permit them access using IP filtering^[42]. You must ensure that your MQTT message broker is accessible for more than one Global Broker to provide resilient transmission of notification messages to WIS2.

If your WIS2 Node is only publishing Core data^[43], you may also restrict access to your data server - instead, relying on the Global Caches to distribute your data. Similarly, Global Caches also operate on fixed IP addresses allowing connections from them to be easily identified. Again, you must ensure that access is given to more than one Global Broker to ensure resilience.

During registration, the WMO Secretariat will provide host names and IP addresses of the Global Services to enable configuration of access control.

Access controls may be implemented for Recommended data. You should use only the "security schemes" for authentication and authorization specified in OpenAPI^[44].

A WIS2 Node must be able to:

If contacted by the Global Montior via GISC for a performance issue, the WIS2 Node should provide metrics to the GISC and Global Monitor when service is restored to indicate resolution of the issue.

The availability of data and performance of system components within WIS2 are actively monitored by GISCs and the Global Monitor service to ensure proactive response to incidents and effective capacity planning for future operations.

WIS2 requires that metrics are provided using OpenMetrics – the de-facto standard ^[45] for transmitting cloud-native metrics at scale. Widely adopted, many commercial and open-source software components already come preconfigured to provide performance metrics using the OpenMetrics standard. Tools such as Prometheus and Grafana provide aggregation and visualisation of metrics provided in this form, making it simple to generate performance insights. The OpenMetrics standard can be found at openmetrics.io ^[46].

The WIS2 Global Services, namely the Global Broker, Global Cache, and Global Discovery Catalogue expose monitoring metrics on their respective service to the Global Monitor.

There is no requirement on WIS2 Nodes to provide monitoring metrics. However their WIS2 interfaces may be queried remotely by Global Services, which in turn can provide metrics on the availability of WIS2 Nodes.

Metrics for the WIS2 monitoring should follow the naming convention:

where program is the name of the responsible WMO Program and name is the name of the metric. Examples for WIS2 metrics can look like

The full set of the WIS2 monitoring metrics is given in WMO: WIS2 Metric Hierarchy ^[47]

In the following sections and for each Global Service, a set of metrics is defined. Each Global Service will provide those metrics. They will then be ingested by the Global Monitor.

To provide a Global Discovery Catalogue, members may use whichever software components they consider most appropriate to comply with WIS2 Technical Regulations.

To assist Members participation in WIS2, a free and open-source Global Discovery Catalogue Reference Implementation is made available for download and use. wis2-gdc builds on mature and robust free and open-source software components that are widely adopted for operational use.

wis2-gdc provides functionality required Global Discovery Catalogue, providing the following technical functions:

wis2-gdc is managed as a free and open source project. Source code, issue tracking and discussions are hosted in the open on GitHub: https://github.com/wmo-im/wis2-gdc.

Meteorological data is an essential input for public weather services and aviation services alike. WIS2 provides the mechanism for data exchange in WMO, while SWIM (System Wide Information Management) is the ICAO initiative to harmonize the provision of aeronautical, meteorological and flight information to support air traffic management (ATM).

Both WIS2 and SWIM support the similar outcomes relating to data exchange. However, there are differences in both approach and implementation.

Specifications for WIS2 are defined in the Manual on WIS (WMO-No. 1060), Volume II, and further elaborated in this Guide. Specifications for SWIM will be defined in the Procedures for Air Navigation Services – Information Management (PANS-IM) (ICAO Doc. 10199)^[50].

During the WIS2 transition phase (2025-2030), meteorological data published via WIS2 will automatically be published to the GTS via the WIS2 WIS2-to-GTS Gateways.

An organisation (e.g., the National Meteorological Service) that is responsible for providing meteorological data to WIS2 may be designated by the ICAO Contracting State as a responsible entity to provide aeronautical meteorological information into SWIM. Where requirements dictate, the organisation may provide regional capability on behalf of a group of countries or territories.

This section of the Guide outlines how such an organisation may efficiently fulfil the requirements in providing required data/information to the two systems. It proposes an interoperability approach between WIS2 and SWIM where meteorological data published via WIS2 can be automatically propagated to SWIM.

This Guide covers only how data from WIS2 can be published into SWIM. Consumption of information from SWIM services is not in scope.

This Guide also does not cover implementation details of the SWIM service - including, but not limited to:

This Guide will be updated as more information is made available from ICAO and/or recommended practices are updated.

Finally, it should be noted that the provision of aeronautical meteorological information and its exchange via the ICAO Aeronautical Fixed Service (AFS) is also out of scope as they are solely defined under the ICAO regulatory framework.

The Ocean Data and Information System (ODIS) is a federation of independent data systems coordinated by the International Oceanographic Data and Information Exchange (IODE) of IOC-UNESCO. This federation includes continental-scale data systems as well as those of small organisations. ODIS partners use Web architectural approaches to share metadata describing their holdings, services, and other capacities. In brief, IODE publishes guidelines on how to share metadata as linked open data, serialised in JSON-LD using schema.org^[63] semantics. ODIS nodes use these guidelines to publish their metadata catalogues on the Web. This allows all systems with Web connectivity to harvest and merge these catalogues, creating a global map of the ocean datascape. IODE harvests all metadata shared by ODIS partners, combines it as a knowledge graph, and processes this to export derivative products (e.g. diagnostic reports and cloud-optimised data products). The Ocean InfoHub (OIH) system is IODE’s reference implementation of a discovery system leveraging ODIS. The ODIS architecture and tools are Free and Open Source (FOSS), with regular releases published for the community.

To reach beyond the oceans domain, ODIS works with other data systems and federations, such as WIS2, to define sustainable data and metadata exchanges and - where needed - translators/converters. The resources needed to convert between such systems are developed in the open and in close collaboration with staff from those systems. These exchanges include ETL functions, to ensure that the bilateral exchange is mutually beneficial.

The efficient and effective provision of services relying on meteorological, climatological, hydrological and oceanographic information depends on a reliable information infrastructure. This infrastructure should be guided by community best practices and standards, including recommendations and requirements on sourcing, securing, managing, archiving, exchanging, and providing easy access to information. These terms and activities can be grouped under the term "information management" and this part of the Guide aims to provide high-level guidance on those activities. This is done by identifying and describing the fundamental principles of good information management and by highlighting the different stages of the information management lifecycle.

Note: The term "information" is used in a general sense and includes data and products.

High-level guidance on information management practices that apply in the context of information related to the Earth system is provided in this part of the Guide. Detailed technical information, such as specification of data formats or quality control and assurance methods, is provided in other parts of the Guide and in other WMO publications. These are referenced where applicable.

The Principles of information management are described below. Five focus areas are described in The information management lifecycle. These are:

This guidance is primarily aimed at personnel within WMO centres, with responsibility for planning and undertaking the creation or acquisition, stewardship, exchange and provision of information related to the Earth system.

Specifically, the guidance has five main target audiences across the information lifecycle:

Note: Information on the FAIR data principles can be found at: FAIR Principles - GO FAIR ^[64]

All information should be subject to a well defined and documented lifecycle. The governance of this process is often referred to as the information management lifecycle and this process helps organizations manage information throughout its full lifecycle, from planning, creation and acquisition through usage and exchange to archival and disposal.

The following sections describe two overarching themes, governance and documentation, that apply to all stages of the information lifecycle and then provides high level guidance split into 5 aspects:

Governance covers the rules that apply to managing information in a secure and transparent manner, documentation covers the act of recording the reasons for, and detail of, all operations in the information management process.

Information managers must be aware of the need to ensure that the technologies, hardware and software used do not become obsolete and must be aware of emerging data issues. This topic is discussed further in the WMO Guidelines on Emerging Data Issues (WMO-No. 1239).

Further information on information security and best practices can be found in the Guide to Information Technology Security (WMO-No. 1115).