• Home
  • Fusion Metadata Registry

  • Date :December 16, 2022
  • Reading Time :13 Minute

Fusion Metadata Registry

SDMX Structural Metadata Registry

Fusion Metadata Registry is a free to use structural metadata registry for SDMX supporting EDI, SDMX 1.0, 2.0, 2.1 and 3.0.

  • Structural Metadata Registry.
  • Author and maintain SDMX structures using the web user interface or programmatically through the REST API.
  • Data validation - validates that data files are correctly structured.
  • SDMX structure mapping - transform SDMX data to a different DSD and coding.
  • SDMX data conversion - convert data files between SDMX data formats and versions.
  • Download and install the Java web application, or deploy as a container.

Sponsored by the Bank for International Settlements

License
Web application: Public License Agreement
Source code: Apache 2.0
License Agreement

Download Docker

BIS Open Tech

Fusion Metadata Registry is published under the BIS Open Tech initiative.

The BIS Open Tech is a platform for sharing statistical and financial software as public goods, by promoting international cooperation and coordination. These software tools are developed according to international best practices and standards and can be reused and further developed in a wide variety of environments

Read More

FMR helps official statistics organisations externalise and gain control of their statistical structural metadata with the benefits of maintainability, re-use, standardisation, harmonisation and improved metadata and data governance.

FMR provides:

  • A controlled repository for SDMX structural metadata, which acts as a single source of truth for: automated metadata-driven data processing systems, metadata needed by data providers in data collections, structures for the purposes of standardisation, statistical data harmonisation, and communal benefit. Implements the SDMX 3.0 Registry specification.
  • Three integrated metadata-driven data processing services: data validation, data transformation (mapping data between structures) and conversion of data between SDMX data formats. Usable through the web UI or integrated into data automation processes using the REST API.
  • Metadata authoring and maintenance web UI, enabling metadata managers to browse, author and maintain their structural metadata model.

Fusion Metadata Registry 11 released

Fusion Metadata Registry version 11 implements the SDMX 3.0 specification.

Download get the Docker image, or read the release notes

Backward compatibility with SDMX 2.1 and FMR 10 has been preserved where possible allowing upgrade to FMR 11 with few changes to operating procedures and intergated systems / processes.
Note that FMR 11 is not compatible with existing FMR 10 databases requiring a fresh install and export / import of the structural metadata content.

introductory video

Features

Feature list
Function Feature Fusion Metadata Registry 10 Fusion Metadata Registry 11
Internal Information Model SDMX 2.1
SDMX 3.0
Structural Metadata Formats SDMX-ML v3.0
SDMX-ML v2.1
SDMX-ML v2.0
SDMX-ML v1.0
SDMX-JSON v2.0 As of FMR 11.1.2: AgencyScheme Categorisation CategoryScheme Codelist ConceptScheme ContentConstraint DataConsumerScheme Dataflow DataProviderScheme DataStructureDefinition ProvisionAgreement RepresentationMap StructureMap
The remainder of the structures are on the roadmap
SDMX-JSON v1.0
EDI
Structural Metadata Management Load structural metadata from file
Load structural metadata from URL
Author and maintain structural metadata using the web interface
Author and maintain structural metadata using FusionXL addin for Excel
ACID structure repository
Structural metadata import / export as SDMX
Structural metadata import / export as Excel
SDMX structures REST API
Structure storage (limited by available memory)
Structure comparison
Structure validation on import
Structure referential integrity enforcement
Tools to correct referential integrity issues (fix 'zombie' structures)
Time dependant constraints
Structure-level validity
Item-level validity
Subscription / notification of changes to structures via email or HTTP POST
Structure update transaction journalling
Rollback structure changes to a specified journalled transaction
Structural metadata time travel On roadmap
SDMX Structures Agency Schemes
Constraints Converted to Data Constraints on import
Categorisations
Category Schemes
Codelists
Concept Schemes
Data Consumer Schemes
Data Provider Schemes
Data Structure Definitions
Dataflows
Hierarchical Codelists Converted to Hierarchies on import
Metadata Structure Definitions
Metadataflows
Organisation Unit Schemes
Processes
Provision Agreements
Reporting Taxonomies
Structure Sets Converted to Structure Maps on import
Structure Map
Representation Map
Organisation Scheme Map
Concept Scheme Map (SDMX 3.0) On roadmap
Category Scheme Map On roadmap
Reporting Taxonomy Map On roadmap
Value List
Hierarchy
Hierarchy Association
Metadata Constraint On roadmap
Data Constraint
Metadata Constraint
Metadata Provision Agreement
Metadata Provider Scheme
Metadataset
'Extended' Structures (structures that are not part of the formal SDMX specification) Validation Schemes
Reporting Templates
Publication Tables
Structural Metadata Storage Options MySQL
Oracle
Microsoft SQL Server
Identity and Access Management Microsoft Active Directory service integration
OpenLDAP directory service integration
Fusion Security directory service integration
Single-user mode (no directory service required)
Certificate authentication
SSO
Data Load data from file
Load data from URL (file or REST API)
SDMX-ML v3.0 structure specific data format (XML) On roadmap
SDMX-ML v2.1 structure specific data format (XML)
SDMX-ML v2.0 structure specific data format (XML)
SDMX-ML v2.1 generic data format (XML)
SDMX-ML v2.0 generic data format (XML)
SDMX-JSON v1.0 data format
SDMX-JSON v2.0 data format On roadmap
SDMX-CSV v1.0 data format
SDMX-CSV v2.0 data format (some limitations)
EDI data format
Excel FusionXL data format
Interactive data load using the web user interface
Data storage
Data processing REST API (validation, conversion, mapping)
Data storage REST API
Plugin interface for adding custom data readers
Data Registration SDMX data source registration
Validation and indexing of registered data sources
Reference Metadata Store Reference Metadata Reports
SDMX-ML v2.1 reference metadata report format read (some limitations)
SDMX-JSON v2.0 reference metadata report format
Reference metadata submission API (POST)
Reference metadata retrieval API (GET)
Reference metadata deletion API (DELETE)
Data Validation Structural compliance
Compliance of data values with the DSD
Compliance with constraint rules
Duplicate observations or series checks
Mandatory attributes
Validation Schemes - basic business rule validation expressions
Validation Schemes - advanced business rule validation expressions
Validation rules configuration - set what rules are applied and whether they prevent conversion or data publication
Plugin interface for adding custom validators
Data validation processing (load data, validate it against the defined structure and business rules, return a validation report plus valid and invalid series as separate output datasets)
Data validation REST API
SDMX-ML v2.1 Structure Specific data XML schema validation
SDMX-ML v3.0 Structure Specific data XML schema
VTL 2.0 Storage of VTL programs as SDMX structural metadata
Execution of VTL programs for validation or transformation of data
SDMX Structure Mapping Concept Scheme Maps (SDMX 2.1)
Codelist Maps
Data Structure Maps
Dataflow Maps
Regex pattern mapping rules
N-to-N data maps
N-to-1 data maps
1-to-N data maps
1-to-1 data maps
Mapped Datasets Mapped Datasets - like SQL views
Virtual Dataflows Virtual Dataflows describe a sub-cube of a dataflow On roadmap
Data Transformation Data transformation processing (load data, transform it to a different structure using SDMX Structure Mapping and return the result)
Data transformation REST API
Data Conversion Between supported SDMX formats and versions (SDMX-ML, SDMX-JSON, SDMX-CSV, EDI)
From Excel Reporting Template to SDMX formats
Quick convert to SDMX-ML 2.1 Structure Specific format
Conversion to and from Fusion Excel format for SDMX datasets On roadmap
Data Reporting Excel Reporting Templates - definition and generation
Excel Reporting Templates - validation and conversion of reported data
Excel Reporting Templates - load and storage of reported data
Excel Reporting Templates - configurable template security rules
Content Security / Access Controls Structure creation and maintenance controls (Agency privileges)
Data load and modification controls (Data Provider privileges)
Data access controls at the dataflow, series or observation level
Structure access controls at the structure or item level
Public mode - Registry content is readable by both unauthenticated guest and authenticated users
Private mode - Registry content is accessible only to authenticated users
Auditing SDMX structure transaction auditing
User activity auditing
Enterprise Deployment Load balancer support (clustering multiple Registry instances behind a load balancer for resilience and load tolerance)
Apache Kafka Integration Kafka 'Structure Notification' Producer for publishing changes to structural metadata on a definable topic
Client-side API IM-JS JavaScript client-side programming library
SDMX-UI TypeScript client-side programming library (higher-level abstraction) On roadmap
Command Line Interface Fusion Command Line Interface for managing structural metadata
2023 Roadmap

Prioritised General Enhancements workstream

Complete
Publication tables
FMR 11.1.4
New presentational metadata artefact describing curated tables of statistics for publication on websites or dashboards (more)
Author, validate, map and transform data using Excel
FMR 11.1.5
Load data from Excel workbooks in FusionXL data format
Use the FusionXL Excel add in for FMR to author, validate and transform SDMX datasets
Improved data processing performance
FMR 11.1.6
Improved performance and reduced memory requirements when loading, validating, mapping and converting large datasets
Reference metadata
FMR 11.2.0
Author Metadata Structure Definitions and Metadataflows
Store and retrieve reference metadata reports (metadatasets)
more...
Certificate authentication
FMR 11.2.0
Use X509 certificates to automatically authenticate with FMR's secure web services without the need to pass username / password credentials
SDMX Data Registration
Support for registering SDMX data sources
On work plan
Metadata time travel
Browse the structural metadata registry at a point of time, or query for structures as they were on a particular date
SDMX-JSON 3.0 structure format
Support for the SDMX 3.0 JSON transmission format for structural metadata
Improved error handling framework
Improved capture, handling and reporting of errors such as referential integrity violations
Structural metadata consistency checks and 'safe mode'
Strengthened checks on Registry start-up with the Registry placed in 'safe mode' if problems are detected
Performance improvements
Improved support for very large structures, improved performance when processing large datasets
Metadata time travel
Retrieve structual metadata as it was at a defined point in the past
Structure mapping default observation value
Option to to set a default ouput observation value for use where the input value is missing
SDMX-JSON 3.0 data format
Support for the SDMX 3.0 JSON transmission format for data excluding hierarchical reference metadata embedded within the data message
Virtual dataflows
Virtual dataflows describe a sub-cube of data similar in concept to a SQL view

Data processing microservices workstream

On work plan
Mapping microservice
Decoupled stateless SDMX mapping microservice
Recent features

See the full changelog for details of all new features and bug fixes.

11.6.0 - March 2023
  • Data Registration support
  • Publication Tables - support a scaling function applied to numerical observations
  • Transformation performance - option to skip the validation phase to improve performance and reduce memory pressure
  • Large codelists performance when using item validity
11.5.0 - December 2022
  • Mapping - Improve Metrics Report with IncUnmappedReport option
  • Publication Tables - support numerical observation formatting.
  • Publication Tables - support set scaling factor.
  • Publication Tables - support categorisation of publication tables via the FMR UI.
  • Support for "references-partial" in regular expressions.
  • Add auditing for structure queries.
11.4.0 - October 2022
  • Dependencies incompatible with the Apache 2.0 open source license removed.
  • Publication Tables - support casting of dates more than once in the same expression.
11.3.0 - September 2022 11.2.0 - July 2022 11.1.7 - July 2022
  • Interactive UI for synchronising structures between FMR environments such as TEST and LIVE - compare the contents of two FMR environments and push / pull selected differences
  • Publication Tables footnotes support
11.1.6 - June 2022
  • New REST API web service parameter to control how duplicate obervations are handled on validation and transformation
  • Introduction of MVStore for temporary storage and consolidation of datasets during processing operations delivering improved performance and reduced memory demand
11.1.5 - June 2022
  • MySQL 8 support
  • Publication Tables: Option to output rows with no data
  • Publication Tables: Dimension Value based on another Dimenisons variable value
  • Publication Tables: CPMI - Dynamic Rows and columns
  • Load data from Excel workbooks in Fusion Excel data format
11.1.4 - May 2022
  • Publication tables - new presentational metadata artefact describing curated tables of statistics for publication on websites or dashboards (more)
11.1.3 - May 2022

maintenance release

11.1.2 - April 2022

maintenance release

11.1.1 - February 2022

maintenance release

11.1.0 - February 2022
  • Code refactored to use the new SDMX 3.0 io.sdmx Java packages
  • New validation rule to check that OBS_PRE_BREAK is consistent with OBS_STATUS
11.0.2 and 11.0.3 - February 2022

maintenance releases

11.0.1 - January 2022
  • SDMX 2.1 categorisations against hierarchical code lists applied to all relevant SDMX 3.0 hierarchies
11.0.0 - December 2021
  • Internal information model changed to SDMX 3.0 (FMR 10 uses the SDMX 2.1 information model internally)
  • Support for storage of all SDMX 3.0 structures with the exception of Metadata Constraints, Concept Scheme Map, Category Scheme Map
  • Support for SDMX 3.0 XML format for structures
  • Support for SDMX 3.0 CSV format for data
  • Automatic conversion of SDMX 2.1 structure sets to the SDMX 3.0 mapping model on load
  • Semantic versioning - support for three-digit version numbers
  • JNDI database connection support
More...
10.7 - May 2021
  • Option to send structures as stubs on Kafka
  • Data validation - new option to supply the SDMX Sender ID
  • Mappings can be applied to multiple datasets if they refer to the same dataset
  • Support the creation of the TIME_SERIES attributes from an EDI file
10.6
  • Improved Apache Kafka integration
    Fusion Metadata Registry acts as an Apache Kafka producer publishing changes to the SDMX structures on definable topics. This is useful in metadata-driven statistics production architectures where distributed applications like data validators need to hold a synchronised copy of the master structural metadata.
    In 10.6, the Kafka interface has been enhanced to support all connection properties, email alerts are sent if the producer terminates, filters have been added to select the structures to publish and publication failure recovery has been strengthened. more
10.5

Inside Fusion Metadata Registry

The heart of Fusion Metadata Registry is its structural metadata repository - essentially a database that implements the SDMX 3.0 information model.

All structural metadata objects are stored according to this model which provides a rigorous dependency graph that defines and enforces the metadata's referential integrity. This strongly enforced referential integrity provides confidence that the metadata model is always complete - a DSD for instance can never exist without its referenced Codelists.

FMR provides two interfaces:

  • Web user interface: a web interface for interactively discovering and retrieving structures, structure authoring and maintenance, and system administration.
  • REST application programming interface: a REST web services API for automation that implements the SDMX v2 structure REST API specification, with additional extensions for creating and modifying structures, data validatio and data transformation.

FMR 10 implements the SDMX 2.1 registry specification.
FMR 11 implements the SDMX 3.0 registry specification.

In addition to its core role as a structural metadata registry, FMR's integrated data processing services support:

  • Data validation: submitted data is checked against nine separate quality rules including consistency with the DSD, and compliance with any Data Constraints
  • Data conversion: conversion of SDMX datasets between transmission formats including EDI, XML, JSON, CSV and Excel
  • Data mapping: transformation of a dataset to a different DSD using SDMX structure mapping

Finally for data collectors, FMR's Excel data collection forms generator generates cross-tabulation data collection forms allowing data reporters to simply fill in the blank cells with the required observation values. The precise content and layout of these forms are defined by creating templates from which a personalised form is generated for each data provider.

Key use cases

Structural metadata management
  • Centralise and control structural metadata
  • Author and maintain structural metadata
  • Drive statistics production using centrally-controlled metadata
  • Provide an SDMX-compliant a REST structure API
Statistics production
  • Validate data for compliance with structural metadata, constraints and business rules (not VTL)
  • Execute complex data mapping (SDMX structure maps, not VTL)
  • Convert data between formats (convert between SDMX formats)
Data collection
  • Collect data using metadata-driven Excel forms
  • Map incoming data into standardised structures
Data reporting
  • Prepare SDMX data for reporting
  • Validate data for compliance with the data collector's specifications prior to reporting
Capacity and Limits
  • No fixed limits on the number of users
  • No fixed limits on the volume of structures - in practice the volume of structures is limited by the available compute platform memory
A self-contained Fusion Metadata Registry image complete with a MySQL database suitable for testing, personal use and light production work loads.
1. Install Docker Desktop
Execute the following commands using the Windows Command App, Windows PowerShell, Linux terminal or Mac terminal.
2. Download the Fusion Metadata Registry Docker image
docker pull metadatatechnology/fmr-mysql:latest
3. Create a container called 'fmr'
docker container create --name fmr --publish 8080:8080 metadatatechnology/fmr-mysql:latest
4. Start the container
docker start fmr

Point your browser at http://localhost:8080

The container will take between 1 and 2 minutes to start - so be patient!

There is a single Root User Account:
Username: root
Password: password

Log in to administer the system, and create, load and maintain SDMX structures.

5. When finished - stop the container
docker stop fmr

More on Docker Hub.

Install on Windows, Linux or Apple Mac

Fusion Metadata Registry is a Java web applications that will run on most Windows, Linux or Apple Mac machines with a minimum of 4GB memory.

There are three key pre-requisities:

  • A Java Runtime Environment (JRE or JDK) - Amazon Corretto 8 is a good free-to-use choice
  • A Java web application server - we recommend Apache Tomcat 8.0, 8.5 or 9.0
  • A database service - MySQL, Oracle or SQL Server are supported

Take a look at the quick start guide

Quick Start Guide