Fusion Metadata Registry

Fusion Metadata Registry

Fusion Metadata Registry is a free to use metadata registry for SDMX supporting EDI, SDMX 2.0, 2.1 and 3.0.

Agile and scalable data operations, interoperable datasets, reduced duplication of effort through metadata sharing and reuse, more collaboration and fewer silos.

Quick start

Get Docker Desktop, run the command below, visit http://localhost:8080

docker run --name fmr -p 8080:8080 sdmxio/fmr-mysql:latest

Features

Metadata classes

  • Schemas - the SDMX core data model including Concepts, Codelists, DSDs, Constraints etc
  • Semantic metadata - names, descriptions, reference metadata, annotations, Hierarchies, Categories etc
  • Transformations - SDMX Structure Maps and VTL transformations
  • Business validation rules - VTL rulesets

Interfaces

  • SDMX REST API with extensions for added functionality
  • Web user interface for metadata browsing, authoring and maintainance, and registry administration

Data processing services

  • Schema validation
  • Transformation using SDMX Structure Maps
  • Conversion between SDMX data serialisation formats including XML, JSON and CSV

Deployment

Sponsored by the Bank for International Settlements

License

Web application: Public License Agreement
Source code: Apache 2.0 License Agreement
License Agreement

BIS Open Tech

Fusion Metadata Registry is published under the BIS Open Tech initiative.

The BIS Open Tech is a platform for sharing statistical and financial software as public goods, by promoting international cooperation and coordination. These software tools are developed according to international best practices and standards and can be reused and further developed in a wide variety of environments.

Why FMR?

Data quality automation

  • Powers ‘schema-aware’ tools including quality testers and transformers in data integration workflows
  • Tools can be code (e.g. Python pysdmx, ‘R’), apps (e.g. SDMX converter), web services or microservices

Scalability and agility

  • Schemas can evolve quickly – users and apps simply consult the registry for the latest iterations
  • Empowers business users to create and maintain schemas, semantic metadata and transformations

Metadata sharing and reuse

  • Single point of reference for useful foundational metadata such as standardised concepts. e.g. unit of measure
  • Reuse fosters consistency, avoids duplication of effort and reduces silos
  • And promotes concept harmonisation making datasets more interoperable

Governance and observability

  • Monitor and enforce data and metadata governance policies e.g. data modelling quality standards

Inside FMR

The heart of Fusion Metadata Registry is its structural metadata repository - essentially a database that implements the SDMX 3.0 information model.

All structural metadata objects are stored according to this model which provides a rigorous dependency graph that defines and enforces the metadata's referential integrity. This strongly enforced referential integrity provides confidence that the metadata model is always complete - a DSD for instance can never exist without its referenced Codelists.

FMR provides two interfaces

  • Web user interface: a web interface for interactively discovering and retrieving structures, structure authoring and maintenance, and system administration.
  • REST application programming interface: a REST web services API for automation that implements the SDMX v2 structure REST API specification, with additional extensions for creating and modifying structures, data validatio and data transformation.
    - FMR 10 implements the SDMX 2.1 registry specification.
    - FMR 11 implements the SDMX 3.0 registry specification.

In addition to its core role as a structural metadata registry, FMR's integrated data processing services support:

  • Data validation: submitted data is checked against nine separate quality rules including consistency with the DSD, and compliance with any Data Constraints
  • Data conversion: conversion of SDMX datasets between transmission formats including EDI, XML, JSON, CSV and Excel
  • Data mapping: transformation of a dataset to a different DSD using SDMX structure mapping

Finally for for data collectors, FMR's Excel data collection forms generator generates cross-tabulation data collection forms allowing data reporters to simply fill in the blank cells with the required observation values. The precise content and layout of these forms are defined by creating templates from which a personalised form is generated for each data provider.

Key Use Cases

Metadata management

  • Centralise and control structural, semantic and transformation metadata
  • Author and maintain metadata
  • Drive statistics production using centrally-controlled metadata
  • Provide an SDMX-compliant a REST structure API

Statistics production

  • Validate data for compliance with structural metadata, constraints and business rules
  • Execute complex data mapping (SDMX structure maps, VTL)
  • Convert data between formats (convert between SDMX formats)

Data collection

  • Collect data using metadata-driven Excel forms
  • Map incoming data into standardised structures

Data reporting

  • Prepare SDMX data for reporting
  • Validate data for compliance with the data collector's specifications prior to reporting

Capacity

  • No fixed limits on the number of users
  • No fixed limits on the volume of structures - in practice the volume of structures is limited by the available compute platform memory

Full feature list

FeatureFMR 10FMR 11
Internal Information Model
SDMX 2.1
SDMX 3.0
Structural Metadata Formats
SDMX-ML v3.0
SDMX-ML v2.1
SDMX-ML v2.0
SDMX-ML v1.0
SDMX-JSON v2.0
SDMX-JSON v1.0
EDI
Structural Metadata Management
Load structural metadata from file
Load structural metadata from URL
Author and maintain structural metadata using the web interface
Author and maintain structural metadata using FusionXL addin for Excel
ACID structure repository
Structural metadata import / export as SDMX
Structural metadata import / export as Excel
SDMX structures REST API
Structure storage (limited by available memory)
Structure comparison
Structure validation on import
Structure referential integrity enforcement
Tools to correct referential integrity issues (fix 'zombie' structures)
Time dependant constraints
Structure-level validity
Item-level validity
Subscription / notification of changes to structures via email or HTTP POST
Structure update transaction journalling
Rollback structure changes to a specified journalled transaction
Structural metadata time travel
SDMX Structures
Agency Schemes
Constraints
Categorisations
Category Schemes
Codelists
Concept Schemes
Data Consumer Schemes
Data Provider Schemes
Data Structure Definitions
Dataflows
Hierarchical Codelists
Metadata Structure Definitions
Metadataflows
Organisation Unit Schemes
Processes
Provision Agreements
Reporting Taxonomies
Structure Sets
Structure Map
Representation Map
Organisation Scheme Map
Concept Scheme Map (SDMX 3.0)
Category Scheme Map
Reporting Taxonomy Map
Value List
Hierarchy
Hierarchy Association
Metadata Constraint
Data Constraint
Metadata Constraint
Metadata Provision Agreement
Metadata Provider Scheme
Metadataset
'Extended' Structures (structures that are not part of the formal SDMX specification)
Validation Schemes
Reporting Templates
Publication Tables
Structural Metadata Storage Options
MySQL
Oracle
Microsoft SQL Server
Identity and Access Management
Microsoft Active Directory service integration
OpenLDAP directory service integration
Fusion Security directory service integration
Single-user mode (no directory service required)
Certificate authentication
SSO
Data
Load data from file
Load data from URL (file or REST API)
SDMX-ML v3.0 structure specific data format (XML)
SDMX-ML v2.1 structure specific data format (XML)
SDMX-ML v2.0 structure specific data format (XML)
SDMX-ML v2.1 generic data format (XML)
SDMX-ML v2.0 generic data format (XML)
SDMX-JSON v1.0 data format
SDMX-JSON v2.0 data format
SDMX-CSV v1.0 data format
SDMX-CSV v2.0 data format
EDI data format
Excel FusionXL data format
Interactive data load using the web user interface
Data storage
Data processing REST API (validation, conversion, mapping)
Data storage REST API
Plugin interface for adding custom data readers
Data Registration
SDMX data source registration
Validation and indexing of registered data sources
Reference Metadata
Store Reference Metadata Reports
SDMX-ML v2.1 reference metadata report format read
SDMX-JSON v2.0 reference metadata report format
Reference metadata submission API (POST)
Reference metadata retrieval API (GET)
Reference metadata deletion API (DELETE)
Data Validation
Structural compliance
Compliance of data values with the DSD
Compliance with constraint rules
Duplicate observations or series checks
Mandatory attributes
Validation Schemes - basic business rule validation expressions
Validation Schemes - advanced business rule validation expressions
Validation rules configuration - set what rules are applied and whether they prevent conversion or data publication
Plugin interface for adding custom validators
Data validation processing (load data, validate it against the defined structure and business rules, return a validation report plus valid and invalid series as separate output datasets)
Data validation REST API
SDMX-ML v2.1 Structure Specific data XML schema validation
SDMX-ML v3.0 Structure Specific data XML schema
VTL 2.0
Storage of VTL programs as SDMX structural metadata
Execution of VTL programs for validation or transformation of data
SDMX Structure Mapping
Concept Scheme Maps (SDMX 2.1)
Codelist Maps
Data Structure Maps
Dataflow Maps
Regex pattern mapping rules
N-to-N data maps
N-to-1 data maps
1-to-N data maps
1-to-1 data maps
Data Transformation
Data transformation processing (load data, transform it to a different structure using SDMX Structure Mapping and return the result)
Data transformation REST API
Data Conversion
Between supported SDMX formats and versions (SDMX-ML, SDMX-JSON, SDMX-CSV, EDI)
From Excel Reporting Template to SDMX formats
Quick convert to SDMX-ML 2.1 Structure Specific format
Conversion to and from Fusion Excel format for SDMX datasets
Data Reporting
Excel Reporting Templates - definition and generation
Excel Reporting Templates - validation and conversion of reported data
Excel Reporting Templates - load and storage of reported data
Excel Reporting Templates - configurable template security rules
Content Security / Access Controls
Structure creation and maintenance controls (Agency privileges)
Data load and modification controls (Data Provider privileges)
Data access controls at the dataflow, series or observation level
Structure access controls at the structure or item level
Public mode - Registry content is readable by both unauthenticated guest and authenticated users
Private mode - Registry content is accessible only to authenticated users
Auditing
SDMX structure transaction auditing
User activity auditing
Enterprise Deployment
Load balancer support (clustering multiple Registry instances behind a load balancer for resilience and load tolerance)
Kafka 'Structure Notification' Producer for publishing changes to structural metadata on a definable topic

2023 Roadmap

Prioritised General Enhancements workstream

Complete
Publication tables
FMR 11.1.4
New presentational metadata artefact describing curated tables of statistics for publication on websites or dashboards (more)
Author, validate, map and transform data using Excel
FMR 11.1.5
Load data from Excel workbooks in FusionXL data format
Use the FusionXL Excel add in for FMR to author, validate and transform SDMX datasets
Improved data processing performance
FMR 11.1.6
Improved performance and reduced memory requirements when loading, validating, mapping and converting large datasets
Reference metadata
FMR 11.2.0
Author Metadata Structure Definitions and Metadataflows
Store and retrieve reference metadata reports (metadatasets)
more...
Certificate authentication
FMR 11.2.0
Use X509 certificates to automatically authenticate with FMR's secure web services without the need to pass username / password credentials
SDMX Data Registration
Support for registering SDMX data sources
On work plan
Metadata time travel
Browse the structural metadata registry at a point of time, or query for structures as they were on a particular date
SDMX-JSON 3.0 structure format
Support for the SDMX 3.0 JSON transmission format for structural metadata
Improved error handling framework
Improved capture, handling and reporting of errors such as referential integrity violations
Structural metadata consistency checks and 'safe mode'
Strengthened checks on Registry start-up with the Registry placed in 'safe mode' if problems are detected
Performance improvements
Improved support for very large structures, improved performance when processing large datasets
Metadata time travel
Retrieve structual metadata as it was at a defined point in the past
Structure mapping default observation value
Option to to set a default ouput observation value for use where the input value is missing
SDMX-JSON 3.0 data format
Support for the SDMX 3.0 JSON transmission format for data excluding hierarchical reference metadata embedded within the data message
Virtual dataflows
Virtual dataflows describe a sub-cube of data similar in concept to a SQL view

Data processing microservices workstream

On work plan
Mapping microservice
Decoupled stateless SDMX mapping microservice

Recent features

See the full changelog for details of all new features and bug fixes.

11.6.0 - March 2023
  • Data Registration support
  • Publication Tables - support a scaling function applied to numerical observations
  • Transformation performance - option to skip the validation phase to improve performance and reduce memory pressure
  • Large codelists performance when using item validity
11.5.0 - December 2022
  • Mapping - Improve Metrics Report with IncUnmappedReport option
  • Publication Tables - support numerical observation formatting.
  • Publication Tables - support set scaling factor.
  • Publication Tables - support categorisation of publication tables via the FMR UI.
  • Support for "references-partial" in regular expressions.
  • Add auditing for structure queries.
11.4.0 - October 2022
  • Dependencies incompatible with the Apache 2.0 open source license removed.
  • Publication Tables - support casting of dates more than once in the same expression.
11.3.0 - September 202211.2.0 - July 202211.1.7 - July 2022
  • Interactive UI for synchronising structures between FMR environments such as TEST and LIVE - compare the contents of two FMR environments and push / pull selected differences
  • Publication Tables footnotes support
11.1.6 - June 2022
  • New REST API web service parameter to control how duplicate obervations are handled on validation and transformation
  • Introduction of MVStore for temporary storage and consolidation of datasets during processing operations delivering improved performance and reduced memory demand
11.1.5 - June 2022
  • MySQL 8 support
  • Publication Tables: Option to output rows with no data
  • Publication Tables: Dimension Value based on another Dimenisons variable value
  • Publication Tables: CPMI - Dynamic Rows and columns
  • Load data from Excel workbooks in Fusion Excel data format
11.1.4 - May 2022
  • Publication tables - new presentational metadata artefact describing curated tables of statistics for publication on websites or dashboards (more)
11.1.3 - May 2022

maintenance release

11.1.2 - April 2022

maintenance release

11.1.1 - February 2022

maintenance release

11.1.0 - February 2022
  • Code refactored to use the new SDMX 3.0 io.sdmx Java packages
  • New validation rule to check that OBS_PRE_BREAK is consistent with OBS_STATUS
11.0.2 and 11.0.3 - February 2022

maintenance releases

11.0.1 - January 2022
  • SDMX 2.1 categorisations against hierarchical code lists applied to all relevant SDMX 3.0 hierarchies
11.0.0 - December 2021
  • Internal information model changed to SDMX 3.0 (FMR 10 uses the SDMX 2.1 information model internally)
  • Support for storage of all SDMX 3.0 structures with the exception of Metadata Constraints, Concept Scheme Map, Category Scheme Map
  • Support for SDMX 3.0 XML format for structures
  • Support for SDMX 3.0 CSV format for data
  • Automatic conversion of SDMX 2.1 structure sets to the SDMX 3.0 mapping model on load
  • Semantic versioning - support for three-digit version numbers
  • JNDI database connection support
More...
10.7 - May 2021
  • Option to send structures as stubs on Kafka
  • Data validation - new option to supply the SDMX Sender ID
  • Mappings can be applied to multiple datasets if they refer to the same dataset
  • Support the creation of the TIME_SERIES attributes from an EDI file
10.6
  • Improved Apache Kafka integration
    Fusion Metadata Registry acts as an Apache Kafka producer publishing changes to the SDMX structures on definable topics. This is useful in metadata-driven statistics production architectures where distributed applications like data validators need to hold a synchronised copy of the master structural metadata.
    In 10.6, the Kafka interface has been enhanced to support all connection properties, email alerts are sent if the producer terminates, filters have been added to select the structures to publish and publication failure recovery has been strengthened. more
10.5
  • Transaction and user activity auditing
    Auditing of changes to SDMX structural metadata artefacts and user activity more
  • Structure transaction rollback
    Journalling of changes to the structural metadata content allowing administrators to rollback to a specific point in time more
  • Server integrity
    Checks the referential integrity of the structural metadata allowing administrators to fix problems found like 'zombie' artefacts that have broken references more
  • Docker image for containerised deployment
    Fusion Metadata Registry is now available as an image on Docker Hub for faster containerised deployment with minimal setup quick start

June 4, 2025

FMR 11.20.0 release

Fusion Metadata Registry update

March 28, 2025

FMR 11.19.5 release

Fusion Metadata Registry update

February 5, 2025

FMR 11.19.4 release

Fusion Metadata Registry update

December 5, 2024

FMR 11.19.3 release

Fusion Metadata Registry update

December 3, 2024

FMR 11.19.2 release

Fusion Metadata Registry update

November 18, 2024

FMR 11.19.1 release

Fusion Metadata Registry update

November 11, 2024

FMR 11.19.0 release

Fusion Metadata Registry update

October 3, 2024

FMR 11.18.1 release

Fusion Metadata Registry update

September 2, 2024

FMR 11.18.0 release

Fusion Metadata Registry update

August 6, 2024

FMR 11.17.2 release

Fusion Metadata Registry update

July 18, 2024

FMR 11.17.1 release

Fusion Metadata Registry update

July 12, 2024

FMR 11.17.0 release

Fusion Metadata Registry update

April 17, 2024

FMR 11.16.0 release

Fusion Metadata Registry update

February 16, 2024

FMR 11.15.0 release

Fusion Metadata Registry update

December 15, 2023

FMR 11.14.0 release

Fusion Metadata Registry update

November 21, 2023

FMR 11.13.0 release

Fusion Metadata Registry update

October 11, 2023

FMR 11.12.1 release

Fusion Metadata Registry update

October 5, 2023

FMR 11.12.0 release

Fusion Metadata Registry update

September 27, 2023

FMR 11.11.1 release

Fusion Metadata Registry update

September 21, 2023

FMR 11.11.0 release

Fusion Metadata Registry update

September 6, 2023

FMR 11.10.1 release

Fusion Metadata Registry update

September 1, 2023

FMR 11.10.0 release

Fusion Metadata Registry update

August 11, 2023

FMR 11.9.0 release

Fusion Metadata Registry update

July 13, 2023

FMR 11.8.1 release

Fusion Metadata Registry update

July 7, 2023

FMR 11.8.0 release

Fusion Metadata Registry update

June 2, 2023

FMR 11.7.0 release

Fusion Metadata Registry update

April 19, 2023

FMR 11.6.1 release

Fusion Metadata Registry update

April 13, 2023

FMR 11.6.0 release

Fusion Metadata Registry update

March 8, 2023

FMR 11.5.2 release

Fusion Metadata Registry update

January 30, 2023

FMR 11.5.1 release

Fusion Metadata Registry update

December 16, 2022

FMR 11.5.0 release

Fusion Metadata Registry update

October 14, 2022

FMR 11.4.0 release

Fusion Metadata Registry update

September 26, 2022

FMR 11.3.0 release

Fusion Metadata Registry update

September 9, 2022

FMR 11.2.1 release

Fusion Metadata Registry update

July 22, 2022

FMR 11.2.0 release

Fusion Metadata Registry update

July 6, 2022

FMR 11.1.7 release

Fusion Metadata Registry update

Docker

A self-contained Fusion Metadata Registry image complete with a MySQL database suitable for testing, personal use and light production work loads.

1. Install Docker Desktop

Execute the following commands using the Windows Command App, Windows PowerShell, Linux terminal or Mac terminal.

2. Start FMR in a container

docker run --name fmr -p 8080:8080 sdmxio/fmr-mysql:latest

Visit http://localhost:8080

The container will take between 1 and 2 minutes to start - so be patient!

There is a single Root User Account:
Username: root
Password: password

Log in to administer the system, and create, load and maintain SDMX structures.

5. When finished - stop the container

docker stop fmr

More on Docker Hub.

Install on Windows, Linux or Apple Mac

Fusion Metadata Registry is a Java web applications that will run on most Windows, Linux or Apple Mac machines with a minimum of 4GB memory.

There are three key pre-requisities:

  • A Java Runtime Environment (JRE or JDK) - Amazon Corretto 17 is a good free-to-use choice
  • A Java web application server - we recommend Apache Tomcat 9.0
  • A database service - MySQL, Oracle or SQL Server are supported

Take a look at the quick start guide

Quick Start Guide