SDMX Metadata AI Assistant (MAIA): enhancing statistical metadata

SDMX Metadata AI Assistant (MAIA): enhancing statistical metadata

The SDMX Metadata AI Assistant (MAIA) is a Python-based tool powered by OpenAI GPT, specifically designed to address the challenges of managing and editing statistical metadata. Leveraging AI, SDMX MAIA simplifies metadata workflows by automating key tasks such as checking syntax, consistency checks and formatting adjustments.

What is the SDMX Metadata AI Assistant (MAIA)?

The SDMX Metadata AI Assistant (MAIA) is a Python-based tool powered by OpenAI GPT, specifically designed to address the challenges of managing and editing statistical metadata. Leveraging AI, SDMX MAIA simplifies metadata workflows by automating key tasks such as checking syntax, consistency checks and formatting adjustments. MAIA provides an efficient Streamlit app interface to edit, validate, and format metadata automatically, simplifying metadata management within statistical production environments. Fully compliant with the Statistical Data and Metadata eXchange (SDMX) standard, MAIA ensures seamless integration into statistical pipelines, supporting the efficient exchange and harmonisation of metadata.

Use Case

The SDMX Metadata AI Assistant (MAIA)’s primary use cases include enhancing the quality and readability of metadata. Users can upload their data through SDMX-ML files along with their Data Structure Definitions (DSDs), select attributes for editing and allow MAIA to automatically validate and refine the content. By offering features such as customisable AI assistants, interactive before/after comparisons and dynamic reporting in SDMX-ML format, MAIA empowers statisticians to maintain high-quality metadata while saving time and resources. Whether addressing metadata consistency issues, aligning metadata with statistical frameworks or preparing data for dissemination, MAIA is a robust and user-friendly solution for modern statistical pipelines.

Metadata quality improvement

  • Detect and correct grammar, syntax, and formatting inconsistencies
  • Standardize terminology, date formats, and textual structure

Metadata transformation

  • Apply large-scale updates to selected attributes (e.g. COMPILATION)
  • Shorten or rephrase text to meet SDMX character constraints

Statistical production support

  • Integrate cleaned metadata back into SDMX pipelines
  • Provide auditable before/after comparison and JSON run logs

Integration with SDMX

MAIA operates directly on SDMX-ML 2.1 metadata and Data Structure Definition (DSD) files. It uses the structural information in the DSD to ensure attribute alignment and proper parsing.

Through the pysdmx library, MAIA can read, modify, and output SDMX files while maintaining schema validity. This tight integration makes it ideal for use in national and international statistical systems that adopt SDMX standards for metadata exchange.

AI engine

At its core, MAIA employs OpenAI’s GPT models to analyze and refine SDMX metadata. The engine applies predefined and customizable prompts to:

  • Fix language and formatting
  • Harmonize content across datasets
  • Log all operations for reproducibility

Each execution generates both before/after metadata views and a performance log capturing timestamps, assistant model, and processing duration.

Features

SDMX MAIA is designed with the following features:

📝 Metadata editing

  • Perform grammar, syntax, and consistency checks to ensure high-quality metadata
  • Streamline processes to improve compilers’ efficiency

🤖 Custom AI assistant

  • Configure your own AI assistant with customisable knowledge bases and instructions
  • Tailor the model to meet specific user requirements

🔗 End-to-end SDMX integration

  • Seamlessly input metadata from SDMX using pysdmx
  • Generate outputs in SDMX-ML 2.1 format

📊 Transparency

  • Automatically save each run as a JSON file
  • Maintain a clear and auditable record of all executions

🔍 Before/after interactive comparison

  • Easily compare in interactive mode the metadata changes before and after edits
  • View results in both terminal and HTML formats for flexibility

🖥️ Streamlit UI

  • An intuitive, web-based workflow execution interface
  • Designed for application designers to simplify processes

📊 Run status and metrics

  • Display logging and performance metrics directly in the UI
  • Save detailed logs to logs/ and performance metrics to metrics_performance.json for further analysis

This application is fully developed in Python and is built upon four key dependencies:

  • Streamlit: Powers the interactive and intuitive web interface
  • Pysdmx: Facilitates the handling and retrieval of metadata from SDMX sources
  • LangChain: Integrates AI workflows, enabling seamless interaction with language models
  • OpenAI: Provides access to OpenAI’s advanced AI assistant models for enhanced natural language processing capabilities

Using SDMX MAIA involves the following workflow:

  1. Upload and configure in three simple steps:
  • Upload SDMX files, including the data and the related Data Structure Definition in SDMX format;
  • Select the attribute to modify available from the list, for example “title”, “compilation” or “methodology”;
  • Configure technical settings, including the OpenAI assistant, API keys and other preferences.
  1. Customise the AI assistant:
  • Customise the AI Assistant by providing specific instructions and uploading any additional files to enhance the model’s knowledge base;
  • To ensure agility and fully leverage OpenAI’s features, this step is performed on the OpenAI platform.
  1. Review and finalise:
  • Once the editing process is complete, the application provides an interactive interface to review changes;
  • The diff-and-diff feature highlights additions in green and deletions in red. Users can directly edit, overwrite, or revert changes for each modified metadata item;
  • Finally, the application generates an updated SDMX file, ready to be ingested back with the new metadata.

Installation

Access to MAIA source code and installation instructions is available upon request by sending an email to contact.sdmx.io@bis.org .

An installable package will be made available in the near future.

Disclaimer

Users must ensure that no confidential or restricted information is uploaded. Files are processed through the OpenAI API, and institutional data-handling policies apply.

This project is licensed under the Apache 2.0 License.