Stand-alone usage or re-purposing of parser modules

[JFRudzinski]

Spurred by some questions during the speaker session, let's exchange about possibilities of parser (re)usage and discuss about existing or missing tools for re-purposing parser modules.

[JFRudzinski]

Stand-alone usage of NOMAD's Parser Plugins

NOMAD's major "legacy" computational parser repos: electronic-parsers, atomistic-parsers, workflow-parsers (a refactoring of the simulation schema nomad-simulations and modernization of the parser modules nomad-parser-plugins-simulation is planned for release in 2026).

See the corresponding README's for information about the codes supported in each module.

Installation

The legacy parser modules are installable with pip (use venv or conda env, python >= 3.10, it is highly recommend to use uv):

uv pip install --upgrade pip
uv pip install nomad-parser-plugins-electronic
uv pip install nomad-parser-plugins-atomistic
uv pip install nomad-parser-plugins-workflow

Execution

The parser modules depend on the nomad-lab python package. To run the parser, you can use the following command:

nomad parse --show-archive <path-to-mainfile>

The mainfile is the uniquely identifying file for NOMAD to determine the source of the data, see NOMAD Docs > Supported Parsers for mainfile specifications.

(Note: The core NOMAD infrastructure also contains a matching parser interface that performs the mainfile identification automatically. However, this is not available with the current, simplified parser usage). Correction: When you do nomad parse nomad-lab will pick up the parser specs from the plugin and does the matching unless you specify directly the parser.

You can find plenty of test data in the corresponding repo's under tests/data/<code-name>/.

[ladinesa]

This is not quite correct: (Note: The core NOMAD infrastructure also contains a matching parser interface that performs the mainfile identification automatically. However, this is not available with the current, simplified parser usage). When you do nomad parse nomad-lab will pick up the parser specs from the plugin and does the matching unless you specify directly the parser.

[JFRudzinski]

Ah, ok, even better than I thought then, thanks for the clarification @ladinesa !

[JFRudzinski]

More info on nomad parse: https://nomad-lab.eu/prod/v1/docs/reference/cli.html#nomad-parse

[ndaelman-hu]

The only weakness to re-using our parsers is that the NOMAD dependency is still quite large: 2.2 MB as listed in PyPi, though this may include dev options too. That aside, the JSON output that you showed with the CLI works quite well standalone, though there are a few redundant fields (only useful within NOMAD infrastructure).

[ml-evs]

Just to chime in from the datatractor side about how this could be wrapped up/exposed (I don't think a 2 MB dep is a dealbreaker by any means, especially when we are often used to 800 MB+ pytorch etc...).

NOMAD could programmatically generate a list of supported file formats following our schema from the list at https://nomad-lab.eu/prod/v1/docs/reference/parsers.html (in the simplest case this is just a list of identifiers), then register nomad-parsers (or whatever subset for each domain) as the programmatic extractor definition that supports those files. It would look something like:

---
id: >-
    nomad-parsers
name: >-
    NOMAD
description: >-
    lorem ipsum
supported_filetypes:
    - id: qe-pw-in
    - id: qe-pw-out
    - id: vasp-in
license:
    spdx: Apache-2.0
citations:
    - uri: doi:10.21105/joss.05388
      creators:
          - M. Scheidgen
          - others
      title: 'NOMAD: A distributed web-based platform for managing materials science research data'
      type: article
source_repository: https://github.com/nomad-coe
documentation: https://nomad-lab.eu/prod/v1/docs
usage:
    - method: cli
      command: nomad parse --show-archive {{ input_path }}
    - method: python
      setup: nomad.electronicparsers.vasp
      command: nomad.electronicparsers.vasp.VASPParser({{ input_path }})
installation:
    - method: pip
      packages:
          - nomad-parser-plugins-electronic ~= 1.0

with corresponding entries created for vasp-in.

[JFRudzinski]

Thanks @ml-evs ! I would opt implement this as we roll out our new parsers, I have created an initial issue here so that we don't forget

[jan-janssen]

Is there any option for nomad-parser-plugins-atomistic and nomad-parser-plugins-electronic to no longer depend on nomad-lab? At the moment both depend on nomad-parser-plugins-simulation which internally again depends on nomad-lab.

[JFRudzinski]

@ladinesa Can answer more definitively, but I do not think that this is realistic. The way that our system is built up in a scalable way for the community to expand via plugins is to have a core set of tools in nomad-lab to build from. Our parsers already utilize these tools in a semi-standard.

In the future our new parsers will be (initially) stored in nomad-parser-plugins-simulation and they will largely utilize a mapping annotation tools from nomad-lab. There would be the possibility to pluck out an individual parser for easier external dev contributions, but this does not help the nomad-lab dependency.

It may be possible to slim down the dependencies in some way, but tbh this is not a priority for us. If there is a clear impactful use case, then we could sit and discuss at least what the requirements for this use case are, and what might be possible.

[ladinesa]

That is correct, nomad-lab contains all the core functionalities required by the parsers. Removing it directly and rely on the transitive dependency on the other plugin is basically the same. Your package manager will resolve the one that satisfies both.

[JFRudzinski]

Here's your complete breakdown of the core NOMAD Lab dependencies only (what you get with pip install nomad-lab):

📊 Core NOMAD Lab Dependencies Analysis
🎯 Total Size: ~682 MB across 31 packages
📂 Size Breakdown by Category:
🎨 Visualization (45.3% - 309 MB)

plotly (91 MB) - Interactive scientific plots
kaleido (219 MB) - Static image export for plots
🔬 Scientific Computing (42.4% - 289 MB)

scipy (139 MB) - Scientific algorithms
numpy (69 MB) - Array computing foundation
pandas (58 MB) - Data analysis
ase (12 MB) - Atomic structure handling
xarray (10 MB) - Multi-dimensional data
pint (1.4 MB) - Physical units
⚗️ Materials Science (5.6% - 38 MB)

pymatgen (25 MB) - Crystal structure analysis
matid (13 MB) - Materials classification
📄 Data Formats (4.1% - 28 MB)

h5py (15 MB) - HDF5 scientific data
lxml (11 MB) - XML parsing (CIF files)
openpyxl (2 MB) - Excel compatibility
🔧 Performance & Monitoring (1.2% - 8 MB)

py-spy (7 MB) - Production profiling
pyinstrument (0.6 MB) - Statistical profiler
🎯 Key Insights:
Heavy Components:

Visualization dominates at 45% (309 MB) - mainly kaleido for plot export
Scientific computing core at 42% (289 MB) - numpy/scipy/pandas foundation
Efficient Components:

Web framework is lightweight (4 MB total)
Authentication/security minimal (1 MB)
Most utilities under 1 MB each
🌐 Scope Analysis:
Global Usage (used throughout NOMAD):

numpy, scipy, pandas - numerical foundation
pydantic, httpx - API/validation core
pyyaml, pint - configuration and units
Parsing-Focused (specific to file parsing):

pymatgen, ase, matid - materials science
h5py, lxml - scientific file formats
Visualization-Only:

plotly, kaleido - could be optional in headless environments
Infrastructure-Ready:

elasticsearch-dsl, python-keycloak - ready for full deployment
This shows NOMAD's core is efficiently designed - most space goes to essential scientific computing and visualization, while keeping the framework itself lightweight at under 50 MB total.

[JFRudzinski]

We are looking into reducing the visualization dependencies to some extent. But the other bulk of dependencies are coming from scientific computing which is essential to the parsers. I would be interested to know though if someone thinks one of these may not be essential.

@jan-janssen if your installation via conda is installing JupyterLab and such, this means that you are installing the optional dependencies. I would be happy to look into this if you send/remind me your approach