Common schemas for workflows: NOMAD PWD Parser/Schema

[JFRudzinski]

Handling Utility Nodes in Python Workflow Definition (PWD) Workflows

The Problem

Python Workflow Definition workflows contain utility nodes (e.g., python_workflow_definition.shared.get_dict, get_list) that serve as data transformation and parameter bundling functions. These nodes are implementation details that don't represent actual scientific computations but are necessary for workflow execution.

Why This Matters for NOMAD

In workflow visualization and analysis tools, these utility nodes create noise that obscures the actual scientific workflow logic. The PWD paper implicitly acknowledges this by omitting these nodes from workflow diagrams, focusing only on meaningful computational steps.

Current Challenge

The NOMAD PWD parser needs to decide how to deal with these utility nodes in order to simultaneously:

• Preserve complete workflow information for reproducibility
• Present clean scientific workflow visualizations for analysis
• Separate technical complexity from scientific complexity (core PWD philosophy)

One Possible Strategy: Hierarchical Workflows

1. Detection Pattern

Identify utility node patterns:

{
  "type": "function", 
  "value": "python_workflow_definition.shared.get_dict"
}

2. Composite Task Creation

Group utility nodes based on their functionality:

Parameter Bundling (get_dict)

Group downstream with target computational node:

Input → [QE Calculation Workflow] → Output
         ├── get_dict (parameter bundling)
         └── calculate_qe (computation)

Data Aggregation (get_list)

Group upstream with source computational nodes:

[Data Collection Workflow] → Output
 ├── calculate_qe_1 (computation)
 ├── calculate_qe_2 (computation) 
 ├── calculate_qe_3 (computation)
 └── get_list (aggregation)

Associated Problem with this approach: Non-unique grouping - The same get_list could be grouped as:

• One workflow with all 3 QE calculations
• Three separate workflows each with 1 QE + partial aggregation
• Mixed groupings based on different semantic criteria

This ambiguity makes consistent hierarchical construction challenging, unless a clear standard is set through associated PWD Node metadata.

[JFRudzinski]

Some other wishlist topics:

• Exchange PWD examples with actual data (minimal identifiers to calc files or paths)

For this Jan has added a interim functionality to attach the output data and path to WD. I will use this to build a prototype of working with PWD in NOMAD

• Additional AIDA-NOMAD, Jobflow-NOMAD mappings?

This is not needed in the short term. We cover interoperability through PWD workflow + parsing of individual (supported) calculation nodes (i.e., from standard simulation software). Non-supported nodes can always be supported in NOMAD, albeit using the native functionalities.

[jan-janssen]

For future reference, we implemented a first proposal for the Python Workflow Definition to include the output objects when they are JSON serializable as well as the directories where the calculations were computed.
https://github.com/pythonworkflow/proposal-for-data-export

[JFRudzinski]

For the main topic above, I need to open an issue to document the idea of tagging certain nodes, although an alternative suggestion from Janine was to utilize the data transfer itself to manage the visualization.

• open issue node tagging