Automated semantic annotations

[ml-evs]

Working group notes building on #18

[edan-bainglass]

We had a discussion regarding how to best introduce semantics at the workflow engine software level. In AiiDA, we've considered introducing semantics at the ORM layer, which has recently been enhanced with pydantic schemas. The idea is to upgrade our pydantic models with semantic support (done here already in the context of PREMISE, but also here as part of OO-LD).

@samwaseda presented the approach of semantikon to introducing semantics at the task and workflow levels. Semantikon moves validation and discovery from the software layer to the semantic layer (or rather opens these doors). This allows validation of workflow step consistency by means of semantic reasoning on input/output connections, as well as discovery by means of SPARCL queries on the connected knowledge graph. The latter enables queries outside of one's own ecosystem, and perhaps even extended beyond one's domain, depending on the level of connectivity made upstream (which is for ontology/domain experts to develop).

@superstar54 and @edan-bainglass poked sufficiently at semantikon. Currently, it appears sound and of sufficient interest to consider in AiiDA. Further testing required!

Potential deliverables

• A demonstration of semantikon integration in AiiDA's ORM models (a yield of semantically annotated schemas would be perfect!)
• A report of pro/cons of using semantikon (or any similar system) and its implications on developers/consumers of workflows

[edan-bainglass]

A comment regarding where the annotation burden lies. In either approach, the burden is on the workflow developer, not the workflow consumer. As one develops the tasks and from them composes workflows, one can introduce semantics much in the same way as introducing type hinting. The difference is that along with types, they now provide meaning, from which the above reasoning features (validation, discovery) are built.

[edan-bainglass]

So.... automation here is w.r.t the workflow consumer, less so for the devs. A win in my opinion.

[edan-bainglass]

@samwaseda let's fix a time this morning/afternoon for a chat. I'd like to show you what we do in AiiDA w.r.t schematics and get your opinion on the integration of semantikon in AiiDA.

[edan-bainglass]

After some discussion with @samwaseda, a somewhat obvious path for Semantikon integration in AiiDA was quickly identified, with a few caveats highlighted by @samwaseda (dict vs. tuple return types). However, there remain concerns in the AiiDA community (at least from @superstar54) if Semantikon's added features can capture the complexity of AiiDA's workflows (or realistic workflows in general). Here it is likely that more discussion is warranted, as such objections may be rooted in a possible disconnect between the understanding of AiiDA mechanics and that of semantic mechanics.

To clear up

• What are the added values of semantic annotations on the tasks and workflows of a WFMS (via any semantics-enabling package)?
• Is the burden on the workflow developer unavoidable? If so, can it be minimized somehow via an abstraction layer?

@samwaseda maybe you can already provide your comments on the above?
@superstar54 good if we can have a chat regarding this in the coming weeks to identify the pain points, also continuing the discussion here to leverage @samwaseda's expertise.

[samwaseda]

• What are the added values of semantic annotations on the tasks and workflows of a WFMS (via any semantics-enabling package)?

This is a similar question to “Why do I ever need to use a WfMS when I can run simulations on my own?”, because semantic annotations are more a question of universality. Ultimately, we are interested in being able to communicate outside of the programming realm, in which case currently we need to convert our data into a format people (e.g. experimentalists) can support. If we can use a more universal language, such as RDF with URI’s, then we don’t need a translation right from the beginning. Since there is no commonly used triple store, the benefits might not be quite obvious, but in the age of AI, which is based on semantic web, it’s a question of time to move to knowledge graphs in conjunction with well defined ontologies instead of local tabular data for us from my point of view.

• Is the burden on the workflow developer unavoidable? If so, can it be minimized somehow via an abstraction layer?

With semantikon, the burden is already a lot less, because there’s no need for conceptualization (assuming there are ontologists working on it), and no need for numerical realization. With this in mind, the only remaining burden, i.e. application of the semantic information on the nodes, has to be carried either by the node developer or the user. So yes, it cannot be avoided, but I don’t think it’s so much of burden. If you have other ideas, let me know!

[superstar54]

First, the idea of automated semantic annotations is great, and the Semantikon package looks interesting. As a workflow manager developer, my main concern is whether Semantikon can handle the dynamic nature of realistic workflows, particularly within AiiDA.

Semantikon seems to require a complete, static "recipe" (kind of a DAG) of the workflow before runtime to generate its knowledge graph. However, many workflow managers like AiiDA are inherently dynamic, meaning the full graph is only known during runtime.
aiida-core provides two entry points to write a workflow:

• WorkFunction: It is fully imperative. The sequence of operations is determined by standard Python logic at runtime, so no static "recipe" is available beforehand.
• WorkChain: it mixes declarative and imperative styles. While a WorkChain has a declarative outline, this only defines the high-level steps. The logic within each step is a black box; As a result, the detailed graph of calculations and their connections is not fully known beforehand.

Thus, Semantikon can not fetch a (complete) knowledge graph from them.

In aiida-workgraph, and the current PWD, we do provide a "recipe". But this is only for a simple workflow (with no flow-control logic). In aiida-workgraph, complex control flow (e.g., if/for/while) is often handled by generating a nested sub-workflow dynamically based on the output of previous calculations at runtime. From the perspective of the parent workflow, this sub-workflow is a black box until it is generated and run.

@samwaseda mentioned that one can avoid certain control flows (e.g., by avoiding if statements). While it could be theoretically possible, I am not convinced this is a practical solution. Requiring users to write more complex code to satisfy a tool's constraints undermines usability, which we want to avoid in a workflow system.

In short, there appears to be a gap between the need for a static, pre-runtime workflow "recipe" and the dynamic, runtime-defined nature of many essential AiiDA workflows.

How would we propose to bridge this gap? I am happy to discuss.

[samwaseda]

What are the ontological validations that AiiDA currently runs for the dynamic nodes?

[superstar54]

AiiDA's ORM data provides the schema (and Pydantic model) for each data node. But AiiDA doesn't perform formal ontological validation in the sense of validating data against a specific semantic schema or ontology.

AiiDA's validation is primarily focused on value-based checks at the level of individual process inputs/outputs. When defining a process, a developer can set the following for each port:

• valid_type: This ensures an input is a specific AiiDA data type (e.g., orm.Float, orm.StructureData).
• validator function: This is a custom function to check the value of the input (e.g., to ensure a orm.Float is positive).

These checks are performed on the inputs/outputs of every process. In the case of WorkChain, one can expose the ports from sub-processes to the top-level process, which helps to validate the inputs for the sub-process, but the actual data that flows between internal steps is only validated during runtime. Please refer to Validation and defaults for details.

I am not familiar with "ontological validations", @edan-bainglass may have more to add on this topic.

[edan-bainglass]

"Ontological validation" (not sure if there is a more formal name for this) would be acting on the meaning of objects, less so their types. Taking the example from Semantikon, suppose the following is integrated as a workflow step:

>>> EX = Namespace("http://example.org/")
>>>
>>> def get_speed(
...     distance: u(float, units="meter", uri=EX.distance),
...     time: u(float, units="second", uri=EX.time),
... ) -> u(float, units="meter/second", label="speed", uri=EX.speed):
...     speed = distance / time
...     return speed

Validating types (AiiDA-level, either via input/output specification, or internal data type annotation) can only validate that, for example, distance is a float.

Validating meaning can go further already here by checking that the input distance has the correct units, given that the output of the previous step is semantically annotated as yielding distance in "meters". Of course, the developer MUST follow the semantic contract requiring the implementation to yield distance in "meters". However, this requirement is not a concern of semantics (likely handled by separate experts).

Semantic validation can go even further. Suppose an input of a process is a molecule. This molecule can semantically be a biological one as part of a drug manufacturing workflow, or one studied for single-electron transistor (SET) application. Though the molecule IsA Molecule, it may have additional semantic properties per use case. AiiDA would require dedicated data types (BioMolecule, SetMolecule, etc.) to cover/encode semantics. Semantikon defers semantic annotation to spec time (for example, when specifying get_speed above), removing the need to establish additional types.