AI/LLM-based tooling and best practices for semantic data pipelines

[ml-evs]

Working group notes building on #3

[ml-evs]

List of initiatives/platforms in the room:

• datalab
• OPTIMADE
• datatractor
• OpenBIS
• PREMISE
• PSDI data conversion service
• SciLog
• SciCat
• Kadi4Mat
• Herbie ELN
• ELNFileFormat
• Chemmotion
• Revvity Signals
• Open Semantic Lab
• pyiron
• AiiDA
• AiiDA lab
• BattINFO
• Battery 2030+

LLM-y stuff:

1. ELN text + ontology -- LLM -> structured/schematised data (Codename: Herbert)
   • sub agents that do chemistry
2. RDM reviewer agent for data recording and publishing (Codename: Robert)
   • potentially uses Herbert's agents
3. Agent tool scaling -- annotating tools vs platform documentation vs MCP?
   • using Herbert and RDM agent as examples
4. RAG Agent for Zenodo battery community
5. Battery use case (Wednesday -- combine with instrument control session)

Non-LLMy stuff:

I. Brainstorming ELN interop use cases -- ELNFileFormat breakout? (Edgar the ELN)
II. Best practices for semantic annotation: does ontology consistency actually matter?

Potential deliverables:

• showcase of structured data from Herbert (@martin-held-hereon)
• 1-page report on ELN Interoperability (w/ ELNFileFormat) suitability/use cases (@samikanza)
• 1-page report on battery use case (RAG Agent for Zenodo Battery 2030+/POLIS Communities + Instrument control + metadata extraction) (Adam)
  • How to zoom out from specific technique annotation -- potentially use round-robin dataset
• GitHub repository with agent implementation of RDM reviewer (@ml-evs)

Breakouts for the week:

• Best practices for semantic annotation [breakout: Wednesday 14:00] (@samikanza)
  • Pragmatic first steps and plateaux for linking data
  • Peer review of show-and-tell of current approach going from free text, structure, semantics, linked data ecosystem for given domain
• datatractor: registering extractors from e.g., PSDI, PSI, NOMAD [breakout: Wednesday 16:30] (@PeterKraus / @ml-evs)
• instrument control [breakout: Wednesday 11:00]

[martin-held-hereon]

All the non-LLMy stuff is here:

• Protocols and guidelines for semantic data exchange #22
• Awesome Links #30

[martin-held-hereon]

All LLM prompts, codes etc are in the "herbert" repo:
https://github.com/MADICES/herbert

[martin-held-hereon]

Summary 21.+22.10.2025:

• Agnes & Tom checked LLM's ability of structuring Wikipedia texts of carbon and oxygen into JSON
  https://github.com/MADICES/herbert/tree/main/agnes
• Martin & Sepideh tested a public standard prompt by Google to structure Chemotion free-text into a JSON
  https://github.com/MADICES/herbert/tree/main/Chemotion%20input%20and%20output/input_free_text
  -- looked good, needs to be thoroughly checked and improved

[martin-held-hereon]

Summary Martin's work 23.10.2025
(including input by others)

• created two very small example texts, one for training und testing
• created a turtle annotation of the training example text
• all of the above matches the example ontology by W3C
• using a google prompt provided by Sepideh
• testing prompts in Copilot for json (v1) & turtle (v2-5) output
• increasing the information of the prompts from (v1 & v2) just key to (v3) with exemplary text and turtle file to (v4) with ontology file to (v5) with ontology link.
• compared it to handwritten turtle output
  • exemplary text + TTL file (v3) improve consistency
  • exemplary text + TTL file + ontology as TTL (v4) are nearly identical to handwritten output and 99% reproducible --> best
  • exemplary text + TTL file + ontology as link (v5) disregards the ontology

[martin-held-hereon]

To Do:

• try simple example with ontology link and ontology file in Hereon's RAG Chatbot
• write prompt for Chemotion entry
• create exemplary owl file for a Chemotion entry
• check whether uploading 7 ontologies works

[ml-evs]

Summary of work on rdm-agent

Simple langchain-based scaffold for "I have some data in a directory that I would like to be reviewed", with the model prompted to consider aspects of FAIR principles, dark/missing data or even missing fields from a schema (e.g., a processing step without including the processing parameters). The agent has "tools" (Python functions) available to it that can perform subtasks, e.g., if a schema is provided, run a schema validation of the data provided.

Next steps:

• new examples of data can be added at https://github.com/MADICES/rdm-agent/tree/main/examples, with "agent runs" committed in as separate files in the responses subdirectory, where ideally the filename indicates the underlying LLM that was used.
• new tools can be provided to the LLM to guide it towards certain review aspects, e.g., resolving links/ontologies, dealing with "real data" from RO-Crates or otherwise, or whatever else people can imagine.

[tomlunderwood]

Summary of work on Herbert 24/10/25

A pull request has been made for the MADICES/herbert repo containing initial code for the 'agent' Herbert. Hopefully this pull request will be merged into main soon.

Key features of this initial version are as follows:

• The current version of Herbert will take a text file and generate JSON-LD corresponding to the scientific content of the text file. The JSON-LD may use schemas or ontologies Herbert has found online. At this point Herbrt probably generates bad JSON-LD; more investigation is needed.
• In principle Herbert can use any of the standard LLMs as a backend. However in the example which accompanies the code Claude is used. Note that to run the example code an API key for Claude is required.
• The code closely resembles that of MADICES/rdm-agent. For instance, it leverages LangChain and the agent can use LangChain tools to perform certain tasks.
• For now only Herbert only has access to one tool: a tool to read a text file and extract its contents as a string - a string which then is passed to the LLM to convert to JSON-LD. Hence any generation of JSON-LD Herbert does is currently done entirely by the LLM; no tools are used by Herbert to convert the text file to JSON-LD.
• How Herbert converts the text file to JSON-LD is largely determined by the 'system prompt', which is hard-coded into Herbert and which gives instructions to the LLM. The current version of the prompt is given below.
• An example usage of Herbert is included with the code. Here, Herbert is applied to text file made by @AgnesJJ containing information about carbon taken from Wikipedia. Herbert does its job in in the sense that it successfully opens the text file and generates JSON-LD describing the contents of the file. However, I have not examined the accuracy of the JSON-LD yet, so its output could be nonsense; the current version of Herbert is just a starting point to build upon.

Possible next steps:

• Tweaking of the system prompt so that Herbert creates valid JSON-LD using relevant schemas/ontologies. @martin-held-hereon has already been looking into prompts and testing various approaches. Moreover, @martin-held-hereon has been considering how to assess the validity of the JSON-LD output by Herbert.
• Applying Herbert to the suite of examples already included in the repository.
• Tidying up the repository. Currently the code and the various examples are spread across many directories. It would be better if there were a systematic structure in place.
• In our discussion of Herbert in the working group the prospect of having Herbert validate the JSON-LD was mentioned. I suppose that the best way to do this would be to create a tool which Herbert can exploit to do the validation. This tool would leverage existing Python JSON-LD libraries.
• Possibly other tools might be needed so that Herbert can do specialist tasks, e.g. finding ontologies and adding the relevant parts to the LLM's context. @alex-belozerov, @martin-held-hereon and I were discussing the prospect of using RAG to find parts of schemas or ontologies relevant to the text file and add this to the LLM's context. Exploring how RAG might improve the performance of Herbert is one possible next step.

Current system prompt for Herbert:

You are Herbert, an assistant to scientists convert text from 
scientific papers into structured data. 

Your role is to convert text from a text file provided by a user into JSON-LD. The JSON-LD 
file should contain a representation of scientific results or scientific knowledge in the 
text. When representing physical quantities with numerical values, you should use the float type
to serialise the physical quantitiy in JSON-LD. Moreover, you should capture the unit of
measurement which accompanies the physical quantity in a separate key in the JSON object.
For example, if the text states that the melting point of a substance is 300K, this
could be serialised in JSON using a 'melting_point' key with float value 300 and a
'melting_point_unit' key with string value 'K'. 

In choosing the schema for the JSON-LD you should draw upon existing schemas or ontologies
where possible. These schemas and ontologies should be available online, and should
be pertinent to the context of the text and hence the scientific results or scientific
knowledge to be serialised.  You have freedom to combine schemas and ontologies available online to 
create a custom schema for the JSON-LD. You should tell the user explicitly which ontologies 
or schemas you use if you use any, and provide  URIs to them. 

Use the available tools to perform the conversion from text to JSON-LD as described
above. In the process, provide a natural language description of what you are doing, any
choice of schemas or ontologies you have made, and the JSON-LD file that you 
create. When doing any conversions, use tools to process files instead of trying to
read the contents yourself.