Controlling instruments with FAIR software

[ml-evs]

Working group notes building on #8

[PeterKraus]

Together with @ml-evs and @adamreupert we have discussed how an uploader from something-like-tomato to something-like-datalab and something-like-kadi4mat could work:

Goal:

Given a ROcrate that defines:

• i) a Dataset representing enumerated data files in the ROcrate,
• ii) a Person representing an existing uniquely-identifiable user in the ELN to be given read-write access to those files, and
• iii) a RepositoryObject representing an existing uniquely-identifiable node in the ELN ("sample"),

attach the files enumerated by the Dataset to (a child of) the RepositoryObject and give an appropriate write permission to that Person.

Recommendations:

• ELNs would ideally expose a POST endpoint where one would authenticate using an appropriate API key and drop the ROcrate in
• the above API key should be of an "instrument user" (i.e. an account specific to that instrument as opposed to a "normal" user account)
• this "instrument user" should have appropriate permissions to manipulate the ELN and assign write permissions
• author property indicates authorship of a Thing by a Person but does not convey permissions

Profile:

The below definitions are based on the RO-crate 1.1 release.

RO-crates implementing this "profile" MUST indicate it as such using conformsTo property of the below Dataset pointing to a Profile with "@id": "https://github.com/MADICES/MADICES-2025/discussions/25" and "version": "0.1".

Nouns

• "@type": "DigitalDocumentPermission" is used to indicate that the ELN should grant permissions of permissionType to a grantee, which is a Person known to be uniquely identifiable by the ELN by the provided @id
• "@type": "RepositoryObject" is used to refer to a node (object, "sample") already present within the ELN, that is uniquely identifibable by the provided @id
• "@type": "Dataset" is used to identify the contents of the ROcrate that are supposed to be actioned; these should be listed as entries in the hasPart property
  • note: a Dataset object MUST have a defined author property to be valid RO-crate 1.1

Actions

• "@type": "UpdateAction" is an action that indicates that the above Dataset should be attached to the above RepositoryObject directly:

  • the object property should point to the @id of the Dataset
  • the target property should point to the @id of the RepositoryObject

• "@type": "CreateAction" is an action that indicates that first, a new RepositoryObject which is a child of the above RepositoryObject should be created; then the above Dataset should be attached to the new child RepositoryObject:

  • the object property should point to the @id of the Dataset
  • the location property should point to the @id of the RepositoryObject where the child is created

Rationale

• this is a minimum working unit of a very rudimentary ROcrate-based mechanism, that can be useful in practice:
  • arbitrary files can now be deposited in an ELN using ROcrate,
  • the UpdateAction mechanism allows for upload of data from "non-destructive" techniques, e.g., pXRD characterisation of a large amount of sample
  • the CreateAction mechanism allows for upload of experimental data used to "create" the new object (although this is not yet annotated), e.g., a catalysis experiment with a meaningful "post-mortem" sample
  • appropriate permissions can be given to a user
• we expect the ELN to be opinionated about what "a child of a RepositoryObject" is; after all the purpose of a "laboratory notebook" is to track sample history

Things (Currently) Out Of Scope

• how do I make a new RepositoryObject without an existing RepositoryObject
• how do I choose the subtype of an RepositoryObject in the ELN ("starting material", "sample", "device" ...)
• how do I semantically annotate the Dataset contents
• how do I find existing users in the ELN
• what do I do with required RO-crate 1.1 metadata (e.g. author property of the Dataset)?
• what do I do with further objects and properties in the RO-crate?
• resolving conflict with current RO-crate example usage of CreateAction

[ml-evs]

Possible example (from memory) of an RO-Crate that an ELN could receive and take to mean "I need to make an entry in my ELN [of some kind] that has the uploaded files attached to them".

{
  "@context": [
    "https://w3id.org/ro/crate/1.1/context",
    {
      "@vocab": "http://schema.org/"
    }
  ],
  "@graph": [
    {
      "@id": "./",
      "@type": "Dataset",
      "name": "An example crate that should be interpreted by an ELN as \"ingest these data files as if they belong to the existing sample with ID <ABCDEF>\" and \"give write permission to existing user with ID <https://orcid.org/0000-0002-1182-9098>\"",
      "author": {
        "@id": "https://orcid.org/0000-0002-1182-9098"
      },
      "hasPart": {
        "@id": "measurement.h5"
      },
      "conformsTo": {
        "@id": "https://github.com/MADICES/MADICES-2025/discussions/25"
      },
      "potentialAction": {
        "@id": "#6"
      },
      "hasDigitalDocumentPermission": {
        "@id": "#7"
      }
    },
    {
      "@id": "https://orcid.org/0000-0002-1182-9098",
      "@type": "Person",
      "name": "M Evans"
    },
    {
      "@id": "measurement.h5",
      "@type": "File",
      "contentSize": "111222333",
      "name": "The measurement recorded on the sample as an HDF5 file"
    },
    {
      "@id": "ro-crate-metadata.json",
      "@type": "CreativeWork",
      "identifier": "ro-crate-metadata.json",
      "about": {
        "@id": "./"
      }
    },
    {
      "@id": "https://github.com/MADICES/MADICES-2025/discussions/25",
      "@type": "Profile",
      "version": "0.1",
      "name": "This RO-crate conforms to the MADICES 2025 proposed dataset action profile."
    },
    {
      "@id": "ABCDEF",
      "@type": "RepositoryObject",
      "name": "The existing ABCDEF sample that these measurements pertain to"
    },
    {
      "@id": "#6",
      "@type": "UpdateAction",
      "name": "The type of action that the ELN should interpret as the task to perform, in this case, update the existing entry",
      "target": {
        "@id": "ABCDEF"
      },
      "object": {
        "@id": "./"
      }
    },
    {
      "@id": "#7",
      "@type": "DigitalDocumentPermission",
      "name": "The grantee should be granted the following permissions over the object by the ELN",
      "grantee": {
        "@id": "https://orcid.org/0000-0002-1182-9098"
      },
      "permissionType": "WritePermission"
    }
  ]
}

[PeterKraus]

[PeterKraus]

After some very helpful discussions today (mainly with Abigail Miller, Andreas Meier, Juan Fuentes, Burkhard Schäfer), it is clear that RO-crate is meant to be used in a descriptive way as opposed to a prescriptive way; i.e. the RO-crate should describe the end state as opposed to give instructions on how to do it.

Compatibility with RO-crate schema plus

The terms and definitions used below are parts of the RO-crate 1.1 definition, therefore they are compatible with RO-crate schema plus. However, openBIS currently expects the exact @type, it's definition (via schema), and @id of objects that already exist within openBIS to be passed via the RO-crate to successfully match the two and process any changes. Partial matches (e.g. @id matches exactly, but @type is a more generic type than that within openBIS) currently result in an error code.

Profile

The below definitions are based on the RO-crate 1.1 release.

RO-crates implementing this "profile" MUST indicate it as such using conformsTo property of the root Dataset pointing to a Profile with "@id": "https://github.com/MADICES/MADICES-2025/discussions/25" and "version": "0.2".

The Repository here refers to the ELN, i.e. the platform consuming the RO-crate.
The concept of a Child and Parent here refers to a directional reference (Parent implies hasPart: [Child] and inverse.)

• "@type": "Dataset" + "@id": "./":
  • used to identify the contents of the RO-crate that are supposed to be actioned
  • these should be listed as entries in the hasPart property of "@type": "File"
  • the Dataset object MUST have an author property to be a valid RO-crate 1.1
    • if the Person behind the @id of the above author is present in the Repository as a valid user, that user should be given the default "write" access to the file
    • if the Person behind the @id of the above author is not present in the Repository, the behaviour is currently undefined; the recommendation is that the Person is created in the Repository and the authorship information thus maintained
  • the Dataset object MAY have an maintainer property and will be a valid RO-crate 1.1
    • the Person behind the @id of the above maintainer MUST be present in the Repository as a valid user; that user should be given the default "write" access to the file
    • note that the preferred property would be requester, not maintainer; however, the requester is not part of RO-crate 1.1
• "@type": "RepositoryObject" + "@id": <unique id>":
  • the <unique id> MUST be a unique ID resolving to an existing object in the Repository
  • the RepositoryObject MUST have a hasPart property with EXACTLY ONE entry:
    • "@type": "Dataset" + "@id": "./": this Dataset is to be uploaded and attached to the existing RepositoryObject
    • "@type": "RepositoryObject" + "@id": "#?", which in turn MUST have a hasPart property with "@type": "Dataset" + "@id": "./": this Dataset is to be uploaded and attached to a new RepositoryObject which is a Child of the existing RepositoryObject
  • the user permissions MUST be assigned based on processing the author and maintainer of the Dataset

[PeterKraus]

Example No. 1: attach `measurement.h5` into `concat:ABCDEF`

{
  "@context": [
    "https://w3id.org/ro/crate/1.1/context",
    {
      "@vocab": "http://schema.org/"
    }
  ],
  "@graph": [
    {
      "@id": "./",
      "@type": "Dataset",
      "name": "An example crate that should be interpreted by an ELN as \"ingest these data files as if they belong to the existing sample with ID <ABCDEF>\" and \"give write permission to existing user with ID <https://orcid.org/0000-0002-1182-9098>\"",
      "author": {
        "@id": "https://orcid.org/0000-0002-1182-9098"
      },
      "hasPart": {
        "@id": "measurement.h5"
      },
      "conformsTo": {
        "@id": "https://github.com/MADICES/MADICES-2025/discussions/25"
      }
    },
    {
      "@id": "https://orcid.org/0000-0002-1182-9098",
      "@type": "Person",
      "name": "M Evans"
    },
    {
      "@id": "measurement.h5",
      "@type": "File",
      "contentSize": "111222333",
      "name": "The measurement recorded on the sample as an HDF5 file"
    },
    {
      "@id": "ro-crate-metadata.json",
      "@type": "CreativeWork",
      "identifier": "ro-crate-metadata.json",
      "about": {
        "@id": "./"
      }
    },
    {
      "@id": "https://github.com/MADICES/MADICES-2025/discussions/25",
      "@type": "Profile",
      "version": "0.1",
      "name": "This RO-crate conforms to the MADICES 2025 proposed dataset action profile."
    },
    {
      "@id": "concat:ABCDEF",
      "@type": "RepositoryObject",
      "name": "The existing ABCDEF sample that these measurements pertain to",
      "hasPart": {
        "@id": "./"
      }
    }
  ]
}

Example No. 2: attach `measurement.h5` into a new child of `concat:ABCDEF`, give user permissions to `https://orcid.org/0000-0002-4359-5003`

{
  "@context": [
    "https://w3id.org/ro/crate/1.1/context",
    {
      "@vocab": "http://schema.org/"
    }
  ],
  "@graph": [
    {
      "@id": "./",
      "@type": "Dataset",
      "name": "An example crate that should be interpreted by an ELN as \"ingest these data files as if they belong to the existing sample with ID <ABCDEF>\" and \"give write permission to existing user with ID <https://orcid.org/0000-0002-4359-5003>\"",
      "author": {
        "@id": "https://orcid.org/0000-0002-1182-9098"
      },
      "maintainer": {
        "@id": "https://orcid.org/0000-0002-4359-5003"
      },
      "hasPart": {
        "@id": "measurement.h5"
      },
      "conformsTo": {
        "@id": "https://github.com/MADICES/MADICES-2025/discussions/25"
      }
    },
    {
      "@id": "https://orcid.org/0000-0002-1182-9098",
      "@type": "Person",
      "name": "M Evans"
    },
    {
      "@id": "https://orcid.org/0000-0002-4359-5003",
      "@type": "Person",
      "name": "P Kraus"
    },
    {
      "@id": "measurement.h5",
      "@type": "File",
      "contentSize": "111222333",
      "name": "The measurement recorded on the sample as an HDF5 file"
    },
    {
      "@id": "ro-crate-metadata.json",
      "@type": "CreativeWork",
      "identifier": "ro-crate-metadata.json",
      "about": {
        "@id": "./"
      }
    },
    {
      "@id": "https://github.com/MADICES/MADICES-2025/discussions/25",
      "@type": "Profile",
      "version": "0.1",
      "name": "This RO-crate conforms to the MADICES 2025 proposed dataset action profile."
    },
    {
      "@id": "concat:ABCDEF",
      "@type": "RepositoryObject",
      "name": "The existing ABCDEF sample that these measurements pertain to",
      "hasPart": {
        "@id": "#8"
      }
    },
    {
      "@id": "#8",
      "@type": "RepositoryObject",
      "name": "The new Child object of  existing ABCDEF sample that these measurements pertain to",
      "hasPart": {
        "@id": "./"
      }
    }
  ]
}

[PeterKraus]

First tomato-RO-crate for testing: results.1.zip

[PeterKraus]

@ml-evs: tomato RO-crate with valid user ID and sample ID on datalab.concatlab.eu: results.3.zip

[PeterKraus]

A note on "interoperability profiles"

This came out of a discussion between @edan-bainglass, myself and Andreas Meier.

Neither the RO-crate 1.1 standard nor the RO-crate schema plus proposal currently define what the platform (Repository in the above) actually does with the RO-crate. There are many ways to deal with those objects, and the practice should be documented somewhere.

One option is that every Repository separately documents their own behaviour upon receiving a standard-compliant RO-crate; for the purposes of this discussion, what happens when a @type conflicts but a unique @id matches. This is very clearly the quicker way to get working solutions, and an argument has been made that it should be done by well-maintained software projects in any case. However, this might not lead towards an interoperable solution in the long term.

Another option is to move towards "interoperability profiles", think HDMI 1.4 vs HDMI 2.0, or the various USB profiles. This would mean a definition of a "strict" profile (i.e. raise on the @type mismatch above) would move into a public space, and we could then decide and agree on further more "relaxed" profiles, such as that proposed in version 0.2 above. Of course, just as not every device that advertises HDMI 2.0 support actually supports it properly, the Repositories would still have to make a best-effort attempt to actually respect such profiles.

In my personal opinion, the latter option was what MADICES should be about.

[juan-fuentes-sis]

Just some extra thoughts regarding how strict is RO-Crate plus as mentioned by @PeterKraus

What is a correct RO-Crate plus Package?

• As far as the objects and types shipped on the object graph match, we could consider this correct. If you don't want to ship properties with an object that you know is already into other system, just an Id and say this is a Thing. From the point of view of the RO-Crate package, this is correct.

What is the correct behavior of a system using RO-Crate plus?

• This is really system dependent, systems are going to not to only validate the package but they are also going to validate and match against their own schema and metadata and there is different levels of strictness and behaviors.

Do they want exactly equal types? Are they happy just matching semantics?
Do they absorb all the data? Only the data they are interested on?

From the point of the specification I'm towards indicating explicitly that the spec doesn't impose a particular behaviors when ingesting a package. This should allow systems ingesting the packages to do what is best for them.

[edan-bainglass]

I agree that "what to do with a crate" cannot be enforced in any way on any platform. If the action is to integrate the data/metadata carried in the crate into the platform, then this is simply an import action. The only thing we can do is specify the some basic dataset metadata that any platform would be "okay" to consume (id, name, ctime, author, has-parts, etc.). The import endpoint MUST support this. Any additional info is ignored unless explicitly handled by the platform. The more it handles, the richer its import feature becomes.