-
Notifications
You must be signed in to change notification settings - Fork 8
Enforce data structure and data type consistency for JSON metadata #1421
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
svogt0511
wants to merge
120
commits into
master
Choose a base branch
from
pb325-json-metadata-validation
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
120 commits
Select commit
Hold shift + click to select a range
1d53e3a
test
svogt0511 8efb138
Implements JSON Schema for titles
codycooperross 2ce744c
Array.wrap titles attribute
codycooperross 20e175b
Linting
codycooperross 19213ec
Addressed issue where a string submitted in the titles attribute woul…
codycooperross 1360a4c
Fix title data structure in test
codycooperross 04497cd
Fix data structure
codycooperross 853dd15
test
svogt0511 77e6295
Update activerecord_json_validator to the latest version.
svogt0511 71143ae
Appease rubocop.
svogt0511 a5ca110
Merge remote-tracking branch 'origin/pb325-json-metadata-validation' …
svogt0511 bdb6327
Update titles.json to use the latest schema and a couple more checks.
svogt0511 42f9f59
Fix test
svogt0511 b4b3451
fix titles schema
svogt0511 c5e8678
Merge pull request #1341 from datacite/titles
svogt0511 8a4a38b
JSON schema for publicationYear.
svogt0511 49c5ff3
JSON schema for publication_year.
svogt0511 e90e088
Add creators schema
svogt0511 3337cc2
Update schema in title_type
svogt0511 1f8dee6
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 ac4864c
Creators test should have failed.
svogt0511 0c343ad
Temporarily comment out test.
svogt0511 c6d150f
Temporarily comment out test.
svogt0511 19a710c
Fix test - creator missing name
svogt0511 da0e51d
Fix test. Creator.affiliation was not an array.
svogt0511 d3d118b
Don't validate json on /dois/validate enpoint. Fix a couple of tests.
svogt0511 d9c046e
Fix some test data - errors detected by validating with json schema.
svogt0511 e5f7e96
Fix test data
svogt0511 0351dc2
Add contributors schema.
svogt0511 9107c7d
Create shared schemas.
svogt0511 2833a4d
Add subjects schema.
svogt0511 e3b1429
Fix test data
svogt0511 de18655
Fix test data.
svogt0511 15c2f2f
Fix test data.
svogt0511 156e56f
Dates schema
svogt0511 c431f7b
Update validation qualifier to only valid schemas.
svogt0511 9036ca7
Appease rubocop.
svogt0511 9663a76
Appease rubocop.
svogt0511 cdd8419
Backing out dates to string rather than verification using regular ex…
svogt0511 a08363c
Remove validations for dates. Date to remain validated as string onl…
svogt0511 d545e4b
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 225deae
Add json-schema validation for resourceType.
svogt0511 c2bba5b
Revert test.
svogt0511 3f6606c
Add json-schema validation for alternageIdentifiers.
svogt0511 6315505
add json-schema validation for relatedIdentifiers.
svogt0511 4bb4b5b
Add json-schema validation for sizes.
svogt0511 2d87449
Add json-schema validation for formats.
svogt0511 329ee8f
Mods.
svogt0511 04de852
Add json-schema validation for version.
svogt0511 44fb68e
Add json-schema validation for descriptions.
svogt0511 598906b
Fix some errors.
svogt0511 acb17d4
More fixes.
svogt0511 ad28046
More fixes.
svogt0511 c6e093f
Fix a test.
svogt0511 da5f9e8
Corrections plus new controlled vocabularies.
svogt0511 111ec55
Restructuring and corrections.
svogt0511 c917c85
Add language validation, make sure it is used in all references to la…
svogt0511 e44889b
Appease rubocop.
svogt0511 743ac08
Fix a test - description.
svogt0511 235c3f9
Fixes - code and test.
svogt0511 fae0e97
Fix errors.
svogt0511 d430aee
Temporarily commenting out some tests having to do with language.
svogt0511 05b301e
Comment out language validation. There is a problem with it.
svogt0511 7a96090
Appease rubocop.
svogt0511 2148812
Fix tests.
svogt0511 8e953cd
Add fundingReference validation.
svogt0511 091b665
Fix test.
svogt0511 f27dcee
geoLocation validation.
svogt0511 fe15778
Add json schema validation for geoLocations.
svogt0511 a0b91fe
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 5e831a9
Fix
svogt0511 d719846
Fixes.
svogt0511 e945040
Fix errors.
svogt0511 f428d24
Fixes.
svogt0511 3c9ee96
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 bb141dc
fixes
svogt0511 681654a
Appease rubocop
svogt0511 7a1071e
Fixes - language
svogt0511 bb739f4
Fix
svogt0511 cf14588
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 74d0375
Fix language validation - add raw_language.
svogt0511 86c22a7
Appease rubocop
svogt0511 d3f2056
Add test.
svogt0511 64206c3
Review comment - fix
svogt0511 1162d93
Types/resourceType validation
svogt0511 3232373
Appease rubocop.
svogt0511 38f9fbd
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 466de93
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 fa2327b
JSON-SCHEMA - metadata-4.7 support.
svogt0511 0cc8590
Use the latest JSON-schema definition in schemas/client/subjects.json…
svogt0511 30cf06e
Remove null from the controlled vocabs.
svogt0511 401d91d
Remove null from the controlled vocabs.
svogt0511 13e2288
Reverse coderabbit comment. Allowing null as alternative to a contro…
svogt0511 95f1c99
Address review comment on affiliation.json - removed field dependency.
svogt0511 9fca65c
Affiliation: require nonempty string in name, as specified in DC meta…
svogt0511 ccda29d
Address review comment: Should both alternateIdentifier and alternate…
svogt0511 95bc9bd
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 f29452b
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 5f28cf8
Affiliations - require nonblank strings when present, require format …
svogt0511 9c2c84b
Changes according to slack discussion - xml nonemptycontentStringType…
svogt0511 1970829
Changes according to slack discussion - xml nonemptycontentStringType…
svogt0511 1920917
Json schema validation: language - using the regular expression that …
svogt0511 a050396
Changes per discussion
svogt0511 b2d425f
Changes per discussion.
svogt0511 37af2cd
Debugging output.
svogt0511 698b6ee
Changes per discussion.
svogt0511 35559f4
Changes per discussion.
svogt0511 555ed9e
Added in case we ever want to use EDTF/W3CDTF data/times, RKMS-ISO860…
svogt0511 72ad7d2
Document regular expressions for date/time patterns.
svogt0511 4e2e7e1
Changes per discussion.
svogt0511 0da7192
Changes per discussion.
svogt0511 675b925
Allow changes per discussion.
svogt0511 58cb3d3
Allow null in xml:lang to be null in app/modles/schemas/client/subjec…
svogt0511 60b5dcd
Appease coderabbitai. remove byebug and pp
svogt0511 d55737c
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 f16990c
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 f579b56
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 315691e
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 059b0f8
Review comments. Creator/name and nameIdentifier/nameIdentifier shoul…
svogt0511 0e1318e
Merge remote-tracking branch 'origin/master' into pb325-json-metadata…
svogt0511 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| { | ||
| "title": "Affiliation", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "description": "Uniquely identifies an affiliation, according to various identifier schemes.", | ||
| "type": "object", | ||
| "properties": { | ||
| "name": { | ||
| "$ref": "definitions.json#/$defs/nonemptycontentStringType" | ||
| }, | ||
| "affiliationIdentifier": { | ||
| "type": "string" | ||
| }, | ||
| "affiliationIdentifierScheme": { | ||
| "type": "string" | ||
| }, | ||
| "schemeUri": { | ||
| "type": "string" | ||
| } | ||
| }, | ||
| "additionalProperties": false | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| { | ||
| "title": "Affiliations", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "array", | ||
| "items": { | ||
| "$ref": "affiliation.json" | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| { | ||
| "title": "AlternateIdentifier", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "description": "An identifier or identifiers other than the primary Identifier applied to the resource being registered. This may be any alphanumeric string which is unique within its domain of issue. May be used for local identifiers. AlternateIdentifier should be used for another identifier of the same instance (same location, same file).", | ||
| "type": "object", | ||
| "properties": { | ||
| "alternateIdentifier": { | ||
| "type": "string" | ||
| }, | ||
| "alternateIdentifierType": { | ||
| "type": "string" | ||
| } | ||
| }, | ||
| "required": ["alternateIdentifierType"], | ||
| "additionalProperties": false | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| { | ||
| "title": "AlternateIdentifiers", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "array", | ||
| "items": { | ||
| "$ref": "alternate_identifier.json" | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| { | ||
| "title": "Contributor", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "description": "The institution or person responsible for collecting, creating, or otherwise contributing to the development of the dataset.", | ||
| "type": "object", | ||
| "$ref": "contributor_base.json", | ||
| "properties": { | ||
| "nameIdentifiers": { | ||
| "$ref": "name_identifiers.json" | ||
| }, | ||
| "affiliation": { | ||
| "$ref": "affiliations.json" | ||
| } | ||
| }, | ||
| "unevaluatedProperties": false | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| { | ||
| "title": "ContributorBase", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "description": "The institution or person responsible for collecting, creating, or otherwise contributing to the development of the dataset.", | ||
| "type": "object", | ||
| "properties": { | ||
| "name": { | ||
| "$ref": "definitions.json#/$defs/nonemptycontentStringType" | ||
| }, | ||
| "nameType": { | ||
| "$ref": "controlled_vocabularies/name_type.json" | ||
| }, | ||
| "lang": { | ||
| "$ref": "definitions.json#/$defs/language" | ||
| }, | ||
| "givenName": { | ||
| "type": "string" | ||
| }, | ||
| "familyName": { | ||
| "type": "string" | ||
| }, | ||
| "contributorType": { | ||
| "$ref": "controlled_vocabularies/contributor_type.json" | ||
| } | ||
| }, | ||
| "required": [ | ||
| "name", | ||
| "contributorType" | ||
| ] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| { | ||
| "title": "Contributors", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "array", | ||
| "items": { | ||
| "$ref": "contributor.json" | ||
| } | ||
| } |
30 changes: 30 additions & 0 deletions
30
app/models/schemas/doi/controlled_vocabularies/contributor_type.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| { | ||
| "title": "ContributorType", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "description": "The type of contributor of the resource.", | ||
| "type": "string", | ||
| "enum": [ | ||
| "ContactPerson", | ||
| "DataCollector", | ||
| "DataCurator", | ||
| "DataManager", | ||
| "Distributor", | ||
| "Editor", | ||
| "HostingInstitution", | ||
| "Producer", | ||
| "ProjectLeader", | ||
| "ProjectManager", | ||
| "ProjectMember", | ||
| "RegistrationAgency", | ||
| "RegistrationAuthority", | ||
| "RelatedPerson", | ||
| "Researcher", | ||
| "ResearchGroup", | ||
| "RightsHolder", | ||
| "Sponsor", | ||
| "Supervisor", | ||
| "Translator", | ||
| "WorkPackageLeader", | ||
| "Other" | ||
| ] | ||
| } |
20 changes: 20 additions & 0 deletions
20
app/models/schemas/doi/controlled_vocabularies/date_type.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| { | ||
| "title": "DateType", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "description": "The type of date. Use RKMS‐ISO8601 standard for depicting date ranges.To indicate the end of an embargo period, use Available. To indicate the start of an embargo period, use Submitted or Accepted, as appropriate.", | ||
| "type": "string", | ||
| "enum": [ | ||
| "Accepted", | ||
| "Available", | ||
| "Copyrighted", | ||
| "Collected", | ||
| "Coverage", | ||
| "Created", | ||
| "Issued", | ||
| "Submitted", | ||
| "Updated", | ||
| "Valid", | ||
| "Withdrawn", | ||
| "Other" | ||
| ] | ||
| } |
14 changes: 14 additions & 0 deletions
14
app/models/schemas/doi/controlled_vocabularies/description_type.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "title": "DescriptionType", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "description": "The type of description of the resource.", | ||
| "type": "string", | ||
| "enum": [ | ||
| "Abstract", | ||
| "Methods", | ||
| "SeriesInformation", | ||
| "TableOfContents", | ||
| "TechnicalInfo", | ||
| "Other" | ||
| ] | ||
| } |
14 changes: 14 additions & 0 deletions
14
app/models/schemas/doi/controlled_vocabularies/funder_identifier_type.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "title": "FunderIdentifierType", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "description": "The type of the funderIdentifier.", | ||
| "type": "string", | ||
| "enum": [ | ||
| "ISNI", | ||
| "GRID", | ||
| "ROR", | ||
| "Crossref Funder ID", | ||
| "Other" | ||
| ] | ||
| } | ||
|
|
9 changes: 9 additions & 0 deletions
9
app/models/schemas/doi/controlled_vocabularies/name_type.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "title": "NameType", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "string", | ||
| "enum": [ | ||
| "Organizational", | ||
| "Personal" | ||
| ] | ||
| } |
11 changes: 11 additions & 0 deletions
11
app/models/schemas/doi/controlled_vocabularies/number_type.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| { | ||
| "title": "NumberType", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "string", | ||
| "enum": [ | ||
| "Article", | ||
| "Chapter", | ||
| "Report", | ||
| "Other" | ||
| ] | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix geolocation validator key mismatch.
Line 175 validates
:geolocations, but this model usesgeo_locations. This typo can prevent geolocation JSON-schema validation from running.🔧 Proposed fix
📝 Committable suggestion
🤖 Prompt for AI Agents