Continue validation and count invalid records #2370

[TobiasNx]

The validation should continue and count the number of invalid file.

[TobiasNx]

@dr0i do you want to have another look?

[dr0i]

I would test and apply @blackwinter suggestions if it was my code in the first plac. Or am I mistaken?

[TobiasNx]

@dr0i thanks I missed the suggestion from @blackwinter but how can I test this locally?

[dr0i]

Test it by adapting like:

UPDATES_FNAME=2026-06-15_to_2026-06-16_lobid-resources-updates.jsonl.gz

# validate schema
cd ~/git/lobid-resources/src/test/resources/schemas

jsonschema validate resource.json ${UPDATES_FNAME} --continue > /tmp/jsonschemaValidationOutput.log 2>&1

if [ -s /tmp/jsonschemaValidationOutput.log ]; then
        NUMBER_OF_INVALID_RECORDS=$(grep -c "fail:" /tmp/jsonschemaValidationOutput.log)
        echo $NUMBER_OF_INVALID_RECORDS
        head -n5 /tmp/jsonschemaValidationOutput.log
fi

I've used a dump which was reported as flawed (see mails) and downloaded it from https://lobid.org/download/dumps/lobid-resources/.
(I've probably an old version of jsonschema locally because it outputs:

/usr/bin/jsonschema:5: DeprecationWarning: The jsonschema CLI is deprecated and will be removed in a future version. Please use check-jsonschema instead, which can be installed from https://pypi.org/project/check-jsonschema/ )

Tested this then on server.
Remarkably, it found only 1 error (don't know if this is coincidence or just bailing out and thus not working as expected).
However, we may just deploy the script and see what happens in the days to come.

Note also that head -n5 results in just a tiny bit of information - the ressource is not dumped in one line but is pretty printed, so we have "one field, one line". Thus I will increase this to 333 and, as usually, will adapt this when we see results.