2357 deletions

[TobiasNx]

@dr0i would it be enough to add a prefix to the title in order to delete the records?

[dr0i]

A prefix in a title field looks like a dirty workaround. If you can guarantee that this prefix will not be a part of a regular title, it would be ok, though. Also it comes with a bit of an extra resource effort (filtering 30 M docs using a simple regex).
Could you delete all fields, beside the id, from that resource and have not just a prefix but a solely title like "DELETED from lobid-resources" ? But then would the schema not be validate, I suppose :/.

[dr0i]

I think this is ok. It's also good to not put too much effort into this issue, so 👍

[blackwinter]

As long as you don't switch the index alias until the deletions have been removed, I guess... Unfortunately, org.metafacture.elasticsearch.JsonToElasticsearchBulk doesn't support the delete action yet, which would certainly be the cleaner option.

[TobiasNx]

@dr0i I adjusted as you suggested. Was a quick adjustment. What do you think about @blackwinter s comment.

[dr0i]

As long as you don't switch the index alias until the deletions have been removed,

does it really hurt if for some minutes there would be titles with the "DELETED"-Prefix in production index @blackwinter ?

[blackwinter]

does it really hurt if for some minutes there would be titles with the "DELETED"-Prefix in production index @blackwinter ?

Yes, I would consider it a serious issue. If, however, the records only contain minimal data (such as in c432b9d), it's less relevant because they won't come up in a casual search. Another option would be to include an internal field which can also be queried efficiently. I'm only worried about the visible effects in the union catalogue.

(And I only noticed after my previous comment that these modified records are immediately live during update indexing; the index switch only happens for the baseline indexing, of course.)

[TobiasNx]

@dr0i you could continue with an apporach to get rid of these records

[dr0i]

So @TobiasNx if it's easily possible to just leave out as much fields as possible (i.e. also title) , we could do an must_not: exist (id) query (if its possible to even let out the id field).

[TobiasNx]

I reduced the number of elements as much as the schema allows for (@context,id,type) and as much the workflow allows for (almaMmsId was necessary otherwise:

o.l.r.AlmaMarc21XmlToLobidJsonMetafixTest - Errored when transforming 
java.lang.NullPointerException: Cannot invoke "Object.toString()" because "value" is null
        at org.lobid.resources.JsonToElasticsearchBulkMap.findId(JsonToElasticsearchBulkMap.java:129)

)

Additionally I added the delete title.