Eliminate hydrus dependency on lyberservices-prod

[peetucket]

We are trying to turn off lyberservices-prod, but Hydrus seems to be making requests to it still. Look at the logs and try to figure out why and also if the depedency is required for Hydrus to continue operating.

Log on lyberservices-prod: /var/log/httpd/access_log

See slack thread: https://stanfordlib.slack.com/archives/C0RK5EM9N/p1596151403493200?thread_ts=1596144800.488800&cid=C0RK5EM9N

See archeology in comments below. Seems to be related to indexing. An attempt was made to add indexing code to Hydrus directly but this caused other problems. See PRs #461 and then #471 which reverted the work.

Next steps to try with the ultimate goal being turn off lyberservices-prod while keeping Hydrus operational.

• coordinate a time with Hydrus service team (e.g. Hannah, Amy, Andrew) and Ops to temporarily turn off lyberservices-prod and see what the side effects are
• same, but instead of turning off lyberservices-prod completely, add a redirect to a new server than can respond to the endpoints hit in the logs with a simple 200, see what side effects on hydrus are

Things to test in Hydrus each time

• create a new object in hydrus
• version an object in hydrus
• view homepage
• view a collection
• edit a collection

[peetucket]

Looks like a request to get workflows from a hydrus object:

10.111.6.163 - - [31/Jul/2020:13:45:35 -0700] "GET /workflow/dor/objects/druid:yh041gt7606/workflows HTTP/1.1" 200 2060

Hydrus Object: https://argo.stanford.edu/view/druid:yh041gt7606

[peetucket]

10.111.6.163 is sul-dor-prod

[andrewjbtw]

When I looked at this yesterday, it looked like the hydrus druid URLs that get hit are

item druid
collection druid (for that item)
APO druid (for that collection)

[peetucket]

The workflows datastream in that Fedora object has the lyberservices-prod URL in it (since it is an "external" databastream). When you view that datastream in Fedora admin, it triggers the call which you can see in the logs

[peetucket]

I bet that URL is embedded in all fedora objects in that workflows datastream (which I believe is now no longer used). Maybe Hydrus shows it somewhere, triggering the request?

[peetucket]

This page is where I thought the request might be coming from, but it is blank for me and doesn't seem to trigger the request in the log: https://sdr.stanford.edu/items/druid:yh041gt7606/datastreams

[andrewjbtw]

Yesterday I made a change to this test collection and saved it: https://sdr.stanford.edu/collections/druid:gs556yg7381 But I didn't change any items in that collection.

That led to the collection druid workflows URL and its APO druid workflows URL showing up in the log. But I didn't check until after I made the edit and saved it, so I don't know when exactly the http requests were made.

[peetucket]

When I looked at this yesterday, it looked like the hydrus druid URLs that get hit are

item druid
collection druid (for that item)
APO druid (for that collection)

Can you clarify what you mean by this? Looking at the logs for lyberservices-prod, it appears that a request for the workflows datastream is what is appearing?

[andrewjbtw]

Sorry, I meant that the druids identified in the workflows datastreams in the log corresponded to those items. Not that URLs for those druid item pages were in the log.

[peetucket]

Opened a new version for this test item and saw the requests come in:

https://sdr.stanford.edu/items/druid:ch858tj0413

10.111.6.163 - - [31/Jul/2020:14:54:28 -0700] "GET /workflow/dor/objects/druid:ch858tj0413/workflows HTTP/1.1" 200 7946
10.111.6.163 - - [31/Jul/2020:14:54:30 -0700] "GET /workflow/dor/objects/druid:ch858tj0413/workflows HTTP/1.1" 200 7946
10.111.6.163 - - [31/Jul/2020:14:54:58 -0700] "GET /workflow/dor/objects/druid:ch858tj0413/workflows HTTP/1.1" 200 7946

So related to opening a new version in Hydrus.

[peetucket]

Saving the object makes the requests appear to. So perhaps it is a save.

[peetucket]

Loading the hydrus item manually on the rails console and saving it triggers the requests in lyberservices-prod logs:

i=Hydrus::Item.find('druid:ch858tj0413')
i.save

leads to this in the logs for lyberservices-prod at /var/log/httpd/access_log

10.111.6.163 - - [31/Jul/2020:14:56:57 -0700] "GET /workflow/dor/objects/druid:ch858tj0413/workflows HTTP/1.1" 200 7946
10.111.6.163 - - [31/Jul/2020:15:02:27 -0700] "GET /workflow/dor/objects/druid:ch858tj0413/workflows HTTP/1.1" 200 7946
10.111.6.163 - - [31/Jul/2020:15:02:29 -0700] "GET /workflow/dor/objects/druid:ch858tj0413/workflows HTTP/1.1" 200 7946

[peetucket]

Saving the object also shows some deprecation warnings...could be unrelated but perhaps we are running older versions of gems that are triggering the calls to lyberservices... will check gem versions:

DEPRECATION WARNING: you provided 2 args, but active_lifecycle now takes kwargs. (called from milestones at /opt/app/hydrus/hydrus/shared/bundle/ruby/2.7.0/gems/dor-workflow-client-3.22.0/lib/dor/workflow/client.rb:47)
DEPRECATION WARNING: passing the repo parameter to active_lifecycle is no longer necessary. This will raise an error in dor-workflow-client version 4. (called from milestones at /opt/app/hydrus/hydrus/shared/bundle/ruby/2.7.0/gems/dor-workflow-client-3.22.0/lib/dor/workflow/client.rb:47)
DEPRECATION WARNING: Dor::ReleaseTagService is deprecated and will be removed in dor-services 9.0. (it's moving to dor-services-app). (called from new at /opt/app/hydrus/hydrus/shared/bundle/ruby/2.7.0/gems/dor-services-8.6.0/lib/dor/services/release_tag_service.rb:9)
DEPRECATION WARNING: Dor::ReleaseTags::Purl is deprecated and will be removed in dor-services 9.0. (it's moving to dor-services-app). (called from new at /opt/app/hydrus/hydrus/shared/bundle/ruby/2.7.0/gems/dor-services-8.6.0/lib/dor/services/release_tag_service.rb:15)

[peetucket]

Using Argo to compare with. Argo is currently on dor-services 9.5.0, but hydrus is locked to ~> 8.0 in the Gemfile and is on 8.6.0

Argo and hydrus are both on the same version for dor-workflow-client and dor-services-client.

So suspicion falls on dor-services v8 doing something with the workflows datastream on the save.