Expands InternalExportDataProvider per dataverse-spi 2.1.0 and refactors the DDI exporter to utilize the new functionality

[landreev]

Un-drafting the PR now!

What this PR does / why we need it:

See issue #11405. Short version: the current export data framework is prohibitively expensive for datasets with very large amounts of tabular data (i.e., "ingested" DataFiles that have associated DataTable entities , with large numbers of chilld DataVariable objects). This PR improves it, via adding code that retrieves the data in manageable chunks. This in trurn relies on the new version of the ExportDataProvider interface from https://github.com/gdcc/dataverse-spi.

Which issue(s) this PR closes:

• Closes Refactor and optimize the Export metadata framework, especially for the data/variable-level metadata #11405

Special notes for your reviewer:

Note that the PR builds with a snapshot release of dataverse-spi:

            <groupId>io.gdcc</groupId>
            <artifactId>dataverse-spi</artifactId>
            <version>2.1.0-SNAPSHOT</version>

I.e., the version 2.1.0 still needs to be properly released under gdcc. But I don't think that should be stopping putting the PR into review and merging it rather sooner than later.

Note specifically the test for the functionality, TabularDataExportIT added in this PR. I have added it to the list of integration tests that are performed automatically by the workflow action (in tests/integration-tests.txt) even though, strictly speaking, it is not a real integration test. I.e., it does not rely on calling a fully-functional Dataverse instance started remotely. It relies instead on the new test framework added in @poikilotherm's #12365 where a local Docker instance of the database is spun up in which some dummy objects - Datasets, Versions, Datatables and Variables etc are created instantly, allowing testing of complex functionality that requires real database access. Without this new framework, the only way to test an export of a dataset with N large tabular files would be to actually ingest these N files via the API in the remote Dataverse instance used by the RestAssured tests. Which would add to the ongoing bloating of the test suite, and still would not allow to test the underlying functionality as thoroughly (for example, there is no easy way to count the actual database queries under such scenario).

You can run the test in question in your own dev. environment via
mvn test -Dtest=TabularDataExportIT#exportTabularMetadata
If you do not have Docker running, the test will be politely skipped. Otherwise it will create a throwaway postgres instance and proceed to test tabular data exports and associated data retrieval, comparing the results produced using the legacy methods vs. the new and optimized versions added in this PR and more.

Suggestions on how to test this:

Please see the info under the "notes for the reviewer" and feel free to play and experiment with the new test described there. Please keep in mind that the test is still somewhat of a reference demonstrator of the new JpaEntityManagerService-based framework from #12365. It can be expanded and improved going forward.

The ultimate test would be to test before vs. after on some real-life monster datasets.
I would suggest these bad boys on qa:
https://qa.dataverse.org/dataset.xhtml?persistentId=doi:10.7910/DVN/KZW8R2 (207 ingested tab. files; 186906 variables total)
https://qa.dataverse.org/dataset.xhtml?persistentId=doi:10.7910/DVN/S5BRBO (134 tab. files; 536006 variables).

"ddi" (aka the "full" ddi) is the export format that needs to be tested.
Note that some extra steps will be required in order to test this on qa:

• You will want to change the storage driver to local filesystem ("file") for the datasets in question - so that Dataverse can write the export files;
• And make sure you are testing the export code vs. looking at a potentially existing cached export; i.e., the file export_ddi.cached will need to be erased between testing runs.
  Happy to assist with the above if needed.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

[landreev]

Note that the PR now builds under Jenkins (with a snapshot release of dataverse-spi that is now published).

[coveralls]

Coverage is 24.88% — 11405-refactor-exports into develop. No base build found for develop.

[github-actions]

🚫 [reviewdog] <com.puppycrawl.tools.checkstyle.checks.blocks.LeftCurlyCheck> _{reported by reviewdog 🐶}
'{' at column 21 should have line break after.

[landreev]

Don't tell me what to do, reviewdog. That was formatted like that on purpose, to emphasize the fact that we don't care about that particular exception at that stage of the test.

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:11405-refactor-exports

ghcr.io/gdcc/configbaker:11405-refactor-exports

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[github-actions]

Test Results

404 tests  +1   389 ✅ +1   24m 2s ⏱️ - 1m 16s
 56 suites +1    15 💤 ±0 
 56 files   +1     0 ❌ ±0 

Results for commit 1eb8420. ± Comparison against base commit df71344.

[landreev]

@poikilotherm Hi, I'm looking at the test results above, and seeing this failure to "Release Snapshot" for Dataverse SPI:

Am I correct assuming this is an old action that simply needs to be removed, now that dataverse-spi lives in https://github.com/gdcc/dataverse-spi?
Similarly, should I go ahead and purge modules/dataverse-spi from the main source tree?
Please stop me if there is any reasons why I should not delete these.
Thank you!

[poikilotherm]

@landreev Yes, the in-tree SPI stuff can be completely eradicated. ☢️ 😄