CUMULUS-4534 -- Ecarton/multi provider db (#4252)

* first-pass WIP

* lint

* testing override of cmr.provider

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* lint

* small name change

* changelog

* linting

* shouldn't have removed test

* pr feedback

* simplify sf-scheduler merge

* lint

* no undefined cmrProvider

* alphabetizing things

* typo in scheduler

* add index

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* lint

* lint

* cmr_provider not optional in types

* keep it optional in types, but mandatory in schedule-sf for  compat with other pieces

* fix tests

* linting

* looking at index creation

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* lint

* default null intemplate

* non-optional cmr_provider in db

* bugfix and testing for bug

* needs fixed infake collection

* passthrough of cmr_provider in test, and  notNullable

* unneeded drop index

* changelog

* fixes in other unit tests

* s3 duck db update for test

* add cmrProviders to these test files

* ad provider in api again

* mistaken name

* more places where the cmr_provider belongs

* linting

* add cmrProviders to integration test collections

* add cmr_provider here too

* adding cmrProvider in more needed places

* lint

* fix producerGranuleId in changeGranuleCollectionPg

* add default cmrProvider to template in integration

* lint

* turn off tests for this

* checking for weirdness at runtime within api

* get hyrax to tell me what it's cmrProvider is

* how about cmr settings coming through

* what about checking this?

* checking CMR IO

* WIP some hardcoded stuff

* lint

* fix hardcode

* lint

* try this half-measure

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* lint

* bugfix

* fixing more unit tests

* fix more tests

* rearrange default CUMULUS integration tests provider

* fix provider hardcode in changeGranuleCollection

* lint

* cleanup

* queue-granules needs a collection cmrprovider in test

* lint

* fix queue test

* mistakencmrProvider in generator

* console logs removed

* addding buildAndStartWorkflow cmrProvider

* lint

* log removed

* dependency not needed

* small cleanups

* remove debug log

* getting some logs to check what's happening in collections api

* invalidSigException?

* collection can be undefined in certain tests

* typo

* lint

* cleanup and pr review

* Revert "typo"

This reverts commit 560e230.

* Revert "collection can be undefined in certain tests"

This reverts commit 4bb3656.

* Revert "cleanup and pr review"

This reverts commit 813e8e5.

* Revert "lint"

This reverts commit 5f5a702.

* lint

* CSD-100: ensure env vars are initialized once per Lambda container by caching initialization promise

* initEnvVars === undefined

* Revert "invalidSigException?"

This reverts commit 22282d2.

* trying to get sf-scheudler workign right

* cleanup and pr review

* check that cmr field is an object

* new and improved changelog

* lets try this in ci, it doesn't work locally

* add upgrade notes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* smal simplification and typing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* linting

* fix update note

* correct partialtype

* typo in changelog

* rework this test

* cmrProvider added to schemas.js

* add sinble quotes for string

* typo

* CSD-100 off master

* small further update toverification

* does this work in ci?

* try this in ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Cumulus Bot <cumulus.bot@gmail.com>
Co-authored-by: jennyhliu <jenny.h.liu@nasa.gov>
Co-authored-by: Nnaga1 <nnagavenkata@yahoo.com>

Author: 5 people

CHANGELOG.md

@@ -10,6 +10,10 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
   ** UPGRADE NOTE: When upgrading the TEA module version, use a two-phase apply to prevent rollback failures
   caused by Terraform destroying old lambda S3 objects before the CloudFormation stack update completes.
 
+### Migration Notes
+
+All core tasks that enqueue messages to launch workflows are updated to use collection defined cmrProvider. Any daac/consolidation tasks which perform the same function need to ensure they do the same.
+
 Phase 1 — upload new S3 objects and update CF stack (keeps old S3 objects intact as rollback targets if the CF update fails):
    ````
    terraform apply \
@@ -25,6 +29,9 @@ Phase 2 — full apply to clean up old S3 objects and apply remaining changes:
 
 ### Added
 
+- **CUMULUS-4534**
+  - collection db model has added non-optional cmr_provider field
+  - update requires db-migration to add cmr_provider to collection model
 - **CUMULUS-4706**
   - Define and serve the iceberg search api routes through the iceberg api server.
 - **CUMULUS-4606**
@@ -37,6 +44,16 @@ Phase 2 — full apply to clean up old S3 objects and apply remaining changes:
 - **async-operations-update**
   - Update Async Operation container to new version 56, `cumuluss/async-operation:56`. Users should update their references to `async-operation` with the new version.
 
+### Changed
+
+- **CUMULUS-4534**
+  - collection translate functions pass cmr_provider/cmrProvider back and forth
+  - sf-scheduler lambda function uses collection cmr_provider to fill provider in cmr section of message template meta
+  - enqueueParsePdrMessage, enqueueGranuleIngestMessage, enqueueWorkflowMessage also use collection cmr_provider to fill provider in cmr section of message template meta
+  - enqueueWorkflowMessage (used in queue-workflow task) uses collection cmr_provider to fill provider in cmr section of message template meta
+  - enqueueParsePdrMessage (used in queue-pdrs task) uses collection cmr_provider to fill provider in cmr section of message template meta
+  - enqueueGranuleIngestMessage (used in queue-granules task) uses collection cmr_provider to fill provider in cmr section of message template meta
+
 ### Fixed
 
 - **Security Vulnerabilities**
@@ -90,6 +107,7 @@ Phase 2 — full apply to clean up old S3 objects and apply remaining changes:
 
 - **CUMULUS-4564**
   - Added private_api_lambda_arn as an output in the Cumulus terraform module
+
 - **CUMULUS-4473**
   - Updated Granules Bulk Operations API endpoints to support `granuleInventoryReportName` and
     `s3GranuleIdInputFile` in the payload. `batchSize` added as an optional parameter for

docs/upgrade-notes/collection_cmr_provider_db_update.md

@@ -0,0 +1,88 @@
+---
+id: collection-cmr-provider-db-update
+title: Update collection schema to include cmr_provider
+hide_title: false
+---
+
+## Background
+
+the updated Collection db model can be found [here](https://github.com/nasa/cumulus/blob/master/packages/db/src/types/collection.ts)
+
+the cmr_provider field now replaces the stack-wide configured cmr provider on a per-collection basis. This field cannot be null, but this presents a problem for existing databases, where collections already exist. as a result an update must be run manually.
+
+## Upgrade Steps
+
+1. Login into EC2 instance with database access
+
+    From AWS console: Go to EC2, pick a `<prefix>-CumulusECSCluster` instance, click Connect, click Session Manager
+    and click the Connect button.
+
+    From AWS CLI: aws ssm start-session --target `EC2 Instance ID`.
+
+    :::note Remember to take a note on which instance you run the commands.
+
+2. Install tmux and postgres client
+
+    ```sh
+    sudo yum install -y tmux
+    # Amazon Linux 2
+    sudo amazon-linux-extras install postgresql17
+    # Amazon Linux 2023
+    sudo dnf install -y postgresql17
+    ```
+
+    Once installed, a tmux session is started with two windows, the Cumulus database is connected to each window
+    using the PostgreSQL client. The primary window is used for running the `CREATE INDEX` commands, while the secondary
+    window is used to monitor the database and `CREATE INDEX` statement. The tmux session can be detached from and
+    reattached to at a later time.
+
+3. Run SQL commands
+
+    The database login credentials can be retrieved from the prefix_db_login secret.
+    When the SQL commands are running, perform step 5 to monitor the commands.
+
+    ```sh
+    tmux new-session -s CumulusUpgrade -n CMRProvider
+
+    psql -h <Endpoint for writer instance> -p <Port for database or 5432> -d <cumulus database name> -U <database admin user> -W
+    #e.g. psql -h cumulus-dev-rds-cluster.cluster-xxx.us-east-1.rds.amazonaws.com -p 5432 -d cumulus_test_db -U cumulus_test -W
+    ```
+
+    The following are the relevant SQL commands. no sql file is provided to prevent confusion as there's a value that must be configured and not confused. Replace <cmr_provider> with your stack's cmr_provider.
+
+    ```sql
+    ALTER TABLE collections ADD COLUMN IF NOT EXISTS cmr_provider TEXT NOT NULL DEFAULT '<cmr_provider>';
+    CREATE INDEX CONCURRENTLY IF NOT EXISTS collections_cmr_provider ON collections (cmr_provider);
+    ```
+
+4. Monitor the running command
+
+    ```sh
+    # From tmux CumulusUpgrade session, open another window
+    Ctrl-b c
+
+    psql -h <Endpoint for writer instance> -p <Port for database or 5432> -d <cumulus database name> -U <database admin user> -W
+
+    select pid, query, state, wait_event_type, wait_event from pg_stat_activity where state = 'active';
+    ```
+
+    These commands should take only minutes, even on large databases.
+
+5. Verify the updates
+
+     We can verify that the tables are updated successfully by checking the `\d collections` results from psql, the column "cmr_provider" should now appear.
+
+     If the concurrent index query fails for any reason, you may have an `invalid` index - if this occurs,
+     make sure to drop and create the index again to avoid resources being used for the invalid index.
+
+6. Close the session
+
+    Close the tmux session after the task is complete by `exit` or `Ctrl-b x`.
+
+## DB Migration notes
+
+During migration of collections over to the consolidated db, or any other migration from a db without this column over to a db with this column, there will need to be a cmr_provider added.
+
+## Task Onboarding notes
+
+Any task which interacts with the message template will find that meta.cmr.provider is now *null*. any such task will need to be updated to fill that value appropriately. see joinCollectionProviderToTemplateCmrMeta [here](https://github.com/nasa/cumulus/blob/master/packages/ingest/src/queue.js)

example/data/collections/ASCATB-L2-Coastal/ASCATB-L2-Coastal.json

@@ -28,5 +28,6 @@
   "name": "ASCATB-L2-Coastal/op",
   "sampleFileName": "ascat_20121029_010301_metopb_00588_eps_o_coa_2101_ovw.l2.nc",
   "url_path": "{cmrMetadata.CollectionReference.ShortName}",
-  "version": "Operational/Near-Real-Time"
+  "version": "Operational/Near-Real-Time",
+  "cmrProvider": "CUMULUS"
 }

example/data/collections/L2_HR_PIXC-000-unique/L2_HR_PIXC-000.json

@@ -25,5 +25,6 @@
   "meta": {
     "uniquifyGranuleId": true,
     "hashLength": 6
-  }
+  },
+  "cmrProvider": "CUMULUS"
 }

example/data/collections/L2_HR_PIXC-000/L2_HR_PIXC-000.json

@@ -21,5 +21,6 @@
       "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.cmr.xml",
       "type": "metadata"
     }
-  ]
+  ],
+  "cmrProvider": "CUMULUS"
 }

example/data/collections/L2_HR_PIXC-099/L2_HR_PIXC-099.json

@@ -25,5 +25,6 @@
   "meta": {
     "uniquifyGranuleId": true,
     "hashLength": 6
-  }
+  },
+  "cmrProvider": "CUMULUS"
 }

example/data/collections/http_testcollection_001/http_testcollection_001.json

@@ -20,5 +20,6 @@
       "bucket": "internal",
       "type": "metadata"
     }
-  ]
+  ],
+  "cmrProvider": "CUMULUS"
 }

example/data/collections/http_testcollection_002/http_testcollection_002.json

@@ -20,5 +20,6 @@
       "bucket": "internal",
       "type": "metadata"
     }
-  ]
+  ],
+  "cmrProvider": "CUMULUS"
 }

example/data/collections/https_testcollection_001/https_testcollection_001.json

@@ -20,5 +20,6 @@
       "bucket": "internal",
       "type": "metadata"
     }
-  ]
+  ],
+  "cmrProvider": "CUMULUS"
 }

example/data/collections/s3_MOD09GQ-AZ_006_full_ingest_move/s3_MOD09GQ-AZ_006.json

@@ -49,5 +49,6 @@
       "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf\\.md5$",
       "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf.md5"
     }
-  ]
+  ],
+  "cmrProvider": "CUMULUS"
 }