Cumulus 4388 add cnm2cma (#4149)

* cnm2cma coreification python code with toml file

* move cnm2cma module into tasks

* added unit test code

* fixed import

* CHANGELOG

* New implementation through Pydantic

* unit test files

* terraform module.  ruff linting/reformatting python code

* added README.  restructure the project

* enhance README.md

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* CHANGELOG

* use SyncGranule's input schema as this component's output.  Added granuleProducerId to cnm schema

* fixed path issue. Simplified build_granule_file and rewrite unit test code

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* write ruff.toml file to ignore generated files.  remove unused model

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ruff format

* added ruff ignore and CHANGELOG

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* build scripts

* package.json and package.sh

* code and tf script enhancements based on review.

* config object schema

* used protocol extraction to simplify the logic.  Enhanced terraform scripts

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* new package.sh

* build script checked in

* eyen ci stack

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove single quotes to overcome eslint warning

* Update lambda.tf to use cnm_to_cma as log group

* move schemas under the src directory so they will be packaged as part of the artifact

* clean up changelog

* fixed the build package.sh path issue

* cleanup old build script

* removed .python-version

* output the lambda cloudwatch loggroup name

* output the lambda function instead of just the arn

---------

Co-authored-by: eyen <david.yen@jpl.nasa.gov>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: 3 people

.python-version

@@ -19,6 +19,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - **CUMULUS-4473**
   - Updated Granules Bulk Operations API endpoints to support `granuleInventoryReportName` and
     `s3GranuleIdInputFile` in the payload.
+- **CUMULUS-4388**
+  - Added cnm_to_cma task (lambda).
+  - Original cnm_to_cma was written in Java.  Converted to Python.
 - **CUMULUS-4382**
   - Migrated the granule-invalidator task to the `tasks` directory as part of a coreification task in support of providing rolling archive functionality.
 - **CUMULUS-4385**

CHANGELOG.md

@@ -23,6 +23,7 @@ function determineIntegrationTestStackName(cb) {
     dhudelson: 'dh-ci',
     etcart: 'ecarton-ci',
     ecarton: 'ecarton-ci',
+    eyen: 'eyen-ci',
     'Filip Graniczny': 'fg-ci',
     'James Norton': 'jn-ci',
     'Jenny Liu': 'jl-rds',

bamboo/select-stack.js

@@ -174,7 +174,7 @@ These steps should be performed in the `example` directory.
 Copy `.env.sample` to `.env`, filling in appropriate values for your deployment.
 
 Set the `DEPLOYMENT` environment variable to match the `prefix` that you
-configured in your `terraform.tfvars` files.
+configured in your `terraform.dtfvars` files.
 
 If you are deploying outside of NGAP, you may need to update `example/config.yml` to override custom settings for your deployment such as the default bucket, your Earth Data Login username for login tests, or the value of `pdrNodeNameProviderBucket`.
 

example/README.md

@@ -35,6 +35,9 @@ cdd-tf:
 ecarton:
   bucket: ecarton-internal
 
+eyen:
+  bucket: eyen-internal
+
 dms-opex-sbx:
   bucket: dms-opex-sbx-internal
 

example/config.yml

@@ -287,6 +287,7 @@ variable "archive_api_users" {
     "dhudelson",
     "dmsorensen",
     "ecarton",
+    "eyen",
     "jasmine",
     "jennyhliu",
     "jmccoy_uat",

example/cumulus-tf/variables.tf

@@ -0,0 +1 @@
+prefix = "eyen-ci-tf"

example/deployments/cumulus/eyen-ci-tf.tfvars

@@ -0,0 +1 @@
+prefix = "eyen-ci-tf"

example/deployments/data-persistence/eyen-ci-tf.tfvars

@@ -0,0 +1 @@
+prefix = "eyen-ci-tf"

example/deployments/db-migration/eyen-ci-tf.tfvars

@@ -0,0 +1,6 @@
+.venv
+.coverage
+build
+__pycache__
+coverage_html
+.nyc_output