add secret plugin for multiple providers aws, gcp, azure

README.md

@@ -45,6 +45,9 @@ Install the extras you need for your project. For example, to install the Flyte
 * *torch:* Utilities for working with PyTorch
 * *monitoring:* Utilities for monitoring training jobs
 * *wandb:* Utilities for logging to Weights and Biases
+* *aws_secrets:* AWS Secret Manager integration for Hydra
+* *gcp_secrets:* GCP Secret Manager integration for Hydra
+* *azure_secrets:* Azure Key Vault integration for Hydra
 * *dev:* Dependencies required for development and running tests
 
 ## 📖 Documentation

docs/usage/subpackages.rst

@@ -8,6 +8,7 @@ Scaffold has several loosely coupled subpackages, which are motivated and their
    :caption: Contents:
 
    subpackages/hydra
+   subpackages/secrets_plugins
    subpackages/iterstream
    subpackages/catalog
    subpackages/artifact_manager

docs/usage/subpackages/secrets_plugins.rst

@@ -0,0 +1,145 @@
+.. _secrets-plugins:
+
+Secrets Plugins
+================
+
+Scaffold provides Hydra plugins for seamless integration with cloud secret managers. These plugins register custom OmegaConf resolvers that fetch secrets on-demand during configuration resolution, allowing secure injection of secrets into your configs without hardcoding values.
+
+**Supported Providers:**
+
+* AWS Secrets Manager (``aws_secrets`` extra)
+* Google Cloud Secret Manager (``gcp_secrets`` extra)
+* Azure Key Vault (``azure_secrets`` extra)
+
+Features
+--------
+
+* **Lazy Evaluation:** Secrets are fetched only when accessed, not at config load time
+* **Automatic Registration:** Plugins auto-register on Hydra initialization via plugin discovery
+* **JSON Support:** Extract specific keys from JSON-formatted secrets
+* **No Hardcoded Values:** Config files show only resolver strings, never actual secrets
+* **Type Safe:** Proper error messages for misconfiguration
+
+Installation
+------------
+
+Install the cloud provider extra you need:
+
+.. code-block:: bash
+
+    # AWS
+    pip install mxm-scaffold[aws_secrets]
+
+    # Google Cloud
+    pip install mxm-scaffold[gcp_secrets]
+
+    # Azure
+    pip install mxm-scaffold[azure_secrets]
+
+Or install all secrets plugins:
+
+.. code-block:: bash
+
+    pip install mxm-scaffold[all]
+
+Google Cloud Secret Manager (GCP)
+----------------------------------
+
+The GCP plugin provides a ``gcp_secret`` resolver for accessing secrets from GCP Secret Manager.
+
+Configuration in YAML
+""""""""""""""""""""""
+
+.. code-block:: yaml
+
+    database:
+      # Simple string secret
+      password: "${gcp_secret:my-project-id/db-password}"
+
+      # Extract a key from a JSON secret
+      username: "${gcp_secret:my-project-id/db-credentials,username}"
+      api_key: "${gcp_secret:my-project-id/db-credentials,api_key}"
+
+    auth:
+      # Using full resource name
+      token: "${gcp_secret:projects/my-project-123/secrets/auth-token/versions/latest}"
+
+Authentication
+"""""""""""""""
+
+GCP Secret Manager uses Application Default Credentials. Ensure your environment is configured:
+
+.. code-block:: bash
+
+    export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
+
+Testing with Real GCP Resources
+"""""""""""""""""""""""""""""""""
+
+To test the GCP plugin with actual secrets:
+
+1. **Install dependencies:**
+
+   .. code-block:: bash
+
+       uv sync --extra gcp_secrets
+
+2. **Update the test configuration** at ``test/hydra_plugins/manual/gcp_secret_real_case.yaml``:
+
+   Replace ``my-project-id`` and secret names with your actual GCP project ID and secret names:
+
+   .. code-block:: yaml
+
+       secrets:
+         service_token: "${gcp_secret:YOUR-GCP-PROJECT-ID/your-secret-name}"
+         db_username: "${gcp_secret:YOUR-GCP-PROJECT-ID/db-credentials,username}"
+         db_password: "${gcp_secret:YOUR-GCP-PROJECT-ID/db-credentials,password}"
+
+3. **Set up GCP credentials:**
+
+   .. code-block:: bash
+
+       export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
+
+4. **Run the integration test:**
+
+   .. code-block:: bash
+
+       python test/hydra_plugins/manual/run_gcp_secret_real_case.py
+
+   This will resolve all ``gcp_secret`` interpolations against real GCP Secret Manager. By default, secret values are redacted for safety. To see full values (on secure terminals only):
+
+   .. code-block:: bash
+
+       python test/hydra_plugins/manual/run_gcp_secret_real_case.py --show-values
+
+AWS Secrets Manager
+--------------------
+
+The AWS plugin provides an ``aws_secret`` resolver for accessing secrets from AWS Secrets Manager.
+
+Configuration in YAML
+""""""""""""""""""""""
+
+.. code-block:: yaml
+
+    database:
+      password: "${aws_secret:my-database-secret}"
+      username: "${aws_secret:my-database-secret,username}"
+
+See ``src/hydra_plugins/aws_secrets_plugin/README.md`` for detailed documentation.
+
+Azure Key Vault
+----------------
+
+The Azure plugin provides an ``azure_secret`` resolver for accessing secrets from Azure Key Vault.
+
+Configuration in YAML
+""""""""""""""""""""""
+
+.. code-block:: yaml
+
+    database:
+      password: "${azure_secret:my-vault,my-secret}"
+
+See ``src/hydra_plugins/azure_secrets_plugin/README.md`` for detailed documentation.

pyproject.toml

@@ -25,6 +25,11 @@ Repository = "https://github.com/merantix-momentum/scaffold-core/"
 Documentation = "https://docs.scaffold.merantix-momentum.cloud/"
 Homepage = "https://merantix-momentum.com"
 
+[project.entry-points."hydra_plugins"]
+aws_secrets = "hydra_plugins.aws_secrets_plugin.aws_secrets_plugin:register_aws_resolver"
+gcp_secrets = "hydra_plugins.gcp_secrets_plugin.gcp_secrets_plugin:register_gcp_resolver"
+azure_secrets = "hydra_plugins.azure_secrets_plugin.azure_secrets_plugin:register_azure_resolver"
+
 [project.optional-dependencies]
 # data
 data = [
@@ -74,6 +79,20 @@ aim = [
     "gcsfs>=2024.12.0",
 ]
 
+# secrets managers
+aws_secrets = [
+    "boto3",
+]
+
+gcp_secrets = [
+    "google-cloud-secret-manager",
+]
+
+azure_secrets = [
+    "azure-identity",
+    "azure-keyvault-secrets",
+]
+
 [dependency-groups]
 # development dependencies
 dev = [

src/hydra_plugins/aws_secrets_plugin/README.md

@@ -0,0 +1,55 @@
+# AWS Secrets Manager Plugin
+
+This plugin registers an OmegaConf custom resolver for AWS Secrets Manager, enabling secure lazy-loaded secret injection into Hydra configurations.
+
+## Installation
+
+### Option 1: Install with optional dependency (recommended)
+
+```bash
+pip install mxm-scaffold[aws_secrets]
+```
+
+### Option 2: Install SDK directly
+
+```bash
+pip install boto3
+```
+
+## Usage in YAML Configs
+
+```yaml
+database:
+  password: "${aws_secret:secret_name, key_name}"
+  host: "prod-db.example.com"
+```
+
+### Arguments
+
+- `secret_id`: The name or ARN of the secret in AWS Secrets Manager
+- `key` (optional): If the secret is JSON-formatted, extract this specific key from it
+
+## Examples
+
+### Simple String Secret
+
+```yaml
+api:
+  token: "${aws_secret:prod/api/token}"
+```
+
+### JSON Secret with Key Extraction
+
+```yaml
+database:
+  password: "${aws_secret:prod/db/credentials, password}"
+  username: "${aws_secret:prod/db/credentials, username}"
+```
+
+## Features
+
+- **Lazy Evaluation**: Secrets are fetched only when accessed, not at config load time
+- **No Hardcoded Values**: Config files show only the resolver string, never the actual secret
+- **Automatic Registration**: The plugin auto-registers on Hydra initialization
+- **JSON Support**: Extract specific keys from JSON-formatted secrets
+- **Type Safe**: Proper error messages for misconfiguration

src/hydra_plugins/aws_secrets_plugin/__init__.py

@@ -0,0 +1,5 @@
+"""AWS Secrets Manager resolver for Hydra.
+
+Registers a custom OmegaConf resolver for AWS Secrets Manager to enable secure,
+lazy-loaded secret injection into Hydra configurations.
+"""

src/hydra_plugins/aws_secrets_plugin/aws_secrets_plugin.py

@@ -0,0 +1,65 @@
+"""AWS Secrets Manager resolver plugin.
+
+This module automatically registers an OmegaConf resolver for AWS Secrets Manager
+when Hydra initializes. Secrets are fetched lazily on-demand.
+
+Usage in YAML configs:
+    password: "${aws_secret:secret_name, key_name}"
+"""
+import json
+from typing import Optional
+
+from omegaconf import OmegaConf
+
+
+def get_aws_secret(secret_id: str, key: Optional[str] = None) -> str:
+    """Fetch a secret from AWS Secrets Manager.
+
+    Args:
+        secret_id: The name or ARN of the secret in Secrets Manager.
+        key: Optional key if the secret is JSON. If provided, returns the value
+             for that key from the JSON secret string.
+
+    Returns:
+        The secret value as a string.
+
+    Raises:
+        ImportError: If boto3 is not installed.
+        Exception: If the secret cannot be retrieved from AWS.
+    """
+    try:
+        import boto3
+    except ImportError:
+        raise ImportError(
+            "boto3 is not installed. Install it with: pip install boto3"
+        )
+
+    client = boto3.client("secretsmanager")
+    response = client.get_secret_value(SecretId=secret_id)
+
+    secret_str = response.get("SecretString") or response.get("SecretBinary", "")
+
+    if key:
+        try:
+            secret_dict = json.loads(secret_str)
+            return secret_dict.get(key, "")
+        except json.JSONDecodeError:
+            raise ValueError(
+                f"Secret '{secret_id}' is not valid JSON. "
+                f"Cannot extract key '{key}'."
+            )
+
+    return secret_str
+
+
+def register_aws_resolver() -> None:
+    """Register the AWS Secrets Manager resolver with OmegaConf.
+
+    This function is called automatically when Hydra initializes due to the
+    plugin discovery mechanism.
+    """
+    OmegaConf.register_new_resolver("aws_secret", get_aws_secret, replace=True)
+
+
+# Automatically register resolver when this module is imported
+register_aws_resolver()