add secret plugin for multiple providers aws, gcp, azure

[MsMatias]

Description

Overview

This PR introduces three independent Hydra plugins that enable secure, lazy-loaded secret injection from AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault directly into Hydra configurations.

Problem Statement

Previously, managing secrets in Hydra configurations was cumbersome:

• Secrets had to be hardcoded or loaded via environment variables
• No native integration with cloud secret managers
• All cloud SDKs had to be installed even if only one was needed
• No lazy evaluation of secrets

Solution

Three new modular plugins that:

• ✅ Register custom OmegaConf resolvers for each cloud provider
• ✅ Support lazy evaluation (secrets fetched only when accessed)
• ✅ Keep secrets out of config files and logs
• ✅ Allow independent installation via optional dependencies
• ✅ Follow Hydra's plugin discovery pattern

Changes

New Plugins

1. AWS Secrets Manager Plugin

• Location: src/hydra_plugins/aws_secrets_plugin/
• Resolver: ${aws_secret:secret_name, key_name}
• Dependencies: boto3
• Installation: pip install mxm-scaffold[aws_secrets]

Example:

database:
  host: "prod-db.example.com"
  username: "admin"
  password: "${aws_secret:prod/db/credentials, password}"

2. GCP Secret Manager Plugin

• Location: src/hydra_plugins/gcp_secrets_plugin/
• Resolver: ${gcp_secret:project_id/secret_name, key_name}
• Dependencies: google-cloud-secret-manager
• Installation: pip install mxm-scaffold[gcp_secrets]

Example:

api:
  endpoint: "https://api.example.com"
  key: "${gcp_secret:my-project/api-credentials, api_key}"

3. Azure Key Vault Plugin

• Location: src/hydra_plugins/azure_secrets_plugin/
• Resolver: ${azure_secret:vault_name, secret_name}
• Dependencies: azure-identity, azure-keyvault-secrets
• Installation: pip install mxm-scaffold[azure_secrets]

Example:

auth:
  region: "us-east-1"
  token: "${azure_secret:my-vault, auth-token}"

Configuration Changes

pyproject.toml:

• Added three new optional dependency groups:
  • aws_secrets: boto3
  • gcp_secrets: google-cloud-secret-manager
  • azure_secrets: azure-identity, azure-keyvault-secrets
• Registered plugin entry points in [project.entry-points."hydra_plugins"]

Testing

Location: test/hydra_plugins/test_resolvers.py

Tests verify:

• Resolver registration with OmegaConf
• Config creation with resolver syntax
• All three providers

Run tests:

.venv/bin/python -m pytest test/hydra_plugins/ -v

Key Features

Lazy Evaluation

Secrets are fetched only when accessed, not at config load time:

# Secret is NOT fetched yet
cfg = OmegaConf.create({"password": "${aws_secret:prod/db}"})

# Secret IS fetched here
password = cfg.password

No Hardcoded Values

Config files show only the resolver string:

# Safe to commit to version control
password: "${aws_secret:prod/db/credentials, password}"

JSON Key Extraction

Extract specific keys from JSON secrets:

# Secret: {"username": "admin", "password": "secret123"}
username: "${aws_secret:db-creds, username}"
password: "${aws_secret:db-creds, password}"

Independent Installation

Users only install what they need:

# AWS only
pip install mxm-scaffold[aws_secrets]

# Multiple providers
pip install mxm-scaffold[aws_secrets,gcp_secrets]

# All three
pip install mxm-scaffold[aws_secrets,gcp_secrets,azure_secrets]

Usage Example

import hydra
from omegaconf import DictConfig

@hydra.main(version_base=None, config_path="conf", config_name="config")
def my_app(cfg: DictConfig) -> None:
    # Secret is fetched lazily when accessed
    db_password = cfg.database.password
    print(f"Connected with password: {db_password}")

if __name__ == "__main__":
    my_app()

Config file (conf/config.yaml):

database:
  host: "prod-db.example.com"
  username: "admin"
  password: "${aws_secret:prod/db/credentials, password}"

Authentication

AWS

Uses standard boto3 credential chain (IAM role, environment variables, ~/.aws/credentials, etc.)

GCP

Uses Application Default Credentials (service account key via GOOGLE_APPLICATION_CREDENTIALS env var or compute instance identity)

Azure

Uses DefaultAzureCredential (environment variables, managed identity, Azure CLI, etc.)

Files Changed

New directories:

• src/hydra_plugins/aws_secrets_plugin/ (3 files)
• src/hydra_plugins/gcp_secrets_plugin/ (3 files)
• src/hydra_plugins/azure_secrets_plugin/ (3 files)
• test/hydra_plugins/ (1 test file)

Modified files:

• pyproject.toml: Added optional dependencies and entry points

Breaking Changes

None. These are additive features.

Migration Guide

No migration needed. Existing code continues to work unchanged.

Testing Recommendations

Unit Tests

pytest test/hydra_plugins/ -v

Integration Tests (with real secrets)

Set up a secret in each cloud provider and test end-to-end:

AWS:

aws secretsmanager create-secret --name test/my-secret --secret-string '{"key": "value"}'

GCP:

echo "my-secret-value" | gcloud secrets create test-secret --data-file=-

Azure:

az keyvault secret set --vault-name my-vault --name test-secret --value my-secret-value

Fixes # issue

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring including code style reformatting
• Other (please describe):

Checklist:

• Lint and unit tests pass locally with my changes
• I have kept the PR small so that it can be easily reviewed
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• All dependency changes have been reflected in the pip requirement files.

[adrianloy]

Docs can be clearer on how to make the resolve work if we execute within the cluster. Overall great idea.

[adrianloy]

would mention the logic of this in the docs

[MsMatias]

I think is self described

[adrianloy]

missing a step on how to put the secret into the gcp secret manager first, fine if we just link there to official docs, but would be nice to have it here.

[MsMatias]

I don't think make sense to put it here

[Pleezon]

Would it make sense to add a logging.Filter to the logger that automatically redacts secrets? I'd imagine it could prevent user error (i.e. logging.debug(cfg)).

[MsMatias]

Would it make sense to add a logging.Filter to the logger that automatically redacts secrets? I'd imagine it could prevent user error (i.e. logging.debug(cfg)).

Wondering if the masking is enough 🤔