Support __FILE suffix for environment variables to load secrets from files

[bdalpe]

Summary

It would be great if any PREFECT_* environment variable could be sourced from a file by setting a corresponding PREFECT_*__FILE variable that points at the file path. This mirrors the convention popularized by Grafana and used by many other container-friendly projects (Postgres, MySQL, Vault, etc.) for handling secrets in containerized environments.

Motivation

When running Prefect in Docker, Kubernetes, Docker Swarm, or other orchestrators, the recommended way to inject secrets is via mounted files, not environment variables:

• Docker / Docker Swarm secrets are exposed at /run/secrets/<name>.
• Kubernetes secrets can be mounted as files via volumeMounts, which is often preferred over env.valueFrom.secretKeyRef because mounted secrets update automatically when the underlying Secret changes and aren't visible in docker inspect / pod env dumps.
• HashiCorp Vault Agent, AWS Secrets Manager CSI driver, External Secrets Operator, and similar tools all deliver secrets as files on disk.

Today, to use Prefect with any of these, users have to write a custom entrypoint or wrapper script that reads the file and exports the variable before launching Prefect. This is boilerplate that every team reinvents, and it's especially painful for PREFECT_API_KEY, which is the most common thing you'd want to keep out of plaintext env.

Desired behavior

For any environment variable named PREFECT_<NAME>__FILE, Prefect would:

1. Read the contents of the file at that path and use it as the value of PREFECT_<NAME>.
2. Error out clearly if both PREFECT_<NAME> and PREFECT_<NAME>__FILE are set (mutually exclusive).
3. Error out clearly if the file doesn't exist or isn't readable.

Example

# Instead of this:
export PREFECT_API_KEY="pnu_abcdef123456..."

# Users could do this:
export PREFECT_API_KEY__FILE=/run/secrets/prefect_api_key

…and PREFECT_API_KEY would be populated automatically.

Prior art

Projects that implement the __FILE / _FILE convention:

• Grafana — GF_*__FILE run.sh
• Postgres official image — POSTGRES_PASSWORD_FILE, POSTGRES_USER_FILE, etc.
• MySQL / MariaDB official images — MYSQL_*_FILE
• HashiCorp Vault — VAULT_*_FILE
• GitLab Omnibus — various *_FILE variants
• Bitnami images broadly support *_FILE for any env var

Reference: Grafana's implementation

For reference, Grafana does this in its container entrypoint as a shell loop:

for VAR_NAME in $(env | grep '^GF_[^=]\+__FILE=.\+' | sed -r "s/([^=]*)__FILE=.*/\1/g"); do
    VAR_NAME_FILE="$VAR_NAME"__FILE
    if [ "${!VAR_NAME}" ]; then
        echo >&2 "ERROR: Both $VAR_NAME and $VAR_NAME_FILE are set (but are exclusive)"
        exit 1
    fi
    echo "Getting secret $VAR_NAME from ${!VAR_NAME_FILE}"
    export "$VAR_NAME"="$(< "${!VAR_NAME_FILE}")"
    unset "$VAR_NAME_FILE"
done

Open questions

• Scope: should this apply to all PREFECT_* settings uniformly, or only a curated subset (API key, DB password, etc.)?
• Where to implement: in the Docker image entrypoint only, or in Prefect's Pydantic settings loading so it also works for non-Docker installs (systemd, bare metal, library use)?
• Trailing newlines: most tools that follow this convention strip a single trailing newline (matching bash's $(< file) behavior).

[harsh21234i]

I opened PR #21889 to address this.

The implementation adds support for PREFECT_*__FILE environment
variables in the settings source, including nested settings like
PREFECT_SERVER_API_PORT__FILE.

It also handles the profile-related cases so PREFECT_PROFILES_PATH__FILE
and PREFECT_PROFILE__FILE are respected when loading profile settings.

The PR includes tests for:

• loading a setting from a file-backed env var
• nested settings
• conflict handling when both PREFECT_* and PREFECT_*__FILE are set
• missing/unreadable files
• profile loading via PREFECT_PROFILES_PATH__FILE
• active profile selection via PREFECT_PROFILE__FILE