Add v2 modularization CI gates, release workflow, and runbook (#1856)

* add v2 modularization CI gates, release workflow, and runbook

Five gate scripts (acyclic deps, single source, SIV placement, no replaces, release-mode build), a checks workflow, a manual lockstep release workflow, and the release runbook. Structural gates skip until the /v2 submodules exist, so they land green now and enforce automatically once the split arrives.

* address review: harden v2 CI gates and release workflow

- v2-release.yml: drop to least-privilege (read default, write only on the tag-push job) and run the release-mode build in pre-flight so an unbuildable commit cannot be tagged.
- check-acyclic-deps.sh: fail closed when a module or imported module is missing from the tier map, instead of defaulting to a permissive top tier.
- check-no-replaces.sh: catch block-form replace directives, not just the single-line form.

* address second review pass on v2 CI scaffolding

- check-release-mode.sh: split the cleanup checkout so a missing go.work.sum can no longer abort the whole revert and leave modules/*/go.mod dirty (git checkout treats multi-pathspec atomically).
- v2-release.yml: validate the version input looks like v2.x before pushing immutable tags; drop a stray whitespace line in the ORDER array.
- runbook: point step 4 at check-no-replaces.sh instead of a single-line grep that misses block-form replaces.

* address review comments on v2 CI scaffolding

- v2-checks.yml: run on pull_request + push to main only (was every branch push, duplicating PR runs).
- Pin actions/checkout and jdx/mise-action to the SHAs already used elsewhere in the repo (both workflows).
- Drop the executable bit on the gate scripts; they are invoked via 'bash scripts/...'.
- check-release-mode.sh: rename B to MODULE_BASE; move function declarations ahead of procedural code; scope the repo-root cd inside main(); keep all scratch files under a single mktemp -d instead of polluting /tmp; derive the consumer import list from the ORDER list instead of duplicating it.

* address overlooked issues from second review pass

Shell (from independent review):
- check-siv-placement.sh: grep -o so the /v2 filter inspects the import token, not the whole source line (a comment or second import could mask a real SIV bug); add the pre-split skip-guard; drop echo|sed.
- check-release-mode.sh: guard the repo-root cd; note core leaf packages must stay in sync.
- check-no-replaces.sh: replace echo|sed with a here-string.
- Document the intentional lack of -e in the failure-accumulating gates.

Workflows/docs (from independent review):
- Fix the checkout pin comment: the SHA is v6, not v4.2.2.
- v2-release.yml: add job timeouts; single-source the module list via a job-level env used by both the tag and proxy steps; tighten the version regex to full semver; make the post-push proxy check a warning so a slow proxy does not fail an already-pushed release.
- v2-checks.yml: add a cancel-in-progress concurrency group.
- Runbook: point at the automated release workflow and clarify the external consumer is generated by check-release-mode.sh, not a committed test-external dir.

* drop the success echo in check-no-replaces

* address CodeRabbit re-review: worktree guard and read-only checkout creds

- check-release-mode.sh: refuse to run against a dirty modules/ or go.work.sum tree, so the cleanup trap cannot discard a developer's uncommitted work on a local run.
- v2-checks.yml and the v2-release verify job: set persist-credentials: false on the read-only checkouts (zizmor artipacked). The tag-push checkout keeps credentials since it pushes tags.

* drop the no-replaces gate

Remove check-no-replaces.sh, the release-prep-guard job in v2-checks, and the pre-flight call in v2-release. go.work keeps go.mods clean during development and replaces are dropped at release-prep, so the separate gate is not needed. The runbook keeps a manual reminder to confirm no replace directives remain before tagging.

* release-mode gate: derive module list from disk to support a partial split

Replaces the hardcoded 16-module ORDER with one derived from which modules have a go.mod, so the gate validates a single-tier split (e.g. core only) as well as the full split. Also exercises all core leaf packages in the consumer and drops the hardcoded '16 modules' wording. Inert until the first submodule lands (the pre-split guard still skips).

Author: james00012

.github/workflows/v2-checks.yml

@@ -0,0 +1,58 @@
+name: v2 Modularization Checks
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+# Cancel superseded runs on the same ref; these gates are cheap and idempotent.
+concurrency:
+  group: v2-checks-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  acyclic-deps:
+    name: Acyclic dependency graph
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+        with:
+          persist-credentials: false
+      - run: bash scripts/check-acyclic-deps.sh
+
+  single-source:
+    name: Single source of truth per package path
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+        with:
+          persist-credentials: false
+      - run: bash scripts/check-single-source.sh
+
+  siv-placement:
+    name: /v2 SIV placement
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+        with:
+          persist-credentials: false
+      - run: bash scripts/check-siv-placement.sh
+
+  consumer-simulation:
+    name: External consumer simulation (GOWORK=off)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+        with:
+          persist-credentials: false
+
+      - name: Set up mise
+        uses: jdx/mise-action@5228313ee0372e111a38da051671ca30fc5a96db # v3
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Release-mode build (each module pins at root versions + external consumer, GOWORK=off)
+        run: bash scripts/check-release-mode.sh

.github/workflows/v2-release.yml

@@ -0,0 +1,121 @@
+name: v2 Release (lockstep tag push)
+
+# Manual-trigger workflow that pushes the full v2 tag set from a release-prep commit.
+# Procedure documented in docs/v2-release-runbook.md.
+#
+# Use this AFTER:
+#   - scripts/release-prep.sh ran on the release-prep branch
+#   - The release-prep commit was reviewed and approved
+
+on:
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Version to tag (e.g. v2.0.0-beta.1). Applied to every submodule.'
+        required: true
+        type: string
+      commit_sha:
+        description: 'SHA of the release-prep commit to tag.'
+        required: true
+        type: string
+      dry_run:
+        description: 'If true, print the tag commands without pushing.'
+        required: false
+        type: boolean
+        default: true
+
+# Default to read-only. Only the tag-pushing job is granted write, so the
+# pre-flight job cannot push even though it runs scripts from a dispatched SHA.
+permissions:
+  contents: read
+
+jobs:
+  verify:
+    name: Pre-flight checks
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+        with:
+          ref: ${{ inputs.commit_sha }}
+          persist-credentials: false
+      - run: bash scripts/check-acyclic-deps.sh
+      - run: bash scripts/check-single-source.sh
+      - run: bash scripts/check-siv-placement.sh
+
+      - name: Set up mise
+        uses: jdx/mise-action@5228313ee0372e111a38da051671ca30fc5a96db # v3
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      - name: Release-mode build (GOWORK=off external consumer)
+        run: bash scripts/check-release-mode.sh
+
+  tag-and-push:
+    name: Push lockstep tags
+    needs: verify
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: write
+    env:
+      VERSION: ${{ inputs.version }}
+      # Single source of truth for the module list and tag order (mirrors ORDER
+      # in scripts/check-release-mode.sh and the runbook tag sequence).
+      MODULE_PATHS: >-
+        modules/core
+        modules/ssh modules/httphelper modules/dnshelper
+        modules/docker modules/packer modules/database modules/opa
+        modules/aws modules/azure modules/gcp modules/k8s modules/helm
+        modules/terraform modules/terragrunt modules/teststructure
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+        with:
+          ref: ${{ inputs.commit_sha }}
+          fetch-depth: 0
+
+      - name: Configure git
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+
+      - name: Tag in dependency order
+        env:
+          DRY_RUN: ${{ inputs.dry_run }}
+        run: |
+          set -e
+
+          if [[ ! "$VERSION" =~ ^v2\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?$ ]]; then
+            echo "::error::version '$VERSION' must look like v2.<minor>.<patch>[-suffix] (e.g. v2.0.0-beta.1)"
+            exit 1
+          fi
+
+          for path in $MODULE_PATHS; do
+            tag="${path}/v2/${VERSION}"
+            if [ "$DRY_RUN" = "true" ]; then
+              echo "DRY-RUN would tag: $tag"
+            else
+              git tag -a "$tag" -m "v2 lockstep release $VERSION"
+              git push origin "$tag"
+              echo "pushed $tag"
+            fi
+          done
+
+      - name: Verify proxy serves each tag
+        if: ${{ inputs.dry_run == false }}
+        run: |
+          # Allow the proxy a moment to populate after push.
+          sleep 30
+          fail=0
+          for path in $MODULE_PATHS; do
+            url="https://proxy.golang.org/github.com/gruntwork-io/terratest/${path}/v2/@v/${VERSION}.info"
+            code=$(curl -s -o /dev/null -w '%{http_code}' "$url")
+            if [ "$code" = "200" ]; then
+              echo "  $path: 200"
+            else
+              echo "::warning::$path: $code from $url"
+              fail=1
+            fi
+          done
+          if [ "$fail" -ne 0 ]; then
+            echo "::warning::Proxy has not served every tag yet. The tags are already pushed and immutable; the proxy usually populates within a few minutes. Re-run this check if needed."
+          fi

docs/v2-release-runbook.md

@@ -0,0 +1,93 @@
+# Terratest v2 Release Runbook
+
+How to cut a coordinated release of the v2 submodules. Read this before tagging.
+
+The tag push and proxy verification are automated by the `v2 Release` workflow
+(`.github/workflows/v2-release.yml`, run via workflow_dispatch). This runbook
+explains the procedure that workflow follows and how to do it by hand if needed.
+
+## Layout
+
+v2 is split into per-domain modules under `modules/<name>/`, each declaring
+`module github.com/gruntwork-io/terratest/modules/<name>/v2`. Local development
+uses the root `go.work`, which resolves every submodule to its local tree, so the
+submodule `go.mod` files do not need internal `require` lines or `replace`
+directives during normal development.
+
+## Why pinning is a release-time step (do not commit it early)
+
+The submodules' `go.mod` files are deliberately left without cross-module
+`require` lines on `main`. Pinning a sibling `require` to the to-be-published
+version (e.g. `core/v2 v2.0.0-beta.1`) BREAKS the workspace build until that tag
+actually exists: `go.work` does not shadow an unpublished required version, so
+`go build` and `go work sync` try to fetch the missing revision and fail. The pin
+must therefore happen on a short-lived release-prep branch, immediately before the
+tags are pushed, never on the modularization PR.
+
+CI validates the release-mode build continuously without committing the pin: the
+`GOWORK=off` check generates the pinned state with throwaway `replace` directives,
+builds a consumer, and discards it (see `scripts/`).
+
+## Pre-flight (on a release-prep branch)
+
+1. Choose the version, e.g. `v2.0.0-beta.1`.
+2. Pin each module, in dependency order (core first, then helpers, tooling,
+   platforms, k8s/helm, IaC). For each module:
+   - Add a temporary `replace` for EVERY sibling it transitively needs, not just
+     its direct imports. Tidy follows transitive edges, so a partial replace set
+     fails with `unknown revision` on a deeper sibling.
+   - Seed the module with the root `go.mod`'s exact dependency versions before
+     tidying (copy its `require` lines in). Otherwise tidy floats deps to the
+     latest compatible release and you get silent version drift, e.g. the Azure
+     SDK advancing to a release that drops a symbol the code uses
+     (`undefined: armmonitor.DiagnosticSettingsClient`). The code is tested
+     against the root's pinned versions; the submodules must inherit them.
+   - `GOWORK=off go mod tidy` to populate external `require`s and `go.sum`.
+
+`scripts/check-release-mode.sh` performs this exact pin (transitive replaces +
+root-version seeding + an all-module external consumer build under `GOWORK=off`)
+in a throwaway and reverts it, and runs in CI on every PR so the release-mode
+build is validated continuously without committing the pin or needing tags.
+3. Set every internal `require` to the exact version being tagged, then DROP all
+   internal `replace` directives. Do not run `go work sync` against the unpinned
+   tree.
+4. Before tagging, confirm no internal `replace` directives remain in any
+   `modules/*/go.mod`. A committed `replace` would publish a module pointing at a
+   local path and break consumers.
+5. Move `test/` to its own module here too if not already done, and pin it the
+   same way (it is test-only, so committed `replace`s are acceptable for it).
+
+## Tag push order
+
+All tags point at the same release commit. Each tag name puts the `/v2` SIV in the
+tag itself: `modules/<name>/v2/<version>` (NOT `modules/<name>/<version>`, which
+the proxy cannot associate with the `/v2` module path). Push in dependency order:
+
+1. `modules/core/v2/v2.0.0-beta.1`
+2. helpers: `ssh`, `httphelper`, `dnshelper`
+3. tooling: `docker`, `packer`, `database`, `opa`
+4. platforms: `aws`, `azure`, `gcp`, then `k8s`, `helm`
+5. IaC: `terraform`, `terragrunt`, `teststructure`
+
+After each tier, probe the proxy before continuing:
+`curl -o /dev/null -w '%{http_code}' https://proxy.golang.org/github.com/gruntwork-io/terratest/modules/<name>/v2/@v/<version>.info`
+should return 200.
+
+## Verify
+
+Build a throwaway external consumer that imports every published module with
+`GOWORK=off`. This is what `scripts/check-release-mode.sh` generates in a temp
+directory, except now it resolves against the real published tags rather than
+local `replace`s. A clean consumer should resolve, build, and test green with
+zero local references.
+
+## If a tag is wrong
+
+Proxy tags are immutable. Recover by cutting the next patch (`v2.0.0-beta.2`),
+never by editing in place. The pre-flight checks exist to keep this rare.
+
+## Beta to GA
+
+After the beta soaks (suggested two weeks minimum), repeat the same procedure at
+`v2.0.0` with no suffix: same release commit shape, same tag sequence, same proxy
+verification, then announce.

scripts/check-acyclic-deps.sh

@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+# check-acyclic-deps.sh — fails CI if any submodule's production code imports a
+# module from a strictly higher tier. Enforces the v2 layering rule:
+# core → helpers → tooling → platforms → IaC, downward-only.
+#
+# Test files (*_test.go) are excluded; cross-module test-only imports are allowed
+# (e.g. modules/core/logger/parser_test imports modules/shell/v2 — legal per the
+# RFC's external _test package rule).
+
+# No -e: we accumulate every tier violation and report them all, rather than
+# aborting on the first one.
+set -uo pipefail
+
+# Tier assignment. Lower number = lower layer.
+declare -A TIER=(
+  [core]=0
+  [ssh]=1
+  [httphelper]=1
+  [dnshelper]=1
+  [docker]=2
+  [packer]=2
+  [database]=2
+  [opa]=2
+  [aws]=3
+  [azure]=3
+  [gcp]=3
+  [k8s]=3
+  [helm]=3
+  [terraform]=4
+  [terragrunt]=4
+  [teststructure]=4
+)
+
+# Pre-split guard: the tier rule only holds once packages are relocated into
+# their /v2 submodules. Until then the flat tree has no tiers to enforce.
+if ! ls modules/*/go.mod >/dev/null 2>&1; then
+  echo "acyclic-deps check: skipped (no /v2 submodules present yet)"
+  exit 0
+fi
+
+fail=0
+
+for dir in modules/*/; do
+  importer=$(basename "$dir")
+  # Only real submodules carry a go.mod and a tier; skip anything else.
+  [ -f "$dir/go.mod" ] || continue
+  if [ -z "${TIER[$importer]+set}" ]; then
+    echo "::error file=${dir}::module '$importer' has no tier; add it to the TIER map in check-acyclic-deps.sh"
+    fail=1
+    continue
+  fi
+  importer_tier="${TIER[$importer]}"
+
+  # Scan all .go files in the submodule recursively, excluding test files.
+  while IFS= read -r gofile; do
+    while IFS= read -r importee; do
+      [ -z "$importee" ] && continue
+      if [ -z "${TIER[$importee]+set}" ]; then
+        echo "::error file=${gofile}::imports unknown module '$importee'; add it to the TIER map in check-acyclic-deps.sh"
+        fail=1
+        continue
+      fi
+      importee_tier="${TIER[$importee]}"
+      if [ "$importee_tier" -gt "$importer_tier" ]; then
+        echo "::error file=${gofile}::tier violation — $importer (tier $importer_tier) imports $importee (tier $importee_tier)"
+        fail=1
+      fi
+    done < <(grep -oE '"github\.com/gruntwork-io/terratest/modules/[a-z][a-z0-9-]*' "$gofile" 2>/dev/null \
+      | awk -F'/' '{print $NF}' \
+      | sort -u)
+  done < <(find "$dir" -name '*.go' -not -name '*_test.go' 2>/dev/null)
+done
+
+if [ "$fail" -ne 0 ]; then
+  echo "::error::Tier-violation imports detected. Imports must flow downward only."
+  exit 1
+fi
+
+echo "acyclic-deps check: OK"