Adding a Secondary Region — Verifying Steps & TGW Setup

[jaguer0]

We're looking at adding the secondary region (currently us-east-2 - adding us-west-2). We have the CIDR for the new region figured out, and I came across this doc, but it seems a bit outdated. I wanted to verify whether these are still the recommended steps.

I'm trying to narrow down the proper sequence and understand how much can be copied from existing files (swapping out the mixins, etc.) from the quickstart we received

If possible, I'd also appreciate more detailed guidance on the secondary-region TGW setup. I found a previous discussion & doc on this topic, but they appear to reference older setups that had connections defined and tgw/spokes with peered_region: true. I'm not sure if the process differs for our current setup, which uses tgw/attachment, tgw/routes, and vpc/route. Is the general workflow something like this?

1. Copy files for the new region and update mixins import refrence
2. Deploy VPCs to the new region (in the network, auto, and platform OUs)
3. Deploy tgw/hub to the new region (no vars?)
4. Deploy hub-connector (setting primary_tgw_hub_region to the new region)
5. Deploy tgw/attachment, tgw/routes, and vpc/route, swapping out the !tfstate references to point to the new region

[aknysh]

@jaguer0 thanks for the question.

The docs at https://docs.cloudposse.com/layers/accounts/tutorials/how-to-add-or-mirror-a-new-region/ are obsolete and needs to be updated (we'll be working on it).

For now, please see the doc describing how to provision all TGW components in another region and connect the two regions together via a TGW peering connection. Please let me know if you have questions or need help.

Transit Gateway (TGW) Components

Obsolete Documentation Notice

WARNING: The TGW section in the Cloud Posse docs at
https://docs.cloudposse.com/layers/accounts/tutorials/how-to-add-or-mirror-a-new-region/
is obsolete and needs to be updated. It references the old tgw/spoke pattern with peered_region: true and
inline connections. The current architecture uses separate tgw/hub, tgw/attachment, tgw/routes, and
vpc-routes components as described below.

Overview

The Transit Gateway (TGW) components provide centralized network connectivity across VPCs in multiple AWS accounts.
TGW acts as a regional cloud router — each VPC attachment connects to the TGW, and route tables control traffic flow
between them.

Components

1. tgw/hub — Transit Gateway Hub

Repo: cloudposse-terraform-components/aws-tgw-hub

Creates the Transit Gateway and its route table in the network account. This is the central router that all VPCs
connect to.

• Creates the Transit Gateway resource
• Creates the TGW route table
• Optionally shares via AWS RAM for cross-account attachments
• Does not create VPC attachments (that's tgw/attachment)

Deploy to: core-{region}-network

2. tgw/attachment — VPC Attachment

**Repo
**: cloudposse-terraform-components/aws-tgw-attachment

Creates a Transit Gateway VPC Attachment for a specific VPC. Each account that needs TGW connectivity deploys this
component.

• Attaches the account's VPC to the Transit Gateway
• Uses private subnets for the attachment ENIs
• Reads transit_gateway_id and transit_gateway_route_table_id from tgw/hub via !terraform.state

Deploy to: Every account that needs TGW connectivity (network, auto, sandbox, dev, staging, prod)

3. tgw/routes — TGW Route Table Routes

**Repo
**: cloudposse-terraform-components/aws-tgw-routes

Manages TGW route table entries — route propagation and static routes. Deployed in the network account to define
which VPC attachments can reach each other through the TGW.

• Creates route propagations from VPC attachments
• Creates route associations
• Supports static routes with optional blackhole

Deploy to: core-{region}-network

4. vpc-routes — VPC Route Table Routes

Not a TGW component, but essential for TGW connectivity. Adds routes to VPC route tables pointing traffic destined
for other VPCs to the Transit Gateway.

Deploy to: Every account that needs TGW connectivity

5. tgw/cross-region-hub-connector (Multi-Region)

**Repo
**: cloudposse-terraform-components/aws-tgw-hub-connector

Creates a TGW Peering Connection between hubs in different regions. TGW does not support cross-region sharing, so each
region needs its own hub. This component peers them together for cross-region connectivity.

• Creates aws_ec2_transit_gateway_peering_attachment
• Accepts the peering in the primary region
• Associates route tables in both regions

Deploy to: Secondary region's network account (e.g., core-usw2-network)

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                        Region: us-east-2                                │
│                                                                         │
│  ┌───────────────────────────────────────────────────────────────────┐  │
│  │ Network Account (core-use2-network)                               │  │
│  │                                                                   │  │
│  │  ┌─────────────────────────────────┐                              │  │
│  │  │     Transit Gateway (Hub)       │                              │  │
│  │  │   + TGW Route Table             │                              │  │
│  │  │   + Route Propagations          │                              │  │
│  │  └──────┬────┬────┬────┬────┬──────┘                              │  │
│  │         │    │    │    │    │                                     │  │
│  │  ┌──────┘    │    │    │    └─────┐                               │  │
│  │  │           │    │    │          │                               │  │
│  └──┼───────────┼────┼────┼──────────┼───────────────────────────────┘  │
│     │           │    │    │          │                                  │
│  ┌──┼──┐  ┌─────┼──┐ │ ┌──┼─────┐ ┌──┼──┐                               │
│  │ VPC │  │  VPC   │ │ │  VPC   │ │ VPC │                               │
│  │     │  │        │ │ │        │ │     │                               │
│  │auto │  │sandbox │ │ │staging │ │prod │                               │
│  └─────┘  └────────┘ │ └────────┘ └─────┘                               │
│                    ┌──┼──┐                                              │
│                    │ VPC │                                              │
│                    │ dev │                                              │
│                    └─────┘                                              │
└─────────────────────────────────────────────────────────────────────────┘

Deployment Order

Single Region

Deploy in this exact order:

Step 1: VPCs (all accounts)
  └── Deploy vpc to every account that needs TGW connectivity

Step 2: TGW Hub (network account only)
  └── atmos terraform apply tgw/hub -s core-use2-network

Step 3: TGW Attachments (all accounts, network LAST)
  ├── atmos terraform apply tgw/attachment -s core-use2-auto
  ├── atmos terraform apply tgw/attachment -s plat-use2-sandbox
  ├── atmos terraform apply tgw/attachment -s plat-use2-dev
  ├── atmos terraform apply tgw/attachment -s plat-use2-staging
  ├── atmos terraform apply tgw/attachment -s plat-use2-prod
  └── atmos terraform apply tgw/attachment -s core-use2-network  ← LAST

Step 4: TGW Routes (network account only)
  └── atmos terraform apply tgw/routes -s core-use2-network

Step 5: VPC Routes (all accounts)
  ├── atmos terraform apply vpc/routes/private -s core-use2-network
  ├── atmos terraform apply vpc/routes/private -s core-use2-auto
  ├── atmos terraform apply vpc/routes/private -s plat-use2-sandbox
  ├── atmos terraform apply vpc/routes/private -s plat-use2-dev
  ├── atmos terraform apply vpc/routes/private -s plat-use2-staging
  └── atmos terraform apply vpc/routes/private -s plat-use2-prod

Why network account attachment is LAST: Connected accounts (non-network) create their TGW attachment first. The
network account's tgw/routes needs all attachment IDs to set up route propagation, so the network attachment and
routes come after all spoke attachments are created.

Adding a Secondary Region

When adding a second region (e.g., us-west-2):

Step 1: Deploy VPCs to the new region

atmos terraform apply vpc -s core-usw2-network
atmos terraform apply vpc -s core-usw2-auto
atmos terraform apply vpc -s plat-usw2-sandbox
atmos terraform apply vpc -s plat-usw2-dev
atmos terraform apply vpc -s plat-usw2-staging
atmos terraform apply vpc -s plat-usw2-prod

Step 2: Deploy TGW Hub in the new region

atmos terraform apply tgw/hub -s core-usw2-network

Step 3: Deploy Cross-Region Hub Connector

This peers the new region's TGW with the primary region's TGW:

atmos terraform apply tgw/cross-region-hub-connector -s core-usw2-network

The connector component must have primary_tgw_hub_region: us-east-2 configured in the stack.

Step 4: Deploy TGW Attachments (connected accounts first, network LAST)

atmos terraform apply tgw/attachment -s core-usw2-auto
atmos terraform apply tgw/attachment -s plat-usw2-sandbox
atmos terraform apply tgw/attachment -s plat-usw2-dev
atmos terraform apply tgw/attachment -s plat-usw2-staging
atmos terraform apply tgw/attachment -s plat-usw2-prod
atmos terraform apply tgw/attachment -s core-usw2-network  # LAST

Step 5: Deploy TGW Routes in the new region

atmos terraform apply tgw/routes -s core-usw2-network

Step 6: Deploy VPC Routes in all accounts

atmos terraform apply vpc/routes/private -s core-usw2-network
atmos terraform apply vpc/routes/private -s core-usw2-auto
atmos terraform apply vpc/routes/private -s plat-usw2-sandbox
atmos terraform apply vpc/routes/private -s plat-usw2-dev
atmos terraform apply vpc/routes/private -s plat-usw2-staging
atmos terraform apply vpc/routes/private -s plat-usw2-prod

Step 7: Update primary region routes for cross-region traffic

Add static routes in the primary region's tgw/routes pointing to the peering attachment for the secondary region's
CIDRs, then re-apply:

atmos terraform apply tgw/routes -s core-use2-network

Key difference from single region: Step 3 adds the cross-region hub connector. After peering, you also need to
add static routes in both regions' tgw/routes configurations to route cross-region traffic through the peering
attachment.

Atmos Configuration

Catalog Defaults

stacks/catalog/tgw/hub/defaults.yaml:

components:
  terraform:
    tgw/hub:
      settings:
        depends_on:
          vpc:
            component: vpc
      vars:
        enabled: true
        name: tgw-hub
        account_map_tenant_name: core

stacks/catalog/tgw/attachment/defaults.yaml:

components:
  terraform:
    tgw/attachment:
      settings:
        depends_on:
          tgw-hub:
            component: tgw/hub
      vars:
        enabled: true
        name: tgw-attachment
        transit_gateway_id: !terraform.state tgw/hub core-use2-network transit_gateway_id
        transit_gateway_route_table_id: !terraform.state tgw/hub core-use2-network transit_gateway_route_table_id

stacks/catalog/tgw/routes/defaults.yaml:

components:
  terraform:
    tgw/routes:
      settings:
        depends_on:
          tgw-hub:
            component: tgw/hub
      vars:
        enabled: true
        name: tgw-routes

Network Account Stack Example

The network account (core-use2-network) is where tgw/hub, tgw/attachment, tgw/routes, and vpc-routes all
come together:

# stacks/orgs/ins/core/network/us-east-2/foundation.yaml
import:
  - catalog/tgw/hub/defaults
  - catalog/tgw/attachment/defaults
  - catalog/tgw/routes/defaults
  - catalog/vpc-routes/defaults

components:
  terraform:
    tgw/routes:
      vars:
        transit_gateway_route_tables:
          - transit_gateway_route_table_id: !terraform.state tgw/hub core-use2-network transit_gateway_route_table_id
            routes:
              # Local VPC (network account)
              - attachment_id: !terraform.state tgw/attachment transit_gateway_vpc_attachment_id
              # core-auto
              - attachment_id: !terraform.state tgw/attachment core-use2-auto transit_gateway_vpc_attachment_id
              # Platform accounts
              - attachment_id: !terraform.state tgw/attachment plat-use2-sandbox transit_gateway_vpc_attachment_id
              - attachment_id: !terraform.state tgw/attachment plat-use2-dev transit_gateway_vpc_attachment_id
              - attachment_id: !terraform.state tgw/attachment plat-use2-staging transit_gateway_vpc_attachment_id
              - attachment_id: !terraform.state tgw/attachment plat-use2-prod transit_gateway_vpc_attachment_id

    vpc/routes/private:
      metadata:
        component: vpc-routes
        inherits:
          - vpc/routes/defaults
      vars:
        route_table_ids: !terraform.state vpc core-use2-network private_route_table_ids
        routes:
          - destination:
              cidr_block: !terraform.state vpc core-use2-auto vpc_cidr
            target:
              type: transit_gateway_id
              value: !terraform.state tgw/hub core-use2-network transit_gateway_id
          - destination:
              cidr_block: !terraform.state vpc plat-use2-dev vpc_cidr
            target:
              type: transit_gateway_id
              value: !terraform.state tgw/hub core-use2-network transit_gateway_id
          # ... repeat for each account

Spoke Account Stack Example

Each spoke account (e.g., plat-use2-dev) only needs tgw/attachment and vpc-routes:

# stacks/orgs/ins/plat/dev/us-east-2/foundation.yaml
import:
  - catalog/tgw/attachment/defaults

# The tgw/attachment catalog default already sets:
#   transit_gateway_id: !terraform.state tgw/hub core-use2-network transit_gateway_id
#   transit_gateway_route_table_id: !terraform.state tgw/hub core-use2-network transit_gateway_route_table_id

Key Concepts

Why Separate Components?

The old tgw/spoke pattern defined all connections inline in the hub. The new pattern uses separate components for
better modularity:

Old Pattern (tgw/spoke)	New Pattern (separate components)
All connections defined in hub	Each account manages its own attachment
Adding an account requires changing hub config	Adding an account = new attachment + route entries
Single deployment for all connections	Granular per-account deployments
peered_region: true for cross-region	Explicit tgw/cross-region-hub-connector

Cross-Account References

All cross-component dependencies use !terraform.state in stack YAML:

• tgw/attachment reads transit_gateway_id from tgw/hub in the network account
• tgw/routes reads transit_gateway_vpc_attachment_id from each tgw/attachment
• vpc-routes reads vpc_cidr from each account's vpc and transit_gateway_id from tgw/hub

TGW Is Regional

Transit Gateway is a regional resource. For multi-region connectivity:

• Deploy a separate tgw/hub in each region
• Use tgw/cross-region-hub-connector to peer hubs
• Deploy tgw/attachment and tgw/routes independently in each region
• Add static routes for cross-region traffic through the peering attachment

Adding a New Account

When adding a new account (e.g., plat-use2-qa) to an existing TGW setup:

Step 1: Deploy VPC in the new account

atmos terraform apply vpc -s plat-use2-qa

Step 2: Deploy TGW Attachment in the new account

atmos terraform apply tgw/attachment -s plat-use2-qa

Step 3: Update TGW Routes in the network account

Add the new attachment ID to the route propagation list in the network account's stack config, then apply:

atmos terraform apply tgw/routes -s core-use2-network

Step 4: Deploy VPC Routes in the new account

atmos terraform apply vpc/routes/private -s plat-use2-qa

Step 5: Update VPC Routes in existing accounts

Add a route for the new account's VPC CIDR in each existing account's vpc-routes config, then re-apply:

atmos terraform apply vpc/routes/private -s core-use2-network
atmos terraform apply vpc/routes/private -s core-use2-auto
atmos terraform apply vpc/routes/private -s plat-use2-sandbox
atmos terraform apply vpc/routes/private -s plat-use2-dev
atmos terraform apply vpc/routes/private -s plat-use2-staging
atmos terraform apply vpc/routes/private -s plat-use2-prod

Important: Steps 3-5 require updating existing stack configurations. Use separate PRs if deploying via Atmos Pro
to respect dependency ordering.

Troubleshooting

Attachment Fails with "Transit Gateway Not Found"

• Verify tgw/hub is deployed and the transit_gateway_id !terraform.state reference points to the correct stack
• Check that AWS RAM sharing is enabled and accepted in the spoke account
• Verify the spoke account is in the same region as the TGW hub

Routes Not Working

• Verify tgw/routes includes the attachment ID for both source and destination accounts
• Check VPC route tables have routes pointing to the TGW for destination CIDRs
• Ensure security groups allow traffic between the VPCs
• Check NACLs if applicable

Cross-Region Peering Not Working

• Verify both tgw/hub instances are deployed (one per region)
• Check the peering attachment is in available state
• Verify static routes are configured in both regions' tgw/routes pointing through the peering attachment

References

• AWS Transit Gateway Documentation
• TGW Peering Attachments
• TGW Route Tables
• cloudposse-terraform-components/aws-tgw-hub
• cloudposse-terraform-components/aws-tgw-attachment
• cloudposse-terraform-components/aws-tgw-routes
• cloudposse-terraform-components/aws-tgw-hub-connector

[jaguer0]

@aknysh can you spot check me , when Adding a Secondary Region we first have to deploy the root backend correct? so like in

stacks/orgs/acme/core/root/us-west-2/foundation.yaml

import:
  - orgs/bloom/core/root/_defaults
  - mixins/region/us-west-2
  - catalog/tfstate-backend/defaults

if so, do we have to do the whole init step like here

any other pre-reqs before I deploy the vpc to new region us-west-2 (just adding not mirroring)

[aknysh]

Prerequisites

Before deploying any components to the new region:

1. Terraform state backend: The tfstate-backend component must be deployed in the new region first. This
   creates the S3 bucket and DynamoDB table for Terraform state storage. Without it, no component can store state.

   atmos terraform apply tfstate-backend -s core-usw2-root

2. Stack files: Create the region directory structure and stack manifests for each account that will have
   resources in the new region. Copy the primary region's stack files and update:

   • The mixins/region import to reference the new region (e.g., mixins/region/us-west-2)
   • Any hardcoded region references in !terraform.state lookups
   • VPC CIDR blocks (must not overlap with the primary region)

3. Region mixin: Ensure a mixins/region/us-west-2.yaml mixin exists that sets the correct region and
   environment variables for the new region.

[aknysh]

i'm going to send you a doc with more info

[jaguer0]

@aknysh Following up — hit an issue deploying tfstate-backend to additional regions.

Context: Ref arch with security/compliance deployed to us-east-2 only. Now doing the cold start for all 17 default-enabled regions — us-west-2 for workloads (VPC, TGW) and the rest for compliance (GuardDuty, Security Hub, Config, Inspector) per the best practice of monitoring all regions.

Problem: tfstate-backend creates a global IAM role (acme-core-gbl-root-tfstate). Primary region already created it, so additional regions fail:

│ Error: creating IAM Role (acme-core-gbl-root-tfstate): EntityAlreadyExists: 
│ Role with name acme-core-gbl-root-tfstate already exists.

Tried overriding access_roles: {} in the additional region stacks — doesn't work because Atmos deep-merges vars, so the catalog's access_roles.default wins.

catalog/tfstate-backend/defaults.yaml

components:
  terraform:
    tfstate-backend:
      settings:
        pro:
          enabled: false
      backend:
        s3:
          profile: null
          role_arn: null
      vars:
        name: tfstate
        force_destroy: false
        prevent_unencrypted_uploads: true
        s3_state_lock_enabled: true
        use_organization_id: true
        access_roles:
          default: &access-template
            write_enabled: true
            allowed_roles: {}
            denied_roles: {}
            allowed_principal_arns:
              - arn:aws:iam::123456789012:user/SuperAdmin
              - "arn:aws:iam::*:role/acme-*-gbl-*-terraform"
              - "arn:aws:iam::*:role/acme-*-gbl-*-planner"
            denied_principal_arns: []
            allowed_permission_sets:
              core-root: &default_permission_sets
                - Terraform*Access
              core-artifacts: *default_permission_sets
              core-audit: *default_permission_sets
              core-auto: *default_permission_sets
              core-dns: *default_permission_sets
              core-network: *default_permission_sets
              core-security: *default_permission_sets
              plat-dev: *default_permission_sets
              plat-prod: *default_permission_sets
              plat-sandbox: *default_permission_sets
              plat-staging: *default_permission_sets
            denied_permission_sets: {}

Additional region stack (attempted override)

# stacks/orgs/acme/core/root/us-east-1/foundation.yaml
import:
  - orgs/acme/core/root/_defaults
  - mixins/region/us-east-1
  - catalog/tfstate-backend/defaults

components:
  terraform:
    tfstate-backend:
      vars:
        access_roles: {}  # ← does not override, deep merge keeps catalog value

Example workflow for deploying all additional region backends

workflows:
  deploy/us-regions:
    steps:
      - command: terraform deploy tfstate-backend -s core-use1-root
      - command: terraform deploy tfstate-backend -s core-usw1-root
      - command: terraform deploy tfstate-backend -s core-usw2-root
  deploy/eu-regions:
    steps:
      - command: terraform deploy tfstate-backend -s core-euw1-root
      - command: terraform deploy tfstate-backend -s core-euw2-root
      - command: terraform deploy tfstate-backend -s core-euw3-root
      - command: terraform deploy tfstate-backend -s core-euc1-root
      - command: terraform deploy tfstate-backend -s core-eun1-root
  deploy/ap-regions:
    steps:
      - command: terraform deploy tfstate-backend -s core-apse1-root
      - command: terraform deploy tfstate-backend -s core-apse2-root
      - command: terraform deploy tfstate-backend -s core-aps1-root
      - command: terraform deploy tfstate-backend -s core-apne1-root
      - command: terraform deploy tfstate-backend -s core-apne2-root
      - command: terraform deploy tfstate-backend -s core-apne3-root
  deploy/other-regions:
    steps:
      - command: terraform deploy tfstate-backend -s core-cac1-root
      - command: terraform deploy tfstate-backend -s core-sae1-root
  all:
    steps:
      - command: workflow deploy/us-regions -f compliance/deploy-tfstate-backends
      - command: workflow deploy/eu-regions -f compliance/deploy-tfstate-backends
      - command: workflow deploy/ap-regions -f compliance/deploy-tfstate-backends
      - command: workflow deploy/other-regions -f compliance/deploy-tfstate-backends

What's the recommended approach / sequence for deploying tfstate-backend to additional regions?

[jaguer0]

@aknysh bumping this if you get a sec —

One follow-up for clarification on the secondary region backend: should us-west-2 use our primary us-east-1 backend like we do for gbl, or does CP create a dedicated TF backend for multi-region no matter what?

If not, it seems like I'd need to account for the backend references and make sure the tfstate resolves correctly. Could be overlooking something but just want to verify before moving on with TGW setups & security stuff