Skip to content

Commit c86bc2c

Browse files
derrickawderrickaw
committed
terraform fmt
1 parent 6672037 commit c86bc2c

4 files changed

Lines changed: 73 additions & 164 deletions

File tree

‎v2/spanner-common/terraform/samples/infra-setup/README.md‎

Lines changed: 64 additions & 155 deletions
Original file line numberDiff line numberDiff line change
@@ -1,179 +1,88 @@
11
# Source Database & Spanner Target Setup for Migration Testing
22

3-
This folder contains Terraform configuration files to automatically set up, configure, and clean up database resources on Google Cloud Platform (GCP).
4-
5-
This setup is designed to help you prepare and test database migration pipelines. It automatically creates:
6-
1. One or more **source database instances** using Google Cloud SQL (either MySQL or PostgreSQL).
7-
2. Inside those database instances, it creates multiple **logical databases (shards)**.
8-
3. It imports a database table structure (your SQL schema) from a local file into all created logical databases.
9-
4. A **target Cloud Spanner database instance**.
10-
5. Two **sharding configuration files** (`shard-config.json` and `bulk-config.json`) that list the host IP, database name, and credentials for all created database shards. You can pass either file directly as an input parameter to your Dataflow migration jobs.
11-
12-
---
13-
14-
## Prerequisites
3+
## Sample Scenario: Sharded Database Infra Setup for Spanner Migrations
4+
5+
> **_SCENARIO:_** This Terraform example illustrates automatically setting up source database instances (MySQL or PostgreSQL) with multiple sharded logical databases and target Cloud Spanner instances for testing migration pipelines.
6+
7+
## Terraform permissions
8+
9+
In order to create the resources in this sample, the service account being used to run Terraform should have the required permissions.
10+
11+
### Using custom role and granular permissions (recommended)
12+
13+
Following permissions are required -
14+
15+
```shell
16+
- cloudsql.instances.create
17+
- cloudsql.instances.delete
18+
- cloudsql.instances.update
19+
- cloudsql.users.create
20+
- cloudsql.databases.create
21+
- spanner.databases.create
22+
- spanner.instances.create
23+
- spanner.instances.delete
24+
- compute.networks.create
25+
- compute.networks.delete
26+
- compute.subnetworks.create
27+
- compute.subnetworks.delete
28+
- compute.globalAddresses.create
29+
- secretmanager.versions.add
30+
- storage.buckets.create
31+
- storage.buckets.delete
32+
- resourcemanager.projects.get
33+
```
1534

16-
Before you begin, make sure your computer has the following installed and configured:
35+
## Assumptions
1736

18-
1. **Terraform CLI** (Version 1.2.0 or newer)
19-
2. **Google Cloud SDK (`gcloud` CLI)**: Installed, logged in, and set up with your project:
20-
```bash
21-
gcloud auth login
22-
gcloud auth application-default login
23-
```
24-
3. **Python 3** (installed and accessible from your command line)
25-
4. **Google Cloud Project** with billing enabled.
37+
1. Service account used for running Terraform can be granted the above permissions.
38+
2. Google Cloud SDK (`gcloud`) and Python 3 are installed on the machine executing Terraform.
2639

27-
---
40+
## Resources Created
2841

29-
## How the Automated Scripts Work
42+
Given these assumptions, it uses the supplied variable configuration and creates the following resources -
3043

31-
This setup includes several helper scripts in the `scripts/` folder to handle database loading, cleanup, and state reconciliation.
44+
1. **Source Database Instances** - One or more Google Cloud SQL (MySQL or PostgreSQL) instances.
45+
2. **Logical Databases (Shards)** - Multiple logical databases across the created physical instances.
46+
3. **Target Spanner Instance** - A target Cloud Spanner database instance.
47+
4. **Networking & Secrets** - VPC network peering and Secret Manager credentials for shard connection configs (`shard-config.json` and `bulk-config.json`).
3248

33-
### 1. Database Schema Loader (`scripts/import_schema.sh`)
34-
Once the Cloud SQL database instances are created, Terraform runs this bash script **once per physical instance** (the import step uses `for_each`), so a failure on one instance only re-imports that instance on the next apply instead of all of them. Each run reads your local SQL structure file (like `schema.sql`) and imports it sequentially into that instance's logical databases (Cloud SQL allows only one import at a time per instance); Terraform runs the instances in parallel.
35-
* **Why the retries are needed:** The bucket grants each Cloud SQL instance's service account read access just before the import runs, but IAM changes take a few seconds to propagate across Google Cloud. An import attempted in that window fails with a permission error. To handle this, the script retries each import up to 6 times (waiting 10 seconds between attempts) until the permission propagates and the schema loads successfully.
49+
## Description
3650

37-
### 2. Spanner Backup Cleanup (`scripts/delete_spanner_backups.sh`)
38-
When you run `terraform destroy` to delete your setup, Google Cloud Spanner will refuse to delete the database instance if there are any automatic database backups present. This script automatically finds and deletes all backups for the Spanner instance right before Terraform deletes the instance.
51+
This sample contains the following files -
3952

40-
### 3. Private Connection Cleanup (`scripts/teardown_vpc_peering.sh`)
41-
If you configure your databases to use private IPs instead of public IPs, Google Cloud creates private networking connections between your network and Cloud SQL. When deleting this infrastructure, Google Cloud occasionally takes time to release these connections. This script cleanly deletes the private network connection using the `gcloud` tool, or safely bypasses it if there are other active resources still using the connection.
53+
1. `main.tf` - This contains the Terraform resources which will be created.
54+
2. `outputs.tf` - This declares the outputs that will be output as part of running this terraform example.
55+
3. `variables.tf` - This declares the input variables that are required to configure the resources.
56+
4. `terraform.tf` - This contains the required providers and APIs/project configurations for this sample.
57+
5. `terraform.tfvars` - This contains the comprehensive dummy inputs that need to be populated to run this example.
58+
6. `terraform_simple.tfvars` - This contains the minimal list of dummy inputs that need to be populated to run this example.
59+
7. `scripts/` - Helper scripts for schema importing and resource cleanup.
4260

61+
## How to run
4362

44-
---
63+
1. Clone this repository locally.
64+
2. Create a local SQL file named `schema.sql` defining your tables and columns.
65+
3. Edit `terraform_simple.tfvars` and replace placeholders with real values.
4566

46-
## Step-by-Step Guide to Deploying
67+
### Initialise Terraform
4768

48-
### Step 1: Prepare Your Local Database Structure
49-
Create a local SQL file named `schema.sql` in this folder. Define the tables and columns you want to load into your source databases. For example:
50-
```sql
51-
CREATE TABLE users (
52-
id INT PRIMARY KEY,
53-
name VARCHAR(100),
54-
email VARCHAR(100)
55-
);
69+
```shell
70+
terraform init
5671
```
5772

58-
### Step 2: Configure Your Variables
59-
There are two variable sample files provided:
60-
1. **`terraform_simple.tfvars` (Recommended for beginners)**: A simple, minimal configuration containing only the most important variables. It leverages the automated prefix generation.
61-
2. **`terraform.tfvars`**: A comprehensive variable template containing all available settings (such as database user, password, network CIDRs, tags, Spanner processing units).
73+
### Run `plan` and `apply`
6274

63-
#### Key Naming Variables:
64-
* **`instance_prefix` (Optional)**: A string prefixed to physical database instances and target Spanner instances. If not provided, a unique random pet name of the form `smt-<word>-<word>` (e.g. `smt-clever-mongoose`) is generated automatically.
65-
* **`migration_prefix` (Optional)**: A string prefixed to other resources like VPC networks, subnets, Secret Manager secrets, and GCS schema buckets. If not provided, a unique random pet name of the form `smt-<word>-<word>` is generated automatically.
66-
* **`spanner_instance_name` / `spanner_database_name` (Optional)**: Overrides the target Spanner instance and database names completely. If left blank, they are dynamically derived from your `instance_prefix` and `migration_prefix` respectively.
67-
68-
Open `terraform_simple.tfvars` or `terraform.tfvars`, replace the placeholders (like `<PROJECT_ID>`) with your actual values, and save the file.
69-
70-
### Step 3: Initialize and Deploy
71-
72-
Run the following commands in your terminal:
73-
74-
```bash
75-
# 1. Download necessary Terraform providers and plugins
76-
terraform init
77-
78-
# 2. Deploy the databases and generate the configuration
79-
# Note: For large scale deployments (e.g., 128 shards), you MUST use the -parallelism flag
80-
# for faster resource creation (default is 10).
75+
```shell
76+
terraform plan --var-file=terraform_simple.tfvars
8177
terraform apply -parallelism=100 --var-file=terraform_simple.tfvars
8278
```
8379

84-
---
85-
86-
## Outputs & Results
87-
88-
Once the deployment completes successfully, Terraform will print the resource details on your screen and generate two sharding configuration files in this directory:
89-
90-
### 1. Regular Shard Config Format (`shard-config.json`) *(Sample output)*
91-
```json
92-
[
93-
{
94-
"logicalShardId": "shard-0",
95-
"host": "198.51.100.5",
96-
"port": "3306",
97-
"user": "migration_user",
98-
"password": null,
99-
"dbName": "shard_db_0",
100-
"namespace": "public",
101-
"secretManagerUri": "projects/my-gcp-project/secrets/smt_clever_mongoose_db_password/versions/latest",
102-
"connectionProperties": "jdbcCompliantTruncation=true"
103-
}
104-
]
105-
```
80+
## FAQ
10681

107-
### 2. Bulk Shard Config Format (`bulk-config.json`) *(Sample output)*
108-
```json
109-
{
110-
"shardConfigurationBulk": {
111-
"dataShards": [
112-
{
113-
"host": "198.51.100.5",
114-
"port": 3306,
115-
"user": "migration_user",
116-
"password": null,
117-
"secretManagerUri": "projects/my-gcp-project/secrets/smt_clever_mongoose_db_password/versions/latest",
118-
"connectionProperties": "jdbcCompliantTruncation=true",
119-
"namespace": "public",
120-
"databases": [
121-
{
122-
"dbName": "shard_db_0",
123-
"databaseId": "shard-0"
124-
},
125-
{
126-
"dbName": "shard_db_1",
127-
"databaseId": "shard-1"
128-
}
129-
]
130-
}
131-
]
132-
}
133-
}
134-
```
82+
### Handling creation timeouts
13583

136-
---
84+
When deploying a high number of physical database instances concurrently (e.g., 128 shards), Google Cloud schedules creation asynchronously. If the client times out, import missing instances using `terraform import`.
13785

138-
## Troubleshooting
86+
### Cleaning up resources
13987

140-
### Handling Creation Timeouts & Operation Dropouts
141-
When deploying a high number of physical database instances concurrently (e.g., 128 shards), you may occasionally encounter a transient timeout or polling connection dropout error from the Google Cloud API:
142-
```
143-
Error: Error waiting for Create Instance: ...
144-
```
145-
Or when running `terraform apply` again after a timeout:
146-
```
147-
Error: Error, failed to create instance ...: googleapi: Error 409: The Cloud SQL instance already exists., instanceAlreadyExists
148-
```
149-
150-
#### Why this happens:
151-
When Terraform requests the creation of 100+ databases, Google Cloud schedules their creation asynchronously in the background. If the local Terraform process loses connection to the GCP Operation API or hits a client-side wait timeout, Terraform aborts the command and **fails to save those specific instances to your local `terraform.tfstate` file**, even though the creation continues successfully in the background on Google's servers.
152-
153-
#### How to resolve this:
154-
1. **Verify creation in GCP**: Run this CLI command to confirm that the instances are active and running on Google Cloud:
155-
```bash
156-
gcloud sql instances list --project="<YOUR_PROJECT_ID>" --filter="name~smt-sharded"
157-
```
158-
2. **Import the affected instances into Terraform State**: For any instances that were successfully created on GCP but are missing from your local state file (causing `409 Already Exists` errors), import them manually back into Terraform. The instances use `for_each`, so the resource address is keyed by the shard index **as a quoted string** (e.g. `["18"]`, not `[18]`):
159-
```bash
160-
terraform import --var-file=terraform_simple.tfvars 'google_sql_database_instance.instances["<INDEX>"]' "projects/<YOUR_PROJECT_ID>/instances/<INSTANCE_NAME>"
161-
```
162-
*Example:*
163-
```bash
164-
terraform import --var-file=terraform_simple.tfvars 'google_sql_database_instance.instances["18"]' "projects/my-gcp-project/instances/smt-sharded-demo-new-physical-shard-18"
165-
```
166-
3. **Resume the Deployment**: Once all missing instances are imported, simply rerun the deployment command with controlled parallelism:
167-
```bash
168-
terraform apply -parallelism=30 --var-file=terraform_simple.tfvars
169-
```
170-
Terraform will successfully refresh the state and complete the configuration setup in minutes!
171-
172-
---
173-
174-
### Cleaning Up Resources
175-
To delete all created Google Cloud resources and avoid ongoing charges, run:
176-
```bash
177-
terraform destroy --var-file=terraform_simple.tfvars
178-
```
179-
All Cloud SQL databases, target Spanner databases, Secret Manager secrets, and networking links will be cleanly removed.
88+
Run `terraform destroy --var-file=terraform_simple.tfvars` to delete all created infrastructure.

‎v2/spanner-common/terraform/samples/infra-setup/main.tf‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,13 +35,13 @@ locals {
3535
default_version = "8_0"
3636
default_port = 3306
3737
# MySQL binds users to connection origins; "%" allows external access.
38-
user_host = "%"
38+
user_host = "%"
3939
}
4040
POSTGRES = {
4141
default_version = "14"
4242
default_port = 5432
4343
# PostgreSQL does not support host-bound users in the GCP API; must be null.
44-
user_host = null
44+
user_host = null
4545
}
4646
}
4747

@@ -127,7 +127,7 @@ resource "null_resource" "private_vpc_connection" {
127127
}
128128

129129
provisioner "local-exec" {
130-
when = destroy
130+
when = destroy
131131
environment = {
132132
NETWORK_NAME = self.triggers.network_name
133133
PROJECT_ID = self.triggers.project_id
@@ -245,8 +245,8 @@ resource "google_storage_bucket_object" "schema_file" {
245245

246246
# Grant IAM permissions to all Cloud SQL service accounts to read schema from the GCS bucket in a single API call to prevent ETag lock collision delays
247247
resource "google_storage_bucket_iam_binding" "sql_gcs_reader" {
248-
bucket = google_storage_bucket.schema_bucket.name
249-
role = "roles/storage.objectViewer"
248+
bucket = google_storage_bucket.schema_bucket.name
249+
role = "roles/storage.objectViewer"
250250
members = [
251251
for inst in google_sql_database_instance.instances :
252252
"serviceAccount:${inst.service_account_email_address}"
@@ -307,7 +307,7 @@ resource "google_spanner_instance" "spanner_instance" {
307307

308308
# Automated teardown of Spanner backups to prevent destroy failures
309309
provisioner "local-exec" {
310-
when = destroy
310+
when = destroy
311311
environment = {
312312
INSTANCE_NAME = self.name
313313
PROJECT_ID = self.project

‎v2/spanner-common/terraform/samples/infra-setup/terraform.tfvars‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ local_schema_file_path = "./schema.sql" # Local SQL schema imported into every s
2222
# ------------------------------------------------------------------------------
2323
# SOURCE DATABASE (Cloud SQL)
2424
# ------------------------------------------------------------------------------
25-
database_provider = "MYSQL" # MYSQL or POSTGRES
25+
database_provider = "MYSQL" # MYSQL or POSTGRES
2626
# database_version = "8_0" # Set specific version (e.g., "14" for Postgres) or leave commented for module default
2727
physical_shards_count = 1 # Number of physical Cloud SQL instances
2828
logical_shards_count = 2 # Logical databases per physical instance
@@ -57,7 +57,7 @@ connection_properties = "jdbcCompliantTruncation=true"
5757
# ------------------------------------------------------------------------------
5858
# spanner_config = "regional-<REGION>" # e.g. "regional-us-central1"; defaults to regional-${var.region}
5959
spanner_display_name = "SMT Spanner Instance"
60-
spanner_processing_units = 100 # Positive multiple of 100 (100 = 0.1 node)
60+
spanner_processing_units = 100 # Positive multiple of 100 (100 = 0.1 node)
6161
spanner_database_dialect = "GOOGLE_STANDARD_SQL" # GOOGLE_STANDARD_SQL or POSTGRESQL
6262
# spanner_instance_name = "my-spanner" # Optional. Unset -> derived from instance_prefix
6363
# spanner_database_name = "my-db" # Optional. Unset -> derived from migration_prefix

‎v2/spanner-common/terraform/samples/infra-setup/terraform_simple.tfvars‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ migration_prefix = "<MIGRATION_PREFIX>"
1515
local_schema_file_path = "./schema.sql"
1616

1717
# Cloud SQL Database Setup
18-
database_provider = "MYSQL"
18+
database_provider = "MYSQL"
1919
# database_version = "8_0"
2020
physical_shards_count = 1
2121
logical_shards_count = 2

0 commit comments

Comments
 (0)