Skip to content

Add README and GCP setup changes with refactored scripts#6

Merged
greg-ogs merged 26 commits into
mainfrom
testing
May 9, 2025
Merged

Add README and GCP setup changes with refactored scripts#6
greg-ogs merged 26 commits into
mainfrom
testing

Conversation

@greg-ogs

@greg-ogs greg-ogs commented May 9, 2025

Copy link
Copy Markdown
Owner
### Description

This pull request includes several updates to enhance the project structure and functionality:

- **Documentation**:
  - Added comprehensive README files for project directories (`shell_scripts`, `terraform`, `tasks`) and the root directory to improve usability and clarity.

- **Terraform and GCP Configurations**:
  - Introduced GCP VPC configuration, compute instance setups, and managed instance groups with autoscaling and health checks.
  - Refactored Terraform structure and variable management for better clarity and maintainability.
  - Added configuration for SSH keys, GCS bucket, application infrastructure, and updated resource handling for regional and single-zone MIGs.

- **Scripts**:
  - Added task automation scripts, including VM creation, MySQL setup, load balancer configuration, and GCP infrastructure management.
  - Refactored existing scripts to simplify and extend functionality, including support for machine images.

- **Docker Enhancements**:
  - Introduced Docker setup for Terraform and sidecar-proxy setups.
  - Streamlined the container registry workflow, improved image handling, and integrated Cloud SQL proxy features.

- **Security**:
  - Updated firewall rules and integrated secret management for database credentials and authentication resources.

### Type of changes

- [x] Documentation updates
- [x] Refactoring/enhancements
- [x] New feature implementation
- [ ] Bug fixes

### Additional Notes

This pull request ensures better clarity, project structure, and flexibility in managing GCP resources and task execution.

greg-ogs added 26 commits April 25, 2025 15:06
Create a managed instance group using an nginx template and configure health checks, autoscaling, and named ports. Include firewall updates for HTTP and health check traffic, and log instance group details for verification.
…proxy setup Kubernetes: Sql pod using embedded proxy (1 container).

Implement a script to create a GCP service account with roles for Artifact Registry, GCS, and Cloud SQL. Add a script for creating a Docker registry in Artifact Registry. Include setup files (Dockerfile, docker-compose, entrypoint) for an embedded SQL proxy Flask application.
Introduced a `Dockerfile` and `entrypoint.sh` for the sidecar-proxy service to handle Flask and Cloud SQL Proxy setup. Updated `docker_registry.sh` script to improve registry handling and add logic for tagging and pushing images. This enhances support for custom workloads and streamlines image deployment.
Added detailed configurations for MySQL instances, Private Services Access (PSA), and secret management into task scripts. Refactored network, subnet, and firewall setups while enabling relevant APIs (SQL Admin, Secret Manager) to streamline the deployment workflow. Included new regional Managed Instance Group for alchemy with autoscaling and additional metadata handling.
Added an environment variable to specify the path for Google Application Credentials. This ensures proper authentication for accessing Google Cloud resources during the script execution.
Uncommented and finalized scripts to set up a MySQL database, create instance templates, and configure a regional Managed Instance Group (MIG) for task automation. Added secret retrieval for database credentials, enabled Private Services Access, and incorporated autoscaling and health checks for reliability. Introduced a new `alchemy-mig.sh` script to manage and configure an additional regional MIG specifically for the "alchemy" service.
Cleaned up the script by removing MySQL-related variables and redundant secret retrievals. Focused the script to streamline Managed Instance Group (MIG) setup for the "alchemy" service. Ensured optimal regional configuration with essential parameters only.
Introduce a new script `task_8.sh` and refine `task_3.sh` to replace standard images with machine images for instance creation and configuration. Unused commented code is removed to improve clarity and maintainability.
Introduce scripts for managing VMs, custom VPCs, private service access, and peering configurations in GCP. Include SQL setup for Alchemy application and related changes in instance groups.
Updated container image versions and ports in `docker-compose.yml` and `Dockerfile`. Enhanced authentication by integrating Google Cloud secrets and adding Cloud SDK installation in the Docker image. Re-enabled and refined the Docker registry creation script for better image handling.
Introduced a new script to automate the creation of a private Google Kubernetes Engine (GKE) cluster with detailed specifications, including network setup, Cloud NAT configuration, and node pool for SQL. Additionally, removed an unnecessary space in an existing script to maintain code consistency.
Modified the Managed Instance Group (MIG) setup to operate within a single zone instead of multiple zones for better alignment with the specified configuration. Updated all relevant commands to replace regional arguments with zonal ones and adjusted associated variables accordingly.
This script sets up an HTTP Load Balancer with path-based routing and a reserved static IP. It routes `/nginx` requests to the nginx Managed Instance Group (MIG) and other requests to the alchemy MIG. The configuration includes backend services, a URL map, an HTTP proxy, and a forwarding rule.
Created a Dockerfile to use the latest Terraform image, setting up a working directory and entry point for Terraform commands. Updated `.gitignore` to exclude specific Terraform-related directories.
Introduce a Terraform setup to manage GCP resources, specifically creating a VPC and subnetworks. Updated the Dockerfile to include a service account configuration comment. Adjusted `.gitignore` to exclude Terraform-related files and credentials for better security.
Introduced a Google Compute Project Metadata resource to manage SSH keys. Added a Google Cloud Storage bucket with a unique name. Refactored variables and outputs into separate files for better structure and maintainability.
Renamed student-related variables across the Terraform configuration. Introduced a new service account resource with IAM binding for Storage Object Creator role. Updated outputs to include the service account email for reference.
Introduce SSH and HTTP firewall rules for secure VPC access. Add new variables for safe IP ranges and load balancer configurations. Implement a GitHub CodeQL workflow for advanced code scanning and security analysis.
These changes add outputs for VPC ID, subnetwork IDs, bucket ID, and project metadata ID. This enhances the configuration's flexibility by exposing these values for use in other modules or external integrations.
This commit introduces a new application.tf file in the base Terraform directory. The file likely contains foundational configurations necessary to set up the application infrastructure. This is a key step in structuring the Terraform codebase.
Moved compute-related configurations to a dedicated directory. Added remote state data source to reference base outputs and defined GCP provider, variables, and necessary configurations for compute resources.
Replaced the `project_id` variable with remote state output for better consistency and removed unused variable definition. Simplified `.gitignore` by consolidating patterns related to `.gcp`.
Introduced instance templates and regional instance group managers for `us-central1` and `us-east1`, along with HTTP health checks and auto-healing policies. Included a startup script to configure nginx, fetch instance metadata, and update the default page. Updated project ID references and refined outputs for compatibility.
# Conflicts:
#	terraform/base/outputs.tf
This update introduces README files for all major directories (`shell_scripts`, `terraform`, `tasks`) and the root directory. These documents provide structure, usage instructions, and detailed explanations for managing GCP resources using scripts, Terraform, and task configurations. These additions aim to improve clarity and usability for users.
@greg-ogs greg-ogs self-assigned this May 9, 2025
@greg-ogs greg-ogs added the documentation Improvements or additions to documentation label May 9, 2025
@greg-ogs greg-ogs merged commit 23942cf into main May 9, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant