ci_cd_pipeline_overview.txt

i have been learning about devops and devsecops a small doubt is the I when I started learning about  I have gone through aws services like ec2 instance creation, vpc, subnets s3 bucket creating attaching the security groups, nacls, configuring cloud trail , cloud watch logs linking it with s3 buckets to store them  this is fine I got understanding about them but when it comes to devops ci/cd they say that it start with git repo or git lab repo , so we create anyone account which is private only for our organisation we ask developers to update it , after that when they are asked for pull request we need to trigger to ci/cd pipeline , so for that some one says if we are using git lab runners it will run the application it or if we configure in aws ec2 instance it will run there like 2 scenarios and in that I know to that if we want to trigger that instance or runner we need to have gitlab-ci.yml file so in that we need to have image called alpine or anything what ever we want and we will configure stages to run  and before, after scripts , to be executed  and we can configure source code analysis, software compositions analysis, and all the type of scanners and once job done some one says that runners shouldn't be terminated but when watching some online tutorials they say that they will be terminated but whether they will be termianted because for some testing purpose only created or in organisation also will they do or not I don't know and I don't know that concept , and once the test cases have been passed in that when they were integration security scans they will use some variables or something to do sonarcube scans and these sonar servers and hosted in different aws but still they will scan there and pull the request to this runner I think or I don't know , and at the end I think they build  the image or push the image to docker , this the process I think but what about the infra created in aws how does they deploy it there or from gitlab when they do deployments how do they work  and one more thing for suppose they is an on going project with stage, uat , dev, prod  so they might test in stage liek that means till does infra will be alive till that time period and does they have 4 repos separately for each build with 4 infra how does they links this i have tried to learn but no one explains propelry can you please explain this to me  dont modify it just convert it to points or single line to save it in github

Complete Breakdown of DevOps & DevSecOps with AWS and GitLab CI/CD
Think of this as a real-world DevOps setup that an organization follows.

Step 1: Setting Up AWS Infrastructure (Base Setup)
What We Need in AWS:
Before CI/CD starts, we need a cloud environment ready where applications will be deployed.

VPC (Virtual Private Cloud)

Why? It creates a private network where our AWS services live.
Components inside VPC:
Subnets (Public for Load Balancers, Private for Backend)
Route Tables (For controlling traffic)
Security Groups (Firewall for instances)
NACLs (Network ACLs for additional security)
EC2 Instances (or Containers in AWS ECS/Kubernetes)

If the app runs on EC2:
Create an EC2 instance (Ubuntu, Amazon Linux, etc.).
Attach a Security Group to control traffic.
Install Docker if using containers.
Use an Elastic IP (static IP) if needed.
If using AWS ECS (Elastic Container Service):
Create an ECS Cluster.
Define a Task Definition (where our app runs).
Set up Fargate or EC2-based ECS service.
Database (Optional, If Needed)

RDS (PostgreSQL, MySQL, etc.) for managed databases.
DynamoDB if using a NoSQL database.
Storage & Logging

S3 Buckets store logs, backups, or deployment artifacts.
CloudWatch Logs to monitor applications.
IAM Roles & Policies

Create IAM roles that allow EC2 or ECS to pull Docker images.
Create IAM users for DevOps teams.
Step 2: Setting Up the Git Repository
GitLab, GitHub, or Bitbucket is used for version control.
Each developer clones the repo and pushes changes.
The repository follows a branching strategy:
develop → Developers commit new code.
stage → Code is tested in a pre-prod environment.
main → Production-ready code.
✅ At this stage, code is stored, but nothing is automated yet.

Step 3: Configuring GitLab CI/CD (.gitlab-ci.yml)
Now, we need to automate everything.

What is .gitlab-ci.yml?
This file tells GitLab how to run the pipeline.
It defines different stages that execute when new code is pushed.
Typical CI/CD Stages
yaml
Copy
Edit
stages:
  - build
  - test
  - security_scan
  - deploy
Step 3.1: Define a Docker Image
yaml
Copy
Edit
image: alpine:latest
This defines what OS our jobs will run inside (e.g., Alpine, Ubuntu).
Step 4: Setting Up Runners
What is a GitLab Runner?
A GitLab Runner is a server that executes the jobs inside the CI/CD pipeline.

Types of Runners
Shared Runners (Temporary)

Provided by GitLab (auto-terminates after the job).
Used for basic jobs like testing.
Self-Hosted Runners (Persistent)

Deployed on AWS EC2 or Kubernetes.
Stays alive to handle ongoing CI/CD jobs.
How to Register a Self-Hosted Runner on AWS EC2?
Create an EC2 instance.
Install GitLab Runner:
sh
Copy
Edit
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh | sudo bash
sudo yum install gitlab-runner
Register the runner:
sh
Copy
Edit
sudo gitlab-runner register
Connect it with your GitLab project.
✅ Now, your pipeline has a dedicated server to execute jobs.

Step 5: Automating the Pipeline
Step 5.1: Build Stage
yaml
Copy
Edit
build-job:
  stage: build
  script:
    - echo "Building the application..."
    - npm install
    - npm run build
  artifacts:
    paths:
      - build/
Compiles the application.
Stores the build artifacts for the next stages.
Step 5.2: Running Tests
yaml
Copy
Edit
test-job:
  stage: test
  script:
    - echo "Running unit tests..."
    - npm test
Runs unit tests.
If tests fail, the pipeline stops here.
Step 5.3: Security Scanning (SonarQube)
yaml
Copy
Edit
security-scan:
  stage: security_scan
  script:
    - sonar-scanner -Dsonar.projectKey=my-project -Dsonar.sources=.
SonarQube scans for vulnerabilities.
Reports findings to the SonarQube server.
If critical issues are found, it blocks deployment.
✅ SonarQube is hosted separately but the runner interacts with it.

Step 5.4: Building & Pushing a Docker Image
yaml
Copy
Edit
docker-build:
  stage: deploy
  script:
    - docker build -t myapp:v1 .
    - docker tag myapp:v1 1234567890.dkr.ecr.us-east-1.amazonaws.com/myapp:v1
    - docker push 1234567890.dkr.ecr.us-east-1.amazonaws.com/myapp:v1
Builds a Docker image of the application.
Pushes it to AWS ECR.
✅ Now, the image is ready to be deployed.

Step 6: Deploying to AWS
Step 6.1: Deploying to an EC2 Instance
yaml
Copy
Edit
deploy-ec2:
  stage: deploy
  script:
    - ssh -o StrictHostKeyChecking=no ec2-user@your-ec2-ip "docker pull 1234567890.dkr.ecr.us-east-1.amazonaws.com/myapp:v1"
    - ssh ec2-user@your-ec2-ip "docker stop myapp || true && docker rm myapp || true"
    - ssh ec2-user@your-ec2-ip "docker run -d --name myapp -p 80:80 1234567890.dkr.ecr.us-east-1.amazonaws.com/myapp:v1"
Connects to EC2 via SSH.
Pulls the latest Docker image.
Runs it inside a container.
✅ Now, the app is running on EC2.

Step 6.2: Deploying to AWS ECS
yaml
Copy
Edit
deploy-ecs:
  stage: deploy
  script:
    - aws ecs update-service --cluster my-cluster --service myapp-service --force-new-deployment
Tells ECS to pull the new Docker image and restart the service.
✅ This works for containerized workloads.

Final Summary
✅ Step 1: Set up AWS infrastructure (VPC, EC2, ECS, IAM).
✅ Step 2: Push code to GitLab repository.
✅ Step 3: .gitlab-ci.yml defines pipeline automation.
✅ Step 4: GitLab Runner executes the jobs.
✅ Step 5: CI/CD Pipeline runs:

Build & Test
Security Scans (SonarQube)
Package & Push Docker Image ✅ Step 6: Deploy to EC2/ECS in AWS.