[minor] Implement Redis-Based Distributed Locking for GitOps Operations by Hardik-Prajapati-10 · Pull Request #2189 · ibm-mas/cli

Hardik-Prajapati-10 · 2026-04-16T09:28:12Z

Issue

https://jsw.ibm.com/browse/MASCORE-13109

Description

This PR implements a comprehensive Redis-based distributed locking mechanism to replace the legacy Git branch-based locking system for GitOps operations. The change addresses critical issues with concurrent pipeline executions that previously caused race conditions, merge conflicts, and branch pollution.

Key Changes

1. Redis Locking Infrastructure (`image/cli/mascli/functions/gitops_utils`)

Added Redis connection management with TLS support
Implemented atomic lock acquisition using Redis SET NX operations
Added lock release with ownership verification using Lua scripts
Included automatic lock expiry (TTL) to prevent orphaned locks
Added comprehensive retry logic with configurable parameters

2. Container Image Updates

Added redis-cli installation script (image/cli/install/install-redis-cli.sh)
Updated Dockerfile to include redis-cli in the CLI container image (v19.6.1+)
Supports multiple package managers (microdnf, dnf, yum, apt-get, apk)

3. Migrated GitOps Functions (10 functions)

All functions now use gitops_lock_and_modify with Redis locking:

gitops_cp4d_service
gitops_suite_app_config
gitops_deprovision_app_config
gitops_mas_config
gitops_suite_workspace
gitops_deprovision_suite_workspace
gitops_deprovision_cp4d_service
gitops_db2u_database
gitops_rds_db2_database
gitops_deprovision_db2u_database

4. Documentation

Comprehensive setup guide (docs/redis-locking-setup.md)
IBM Toolchain configuration script (docs/ibm-toolchain-redis-setup.sh)
Migration examples and troubleshooting guide

Impact

Reliability Improvements:

Eliminates race conditions through atomic Redis operations
Prevents merge conflicts in concurrent updates
Removes temporary Git branch pollution
Automatic lock expiry prevents orphaned locks
Fail-fast behavior when Redis is unavailable

Breaking Change:

Redis is now required for migrated GitOps flows
No fallback to Git branch locking
Operations fail immediately if Redis is unavailable or misconfigured
GITOPS_USE_REDIS_LOCKING=true must be set

Test Results

Manual Testing

✅ Redis connection validation with TLS
✅ Lock acquisition and release operations
✅ Concurrent pipeline execution without conflicts
✅ Lock TTL expiry behavior
✅ Retry logic under contention
✅ All 10 migrated GitOps functions tested

Integration Testing

✅ IBM Cloud Redis instance connectivity
✅ IBM Toolchain pipeline execution
✅ Multi-pipeline concurrent operations
✅ Error handling for Redis unavailability

Configuration Testing

✅ TLS certificate handling
✅ Authentication (username/password)
✅ Environment variable configuration
✅ Setup script validation (ibm-toolchain-redis-setup.sh)

Backporting

Related Pull Requests

https://github.ibm.com/maximoappsuite/saas-deploy-py/pull/262

⚠️ Notes for Reviewers

Ensure you have understood the guidelines before proceeding with a review.
Ensure all sections in the PR template are appropriately completed.
Critical: This is a breaking change - Redis is now mandatory for migrated GitOps operations
Review the migration strategy in docs/redis-locking-setup.md
Verify the setup script docs/ibm-toolchain-redis-setup.sh for IBM Cloud integration
Check that all 10 migrated functions follow the new locking pattern consistently
Validate error handling for Redis unavailability scenarios
Review security aspects: TLS configuration, credential handling, and password masking

image/cli/mascli/functions/gitops_suite_app_config - Migrated image/cli/mascli/functions/gitops_mas_config - Migrated image/cli/mascli/functions/gitops_deprovision_app_config - Migrated

…ging

whitfiea · 2026-04-17T08:07:00Z

+### Required Runtime Requirement
+
+For migrated GitOps flows:
+- `GITOPS_USE_REDIS_LOCKING` must be `true`


It seems that GITOPS_USE_REDIS_LOCKING has to be true, so i am not sure what the value is of having an env var anyway?

The env var acts as a "safety check" - even though Redis is required, the explicit flag prevents accidental misconfiguration and provides clear error messages when Redis setup is incomplete.

Originally, GITOPS_USE_REDIS_LOCKING served two purposes:

Feature switch during development
Redis locking was the preferred mechanism
Git branch locking was a fallback when Redis wasn’t available

Safety / explicit opt‑in
Prevent accidental startup with partial or missing Redis configuration
Force developers/operators to make an intentional choice

Thinking about this more we could keep the GITOPS_USE_REDIS_LOCKING flag and have it use the branch locking mechanism when this is false. This would allow external customers using gitops to continue with the branch locking and not force a redis instance on them.

we are going with following approach :
If Redis is available → Uses Redis locking
If Redis is unavailable → Automatically falls back to Git locking

whitfiea · 2026-04-17T08:09:11Z

+### Step 2: Configure IBM Toolchain Environment Variables
+
+Add these environment variables to your IBM Toolchain pipeline.
+
+#### In IBM Toolchain → Pipeline → Environment Properties:
+
+| Variable Name | Type | Value | Description |
+|--------------|------|-------|-------------|
+| `REDIS_USERNAME` | Secure | `ibm_cloud_user` | Redis username from credentials |
+| `REDIS_HOST` | Text | `c-abc123.databases.appdomain.cloud` | Redis hostname |
+| `REDIS_PORT` | Text | `31234` | Redis port |
+| `REDIS_PASSWORD` | Secure | `your-redis-password` | Redis password |
+| `REDIS_TLS_CA_CERT_B64` | Secure | `LS0tLS1CRUdJTi...` | Base64-encoded TLS certificate |
+| `GITOPS_USE_REDIS_LOCKING` | Text | `true` | Required for migrated locking flows |
+| `REDIS_TLS` | Text | `true` | Enable TLS (required for IBM Cloud) |
+| `REDIS_DB` | Text | `0` | Redis database number |
+
+#### Optional Tuning Parameters
+
+| Variable Name | Default | Description |
+|--------------|---------|-------------|
+| `GITOPS_LOCK_TTL` | `300` | Lock expires after N seconds |
+| `GITOPS_LOCK_RETRY_MAX` | `100` | Maximum retry attempts |
+| `GITOPS_LOCK_RETRY_DELAY` | `20` | Seconds between retry attempts |


Using the IBM toolchain to call the functions is optional. We should update the docs in https://ibm-mas.github.io/gitops/main/ so state these env vars should be set when executing the functions, and also add that to this file.

doc updated

clarify that Redis env vars must be set whenever migrated GitOps functions are executed, not only in IBM Toolchain note that IBM Toolchain is optional and document pipeline environment properties as one way to provide runtime variables remove the custom CLI image/container setup section because custom CLI images are not supported

whitfiea · 2026-04-20T08:45:23Z

+### Required Runtime Requirement
+
+For migrated GitOps flows:
+- `GITOPS_USE_REDIS_LOCKING` must be `true`


Thinking about this more we could keep the GITOPS_USE_REDIS_LOCKING flag and have it use the branch locking mechanism when this is false. This would allow external customers using gitops to continue with the branch locking and not force a redis instance on them.

If Redis is available → Uses Redis locking If Redis is unavailable → Automatically falls back to Git locking

… updated behavior to error when Redis is unavailable with explicit Redis locking enabled.

Hardik-Prajapati-10 and others added 28 commits April 6, 2026 16:51

MASCORE-13109

81e5c94

gitops-cp4d-service

8c0ad0f

Migrated 3 functions.

af353a0

image/cli/mascli/functions/gitops_suite_app_config - Migrated image/cli/mascli/functions/gitops_mas_config - Migrated image/cli/mascli/functions/gitops_deprovision_app_config - Migrated

merge

7bd494a

Fix Redis locking: correct function scoping and add configuration log…

a32da51

…ging

dummy

2d42cb0

dummy

8659c1a

redis-cli

2d32a90

redis-cli installed

3e97b81

Update install-redis-cli.sh

cf69f69

Enable redis module stream for RHEL 9 UBI compatibility

a059222

fix error handling to filter timestamp output

b7ffb33

Update gitops_utils

842eeeb

Update gitops_utils

975058b

Update gitops_utils

ef2b7c6

build

5c76068

New Lock Key Format

22796ff

Redis Lock Key

5e7e6b4

Without any password warnings or the -a **** flag in the display.

a1930a1

build

d22442f

Merge branch 'master' into mascore-13109-branch-lock

da35f00

Idempotent cleanup

b95f77a

Handles "No Changes" Scenario

e1bce1e

Redis Authentication and Exit Code Issues

36230ef

Update gitops_suite_workspace

2cfd021

migrated other functions

26ceff9

Removed fallback-to-Git mechanism

7b8f515

Redis Setup Documentation

74f7de3

Hardik-Prajapati-10 requested a review from a team as a code owner April 16, 2026 09:28

Hardik-Prajapati-10 requested a review from whitfiea April 16, 2026 09:32

Hardik-Prajapati-10 requested review from amitpandey0217, mnivedithaa and rbinns April 16, 2026 09:33

Hardik-Prajapati-10 self-assigned this Apr 16, 2026

removed debug logs

435b8cf

whitfiea changed the title ~~Implement Redis-Based Distributed Locking for GitOps Operations~~ [major] Implement Redis-Based Distributed Locking for GitOps Operations Apr 17, 2026

whitfiea requested changes Apr 17, 2026

View reviewed changes

Hardik-Prajapati-10 added 3 commits April 17, 2026 17:26

Merge branch 'master' into mascore-13109-branch-lock

087e74f

Update ibm-toolchain-redis-setup.sh

3f295e8

whitfiea requested changes Apr 20, 2026

View reviewed changes

Hardik-Prajapati-10 added 5 commits April 20, 2026 15:55

fallback-to-Git mechanism

8ebeb15

If Redis is available → Uses Redis locking If Redis is unavailable → Automatically falls back to Git locking

Update redis-locking-setup.md

6a6869f

Merge branch 'master' into mascore-13109-branch-lock

aa30b04

docs updated

a59f24b

Merge branch 'master' into mascore-13109-branch-lock

cee7f22

whitfiea requested changes Apr 21, 2026

View reviewed changes

Comment thread image/cli/mascli/functions/gitops_utils Outdated

Comment thread image/cli/mascli/functions/gitops_utils Outdated

Hardik-Prajapati-10 requested a review from whitfiea April 21, 2026 08:22

changed GITOPS_USE_REDIS_LOCKING default value from true to false and…

1f67dac

… updated behavior to error when Redis is unavailable with explicit Redis locking enabled.

whitfiea changed the title ~~[major] Implement Redis-Based Distributed Locking for GitOps Operations~~ [minor] Implement Redis-Based Distributed Locking for GitOps Operations Apr 21, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[minor] Implement Redis-Based Distributed Locking for GitOps Operations#2189

[minor] Implement Redis-Based Distributed Locking for GitOps Operations#2189
Hardik-Prajapati-10 wants to merge 38 commits intomasterfrom
mascore-13109-branch-lock

Hardik-Prajapati-10 commented Apr 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

whitfiea Apr 17, 2026

Uh oh!

Hardik-Prajapati-10 Apr 17, 2026 •

edited

Loading

Uh oh!

whitfiea Apr 20, 2026

Uh oh!

Hardik-Prajapati-10 Apr 20, 2026

Uh oh!

whitfiea Apr 17, 2026

Uh oh!

Hardik-Prajapati-10 Apr 17, 2026

Uh oh!

Uh oh!

whitfiea Apr 20, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Hardik-Prajapati-10 commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Description

Key Changes

1. Redis Locking Infrastructure (image/cli/mascli/functions/gitops_utils)

2. Container Image Updates

3. Migrated GitOps Functions (10 functions)

4. Documentation

Impact

Test Results

Manual Testing

Integration Testing

Configuration Testing

Backporting

Related Pull Requests

⚠️ Notes for Reviewers

Uh oh!

Uh oh!

whitfiea Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Hardik-Prajapati-10 Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

The env var acts as a "safety check" - even though Redis is required, the explicit flag prevents accidental misconfiguration and provides clear error messages when Redis setup is incomplete.

Uh oh!

whitfiea Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Hardik-Prajapati-10 Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

whitfiea Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Hardik-Prajapati-10 Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

whitfiea Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Hardik-Prajapati-10 commented Apr 16, 2026 •

edited

Loading

1. Redis Locking Infrastructure (`image/cli/mascli/functions/gitops_utils`)

Hardik-Prajapati-10 Apr 17, 2026 •

edited

Loading