Skip to content

fix: propagate shared resource values and rerun mirror postgres init#4650

Open
JeffreyDallas wants to merge 12 commits into
mainfrom
fix-shared-resouce-value-file
Open

fix: propagate shared resource values and rerun mirror postgres init#4650
JeffreyDallas wants to merge 12 commits into
mainfrom
fix-shared-resouce-value-file

Conversation

@JeffreyDallas

@JeffreyDallas JeffreyDallas commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Propagate mirror-node values into the shared-resources Helm chart so Postgres/Redis inherit scheduling values such as tolerations and node selectors.
  • Add a Redis role scheduling fallback so shared Redis inherits solo.hashgraph.io/role scheduling from mirror-node component values when Redis does not define it explicitly.
  • Propagate top-level scheduling values into mirror pinger and the separate mirror/explorer HAProxy ingress controller charts.
  • Apply explorer --ingress-controller-value-file consistently for explorer ingress controller installs.
  • Run internal mirror-node Postgres secret priming and database initialization on reruns even when the shared-resources Helm release already exists.
  • Add unit coverage for shared-resource scheduling propagation, Helm scheduling propagation, and rerun skip behavior.

Root Cause

There were four related failure modes:

  1. Mirror-node values were not propagated to the shared-resources chart, so shared Postgres/Redis pods could miss cluster scheduling values and remain Pending on tainted performance clusters.
  2. The performance cluster also taints nodes by solo.hashgraph.io/role. The mirror-node values put that role scheduling on mirror-node components, but Redis did not define role scheduling directly. Redis therefore still lacked the role toleration after owner/network propagation was fixed.
  3. Mirror pinger is a mirror chart component that does not inherit top-level nodeSelector/tolerations, and mirror/explorer HAProxy ingress controllers are installed as separate Helm releases. Those pods could remain Pending even after shared resources were fixed.
  4. After a partial failure, a retry set installSharedResources=false because the shared-resources release already existed. The mirror-node command used that flag to skip Postgres initialization, even though the mirror-node DB roles/users may not have been created yet.

installSharedResources only tells us whether the shared-resources chart needs to be installed in this run. It is not a valid signal that mirror-node Postgres has already been initialized. The existing initialization code is idempotent through the solo-initialized database comment sentinel, so it is safe to invoke for every internal DB deployment.

Closes #4649.

Validation

  • npx mocha 'test/unit/core/shared-resources/shared-resource-manager.test.ts'
  • MOCHA_SUITE_NAME="MirrorNodeCommand" npx mocha 'test/unit/commands/mirror-node.test.ts'
  • npx mocha test/unit/core/util/helm-scheduling-values.test.ts test/unit/commands/mirror-node.test.ts test/unit/commands/explorer.test.ts
  • npx tsc --noEmit --pretty false
  • npx eslint src/core/shared-resources/shared-resource-manager.ts test/unit/core/shared-resources/shared-resource-manager.test.ts
  • npx eslint src/commands/mirror-node.ts test/unit/commands/mirror-node.test.ts
  • task build
  • task format
  • task check

task build, task format, and task check completed successfully with the repository's existing lint/typedoc warnings.

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>
@trunk-io

trunk-io Bot commented Jun 13, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@github-actions

github-actions Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Unit Test Results - Linux

38 tests  +2   38 ✅ +2   0s ⏱️ ±0s
17 suites ±0    0 💤 ±0 
 1 files   ±0    0 ❌ ±0 

Results for commit 4168261. ± Comparison against base commit de23795.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Unit Test Results - Windows

    1 files  ± 0    333 suites  +1   9s ⏱️ ±0s
1 051 tests +11  1 051 ✅ +11  0 💤 ±0  0 ❌ ±0 
1 055 runs  +11  1 055 ✅ +11  0 💤 ±0  0 ❌ ±0 

Results for commit 4168261. ± Comparison against base commit de23795.

♻️ This comment has been updated with latest results.

@github-actions

Copy link
Copy Markdown
Contributor

E2E Test Report

 10 files  ±0   94 suites  ±0   1h 23m 27s ⏱️ -33s
301 tests ±0  301 ✅ ±0  0 💤 ±0  0 ❌ ±0 
320 runs  ±0  320 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 110e9da. ± Comparison against base commit 20b52d3.

@github-actions

github-actions Bot commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

E2E Test Report

 10 files  ±0   94 suites  ±0   1h 24m 39s ⏱️ + 1m 21s
301 tests ±0  301 ✅ ±0  0 💤 ±0  0 ❌ ±0 
320 runs  ±0  320 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 88b7004. ± Comparison against base commit de23795.

♻️ This comment has been updated with latest results.

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>
@JeffreyDallas JeffreyDallas force-pushed the fix-shared-resouce-value-file branch from 110e9da to 60d08e4 Compare June 13, 2026 04:41
@JeffreyDallas JeffreyDallas marked this pull request as ready for review June 14, 2026 15:45
@JeffreyDallas JeffreyDallas requested a review from a team as a code owner June 14, 2026 15:45
@JeffreyDallas JeffreyDallas self-assigned this Jun 14, 2026
@JeffreyDallas JeffreyDallas added the P1-💎 Current Milestone & Goals label Jun 14, 2026
@jan-milenkov jan-milenkov added the PR: Checks Failed A pull request where the checks have failed. label Jun 15, 2026
Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>
Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>
@JeffreyDallas

Copy link
Copy Markdown
Contributor Author

Migration CI follow-up: the failure was caused by the Postgres init wrapper reading the database sentinel with obj_description(oid, 'pg_database'). Database comments are shared-object comments in PostgreSQL, so this returned empty even after Solo wrote COMMENT ON DATABASE mirror_node IS 'solo-initialized'. During mirror node upgrade, Solo treated the initialized DB as partial, dropped/recreated the DB, left the legacy mirror_rest role behind, and the importer later failed Flyway V1 with ERROR: role "mirror_rest" already exists.

Commit d730739a2 fixes this by reading the sentinel with shobj_description(oid, 'pg_database') and including mirror_rest in partial-cleanup role handling. Validation passed in a clean PR worktree: focused Postgres shared-resource unit test, npx tsc --noEmit --pretty false, task build, task format, and task check all completed successfully with existing repository warnings.

@JeffreyDallas JeffreyDallas added PR: Needs Team Approval A pull request that needs review from a team member. and removed PR: Checks Failed A pull request where the checks have failed. labels Jun 15, 2026
Comment thread src/core/shared-resources/shared-resource-manager.ts
@jan-milenkov jan-milenkov added the PR: Unresolved Comments A pull request where there are comments and they need to be resolved. label Jun 16, 2026
@JeffreyDallas JeffreyDallas removed the PR: Unresolved Comments A pull request where there are comments and they need to be resolved. label Jun 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cleanup rogue function exports, look for opportunity to make the code more readable, avoid unnecessary complexion

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

@jan-milenkov jan-milenkov added PR: Unresolved Comments A pull request where there are comments and they need to be resolved. and removed PR: Needs Team Approval A pull request that needs review from a team member. labels Jun 17, 2026
Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>
Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>
@JeffreyDallas JeffreyDallas marked this pull request as draft June 17, 2026 17:08
@JeffreyDallas JeffreyDallas added PR: Needs Team Approval A pull request that needs review from a team member. and removed PR: Unresolved Comments A pull request where there are comments and they need to be resolved. labels Jun 17, 2026
@JeffreyDallas JeffreyDallas marked this pull request as ready for review June 17, 2026 20:32
Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Files exporting rogue functions are unwanted, if you need help with figuring out how to where/and how to refactor this, feel free to reach-out to teammembers

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated.
Added new rule to coding style to enforce the rogue function requirment

@jan-milenkov jan-milenkov added the PR: Unresolved Comments A pull request where there are comments and they need to be resolved. label Jun 18, 2026
Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>
@JeffreyDallas JeffreyDallas requested a review from a team as a code owner June 18, 2026 13:20
@JeffreyDallas JeffreyDallas removed the PR: Unresolved Comments A pull request where there are comments and they need to be resolved. label Jun 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P1-💎 Current Milestone & Goals PR: Needs Team Approval A pull request that needs review from a team member.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mirror node add rerun skips shared Postgres initialization after partial install failure

2 participants