[SPARK-58776][K8S] Propagate spark.ssl.rpc.* passwords to executors by sweb · Pull Request #58010 · apache/spark

sweb · 2026-08-14T18:16:42Z

What changes were proposed in this pull request?

BasicExecutorFeatureStep now populates the executor environment from SecurityManager.getEnvironmentForSslRpcPasswords, immediately alongside the existing _SPARK_AUTH_SECRET injection. This is the same call the standalone worker already makes in CommandUtils.

Variable names the user already binds through spark.kubernetes.executor.secretKeyRef are skipped, so an explicit secret reference does not end up with a literal password beside it in the pod spec.

Scope note: this fixes Kubernetes only. YARN has the same issue and requires a follow up. This is tracked in https://issues.apache.org/jira/browse/SPARK-59408

Why are the changes needed?

spark.ssl.rpc.enabled=true cannot work on Kubernetes today. SparkConf.isExecutorStartupConf deliberately withholds spark.ssl.* keys containing Password from the executor startup conf, with the comment "Passwords are propagated separately though". That separate channel is the _SPARK_SSL_RPC_* environment variables, read back by SSLOptions.parse, and the Kubernetes backend never wrote them. Executors thus start with no keystore password and die building their RpcEnv:

java.lang.RuntimeException: SSLFactory creation failed
    at org.apache.spark.network.ssl.SSLFactory.<init>(SSLFactory.java:84)
    at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:69)
Caused by: java.security.UnrecoverableKeyException: Get Key failed:
    Cannot read the array length because "password" is null

The application then aborts on Max number of executor failures (4) reached without submitting a job. Using RPC SSL on Kubernetes currently requires setting the four variables by hand, through spark.kubernetes.executor.secretKeyRef, spark.executorEnv, or an executor pod template.

Does this PR introduce any user-facing change?

Yes. Previously, setting spark.ssl.rpc.enabled=true on Kubernetes without also setting spark.kubernetes.executor.secretKeyRef._SPARK_SSL_RPC_* caused every executor to fail during RpcEnv construction and the application to abort. Now the passwords reach the executors from the driver's configuration and the application runs.

Users already applying the secretKeyRef workaround are unaffected: those bindings still win, and no literal password is added beside them.

The _SPARK_SSL_RPC_* fallbacks in SSLOptions.parse are not scoped to the rpc namespace. Once RPC SSL is enabled, another enabled namespace on the executor can therefore resolve the RPC password.

How was this patch tested?

Three new tests in BasicExecutorFeatureStepSuite:

SSL RPC password propagation
SSL RPC passwords shouldn't propagate if RPC SSL is disabled
SSL RPC passwords shouldn't override an explicit secretKeyRef

Also verified end-to-end on minikube, k8s v 1.35.1

Was this patch authored or co-authored using generative AI tooling?

Claude Code was used when debugging the original issue that lead to this PR - the first version of this PR was drafted in that session as well.

uros-b · 2026-08-15T10:22:29Z

Thank you @sweb! Adding @pan3793 for additional review here (K8s)

sweb · 2026-08-24T09:36:54Z

@pan3793 may I ask you for a review? Thanks a lot!

uros-b · 2026-08-26T12:00:43Z

Yes, I'm not expert in this part - so PTAL @pan3793 ^^

sweb · 2026-09-04T11:01:28Z

Hey @uros-b anyone else we could ask for a review? :)

uros-b · 2026-09-04T15:06:32Z

@LuciferYang and @dongjoon-hyun are also experts in K8S

pan3793 · 2026-09-04T15:15:12Z

The fix looks correct from the functional perspective, but sorry, I don't use this feature, and given that it involves security parts, I think I'm not the right person to review it.

uros-b

perhaps we should update the docs a bit too

uros-b · 2026-09-04T15:15:16Z

+      val sslRpcPasswords = secMgr.getEnvironmentForSslRpcPasswords.filterNot {
+        case (name, _) => kubernetesConf.secretEnvNamesToKeyRefs.contains(name)
+      }.toSeq
+


docs/security.md already says that on Kubernetes the auth secret is injected as an env var, so anyone who can list pods can read it. This PR puts keystore / truststore / key passwords on that same channel. That’s the intended standalone design, but it is a user-facing security change on K8s and should be called out next to the existing auth-secret paragraph. Users who don’t want literals can keep using spark.kubernetes.executor.secretKeyRef.SPARK_SSL_RPC*.

thank you for your help! I'll take a look at the docs.

@uros-b I added it here: 2285333

I placed it in Network Encryption -> SSL Encryption but I am happy to move it to the explicit Kubernetes section within authentication.

LuciferYang · 2026-09-05T02:05:12Z

+      // spark.kubernetes.executor.secretKeyRef are skipped, so that an explicit
+      // secret reference is not shadowed by a literal password in the pod spec.
+      val sslRpcPasswords = secMgr.getEnvironmentForSslRpcPasswords.filterNot {
+        case (name, _) => kubernetesConf.secretEnvNamesToKeyRefs.contains(name)


The filter only skips names bound via spark.kubernetes.executor.secretKeyRef. A spark.executorEnv._SPARK_SSL_RPC_KEY_STORE_PASSWORD enters the same env Seq earlier via kubernetesConf.environment, buildEnvVars does not dedup, and a same-name env predefined on a pod template (including valueFrom) is unchecked. Kubernetes validation does not reject duplicate env names; runtimes resolve them last-wins, and the literal is appended after the user entry, so it silently overrides the value the user set. A template secretKeyRef password effectively becomes a plaintext literal, opposite of the precedence this PR intends for secretKeyRef.

Please extend the skip-list to kubernetesConf.environment keys plus env names already on the container (with a template, that is the predefined set); skipping means the user value wins. For the test, group the container env by name and assert no duplicates — checkEnv collapses to a Map and cannot detect them. The description's "the only way is a Secret" claim needs a correction too.

Addressed in 2285333 - I also adjusted the PR description.

LuciferYang · 2026-09-05T02:05:13Z

          ++ attributes
          ++ kubernetesConf.environment
          ++ sparkAuthSecret
+          ++ sslRpcPasswords


This fix only wires Kubernetes. On YARN, ExecutorRunnable.prepareCommand filters executor java opts with the same SparkConf.isExecutorStartupConf (passwords withheld there too), and prepareEnvironment only forwards env vars starting with SPARK plus spark.executorEnv; it never calls getEnvironmentForSslRpcPasswords. Under --master yarn the same configuration fails executors with exactly the symptom this PR describes.

ExecutorRunnable already receives securityMgr as a constructor param, so appending the same map at the end of prepareEnvironment, mirroring CommandUtils on standalone, is a one-liner. The HashMap overwrite semantics deduplicate naturally; note the driver-side value then overrides a same-name spark.executorEnv entry — the standalone behavior, opposite of the k8s-side user-value precedence. If YARN is out of scope, please say so in the description and open a follow-up JIRA.

If it's okay with you, I'd keep YARN out of this PR? I am happy to create a follow up JIRA / PR it that works for you. I added a scope note to the PR description and created this ticket: https://issues.apache.org/jira/browse/SPARK-59408

LuciferYang · 2026-09-05T02:05:15Z

+      // does in CommandUtils. Names the user already supplies via
+      // spark.kubernetes.executor.secretKeyRef are skipped, so that an explicit
+      // secret reference is not shadowed by a literal password in the pod spec.
+      val sslRpcPasswords = secMgr.getEnvironmentForSslRpcPasswords.filterNot {


One behavioral edge, no code change requested: the _SPARK_SSL_RPC_* env fallback lives in the shared SSLOptions.parse, so every spark.ssl.* namespace reads it, ahead of the defaults fallback. Global spark.ssl.* passwords are withheld from executors too, so once rpc is enabled another enabled namespace resolves the rpc password on executors while the driver uses its own, and the two sides disagree. Standalone injects the same env vars and behaves identically, and before this PR executors had no password on that path at all, so this breaks no working setup. Worth one sentence in the PR description or the docs.

I added a paragraph to the PR description

LuciferYang · 2026-09-05T02:05:16Z

    }
  }

+  test("SSL RPC password propagation") {


The three new tests all set spark.ssl.rpc.* passwords explicitly. The shape where a user sets spark.ssl.enabled=true with the global spark.ssl.keyStorePassword, sets no rpc-side password, and separately enables spark.ssl.rpc.enabled=true also works: rpcSSLOptions inherits the global password through the defaults fallback on the driver, and the env carries the inherited value (spark.ssl.enabled must be on, or nothing is inherited). That is one of the common usages this PR claims to fix, and right now no test guards the inheritance path — could you add one for this shape?

Addressed in 2285333 in test SSL RPC passwords inherited from the global spark.ssl.* namespace propagate

LuciferYang · 2026-09-05T02:05:18Z

 upgrading Spark versions.

+On Kubernetes, the passwords used by RPC SSL are propagated to executor pods using environment
+variables, the same way as the authentication mechanics described under [Kubernetes](#Kubernetes)


The [Kubernetes](#Kubernetes) link in this new paragraph does not resolve: the target heading is ### Kubernetes, whose generated slug is lowercase kubernetes, and fragment matching is case-sensitive. Changing the link to (#kubernetes) fixes it.

fixed in 2285333

sweb · 2026-09-05T10:53:02Z

@LuciferYang thank you for your review! New commit was added, addressing your feedback.

…n BasicExecutorFeatureStep

sweb · 2026-09-12T07:49:12Z

@LuciferYang may I ask for another review? Thanks!

sweb force-pushed the SPARK-58776 branch from 8b82883 to acdb078 Compare August 14, 2026 19:42

uros-b approved these changes Aug 15, 2026

View reviewed changes

uros-b requested a review from pan3793 August 15, 2026 10:22

uros-b reviewed Sep 4, 2026

View reviewed changes

uros-b requested review from LuciferYang, dongjoon-hyun and uros-b September 4, 2026 15:17

LuciferYang reviewed Sep 5, 2026

View reviewed changes

sweb added 3 commits September 8, 2026 10:43

[SPARK-58776][K8S] Propagate spark.ssl.rpc.* passwords to executors i…

2dc071b

…n BasicExecutorFeatureStep

docs: add section on rpc ssl passwords on kubernetes via env vars

1da8deb

address review comments

2285333

sweb force-pushed the SPARK-58776 branch from 22e8bbb to 2285333 Compare September 8, 2026 08:44

sweb requested a review from LuciferYang September 8, 2026 10:34

Uh oh!

Conversation

sweb commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

uros-b commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sweb commented Aug 24, 2026

Uh oh!

uros-b commented Aug 26, 2026

Uh oh!

sweb commented Sep 4, 2026

Uh oh!

uros-b commented Sep 4, 2026

Uh oh!

pan3793 commented Sep 4, 2026

Uh oh!

uros-b left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sweb Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sweb Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sweb Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sweb Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sweb Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sweb commented Sep 5, 2026

Uh oh!

sweb commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sweb commented Aug 14, 2026 •

edited

Loading

uros-b commented Aug 15, 2026 •

edited

Loading

sweb Sep 4, 2026 •

edited

Loading

sweb Sep 5, 2026 •

edited

Loading

sweb Sep 5, 2026 •

edited

Loading

sweb Sep 5, 2026 •

edited

Loading

sweb Sep 5, 2026 •

edited

Loading