[SPARK-59296][CORE][FOLLOWUP] Address review: bind local credential callback to this SparkEnv and document the driver-side early-startup window

Addresses review feedback on the local-mode parity change:

- LocalSchedulerBackend.setupUserCredentialManager now captures this backend's SparkEnv once and
  uses it in both the propagation callback and the initial credential store, instead of looking up
  the global SparkEnv.get on every callback. stop() only waits a bounded time for the renewal
  thread, so a renewal that outlives this SparkContext could otherwise write this application's
  credentials into a different SparkContext's store created later in the same JVM (notebook, test,
  or Spark Connect session), where the restarted version counter would then reject the new
  application's own renewals. Binding to this env confines a late renewal to this (stopped)
  application's store.

- Correct the applyProviderProperties scaladoc: running the selection phase in local mode extends
  the same driver-side early-startup window that already exists in cluster mode. Provider wiring is
  applied during SparkContext construction but credentials are not resolved until scheduler-backend
  start, so driver-side access to a wired scheme during construction (spark.jars / spark.files /
  spark.archives / spark.checkpoint.dir on e.g. s3a://) runs before credentials exist. The earlier
  wording ("points at credentials that are actually populated") was inaccurate for this window. The
  OIDC security docs (separate docs PR) document this for local mode alongside cluster mode.

This keeps credential resolution late (parity with HadoopDelegationTokenManager and with cluster
mode); moving resolution earlier in local mode only would diverge local from cluster and break that
design intent, so the window is documented rather than closed.

Author: sarutak

core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala

@@ -541,10 +541,16 @@ private[spark] object UserCredentialManager extends Logging {
    *
    * This phase runs in local mode as well: `LocalSchedulerBackend` starts a
    * [[UserCredentialManager]] (for parity with `HadoopDelegationTokenManager`, which also runs
-   * in `LocalSchedulerBackend`), so a resolution phase follows and the wiring applied here
-   * points at credentials that are actually populated. (Before SPARK-59296's follow-up, local
-   * mode had no resolution phase, so this selection phase was skipped there to avoid wiring a
-   * provider whose credentials would never be resolved.)
+   * in `LocalSchedulerBackend`), so a resolution phase follows and populates the credentials the
+   * wiring points at. As in cluster mode, the resolution phase runs later than this selection
+   * phase (at scheduler-backend start), so there is a driver-side early-startup window: any
+   * driver-side access to a wired scheme during `SparkContext` construction -- for example
+   * fetching `spark.jars` / `spark.files` / `spark.archives`, or a `spark.checkpoint.dir` on such
+   * a scheme -- happens before the credentials exist and therefore cannot use them. This is the
+   * same driver-side window that exists in cluster mode; prefer `local://` for such resources.
+   * (Before SPARK-59296's follow-up, local mode had no resolution phase; the wiring applied here
+   * would never have been backed by resolved credentials, so this phase returned early in local
+   * mode.)
    *
    * Scheme selection is limited to schemes for which a provider is UNAMBIGUOUSLY selected:
    * either an explicitly-configured scheme (`spark.security.oidc.provider.<scheme>`) or a

core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala

@@ -155,22 +155,29 @@ private[spark] class LocalSchedulerBackend(
    * driver's Hadoop Configuration.
    */
   private def setupUserCredentialManager(): Unit = {
+    // Capture this backend's SparkEnv once, rather than looking up the global SparkEnv.get on
+    // every callback. stop() only waits a bounded time for the renewal thread, so a renewal that
+    // outlives this SparkContext must not write into a *different* SparkContext's credential
+    // store (e.g. a new context created in the same JVM by a notebook, test, or Spark Connect
+    // session). Binding to this env ensures a late renewal updates only this application's store,
+    // which is harmless once this env is stopped. (CoarseGrainedSchedulerBackend avoids the issue
+    // differently, by routing updates through its own already-stopped driverEndpoint.)
+    val env = SparkEnv.get
     // Reuse the loader from SparkContext's selection phase (Some when OIDC is enabled, None
     // otherwise). Passing the Option straight through keeps SparkContext as the single owner of
     // the loader: create() enforces that an enabled configuration has a loader rather than
     // silently allocating one here that no one would close.
     userCredentialManager = UserCredentialManager.create(conf, { (version, credentials) =>
       // No remote executors in local mode; update the shared credential store directly so that
       // subsequently dispatched tasks (and driver-side access) observe the new credentials.
-      VersionedCredentials.updateIfNewer(SparkEnv.get.userCredentials, version, credentials)
+      VersionedCredentials.updateIfNewer(env.userCredentials, version, credentials)
     }, scheduler.sc.userCredentialProviderLoader)
     userCredentialManager.foreach { manager =>
       val (version, initialCredentials) = manager.start()
       // Store initial credentials synchronously so they are available for TaskDescription
       // (task dispatch) immediately. The onCredentialsUpdate callback above also runs the same
       // updateIfNewer, so this is idempotent.
-      VersionedCredentials.updateIfNewer(
-        SparkEnv.get.userCredentials, version, initialCredentials)
+      VersionedCredentials.updateIfNewer(env.userCredentials, version, initialCredentials)
     }
   }