[SPARK-57900][K8S][TEST] Add OIDC credential propagation E2E tests on Minikube with moto

### What changes were proposed in this pull request?
Add a new optional integration-test module, `connector/credential-aws-integration-tests`, that validates the end-to-end OIDC credential propagation pipeline on a real Kubernetes cluster (Minikube). This is Sub-task 11 of the OIDC Credential Propagation SPIP ([SPARK-57703](https://issues.apache.org/jira/browse/SPARK-57703)), and it exercises the
whole feature together: projected ServiceAccount token -> `FileTokenIngestor` -> `AwsStsCredentialProvider` -> STS -> S3A read/write, plus mid-job token rotation and late-registering executors.

The tests use [moto](https://github.com/getmoto/moto) (Apache 2.0-licensed) as a lightweight S3 + STS backend. The original SPIP mentioned LocalStack, but both LocalStack and MinIO have moved away from freely usable OSS distributions and are incompatible with the ASF license policy. moto runs as a plain HTTP server (no extra container) and does not
verify the OIDC JWT, keeping the test focused on Spark's credential propagation logic.

**Three scenarios are implemented:**

1. **Basic flow** (`OidcS3ReadWriteJob`): a Spark job on Minikube exchanges the identity token for STS credentials and reads/writes S3 via S3A.
2. **Mid-job token rotation** (`OidcTokenRotationJob`): a long-running job writes to S3 repeatedly while the test rewrites the identity token file in the driver pod. The initial token is supplied by an init container into an emptyDir (an externally-provided, rotatable token file, as the SPIP assumes). The rotated token carries a *different* principal; with a short renewal interval, `UserCredentialManager` re-reads it, re-exchanges it via STS, and propagates fresh credentials. The test asserts the driver logged the rotated principal (proving the new token was actually read, not a no-op) and that S3 output for all iterations spanning the rotation is present.
3. **Late-registering executor** (`OidcLateExecutorJob`): with dynamic allocation and a short idle timeout, a job warms up, idles until executors scale down, then runs a wider stage that forces new executors to register *after* credentials were acquired. The test asserts more than one distinct executor registered over the run (evidence of a genuinely late-registering executor) and that the wide stage produced all outputs — an executor that did not receive credentials via the `SparkAppConfig` registration response would have failed its task.

**Structure and design:**

- The module is gated behind the `-Poidc-e2e` Maven profile (and requires `-Pkubernetes`), so it is skipped by default.
- Jobs that run on the cluster live in `src/main` so they are packaged into the module jar and baked into the Spark image; test classes are not packaged.
- S3A support (hadoop-aws + AWS SDK) is provided by building the image with `-Phadoop-cloud`.
- Image building is handled by an explicit step (`docker-image-tool.sh`) in CI and by `dev-run-integration-tests.sh` locally, rather than being bound to the sbt test task.
- The spark-submit helpers (`SparkAppLauncher`, `SparkAppConf`, `SparkAppArguments`, `ProcessUtils`) are implemented locally instead of depending on the `spark-kubernetes-integration-tests` test-jar, which sbt could not resolve as an inter-project reference. They mirror the equivalents there.
- moto is reached from two vantage points: pods use the host gateway IP (`spark.oidc.test.s3Endpoint` / `stsEndpoint`), while the test process uses loopback (`spark.oidc.test.s3ClientEndpoint`). In CI, moto is installed into an isolated virtualenv
(to avoid the OS-provided urllib3/pyOpenSSL that crashes moto on startup) and started inside the same workflow step that runs the tests.

**New files:**
- `connector/credential-aws-integration-tests/` — module with the test suite (`OidcCredentialE2ESuite`), the three Spark jobs, spark-submit helpers, `pom.xml`, `log4j2.properties`, a local runner script (`dev-run-integration-tests.sh`), and `README.md`.

**Modified files:**
- `pom.xml` (root) — add the `oidc-e2e` profile / module.
- `project/SparkBuild.scala` — register `credentialAwsIntegrationTests`.
- `.github/workflows/build_and_test.yml` — add the `oidc-e2e` job (moto + Minikube).

### Why are the changes needed?
The SPIP calls for an end-to-end test that validates the full credential propagation pipeline in a realistic Kubernetes environment. The prior sub-tasks each cover a slice with unit/integration tests, but nothing exercised the entire flow — token ingestion, STS exchange, RPC + SparkAppConfig propagation, S3A read/write, and mid-job refresh — against a real cluster. This module provides that coverage and guards against regressions in how the pieces fit together.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
This *is* the test. The suite was run on a local Minikube (with moto) under both build tools and all three scenarios passed:

- sbt: `build/sbt -Phadoop-3 -Pkubernetes -Pcredential-aws -Poidc-e2e ... credential-aws-integration-tests/test`
- Maven: `build/mvn integration-test -pl connector/credential-aws-integration-tests -Phadoop-3 -Pkubernetes -Pcredential-aws -Poidc-e2e ...`

The new `oidc-e2e` GitHub Actions job (Minikube + moto) is green. `dev-run-integration-tests.sh` was also verified to build the image, start/stop moto, and run the suite end-to-end.

### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude

Closes #58426 from sarutak/oidc-propagation/e2e-tests.

Authored-by: Kousuke Saruta <sarutak@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Author: sarutak

.github/workflows/build_and_test.yml

@@ -116,6 +116,10 @@ jobs:
           if [[ "${{ github.repository }}" != 'apache/spark' ]]; then
             yarn=`./dev/is-changed.py -m yarn`
             kubernetes=`./dev/is-changed.py -m kubernetes`
+            # The OIDC E2E job covers connector/credential-aws end to end on K8s, so it
+            # must run when either the Kubernetes support or the credential-aws module
+            # (including its integration-tests module, mapped to credential-aws) changes.
+            oidc_e2e=`./dev/is-changed.py -m kubernetes,credential-aws`
             sparkr=`./dev/is-changed.py -m sparkr`
             tpcds=`./dev/is-changed.py -m sql`
             docker=`./dev/is-changed.py -m docker-integration-tests`
@@ -156,6 +160,7 @@ jobs:
             pandas=false
             yarn=false
             kubernetes=false
+            oidc_e2e=false
             sparkr=false
             tpcds=false
             docker=false
@@ -182,6 +187,7 @@ jobs:
               \"docs\" : \"$docs\",
               \"yarn\" : \"$yarn\",
               \"k8s-integration-tests\" : \"$kubernetes\",
+              \"oidc-e2e\" : \"$oidc_e2e\",
               \"buf\" : \"$buf\",
               \"ui\" : \"$ui\",
               \"transpile\": \"$transpile\",
@@ -541,6 +547,7 @@ jobs:
         fromJson(needs.precondition.outputs.required).sparkr == 'true' ||
         fromJson(needs.precondition.outputs.required).docker-integration-tests == 'true' ||
         fromJson(needs.precondition.outputs.required).k8s-integration-tests == 'true' ||
+        fromJson(needs.precondition.outputs.required).oidc-e2e == 'true' ||
         fromJson(needs.precondition.outputs.required).tpcds-1g == 'true')
     name: "Precompile Spark"
     runs-on: ubuntu-latest
@@ -582,7 +589,7 @@ jobs:
       run: |
         ./build/sbt -Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud -Phive \
           -Pkubernetes -Pjvm-profiler -Pkinesis-asl -Pcredential-aws -Phive-thriftserver \
-          -Pdocker-integration-tests -Pkubernetes-integration-tests -Pvolcano \
+          -Pdocker-integration-tests -Pkubernetes-integration-tests -Pvolcano -Poidc-e2e \
           Test/package streaming-kinesis-asl-assembly/assembly connect/assembly assembly/package
     - name: Package compile output
       run: |
@@ -1447,6 +1454,171 @@ jobs:
           name: spark-on-kubernetes-it-log
           path: "**/target/integration-tests.log"
 
+  oidc-e2e:
+    needs: [precondition, precompile]
+    if: fromJson(needs.precondition.outputs.required).oidc-e2e == 'true'
+    name: Run OIDC E2E integration tests (Minikube + moto)
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    steps:
+      - name: Bootstrap composite actions
+        uses: actions/checkout@v6
+      - name: Checkout and sync Spark repository
+        uses: ./.github/actions/checkout-and-sync
+        with:
+          ref: ${{ needs.precondition.outputs.head_sha }}
+      - name: Cache SBT and Maven
+        uses: actions/cache@v5
+        with:
+          path: |
+            build/apache-maven-*
+            build/*.jar
+            ~/.sbt
+          key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+          restore-keys: |
+            build-${{ runner.os }}-
+      - name: Restore Coursier local repository
+        uses: actions/cache/restore@v5
+        with:
+          path: ~/.cache/coursier
+          key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+          restore-keys: |
+            coursier-${{ runner.os }}-
+      - name: Free up disk space
+        timeout-minutes: 10
+        continue-on-error: true
+        run: |
+          if [ -f ./dev/free_disk_space ]; then
+            ./dev/free_disk_space
+          fi
+      - name: Install Java ${{ inputs.java }}
+        uses: actions/setup-java@v5
+        with:
+          distribution: zulu
+          java-version: ${{ inputs.java }}
+      - name: Download precompiled artifact
+        uses: actions/download-artifact@v8
+        with:
+          name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
+      - name: Extract precompiled artifact
+        run: |
+          zstd -dc compile-artifact.tar.zst | tar -xf -
+          rm compile-artifact.tar.zst
+      - name: Install Python and moto
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y python3-venv
+          # Install moto into an isolated virtualenv. A plain "pip install --user"
+          # leaves the OS-provided packages under /usr/lib/python3/dist-packages on
+          # the import path, and that copy of pyOpenSSL is incompatible with the
+          # cryptography build pulled in transitively, which crashes moto on startup
+          # (AttributeError: module 'lib' has no attribute 'GEN_EMAIL'). A venv gives
+          # moto a clean, self-consistent dependency set.
+          python3 -m venv "${RUNNER_TEMP}/moto-venv"
+          "${RUNNER_TEMP}/moto-venv/bin/pip" install --upgrade pip
+          "${RUNNER_TEMP}/moto-venv/bin/pip" install "moto[server,s3,sts]>=5.0.0,<6.0.0"
+          echo "MOTO_PYTHON=${RUNNER_TEMP}/moto-venv/bin/python" >> $GITHUB_ENV
+      - name: Start Minikube
+        uses: medyagh/setup-minikube@e9e035a86bbc3caea26a450bd4dbf9d0c453682e # v0.0.21
+        with:
+          kubernetes-version: "1.36.0"
+          cpus: 2
+          memory: 6144m
+      - name: Print K8S pods and nodes info
+        run: |
+          kubectl get pods -A
+          kubectl describe node
+      - name: Resolve moto host IP reachable from Minikube pods
+        run: |
+          # The host gateway IP as seen from inside Minikube
+          HOST_IP=$(minikube ssh "ip route | grep default | awk '{print \$3}'" | tr -d '[:space:]')
+          echo "MOTO_HOST_IP=${HOST_IP}" >> $GITHUB_ENV
+          echo "Moto reachable from Minikube at: http://${HOST_IP}:5000"
+      - name: Build and load Spark image into Minikube
+        run: |
+          eval $(minikube docker-env)
+          # Build the base Spark Docker image using the precompiled artifact. The
+          # precompile step builds with -Phadoop-cloud, so hadoop-aws and the AWS SDK
+          # (needed for S3A) are included in /opt/spark/jars.
+          ./bin/docker-image-tool.sh \
+            -r docker.io/kubespark \
+            -t oidc-e2e-test \
+            build
+          # The job class (OidcS3ReadWriteJob) lives in this module's jar, which is not
+          # part of the assembly and is not copied into the image by docker-image-tool.sh
+          # (it only copies examples/jars). SparkSubmit already puts a local:// primary
+          # resource on the driver classpath, so this is not a classpath gap -- the jar
+          # simply has to be present in the container, so bake it into /opt/spark/jars.
+          JOB_JAR=$(ls connector/credential-aws-integration-tests/target/scala-*/spark-credential-aws-integration-tests_*.jar | grep -v -- '-tests.jar' | head -1)
+          JOB_JAR_NAME=$(basename "$JOB_JAR")
+          BUILD_CTX=$(mktemp -d)
+          cp "$JOB_JAR" "$BUILD_CTX/"
+          cat > "$BUILD_CTX/Dockerfile" <<EOF
+          FROM docker.io/kubespark/spark:oidc-e2e-test
+          COPY ${JOB_JAR_NAME} /opt/spark/jars/${JOB_JAR_NAME}
+          EOF
+          docker build -t docker.io/kubespark/spark:oidc-e2e-test-job "$BUILD_CTX"
+          rm -rf "$BUILD_CTX"
+          export SPARK_IMAGE="docker.io/kubespark/spark:oidc-e2e-test-job"
+          echo "SPARK_IMAGE=${SPARK_IMAGE}" >> $GITHUB_ENV
+      - name: Run OIDC E2E integration tests
+        run: |
+          # moto must be started in the same step as the test run: GitHub Actions
+          # terminates background processes when a step's shell exits, so a moto
+          # started in a previous step would already be dead here.
+          # Use the isolated venv python so moto runs against a clean, self-consistent
+          # dependency set (avoids the broken OS-provided pyOpenSSL).
+          "${MOTO_PYTHON}" -m moto.server -H 0.0.0.0 -p 5000 > /tmp/moto.log 2>&1 &
+          MOTO_PID=$!
+          trap "kill ${MOTO_PID} 2>/dev/null || true" EXIT
+          # Wait for moto to be ready, and fail loudly (with logs) if it never comes up.
+          ready=false
+          for i in $(seq 1 30); do
+            if curl -sf http://127.0.0.1:5000/ >/dev/null 2>&1; then
+              ready=true
+              break
+            fi
+            if ! kill -0 ${MOTO_PID} 2>/dev/null; then
+              echo "ERROR: moto server process exited early." >&2
+              cat /tmp/moto.log >&2
+              exit 1
+            fi
+            sleep 1
+          done
+          if [ "${ready}" != "true" ]; then
+            echo "ERROR: moto server did not become ready within 30s." >&2
+            cat /tmp/moto.log >&2
+            exit 1
+          fi
+          echo "moto server is ready on 127.0.0.1:5000 (PID ${MOTO_PID})."
+          # Pods inside Minikube reach moto via the host gateway IP; the test
+          # process (on the host) reaches the same moto server on loopback.
+          MOTO_ENDPOINT="http://${MOTO_HOST_IP}:5000"
+          MOTO_CLIENT_ENDPOINT="http://127.0.0.1:5000"
+          # Grant cluster-admin to all service accounts. This is intentionally broad
+          # but scoped to this ephemeral, single-run CI Minikube cluster (destroyed
+          # with the runner). It mirrors kubernetes-integration-tests and spares the
+          # test from provisioning fine-grained RBAC (pod create/exec/log, namespace
+          # management) for the many service accounts Spark-on-K8s uses. Do NOT copy
+          # this into any shared or long-lived cluster; there, create a minimal Role
+          # limited to the test namespace instead.
+          kubectl create clusterrolebinding serviceaccounts-cluster-admin \
+            --clusterrole=cluster-admin --group=system:serviceaccounts || true
+          build/sbt \
+            -Phadoop-3 -Pkubernetes -Pcredential-aws -Poidc-e2e \
+            -Dspark.kubernetes.test.deployMode=minikube \
+            -Dspark.oidc.test.stsEndpoint="${MOTO_ENDPOINT}" \
+            -Dspark.oidc.test.s3Endpoint="${MOTO_ENDPOINT}" \
+            -Dspark.oidc.test.s3ClientEndpoint="${MOTO_CLIENT_ENDPOINT}" \
+            -Dspark.oidc.test.sparkImage="${SPARK_IMAGE}" \
+            "credential-aws-integration-tests/test"
+      - name: Upload OIDC E2E integration tests log files
+        if: ${{ !success() }}
+        uses: actions/upload-artifact@v7
+        with:
+          name: oidc-e2e-it-log
+          path: "**/target/integration-tests.log"
+
   ui:
     needs: [precondition]
     if: fromJson(needs.precondition.outputs.required).ui == 'true'

.github/workflows/build_java17.yml

@@ -45,5 +45,6 @@ jobs:
           "tpcds-1g": "true",
           "docker-integration-tests": "true",
           "k8s-integration-tests": "true",
+          "oidc-e2e": "true",
           "lint" : "true"
         }

.github/workflows/build_java21.yml

@@ -52,6 +52,7 @@ jobs:
           "docker-integration-tests": "true",
           "yarn": "true",
           "k8s-integration-tests": "true",
+          "oidc-e2e": "true",
           "buf": "true",
           "ui": "true"
         }

.github/workflows/build_java25.yml

@@ -52,6 +52,7 @@ jobs:
           "docker-integration-tests": "true",
           "yarn": "true",
           "k8s-integration-tests": "true",
+          "oidc-e2e": "true",
           "buf": "true",
           "ui": "true"
         }