test(uat): enable Azure serve now that GCP has served

GCP run 32825515255 is the served completion this lane was gated on.
AKS also checks the worker's LD_LIBRARY_PATH append from the other
side: /usr/local/nvidia is absent under the GPU Operator toolkit, so
the append must be an inert no-op there.

Signed-off-by: Rohit Rajani <rorajani@nvidia.com>

Author: rorajani

.github/workflows/uat-azure.yaml

@@ -775,10 +775,9 @@ jobs:
         run: ./tests/uat/azure/run conformance "${TEST_CONFIG}"
 
       # The CUJ phase is intent-selected, mirroring the runner's intent-aware
-      # `all`: training runs the Kubeflow TrainJob; inference would run the
-      # served DynamoGraphDeployment (phase_serve, DC3). For training, the
-      # TrainJob fires; for inference the serve step is currently disabled (see
-      # below), so verify/evidence cover the deployed stack either way.
+      # `all`: training runs the Kubeflow TrainJob, inference the served
+      # DynamoGraphDeployment (phase_serve, DC3). Exactly one fires per run;
+      # verify/evidence cover the deployed stack either way.
       - name: Refresh Azure credentials before CUJ
         if: steps.conformance.outcome == 'success'
         uses: azure/login@f5d393ae46f8fde4be8b75f32e3fc50e654ad0ca  # v3.0.1
@@ -797,28 +796,29 @@ jobs:
           RUN_ID: ${{ github.run_id }}
         run: ./tests/uat/azure/run train "${TEST_CONFIG}"
 
-      # STILL DISABLED (step commented out below). phase_serve already selects the
-      # GPU pool for both graph components (#1644), so the Frontend no longer
-      # cold-pulls the ~12GB vllm-runtime image on a small CPU-pool node. Only
-      # uat-gcp.yaml re-enables the serve *step* for now (GKE x inference is what
-      # #1644 tracks); re-enable this one after that lane has a served completion,
-      # and restore the Serve summary row below to `${{ steps.serve.outcome }}`.
-      # Until then the inference STACK (dynamo + KAI scheduler + DRA driver) is
-      # still stood up and validated by prep/install/conformance/verify; only the
-      # served-workload run is skipped (Serve summary row is a static `disabled`).
+      # Live as of the GCP served completion this lane was gated on (run
+      # 32825515255, #1644): phase_serve selects the GPU pool for both graph
+      # components, so the Frontend no longer cold-pulls the ~12GB
+      # vllm-runtime image on a small CPU-pool node.
       #
-      # - name: UAT - serve (DynamoGraphDeployment + endpoint)
-      #   id: serve
-      #   if: steps.conformance.outcome == 'success' && inputs.intent == 'inference'
-      #   # Larger budget than train: a cold pull of the multi-GB vllm-runtime
-      #   # image + model download + engine warmup precedes the first served
-      #   # token (phase_serve's SERVE_READY_TIMEOUT_SECONDS is 30m).
-      #   timeout-minutes: 40
-      #   shell: bash
-      #   env:
-      #     AICR_BIN: ${{ github.workspace }}/aicr
-      #     RUN_ID: ${{ github.run_id }}
-      #   run: ./tests/uat/azure/run serve "${TEST_CONFIG}"
+      # AKS is also the non-GKE check on the worker's driver-library append.
+      # phase_serve wraps the vLLM command with an LD_LIBRARY_PATH append for
+      # /usr/local/nvidia/lib64, which GKE needs and the GPU Operator's
+      # container toolkit makes unnecessary here. That path is absent on AKS,
+      # so the append is expected to be an inert no-op — this lane is what
+      # proves the shared script did not regress a working platform.
+      - name: UAT - serve (DynamoGraphDeployment + endpoint)
+        id: serve
+        if: steps.conformance.outcome == 'success' && inputs.intent == 'inference'
+        # Larger budget than train: a cold pull of the multi-GB vllm-runtime
+        # image + model download + engine warmup precedes the first served
+        # token (phase_serve's SERVE_READY_TIMEOUT_SECONDS is 30m).
+        timeout-minutes: 40
+        shell: bash
+        env:
+          AICR_BIN: ${{ github.workspace }}/aicr
+          RUN_ID: ${{ github.run_id }}
+        run: ./tests/uat/azure/run serve "${TEST_CONFIG}"
 
       - name: UAT - verify (evidence verify)
         id: verify
@@ -936,7 +936,7 @@ jobs:
             echo "| Install | ${{ steps.install.outcome }} |"
             echo "| Validate (all phases) | ${{ steps.conformance.outcome }} |"
             echo "| Train | ${{ steps.train.outcome }} |"
-            echo "| Serve | disabled (vLLM excluded from UAT) |"
+            echo "| Serve | ${{ steps.serve.outcome }} |"
             echo "| Verify | ${{ steps.verify.outcome }} |"
             if [[ -n "${{ steps.evidence_ref.outputs.ref }}" ]]; then
               echo ""

.github/workflows/uat-gcp.yaml

@@ -698,10 +698,11 @@ jobs:
       # Frontend cold-pulled the ~12GB vllm-runtime image on a CPU-pool node and
       # wedged in ContainerCreating past the readiness budget; phase_serve now
       # selects the GPU pool for both graph components, whose a3-megagpu-8g class
-      # pulls that image in a fraction of the budget. The first GCP inference
-      # nightly (or manual `run serve`) after this is the proving run — serving
-      # was never verified end-to-end. AWS/Azure serve steps stay commented until
-      # this lane demonstrates a served completion.
+      # pulls that image in a fraction of the budget. Proven end-to-end by run
+      # 32825515255, which also needed the worker's LD_LIBRARY_PATH append (GKE
+      # mounts the driver at /usr/local/nvidia without putting it on the search
+      # path) — that run unblocked uat-azure.yaml's serve step. AWS stays
+      # commented while its lane fails in Bringup Infra, before any CUJ.
       - name: UAT - serve (DynamoGraphDeployment + endpoint)
         id: serve
         if: steps.conformance.outcome == 'success' && inputs.intent == 'inference'