Fix duplicate containerStatuses in Kubernetes app diagnose error output

[turboFei]

Why are the changes needed?

When a Spark-on-Kubernetes application fails, getApplicationStateAndErrorFromPod builds a diagnostic JSON map containing both PodStatus (the full Kubernetes PodStatus object) and ContainerStatus (the specific driver container's status extracted from pod.getStatus.getContainerStatuses).

The problem is that PodStatus already embeds containerStatuses[] internally. For a single-container pod (the typical Spark driver), PodStatus.containerStatuses[0] and the top-level ContainerStatus are identical, producing duplicated data in the error output.

Before — same container error appears twice:

{
  "PodStatus": {
    "conditions": [...],
    "containerStatuses": [{ "name": "spark-kubernetes-driver", "state": { "terminated": { "exitCode": 1, "message": "..." } } }]
  },
  "ContainerStatus":    { "name": "spark-kubernetes-driver", "state": { "terminated": { "exitCode": 1, "message": "..." } } }
}

After — clean separation of pod-level and container-level info:

{
  "PodStatus": { "conditions": [...], "phase": "Failed", "hostIP": "...", "podIP": "..." },
  "ContainerStatus": { "name": "spark-kubernetes-driver", "state": { "terminated": { "exitCode": 1, "message": "..." } } }
}

How was this patch tested?

Manual verification with a failing Spark-on-Kubernetes job. The diagnostic JSON no longer contains duplicate container status data. PodStatus retains pod-level fields (conditions, phase, hostIP, podIP, qosClass) and ContainerStatus retains the full driver container state with exit code and error message.

The POD-mode fallback path (no ContainerStatus sibling) is unaffected — it continues to emit the full PodStatus including containerStatuses.

Related: https://github.corp.ebay.com/hadoop/kyuubi/pull/868

Was this patch authored or co-authored using generative AI tooling?

Assisted-by: Claude:claude-sonnet-4-6