Hypervisor JWT authentication failure

[EdwardZhang88]

Followed the on-premise install instructions and got stuck with the errors below.

1. worker v1.68.1 pod log
   ERROR cuda_limiter::config: HTTP request failed with status: 401 Unauthorized
   ERROR cuda_limiter: failed to get device configs: HTTP request failed

2. hypervisor v1.42.3 pod lod
   request{remote_addr=xx method=GET uri=/api/v1/pod?container_pid=26&&container_name=python}:hypervisor::controller::auth: JWT authentication failed: Invalid JWT token: Failed to parse payload: missing field 'kubernetes.io' at line 1 column 369

Tried with several different versions of worker images but to no avail. Please help!!!

[EdwardZhang88]

Also tried setting SA_TOKEN_VERIFY_ENABLED=false in hypervisor pod env but no luck.
@Code2Life Appreciate if you could shed some light.

[Mishaoyue]

@0x5457 hi carl Could you please take a look at this problem when you have time?

[0x5457]

Hi @EdwardZhang88 You could try this combination of versions; if you still encounter issues, I’ll check again.

operator:
  image: tensorfusion/tensor-fusion-operator
  version: 1.59.5
worker:
  image: tensorfusion/tensor-fusion-worker
  version: 1.71.0
nodeDiscovery:
  image: tensorfusion/tensor-fusion-node-discovery
  version: 1.59.5
hypervisor:
  image: tensorfusion/tensor-fusion-hypervisor
  version: 1.51.9
client:
  embeddedModeImage: tensorfusion/tensor-fusion-ngpu
  embeddedModeVersion: 1.8.39

[EdwardZhang88]

@0x5457 Thanks for your reply. However, the same problem still persists after upgrade. Could you further clarify on the questions below?

1. Does hypervisor follow the same code as operator in dealing with JWT authentication?
2. Is k8s 1.18.15 supported?

[0x5457]

@EdwardZhang88 Thanks, that explains it.

This is caused by BoundServiceAccountTokenVolume. Kubernetes 1.18.15 does not use that token format by default, while the current hypervisor expects a bound/projected ServiceAccount token with the kubernetes.io claim. That is why the request is rejected with 401.

[EdwardZhang88]

@0x5457 Thanks for your reply.
After turning on BoundServiceAccountTokenVolume in current k8s 1.18 cluster, worker is now able to call hypervisor. But when I run an actual pytorch test，worker crashes with the following error.

[TF Code] Error Func: TraceKernelParamsInfoOffsetTest Line: 185 failed to load test module: 209
[TF Code] Error Func: operator() Line: 253 TraceKernelParamsInfoOffsetTest failed
[TF] Panic Func: RegisterCUFunction Line:99 failed to retrieve kernel params info

strings -a tensor-fusion-worker | grep -E "sm_[0-9]+" returns sm_100 and sm_120. Does it mean that Tesla series GPU（V100）are not supported?

Also, there are perioidc errors in both worker and hypervisor logs. Not sure if this is relevant to the error above.

worker: ERROR cuda_limiter::detour:gpu failed to initialize worker via cu_init_detour：Invalid response from hypervisor, will try on next call
hypervisor: WARN request{remote_addr=xx method=POST uri=/api/v1/process?container_pid=1&&container_name=tensor-fusion-worker}:hypervisor::controller::handlers: Process initialization timeout timeout_sec=5s

[andystime]

@0x5457 Thanks for your reply. After turning on BoundServiceAccountTokenVolume in current k8s 1.18 cluster, worker is now able to call hypervisor. But when I run an actual pytorch test，worker crashes with the following error.

[TF Code] Error Func: TraceKernelParamsInfoOffsetTest Line: 185 failed to load test module: 209 [TF Code] Error Func: operator() Line: 253 TraceKernelParamsInfoOffsetTest failed [TF] Panic Func: RegisterCUFunction Line:99 failed to retrieve kernel params info

strings -a tensor-fusion-worker | grep -E "sm_[0-9]+" returns sm_100 and sm_120. Does it mean that Tesla series GPU（V100）are not supported?

Also, there are perioidc errors in both worker and hypervisor logs. Not sure if this is relevant to the error above.

worker: ERROR cuda_limiter::detour:gpu failed to initialize worker via cu_init_detour：Invalid response from hypervisor, will try on next call hypervisor: WARN request{remote_addr=xx method=POST uri=/api/v1/process?container_pid=1&&container_name=tensor-fusion-worker}:hypervisor::controller::handlers: Process initialization timeout timeout_sec=5s

"TraceKernelParamsInfoOffsetTest failed [TF] Panic Func: RegisterCUFunction Line:99 failed to retrieve kernel params info" was called by old version of tensor-fusion-worker, the latest version has solved this problem and truly support V100.