reservation lost mid run

[HDCharles]

Haven't seen this one before:

running a distributed process on 2 gpus and got:

ERROR: Failed to send heartbeat: GPU 4 reservation lost: expected user=HDCharles, type=run but found user=, type=
GPU reservations may be at risk of expiring!

on one of the gpus repeatedly every minute or so and it looks like it shows up as unreserved

 GPU │ STATUS       │ USER                  │ DURATION │ TYPE │ DETAILS                     │ VALIDATION │ MODEL              │ NOTE 
─────┼──────────────┼───────────────────────┼──────────┼──────┼─────────────────────────────┼────────────┼────────────────────┼──────
...
 2   │ ● AVAILABLE  │ -                     │ -        │ -    │ never used                  │ 0MB used   │ -                  │ -    
 3   │ ● AVAILABLE  │ -                     │ -        │ -    │ never used                  │ 0MB used   │ -                  │ -    
 4   │ ⚠ UNRESERVED │ HDCharles             │ -        │ -    │ 27803MB used by 1 process   │ -          │ -                  │ -    
 5   │ ⚠ UNRESERVED │ HDCharles             │ -        │ -    │ 27035MB used by 1 process   │ -          │ -                  │ -    
 6   │ ● AVAILABLE  │ -                     │ -        │ -    │ never used                  │ 0MB used   │ -                  │ -    
 7   │ ● AVAILABLE  │ -                     │ -        │ -    │ never used 

the script was run with chg run --gpus 2 --

[russellb]

@HDCharles can you confirm which version you were using? Some relevant code changed significantly in 1.0.

[russellb]

I pushed a change to make the supervisor process a bit more resilient to Redis connection issues, which could possibly fix the situation. There's no way to really know based on these logs. Possibilities include:

1. redis-server died for some reason.
2. the canhazgpu connection to redis broke. I would have expected some more logs if this were the case though, but this could still be it.
3. The server was SO overloaded, that the process got starved for several minutes? seems unlikely ...

[HDCharles]

yeah the process was still continuing without issue and there were open GPUs

it was on canhazgpu version 1.0.2

[russellb]

have you seen it happen more than once?

[HDCharles]

No, just the once