WMAgent: Unhandled exceptions at JobStatusLite component

[todor-ivanov]

Impact of the bug
WMAgent

Describe the bug
While monitoring the agents during the regular WMCore dev on shift I found few of them with both components down:

• JobStatusLite
• JobUpdater

With the following errors in their logs [1], [2]. both of them stemming from BosAir and HtcondorPlugin interaction. And in the case of the former (JobStatusLite) the component was completely down. Its threads terminated, which should not happen.

How to reproduce it
So far the hypothesis is it is a combination of events related to priority updates and high granularity workflows in the system, which may result in increased pressure on Condor.

Expected behavior
To not crash the component's thread.

Additional context and error message

[1]

2025-04-07 20:40:17,176:139777553495808:INFO:BossAirAPI:About to look for 1010 loadedJobs.
2025-04-07 20:40:17,190:139777553495808:ERROR:BossAirAPI:Unhandled exception while tracking jobs for plugin SimpleCondorPlugin!
Unable to locate local daemon
2025-04-07 20:40:17,236:139777553495808:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMCore.BossAir.StatusPoller.StatusPoller object at 0x7f2085062dc0> <@========== WMException Start ==========@>
Exception Class: BossAirException
Message: Unhandled exception while tracking jobs for plugin SimpleCondorPlugin!
Unable to locate local daemon
        ClassName : None
        ModuleName : WMCore.BossAir.BossAirAPI
        MethodName : track
        ClassInstance : None
        FileName : /usr/local/lib/python3.8/site-packages/WMCore/BossAir/BossAirAPI.py
        LineNumber : 486
        ErrorNr : 0

Traceback: 
  File "/usr/local/lib/python3.8/site-packages/WMCore/BossAir/BossAirAPI.py", line 473, in track
    localRunning, localChanges, localCompletes = pluginInst.track(jobs=jobsToTrack[plugin])

  File "/usr/local/lib/python3.8/site-packages/WMCore/BossAir/Plugins/SimpleCondorPlugin.py", line 208, in track
    schedd = htcondor.Schedd()

  File "/usr/local/lib/python3.8/site-packages/htcondor/_lock.py", line 70, in wrapper
    rv = func(*args, **kwargs)

<@---------- WMException End ----------@>  File "/usr/local/lib/python3.8/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 183, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/usr/local/lib/python3.8/site-packages/WMCore/Database/DBExceptionHandler.py", line 41, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/Utils/Timers.py", line 57, in wrapper
    res = func(*arg, **kw)
  File "/usr/local/lib/python3.8/site-packages/WMCore/BossAir/StatusPoller.py", line 70, in algorithm
    self.checkStatus()
  File "/usr/local/lib/python3.8/site-packages/WMCore/BossAir/StatusPoller.py", line 94, in checkStatus
    runningJobs = self.bossAir.track()
  File "/usr/local/lib/python3.8/site-packages/WMCore/BossAir/BossAirAPI.py", line 486, in track
    raise BossAirException(msg)

2025-04-07 20:40:17,237:139777553495808:INFO:Harness:>>>Terminating worker threads
2025-04-07 20:40:17,237:139777553495808:ERROR:BaseWorkerThread:Error in event loop (2): <WMCore.BossAir.StatusPoller.StatusPoller object at 0x7f2085062dc0> <@========== WMException Start ==========@>
Exception Class: BossAirException
Message: Unhandled exception while tracking jobs for plugin SimpleCondorPlugin!
Unable to locate local daemon
        ClassName : None
        ModuleName : WMCore.BossAir.BossAirAPI
        MethodName : track
        ClassInstance : None
        FileName : /usr/local/lib/python3.8/site-packages/WMCore/BossAir/BossAirAPI.py
        LineNumber : 486
        ErrorNr : 0

Traceback: 
  File "/usr/local/lib/python3.8/site-packages/WMCore/BossAir/BossAirAPI.py", line 473, in track
    localRunning, localChanges, localCompletes = pluginInst.track(jobs=jobsToTrack[plugin])

  File "/usr/local/lib/python3.8/site-packages/WMCore/BossAir/Plugins/SimpleCondorPlugin.py", line 208, in track
    schedd = htcondor.Schedd()

  File "/usr/local/lib/python3.8/site-packages/htcondor/_lock.py", line 70, in wrapper
    rv = func(*args, **kwargs)

<@---------- WMException End ----------@>
Backtrace:
  File "/usr/local/lib/python3.8/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 209, in __call__
    raise ex
  File "/usr/local/lib/python3.8/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 183, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/usr/local/lib/python3.8/site-packages/WMCore/Database/DBExceptionHandler.py", line 41, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/Utils/Timers.py", line 57, in wrapper
    res = func(*arg, **kw)
  File "/usr/local/lib/python3.8/site-packages/WMCore/BossAir/StatusPoller.py", line 70, in algorithm
    self.checkStatus()
  File "/usr/local/lib/python3.8/site-packages/WMCore/BossAir/StatusPoller.py", line 94, in checkStatus
    runningJobs = self.bossAir.track()
  File "/usr/local/lib/python3.8/site-packages/WMCore/BossAir/BossAirAPI.py", line 486, in track
    raise BossAirException(msg)

2025-04-07 20:40:17,616:139777553495808:INFO:BaseWorkerThread:Worker thread <WMCore.BossAir.StatusPoller.StatusPoller object at 0x7f2085062dc0> terminated

[2]

2025-04-05 10:37:34,153:140153345967872:INFO:JobUpdaterPoller:Updating condor jobs priority for request: cmsunified_task_BPH-RunIII2024Summer24GS-00045__v1_T_250320_102403_440
2025-04-05 10:37:35,040:140153345967872:ERROR:SimpleCondorPlugin:Failed to update JobPrio for WMAgent_RequestName=cmsunified_task_BPH-RunIII2024Summer24GS-00045__v1_T_250320_102403_440
2025-04-05 10:37:35,041:140153345967872:ERROR:SimpleCondorPlugin:Unable to edit jobs matching constraint
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/WMCore/BossAir/Plugins/SimpleCondorPlugin.py", line 486, in updateJobInformation
    schedd.edit(constraint, 'JobPrio', classad.Literal(newPriority))
  File "/usr/local/lib/python3.8/site-packages/htcondor/_lock.py", line 70, in wrapper
    rv = func(*args, **kwargs)
htcondor.HTCondorIOError: Unable to edit jobs matching constraint

[mapellidario]

I saw the same problem for JobSubmitter. My gut feeling is that when condor has a looot of things to do, it simply refuses new connections from client applications. that exception kind of correlates with the end of a long period of high "resource requests sent" [1] for all the schedds.

i think that we should simply re-try the connection to local condor (and more in general to rucio and dbs and other external services).

In any case, I think and hope that #12172 should cover this issue as well

---

[1] https://monit-grafana.cern.ch/goto/OmrYw0ANR?orgId=11

[amaltaro]

As discussed in the meeting today, this issue is a duplicate of #9703 and we can close it out as a dup. Doing so.