WorkflowUpdater failing to update pileupconf.json

[hassan11196]

Impact of the bug
WMAgent (WorkflowUpdater), WMA Jobs submitted without proper pileupconf.json

Describe the bug

2025-08-25 10:07:46,535:140682525681344:ERROR:WorkflowUpdaterPoller:Caught unexpected exception in WorkflowUpdater. Details:
'logical_file_name'
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py", line 310, in algorithm
    self.adjustJSONSpec(puWflows, msPileupList)
  File "/usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py", line 396, in adjustJSONSpec
    puNewJsonContent = updateBlockInfo(puJsonContent, msPUBlockLoc, self.dbsUrl, self.logger)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py", line 185, in updateBlockInfo
    binfo = getBlockInfo4PU(blocksToUpdate, dbsUrl, ckey(), cert())
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/WMCore/Services/DBS/DBSConcurrency.py", line 41, in getBlockInfo4PU
    files = [r['logical_file_name'] for r in data]
             ~^^^^^^^^^^^^^^^^^^^^^
KeyError: 'logical_file_name'
2025-08-25 10:07:46,567:140682525681344:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMComponent.WorkflowUpdater.WorkflowUpdaterPoller.WorkflowUpdaterPoller object at 0x7ff337b67a40> <@========== WMException Start ==========@>
Exception Class: WorkflowUpdaterException
Message: Caught unexpected exception in WorkflowUpdater. Details:
'logical_file_name'
        ClassName : None
        ModuleName : WMComponent.WorkflowUpdater.WorkflowUpdaterPoller
        MethodName : algorithm
        ClassInstance : None
        FileName : /usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py
        LineNumber : 314
        ErrorNr : 0

Traceback: 
  File "/usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py", line 310, in algorithm
    self.adjustJSONSpec(puWflows, msPileupList)

  File "/usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py", line 396, in adjustJSONSpec
    puNewJsonContent = updateBlockInfo(puJsonContent, msPUBlockLoc, self.dbsUrl, self.logger)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py", line 185, in updateBlockInfo
    binfo = getBlockInfo4PU(blocksToUpdate, dbsUrl, ckey(), cert())
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/site-packages/WMCore/Services/DBS/DBSConcurrency.py", line 41, in getBlockInfo4PU
    files = [r['logical_file_name'] for r in data]
             ~^^^^^^^^^^^^^^^^^^^^^

<@---------- WMException End ----------@>  File "/usr/local/lib/python3.12/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 183, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/WMCore/Database/DBExceptionHandler.py", line 43, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/Utils/Timers.py", line 57, in wrapper
    res = func(*arg, **kw)
          ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py", line 314, in algorithm
    raise WorkflowUpdaterException(msg) from None

2025-08-25 10:07:46,567:140682525681344:INFO:Harness:>>>Terminating worker threads
2025-08-25 10:07:46,567:140682525681344:ERROR:BaseWorkerThread:Error in event loop (2): <WMComponent.WorkflowUpdater.WorkflowUpdaterPoller.WorkflowUpdaterPoller object at 0x7ff337b67a40> <@========== WMException Start ==========@>
Exception Class: WorkflowUpdaterException
Message: Caught unexpected exception in WorkflowUpdater. Details:
'logical_file_name'
        ClassName : None
        ModuleName : WMComponent.WorkflowUpdater.WorkflowUpdaterPoller
        MethodName : algorithm
        ClassInstance : None
        FileName : /usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py
        LineNumber : 314
        ErrorNr : 0

Traceback: 
  File "/usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py", line 310, in algorithm
    self.adjustJSONSpec(puWflows, msPileupList)

  File "/usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py", line 396, in adjustJSONSpec
    puNewJsonContent = updateBlockInfo(puJsonContent, msPUBlockLoc, self.dbsUrl, self.logger)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py", line 185, in updateBlockInfo
    binfo = getBlockInfo4PU(blocksToUpdate, dbsUrl, ckey(), cert())
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/site-packages/WMCore/Services/DBS/DBSConcurrency.py", line 41, in getBlockInfo4PU
    files = [r['logical_file_name'] for r in data]
             ~^^^^^^^^^^^^^^^^^^^^^

<@---------- WMException End ----------@>
Backtrace:
  File "/usr/local/lib/python3.12/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 209, in __call__
    raise ex
  File "/usr/local/lib/python3.12/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 183, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/WMCore/Database/DBExceptionHandler.py", line 43, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/Utils/Timers.py", line 57, in wrapper
    res = func(*arg, **kw)
          ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/WMComponent/WorkflowUpdater/WorkflowUpdaterPoller.py", line 314, in algorithm
    raise WorkflowUpdaterException(msg) from None

2025-08-25 10:07:46,909:140682525681344:INFO:BaseWorkerThread:Worker thread <WMComponent.WorkflowUpdater.WorkflowUpdaterPoller.WorkflowUpdaterPoller object at 0x7ff337b67a40> terminated
^[[A

WorkflowUpdater is failing to update pileupconf.json due to keyError caused by data fetched from DBS,

  File "/usr/local/lib/python3.12/site-packages/WMCore/Services/DBS/DBSConcurrency.py", line 41, in getBlockInfo4PU
    files = [r['logical_file_name'] for r in data]
             ~^^^^^^^^^^^^^^^^^^^^^
KeyError: 'logical_file_name'

I would expect each file to have a logical_file_name but it seems like that not the cause.

How to reproduce it
Steps to reproduce the behavior:
Inject a workflow reading the pileup /Neutrino_E-10_gun/Run3Summer21PrePremix-Summer23_130X_mcRun3_2023_realistic_v13-v1/PREMIX and the WorkflowUpdater will not be able to create the pileupconf.json

Expected behavior
pileupconf.json to be created properly and WorkflowUpdater to not crash becuase of it.

Additional context and error message
I will try to find the DBS file or block responsible and see if its a DBS issue.

FYI @anpicci @amaltaro

[vkuznet]

@hassan11196 I discussed this very issue in WM Mattermost and outlined all details over here: https://gist.github.com/vkuznet/f42a31fcb3d3e349f6abcedf3db076ad I provided DBS block and shown full DBS error. There may be plenty of potential explanations from misbehaving DBS clusters, and or DBS pods, but I lean to conclude that the issue is dimensional and related to the nature of components, our apache FE and run-time conditions overlapping with polling cycles. I'll discuss it in details in our extended and dev WM meetings this week.

[khurtado]

The patched agent is vocms0282

There was a bug in the original PR which I just fixed.
@hassan11196 modified #12431 to add extra debugging, but it sets the url back to use -prod

https://github.com/dmwm/WMCore/pull/12433/files#diff-0f2f04c342289791ed9a6b465578ccdf229ad9e28bdb98a93651e599ae5b0c1cR92

My latest version of the PR
#12431

should get them back to cmsweb, plus modifies Emulator related code (so jenkins doesn't complain), but it is not atm incorporating hassan's extra lines for debugging

[vlimant]

is that #12479 and #12481 ?