do not add walltime over resubmits for maxtime check

[belforte]

In the way that we user resubmissions and retries now, it makes no sense to add time spend in previous submissions in order to check for job having exceeded MAX_WALLTIME.
In particular we want to automatically increase walltime request in subsequent resubmissions.

This part must be reviewed

CRABServer/src/python/TaskWorker/Actions/RetryJob.py

Lines 375 to 390 in 872622c

	total_job_time = self.report['steps']['cmsRun']['performance']['cpu']['TotalJobTime']
	try:
	total_job_time = float(total_job_time)
	except ValueError:
	return
	integrated_job_time = 0
	for ad in self.ads:
	if 'RemoteWallClockTime' in ad:
	integrated_job_time += ad['RemoteWallClockTime']
	self.integrated_job_time = integrated_job_time
	if total_job_time > self.MAX_WALLTIME:
	exitMsg = "Not retrying a long running job (job ran for %d hours)" % (total_job_time / 3600)
	self.create_fake_fjr(exitMsg, 50664) # this raises FatalError
	if integrated_job_time > (1.5 * self.MAX_WALLTIME):
	exitMsg = "Not retrying a job because the integrated time (across all retries) is %d hours." % (integrated_job_time / 3600)
	self.create_fake_fjr(exitMsg, 50664) # this raises FatalError

• remove integrated_job_time
• check is it is worth to look at Committed* ad
• improve logging so that PostJob log shows actual usage and limit

[belforte]

current PostJonb log is like this

Wed, 01 Jul 2026 14:55:04 CEST(+0200):INFO:PostJob ====== Starting to analyze job exit status.
Wed, 01 Jul 2026 14:55:04 CEST(+0200):INFO:PostJob        -----> RetryJob log start -----
Wed, 01 Jul 2026 14:55:04 CEST(+0200):DEBUG:RetryJob Job ads already present. Will not use condor_q, but will load previous jobs ads.
Wed, 01 Jul 2026 14:55:04 CEST(+0200):INFO:RetryJob New job_fjr.2180.2.json file content: {"exitMsg": "Not retrying job due to wall clock limit (job automatically killed on the worker node)", "jobExitCode": 50664, "exitCode": 50664}
Wed, 01 Jul 2026 14:55:04 CEST(+0200):ERROR:RetryJob Original error: Job did not produce a usable framework job report.
Wed, 01 Jul 2026 14:55:04 CEST(+0200):ERROR:RetryJob Not retrying job due to wall clock limit (job automatically killed on the worker node)
Wed, 01 Jul 2026 14:55:04 CEST(+0200):INFO:PostJob        <----- RetryJob log finish ----
Wed, 01 Jul 2026 14:55:04 CEST(+0200):INFO:PostJob The retry handler indicated this was a fatal error.

[belforte]

can compute wall clock time using the ads JobCurrentStartDate and EnteredCurrentStatus (the latter is the time when job was removed)
Note: cannot use CommittedTime since when time limit is exceeded, job is evicted, so CommittedTime is reset to 0

Here's an example or relevant classAds from a job which hit the time limit from https://cmsweb.cern.ch/scheddmon/0121/nladams/260619_134211:nladams_crab_ScoutingPFRun3F/postjob.2180.2.txt

Ad	Value
RemoteWallClockTime	217638.0	seconds = 3627 min = 60.4 h
LastRemoteWallClockTime	144053.0	seconds = 2400.88 min = 40.01 h
MaxWallTimeMinsRun	2400	what is used in PeriodicRemove
LastMatchTime	1782766448
JobCurrentStartDate	1782766448
JobCurrentStartExecutingDate	1782766453
EnteredCurrentStatus	1782910501	delta from JobCurrStartDate = 2400.88 min
CommittedTime	0
NumJobStarts	2
NumShadowStarts	2

Note: RemoteWallCLockTime includes previous attempts, indeed we not even have 60h pilots !
This means that this line is also wrong https://github.com/dmwm/cms-htcondor-es/blob/5fe8acbeaa03c73a8321a6570423857a870241d0/src/htcondor_es/convert_to_json.py#L698

Note also the description of RemoteWallClockTime vs. LastRemoveWallClockTime in https://htcondor.readthedocs.io/en/latest/classad-attributes/job-classad-attributes.html#RemoteWallClockTime and
https://htcondor.readthedocs.io/en/latest/classad-attributes/job-classad-attributes.html#LastRemoteWallClockTime

[belforte]

Note that the job in the example above had two starts, the first time ran for about 20h, then was killed for unknown reasons and restarted by the schedd) is summed to the second start (which lasted 40h and was ended by periodic remove).