Skip to content

fix: set end_time on finished job #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
jrsitabi wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

fix: set end_time on finished job #56

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,6 @@ Lists spiders from the spider image's `org.scrapy.spiders` label.
### `listjobs.json` ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#listjobs-json))

Lists current jobs by looking at Docker containers or Kubernetes jobs.
Note that `end_time` is not yet supported for Docker.

### ~~`delversion.json`~~ ([➽](https://scrapyd.readthedocs.io/en/latest/api.html#delversion-json))

Expand Down
2 changes: 1 addition & 1 deletion scrapyd_k8s/launcher/docker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ def _parse_job(self, c):
'project': c.labels.get(self.LABEL_PROJECT),
'spider': c.labels.get(self.LABEL_SPIDER),
'start_time': format_iso_date_string(c.attrs['State']['StartedAt']) if state in ['running', 'finished'] else None,
'end_time': None, # Not available using Docker's API. Add to the job representation to keep it the same as K8s jobs listing.
'end_time': format_iso_date_string(c.attrs['State']['FinishedAt']) if state in ['finished'] else None,
}

def _get_container(self, project_id, job_id):
Expand Down
4 changes: 2 additions & 2 deletions scrapyd_k8s/tests/integration/test_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ def test_scenario_cancel_running_finished_ok():
jobinfo = assert_listjobs(finished=jobid)
start_time, end_time = jobinfo.pop('start_time'), jobinfo.pop('end_time')
assert datetime.strptime(start_time, '%Y-%m-%d %H:%M:%S.%f')
assert end_time is None
assert datetime.strptime(end_time, '%Y-%m-%d %H:%M:%S.%f')

@wvengen wvengen Feb 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the job cancel scenario: it seems here that for Kubernetes, the job end time is not available when the job is canceled. For Docker, it seems that it is available. So this is a behaviour difference between the two, and that is not desired.

Can you look at scrapyd to see what it emits? We want to mirror that behaviour (as far as possible).

assert jobinfo == { 'id': jobid, 'project': RUN_PROJECT, 'spider': RUN_SPIDER, 'state': 'finished' }
# then cancel it again, though nothing would happen
response = requests.post(BASE_URL + '/cancel.json', data={ 'project': RUN_PROJECT, 'job': jobid })
Expand Down Expand Up @@ -191,7 +191,7 @@ def scenario_regular(schedule_args):
jobinfo = assert_listjobs(finished=jobid)
start_time, end_time = jobinfo.pop('start_time'), jobinfo.pop('end_time')
assert datetime.strptime(start_time, '%Y-%m-%d %H:%M:%S.%f')
assert datetime.strptime(end_time, '%Y-%m-%d %H:%M:%S.%f') if WITH_K8S else end_time is None
assert datetime.strptime(end_time, '%Y-%m-%d %H:%M:%S.%f')
assert jobinfo == { 'id': jobid, 'project': RUN_PROJECT, 'spider': RUN_SPIDER, 'state': 'finished' }

def assert_response_ok(response):
Expand Down
Loading