[Bug][Github GraphQL] Deleted or transferred issues leave stale local rows after #8637

[squatboy]

Search before asking

• I had searched in the issues and found the related transferred-issue fix [Bug][Github] GraphQL collector fails on transferred issues  #7969 / PR fix: skip transferred issue for Github source #8637.

What happened

GitHub GraphQL issue refresh no longer fails immediately on Could not resolve to an Issue after #8637, but deleted or transferred issues can still leave stale local rows in DevLake.

Those stale rows remain eligible for future refresh attempts because the local _tool_github_issues and raw rows are not cleaned up when the source issue no longer resolves.

In the reproduced case, a previously collected GitHub issue was deleted from the source repository. DevLake had already collected it into local tables, and later refresh attempts still tried to resolve it through GitHub GraphQL.

What do you expect to happen

When GitHub GraphQL returns Could not resolve to an Issue for an issue in the refresh-open-issues path, DevLake should:

1. continue processing the rest of the batch
2. remove or tombstone the stale local issue rows that keep the missing issue in the refresh path
3. avoid retrying the same orphaned issue forever

How to reproduce

1. Create a GitHub issue in repository A
2. Run a DevLake pipeline that collects GitHub issues for repository A
3. Delete the GitHub issue from repository A, or transfer it to another repository
4. Re-run the DevLake pipeline
5. Observe that DevLake still has stale local rows for the old issue and continues to encounter the missing issue in the refresh path

Anything else

Related upstream fix:

• [Bug][Github] GraphQL collector fails on transferred issues  #7969
• fix: skip transferred issue for Github source #8637

The remaining gap seems to be stale local data cleanup rather than just skipping the GraphQL error.

Version

Reproduced on v1.0.3-beta9@1410838

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hi @squatboy! I'm Dosu and I’m helping the incubator-devlake team.

This is a valid gap. After digging into the code, here's what's happening and a couple of additional things worth noting for whoever picks up the fix:

Current behavior after PR #8637: The fetchAsync() in graphql_collector.go now catches "Could not resolve to an Issue" errors and logs a warning instead of crashing the task — but the stale row in _tool_github_issues is never cleaned up. The Phase 2 refresh path queries all local issues with state='OPEN' [1], so the orphaned issue keeps appearing in every future refresh cycle indefinitely.

The _tool_github_issues model has no tombstone or soft-delete fields — no deleted_at, is_stale, etc. [2], so a fix would need to either hard-delete the row, update State to CLOSED (or a new sentinel value), or add a new field.

Batch-collateral issue: There's a subtle additional problem — after the dataErrors loop in fetchAsync(), there's an early return [3]. This means if one issue in a batch of 100 triggers "Could not resolve", the ResponseParser is skipped for the entire batch, so valid issues in the same batch also miss their refresh update for that cycle.

A fix would likely need to address at least two things:

1. Stale row cleanup — when the error is detected, delete or mark the corresponding _tool_github_issues row so it's excluded from future Phase 2 refresh queries.
2. Batch handling — avoid the early return so that valid issues in the same batch still get processed by ResponseParser. One approach could be to set IgnoreQueryErrors: true (like account_collector.go and job_collector.go already do) and handle the error inside the ResponseParser instead, or to filter out the errored entries and continue processing the rest of the batch.

Since you indicated you're willing to submit a PR — the key files to look at are backend/plugins/github_graphql/tasks/issue_collector.go (Phase 2 refresh logic) and backend/helpers/pluginhelper/api/graphql_collector.go (the fetchAsync error handling).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}