[Question][GitLab] GitLab data collection is incomplete

[yu501954162]

Why does using normal Collect Data result in missing some commit data, but using Collect Data in Full Refresh Mode collects everything? How can I configure the scheduled task to always use Full Refresh Mode for each collection?

Version

v1.0.3-beta9@1410838

[yu501954162]

@dosubot

[dosubot]

This is a great question — here's what's happening and what you can do about it.

Why normal collection misses commits: In incremental mode, the gitextractor plugin performs a shallow git clone (git fetch --shallow-since=<timeAfter>) instead of a full clone [1]. Commits whose parent commits fall outside the timeAfter window are intentionally skipped to avoid writing incorrect diff statistics [2]. This means boundary commits get dropped from the repo_commits and commit_parents tables entirely. This behavior was reported in Issue #8434 [3].

Why Full Refresh Mode works: When FullSync = true, DevLake performs a complete git clone --bare with no shallow clone or time boundary, so all commits and their parents are available [4].

Configuring scheduled tasks for Full Refresh: Unfortunately, there is currently no built-in way to make a cron-scheduled blueprint always run in Full Refresh Mode. The FullSync flag is a trigger-time override only, not a persistent blueprint setting [5]. Your options are:

1. Trigger via API each time with Full Refresh:

   POST /api/blueprints/{blueprintId}/trigger
   {"skipCollectors": false, "fullSync": true}
   

   You could automate this with an external cron job calling this API endpoint [6].

2. Extend timeAfter generously — set it far enough in the past that parent commits of all relevant commits fall within the window. The maintainer suggested "increasing the time range" as a workaround [3].

3. Periodically update timeAfter via the UI to keep the collection window appropriate for your needs [7].

Note: Be aware that timeAfter is stored as a fixed timestamp from when the blueprint was configured — it does not automatically slide forward on cron runs [7]. Also, avoid setting timeAfter to null via the PATCH API, as there's an open bug (#8516) where it gets stored as 0001-01-01 instead [8].

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}