ci: add sync-params workflow

[paulsohn]

Description

Introduce sync-params workflow that updates configurated *.param.yaml with custom overrides.

As described in sync-params.yaml in this PR, the configuration goes like this.

perception: # category
  - repository: autowarefoundation/autoware_universe
    ref: main
    files:
      - # Source param path in source repository (i.e. in autoware_universe).
        source: perception/autoware_image_object_locator/config/bbox_object_locator.param.yaml
        # List of param paths in current repository that are variants of the source param path.
        # Variant files should have the same yaml file layout as the source file and only differ in the parameters.
        variants:
          - path: autoware_launch/config/perception/object_recognition/detection/camera_vru_detection/near_range_camera_vru_detector.param.yaml

In this case, the python run with category python3 ./.github/scripts/sync_params.py perception clones the upstream repository to a temp directory, copies source (autoware_universe)/perception/autoware_image_object_locator/config/bbox_object_locator.param.yaml to each variant path - in this case only (autoware_launch)/autoware_launch/config/perception/object_recognition/detection/camera_vru_detection/near_range_camera_vru_detector.param.yaml with a comment header, and duplicates the original file content as a comment block at the bottom of the

In variant files, leaf fields (i.e. scalar or list of scalars) can be marked overridden by having a comment starting with {OVERRIDE} or {OVERRIDE: reason to override} (the reason can be any text without newlines or braces). In this case, the marked leaf field is overlaid upon the original. So if you want to override a certain value, add # {OVERRIDE} or # {OVERRIDE: reason} comment (or prepend the existing one with it).

Example

suppose the upstream param file looks like:

foo: 42
bar: hoge
baz: 1

and the corresponding downstream variant:

foo: 42
bar: hogehoge # {OVERRIDE}
baz: 2 # {OVERRIDE}

now suppose the upstream param file changes:

-foo: 42
+foo: 40 # updated
-bar: hoge
+bar: fuga
-baz: 1 # removed
+qux: piyo # added

then after syncing, the downstream will be:

-foo: 42
+foo: 40 # updated
 bar: hogehoge # {OVERRIDE} # not updated
- baz: 2 # {OVERRIDE} # dropped (upstream is gone)
+qux: piyo # added

Check mode

The check-params workflow runs sync_params.py --check <category> on every PR. In check mode the script performs the same computation as update mode - cloning upstream, applying overrides, building the expected variant content - but writes nothing to disk, and exits non-zero if any variant has drifted from its expected state.

Check mode uses the SHA pinned in each variant's header (the # Source: line) rather than the current upstream HEAD. Upstream changes between the pinned SHA and HEAD are intentionally ignored, because those are the responsibility of update-params. A freshly synced variant will always pass check mode until someone edits it by hand without going through the sync workflow.

Rationales on designs

Q. Why do we have categories instead of bumping all param files at once?

perception / localization / etc have different codeowners, so one large PR updating all param files across all categories will need many approvals to sync. The exact granularity is a subject to change (e.g. we can have perception/object_recognition and perception/prediction categories split instead of one large perception category) but it is beyond the scope of this PR.

Q. Why should we duplicate a full copy footer of the upstream param file?

This is for visibility of updates of overridden leaf fields.

We can of course sync upstream source to variants directly, without duplicating the original as a comment footer. However when the upstream changes a leaf field, then the syncing PR might be ignorant to the context of the upstream change because the diff is suppressed by overriding mechanism.

For example, if the upstream param file has a field

some_list: [1, 2]

and the downstream overrides it

some_list: [10, 20] # {OVERRIDE}

and later the upstream changes the field to some_list: [1,2,3,4] because the list size should be four, the sync PR will be signaled that the upstream has been changed, but that exact diff is suppressed so the reviewer might not realize or forget that the override value should be updated to length-4 list e.g. [10, 20, 30, 40]. Having a local copy on this repository exposes the exact diff to the sync PR.

TODOs before getting approval of this PR

• create an actual workflow to sync (both periodically or manually triggered. different category corresponds to different PR branch.)
• create a verification workflow that validates each variant: original file + override = variant, and the original file footer matches to the source (version pinned in the header, not the branch head)
  • Run python3 .github/scripts/sync_params.py --check perception for this.
  • I deferred 'shape consistency check' for now, i.e. whether the list of leaf field keys match between the original content and the variant. But on the second thought I concluded that the removal can be easily noticed, and it is only useful when we MUST remove a parameter which is overridden in the downstream, which should be a rare case (for backward compatibility purpose, the obsolete param should be kept unused)
• get confirmation from perception owners that the configured correspondences between autoware_universe and autoware_launch is correct
• get confirmation from perception owners that the applied override markers match to their intent

Future works beyond the scope of this PR

• Apply this workflow to the rest of perception, localization, map, etc.
• Make the workflow reusable in https://github.com/autowarefoundation/autoware-github-actions

How was this PR tested?

Notes for reviewers

Effects on system behavior

None.

[github-actions]

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

• You've checked our contribution guidelines.
• Your PR follows our pull request guidelines.
• All required CI checks pass before marking the PR ready for review.

[paulsohn]

memo: this aligns with autowarefoundation/sync-file-templates#56 .

[isamu-takagi]

@paulsohn This action looks very useful. I'm not sure if this is possible, but how about saving the original parameters as comments, like OVERRIDE: <original-parameter>? That way, this action wouldn't need to copy the original file, and the differences would be displayed very clearly.

- some_list: [10, 20] # OVERRIDE: [1, 2]
+ some_list: [10, 20] # OVERRIDE: [1, 2, 3, 4]

[paulsohn]

This is the only field breaking changes so far.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

.pre-commit-config.yaml is marked as automatically synced from autowarefoundation/sync-file-templates (see header). Local edits like adding Prettier args are likely to be overwritten by the sync workflow; apply this change in the template repo instead (or update whatever sync mechanism manages this file).

Suggested change

args: [--no-error-on-unmatched-pattern]

[paulsohn]

This should be actually merged with #1338 but currently no approvals are out there.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sasakisasaki]

Tomorrow I'll continue my review. I guess I'll be understanding the behavior of the workflow tomorrow after doing some minor tests. Let me put a quick comment.

[sasakisasaki]

Personally, I prefer to use gh utility as like this workflow rather than using a third party workflow.

• https://github.com/autowarefoundation/autoware-github-actions/blob/f3e70631ee2e625f4764f87b05759e6183001950/bump-new-version/action.yaml#L200-L218

[sasakisasaki]

Still reviewing, need more time to understand -> let me understand on Monday. I guess in sync_params.py, it would be helpful to add a brief description (comment) for each function (for code readability).

Let me put a comment tomorrow again 🙏