-
Notifications
You must be signed in to change notification settings - Fork 448
ci: add sync-params workflow #1812
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
paulsohn
wants to merge
35
commits into
autowarefoundation:main
Choose a base branch
from
paulsohn:sohn/sync-params
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 29 commits
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
9955171
remove stale update-sync-param-file
paulsohn d72a241
add sync params script
paulsohn ed4a877
update for optional ref and filtering
paulsohn d17ca81
text change
paulsohn 48bb0c4
update prettier config --no-error-on-unmatched-pattern
paulsohn 6676a7c
bugfix
paulsohn 6db6848
add actual file changes
paulsohn 658cedf
Merge branch 'main' into sohn/sync-params
paulsohn 52daed3
add category based sync params instead of filtering
paulsohn 3e316ec
update checking logic
paulsohn cc10d5d
update param files
paulsohn f20ca45
improve edge case handling logic
paulsohn 211a8dc
allow annotation inside override marker
paulsohn a37b002
update param file with new annotation format
paulsohn 1ac12cb
add actual github workflows
paulsohn 1a09d67
remove stale update-sync-param-files workflow
paulsohn 3c6284b
fix yq to output oneline
paulsohn 002ceeb
refine workflows
paulsohn 95bfd35
fix misconfigured param updates
paulsohn bfc027e
avoid marker capture the existing comment
paulsohn e057898
return dict instead of empty array
paulsohn ab538ee
guide to run directly
paulsohn e9830c2
fix SHA for yq to prevent supply chain risk
paulsohn 4c79b04
copilot review round
paulsohn 8520572
Merge branch 'main' into sohn/sync-params
paulsohn 6afaa62
drop the stale field with warning instead of raising an error
paulsohn 2bf4291
disable cspell for test
paulsohn ca68986
add cspell ignore for script
paulsohn daa85e7
hardcode input_category instead of strenv
paulsohn b18e6e8
run test in check-params.yaml
paulsohn d623601
remove redundant re-read
paulsohn 3402d86
adjust the correct behavior for check mode
paulsohn fda253e
update an integration test to ensure check after update passes
paulsohn 11ce71b
replying to copilot comment
paulsohn 42b3f55
provide empty string category on scheduled run
paulsohn File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| perception: | ||
| - repository: autowarefoundation/autoware_universe | ||
| ref: main | ||
| files: | ||
| # Perception | ||
| - # Source param path in source repository. | ||
| source: perception/autoware_image_object_locator/config/bbox_object_locator.param.yaml | ||
| # List of param paths in current repository that are variants of the source param path. | ||
| # Variant files should have the same yaml file layout as the source file and only differ in the parameters. | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/camera_vru_detection/near_range_camera_vru_detector.param.yaml | ||
| - source: perception/autoware_euclidean_cluster/config/euclidean_cluster.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/clustering/euclidean_cluster.param.yaml | ||
| - source: perception/autoware_detected_object_validation/config/obstacle_pointcloud_based_validator.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/detected_object_validation/obstacle_pointcloud_based_validator.param.yaml | ||
| - source: perception/autoware_detected_object_validation/config/occupancy_grid_based_validator.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/detected_object_validation/occupancy_grid_based_validator.param.yaml | ||
| - source: perception/autoware_detection_by_tracker/config/detection_by_tracker.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/detection_by_tracker/detection_by_tracker.param.yaml | ||
| - source: perception/autoware_image_projection_based_fusion/config/fusion_common.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/image_projection_based_fusion/fusion_common.param.yaml | ||
| - source: perception/autoware_image_projection_based_fusion/config/pointpainting_common.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/image_projection_based_fusion/pointpainting_common.param.yaml | ||
| - source: perception/autoware_image_projection_based_fusion/config/roi_cluster_fusion.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/image_projection_based_fusion/roi_cluster_fusion.param.yaml | ||
| - source: perception/autoware_image_projection_based_fusion/config/roi_detected_object_fusion.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/image_projection_based_fusion/roi_detected_object_fusion.param.yaml | ||
| - source: perception/autoware_image_projection_based_fusion/config/roi_pointcloud_fusion.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/image_projection_based_fusion/roi_pointcloud_fusion.param.yaml | ||
| # | ||
| # irregular_object_detector.param.yaml omitted; | ||
| # It is actually a mix of 3 different param files. Needs refactoring to track via this workflow. | ||
| # | ||
| - source: perception/autoware_bevfusion/config/bevfusion_camera_lidar.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/lidar_model/bevfusion_camera_lidar.param.yaml | ||
| - source: perception/autoware_bevfusion/config/common_bevfusion.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/lidar_model/bevfusion_common.param.yaml | ||
| - source: perception/autoware_bevfusion/config/bevfusion_lidar.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/lidar_model/bevfusion_lidar.param.yaml | ||
| - source: perception/autoware_lidar_centerpoint/config/centerpoint_common.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/lidar_model/centerpoint_common.param.yaml | ||
| - source: perception/autoware_lidar_centerpoint/config/centerpoint_sigma.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/lidar_model/centerpoint_sigma.param.yaml | ||
| - source: perception/autoware_lidar_centerpoint/config/centerpoint_tiny.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/lidar_model/centerpoint_tiny.param.yaml | ||
| - source: perception/autoware_lidar_centerpoint/config/centerpoint.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/lidar_model/centerpoint.param.yaml | ||
| - source: perception/autoware_image_projection_based_fusion/config/pointpainting.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/lidar_model/pointpainting.param.yaml | ||
| - source: perception/autoware_lidar_transfusion/config/transfusion_common.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/lidar_model/transfusion_common.param.yaml | ||
| - source: perception/autoware_lidar_transfusion/config/transfusion.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/lidar_model/transfusion.param.yaml | ||
| - source: perception/autoware_detected_object_validation/config/object_lanelet_filter.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/object_filter/object_lanelet_filter.param.yaml | ||
| - path: autoware_launch/config/perception/object_recognition/detection/object_filter/radar_lanelet_filter.param.yaml | ||
| - source: perception/autoware_detected_object_validation/config/object_position_filter.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/object_filter/object_position_filter.param.yaml | ||
| - source: perception/autoware_object_merger/config/data_association_matrix.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/object_merger/data_association_matrix.param.yaml | ||
| - source: perception/autoware_object_merger/config/object_association_merger.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/object_merger/ml_camera_lidar_object_association_merger.param.yaml | ||
| - source: perception/autoware_object_merger/config/overlapped_judge.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/object_merger/overlapped_judge.param.yaml | ||
| # | ||
| # pointcloud_map_filter.param.yaml omitted. The 'param file' is completely interpreted by launcher. | ||
| # | ||
| - source: perception/autoware_object_sorter/config/detected_object_sorter.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/detection/radar_filter/detected_object_sorter.param.yaml | ||
| - source: perception/autoware_map_based_prediction/config/map_based_prediction.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/prediction/map_based_prediction.param.yaml | ||
| - source: perception/autoware_multi_object_tracker/config/data_association_matrix.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/tracking/multi_object_tracker/data_association_matrix.param.yaml | ||
| - source: perception/autoware_multi_object_tracker/config/input_channels.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/tracking/multi_object_tracker/input_channels.param.yaml | ||
| - source: perception/autoware_multi_object_tracker/config/multi_object_tracker_node.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/tracking/multi_object_tracker/multi_object_tracker_node.param.yaml | ||
| - source: perception/autoware_detected_object_validation/config/tracked_object_lanelet_filter.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/tracking/radar_object_tracker/tracked_object_lanelet_filter.param.yaml | ||
| - source: perception/autoware_object_sorter/config/tracked_object_sorter.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/tracking/radar_object_tracker/tracked_object_sorter.param.yaml | ||
| - source: perception/autoware_tracking_object_merger/config/data_association_matrix.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/tracking/tracking_object_merger/data_association_matrix.param.yaml | ||
| - source: perception/autoware_tracking_object_merger/config/decorative_tracker_merger_policy.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/tracking/tracking_object_merger/decorative_tracker_merger_policy.param.yaml | ||
| - source: perception/autoware_tracking_object_merger/config/decorative_tracker_merger.param.yaml | ||
| variants: | ||
| - path: autoware_launch/config/perception/object_recognition/tracking/tracking_object_merger/decorative_tracker_merger.param.yaml | ||
| # Perception/obstacle_segmentation | ||
| # Perception/occupancy_grid_map | ||
| # Perception/traffic_light_recognition | ||
|
|
||
| localization: [] | ||
| map: [] | ||
| planning: [] | ||
| control: [] | ||
| system: [] | ||
| vehicle: [] | ||
| simulator: [] |
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| name: check-params | ||
|
|
||
| on: | ||
| pull_request: | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| resolve-categories: | ||
| runs-on: ubuntu-22.04 | ||
| outputs: | ||
| matrix: ${{ steps.build-matrix.outputs.matrix }} | ||
| steps: | ||
| - name: Check out repository | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Install yq | ||
| run: | | ||
| sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/v4.49.2/yq_linux_amd64 | ||
| echo "be2c0ddcf426b6a231648610ec5d1666ae50e9f6473e82f6486f9f4cb6e3e2f7 /usr/local/bin/yq" | sha256sum -c - | ||
| sudo chmod +x /usr/local/bin/yq | ||
| - name: Build category matrix from config | ||
| id: build-matrix | ||
| run: | | ||
| set -euo pipefail | ||
| matrix_json="$(yq -o=json -I=0 '. | keys' .github/sync-params.yaml)" | ||
| echo "matrix=${matrix_json}" >> "${GITHUB_OUTPUT}" | ||
| echo "Selected categories: ${matrix_json}" | ||
| check-params-per-category: | ||
| needs: resolve-categories | ||
| runs-on: ubuntu-22.04 | ||
| strategy: | ||
| fail-fast: false | ||
| max-parallel: 4 | ||
| matrix: | ||
| category: ${{ fromJson(needs.resolve-categories.outputs.matrix) }} | ||
| steps: | ||
| - name: Check out repository | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Install dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| python -m pip install ruamel.yaml | ||
| - name: Check params | ||
| run: python .github/scripts/sync_params.py --check ${{ matrix.category }} | ||
|
paulsohn marked this conversation as resolved.
|
||
|
|
||
| check-params: | ||
| needs: check-params-per-category | ||
| if: ${{ always() }} | ||
| runs-on: ubuntu-22.04 | ||
| steps: | ||
| - name: Evaluate per-category results | ||
| run: | | ||
| if [[ "${{ needs.check-params-per-category.result }}" != "success" ]]; then | ||
| echo "At least one category check failed." | ||
| exit 1 | ||
| fi | ||
| echo "All category checks passed." | ||
|
paulsohn marked this conversation as resolved.
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.