[ABLD-387] Use hermetic Bazel-managed protobuf toolchain#48970
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits intomainfrom Apr 9, 2026
Merged
[ABLD-387] Use hermetic Bazel-managed protobuf toolchain#48970gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits intomainfrom
protobuf toolchain#48970gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits intomainfrom
Conversation
rdesgroppes
added
changelog/no-changelog
dd-octo-sts
bot
added
internal
dd-octo-sts
bot
added
team/agent-devx
team/agent-security
team/container-platform
Contributor
Files inventory check summaryFile checks results against ancestor 3a4e6cd5: Results for datadog-agent_7.79.0~devel.git.556.660a508.pipeline.106871821-1_amd64.deb:No change detected |
rdesgroppes
force-pushed
the
regis.desgroppes/use-hermetic-protoc
branch
from
3c2fa8a to
7c6783d
Compare
Contributor
Static quality checks✅ Please find below the results from static quality gates 31 successful checks with minimal change (< 2 KiB)
On-wire sizes (compressed)
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: b255ecc Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | +2.23 | [-0.79, +5.24] | 1 | Logs |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | +2.23 | [-0.79, +5.24] | 1 | Logs |
| ➖ | quality_gate_logs | % cpu utilization | +2.05 | [+0.38, +3.73] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | +0.40 | [+0.37, +0.44] | 1 | Logs bounds checks dashboard |
| ➖ | ddot_metrics | memory utilization | +0.33 | [+0.15, +0.51] | 1 | Logs |
| ➖ | ddot_metrics_sum_cumulative | memory utilization | +0.26 | [+0.11, +0.40] | 1 | Logs |
| ➖ | ddot_metrics_sum_delta | memory utilization | +0.19 | [+0.01, +0.37] | 1 | Logs |
| ➖ | ddot_logs | memory utilization | +0.17 | [+0.10, +0.24] | 1 | Logs |
| ➖ | otlp_ingest_metrics | memory utilization | +0.09 | [-0.07, +0.25] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | +0.06 | [-0.33, +0.44] | 1 | Logs |
| ➖ | file_tree | memory utilization | +0.06 | [-0.00, +0.12] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | +0.05 | [-0.37, +0.48] | 1 | Logs |
| ➖ | docker_containers_memory | memory utilization | +0.03 | [-0.05, +0.11] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_v3 | ingress throughput | +0.02 | [-0.19, +0.22] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | +0.01 | [-0.12, +0.14] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.00 | [-0.10, +0.10] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | -0.01 | [-0.22, +0.19] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | -0.09 | [-0.59, +0.42] | 1 | Logs |
| ➖ | otlp_ingest_logs | memory utilization | -0.09 | [-0.20, +0.02] | 1 | Logs |
| ➖ | uds_dogstatsd_20mb_12k_contexts_20_senders | memory utilization | -0.16 | [-0.22, -0.10] | 1 | Logs |
| ➖ | ddot_metrics_sum_cumulativetodelta_exporter | memory utilization | -0.25 | [-0.47, -0.02] | 1 | Logs |
| ➖ | quality_gate_idle | memory utilization | -0.29 | [-0.34, -0.23] | 1 | Logs bounds checks dashboard |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -0.43 | [-0.62, -0.23] | 1 | Logs |
| ➖ | quality_gate_metrics_logs | memory utilization | -0.47 | [-0.70, -0.23] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | docker_containers_cpu | simple_check_run | 10/10 | 579 ≥ 26 | |
| ✅ | docker_containers_memory | memory_usage | 10/10 | 274.77MiB ≤ 370MiB | |
| ✅ | docker_containers_memory | simple_check_run | 10/10 | 684 ≥ 26 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | 0.19GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_0ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | 0.23GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_1000ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | 0.20GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_100ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | 0.21GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_500ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | quality_gate_idle | intake_connections | 10/10 | 3 = 3 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 175.24MiB ≤ 181MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 3 = 3 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 487.66MiB ≤ 550MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 3 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 202.14MiB ≤ 220MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 345.91 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 4 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 403.05MiB ≤ 475MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
rdesgroppes
force-pushed
the
regis.desgroppes/use-hermetic-protoc
branch
from
7c6783d to
a932343
Compare
rdesgroppes
changed the title
protocprotobuf toolchain
rdesgroppes
force-pushed
the
regis.desgroppes/use-hermetic-protoc
branch
from
a932343 to
d08cbd7
Compare
Contributor
Gitlab CI Configuration ChangesModified Jobsprotobuf_test protobuf_test:
before_script:
- mkdir -p $GOPATH/bin $GOPATH/pkg/mod/cache && zstd -dc modcache_tools.tar.zst
| tar xf - -C $GOPATH
- rm -f modcache_tools.tar.zst
- export PATH=$PATH:$GOPATH/bin
+ cache:
+ - key:
+ files:
+ - .bazelversion
+ prefix: bazelversion-$CI_RUNNER_DESCRIPTION
+ paths:
+ - .cache/bazelisk
+ - .cache/bazel/*/install
+ policy: pull$BAZEL_CACHE_POLICY_SUFFIX
+ when: on_success
+ - key:
+ files:
+ - .go-version
+ - .python-version
+ prefix: bazel-$CI_JOB_NAME
+ paths:
+ - .cache/bazel/*/cache
+ - .cache/bazel/disk-cache
+ - .cache/go
+ - .cache/ms-go
+ - .cache/pip
+ policy: pull$BAZEL_CACHE_POLICY_SUFFIX
+ when: on_success
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
needs:
- go_tools_deps
script:
- dda inv -- -e protobuf.generate --pre-commit
stage: source_test
tags:
- arch:amd64
- specific:true
+ variables:
+ BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
+ XDG_CACHE_HOME: $CI_PROJECT_DIR/.cachesecurity_go_generate_check security_go_generate_check:
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && zstd -dc modcache.tar.zst | tar xf - -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.zst
- mkdir -p $GOPATH/bin $GOPATH/pkg/mod/cache && zstd -dc modcache_tools.tar.zst
| tar xf - -C $GOPATH
- rm -f modcache_tools.tar.zst
- export PATH=$PATH:$GOPATH/bin
+ cache:
+ - key:
+ files:
+ - .bazelversion
+ prefix: bazelversion-$CI_RUNNER_DESCRIPTION
+ paths:
+ - .cache/bazelisk
+ - .cache/bazel/*/install
+ policy: pull$BAZEL_CACHE_POLICY_SUFFIX
+ when: on_success
+ - key:
+ files:
+ - .go-version
+ - .python-version
+ prefix: bazel-$CI_JOB_NAME
+ paths:
+ - .cache/bazel/*/cache
+ - .cache/bazel/disk-cache
+ - .cache/go
+ - .cache/ms-go
+ - .cache/pip
+ policy: pull$BAZEL_CACHE_POLICY_SUFFIX
+ when: on_success
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux$CI_IMAGE_LINUX_SUFFIX:$CI_IMAGE_LINUX
needs:
- go_deps
- go_tools_deps
script:
- dda inv -- -e security-agent.go-generate-check
stage: source_test
tags:
- arch:amd64
- specific:true
variables:
+ BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
KUBERNETES_CPU_REQUEST: 4
+ XDG_CACHE_HOME: $CI_PROJECT_DIR/.cacheChanges Summary
ℹ️ Diff available in the job log. |
rdesgroppes
changed the title
protobuf toolchainprotobuf toolchain
rdesgroppes
force-pushed
the
regis.desgroppes/use-hermetic-protoc
branch
3 times, most recently
from
183f9dd to
a2d578c
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a2d578ca44
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
rdesgroppes
force-pushed
the
regis.desgroppes/use-hermetic-protoc
branch
from
a2d578c to
057f727
Compare
davidor
approved these changes
Apr 9, 2026
Member
davidor
left a comment
davidor
left a comment
There was a problem hiding this comment.
👍 for container-platform files
rdesgroppes
force-pushed
the
regis.desgroppes/use-hermetic-protoc
branch
from
057f727 to
7e3e21b
Compare
JSGette
reviewed
Apr 9, 2026
dd-gplassard
approved these changes
Apr 9, 2026
Contributor
dd-gplassard
left a comment
dd-gplassard
left a comment
There was a problem hiding this comment.
For actionplatform files 👍
YoannGh
approved these changes
Apr 9, 2026
Contributor
YoannGh
left a comment
YoannGh
left a comment
There was a problem hiding this comment.
Good for agent-security owned files!
pgimalac
approved these changes
Apr 9, 2026
rdesgroppes
force-pushed
the
regis.desgroppes/use-hermetic-protoc
branch
from
7e3e21b to
a56f4fd
Compare
KevinFairise2
approved these changes
Apr 9, 2026
Member
KevinFairise2
left a comment
KevinFairise2
left a comment
There was a problem hiding this comment.
Looks good for the files owned by Agent DevX
domodwyer
approved these changes
Apr 9, 2026
### What does this PR do?
Replace all PATH-dependent protobuf tool invocations in
`tasks/protobuf.py` and `tasks/security_agent.py` with hermetic
Bazel-managed equivalents (`protoc`, `protoc-gen-go`,
`protoc-gen-go-grpc`, `protoc-gen-go-vtproto`,
`protoc-go-inject-tag`, `mockgen`, `msgp`).
Introduce a `Tools` class that lazily resolves binary paths via
`bazel cquery` on first use, caching results for the duration of
the invoke session.
Remove the `install_protoc` task, `.protoc-version` pin file,
`check_tools` gating, the `go install` step from
`generate_cws_proto`, and the associated Renovate rule.
### Motivation
Part of the ongoing Bazelification of the agent build. Hermetic
tools guarantee identical output across all developer environments
and CI, eliminating version drift and the manual setup steps
(`dda inv install-tools`, `dda inv install-protoc`).
This is a prerequisite to migrating the generation of proto
language bindings to BUILD.bazel files: by establishing a
homogeneous, pinned toolchain now, subsequent PRs won't be
encumbered with changes in generated content.
### Describe how you validated your changes
- `dda inv protobuf.generate` produces output identical to the
committed generated files.
- `dda inv security-agent.generate-cws-proto` regenerates the CWS
API proto files successfully.
### Additional Notes
Reviewers will notice two categories of changes in the generated
files:
- version strings in headers: `// protoc` (previously stripped by
a regex post-processing step in `generate_cws_proto`) now reads
`// protoc v7.34.1`; this is intentional — the hermetic
toolchain guarantees consistency everywhere.
- descriptor representation (`api.pb.go` shrinks from 171 KB to
129 KB): `protoc-gen-go` v1.36.8 switched the raw file descriptor
from `var []byte{0x0a, ...}` to `const string = "\x0a..."` +
`unsafe.Slice(unsafe.StringData(...))`, a source-level
compaction with identical runtime semantics.
rdesgroppes
force-pushed
the
regis.desgroppes/use-hermetic-protoc
branch
from
a56f4fd to
660a508
Compare
ihssane-yb
approved these changes
Apr 9, 2026
kkhor-datadog
approved these changes
Apr 9, 2026
ichinaski
approved these changes
Apr 9, 2026
gh-worker-dd-mergequeue-cf854d
bot
deleted the
regis.desgroppes/use-hermetic-protoc
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Replace all host-dependent protobuf tool invocations in
tasks/protobuf.pyandtasks/security_agent.pywith hermetic Bazel-managed equivalents (protoc,protoc-gen-go,protoc-gen-go-grpc,protoc-gen-go-vtproto,protoc-go-inject-tag,mockgen,msgp).Introduce a
BazelToolsclass that lazily resolves binary paths viabazel cqueryon first use, caching results for the duration of theinvokesession.Remove the
install_protoctask,.protoc-versionpin file,check_toolsgating, thego installstep fromgenerate_cws_proto, and the associated Renovate rule.Motivation
This is a prerequisite to migrating the generation of proto language bindings to
BUILD.bazelfiles: by establishing a homogeneous, pinned toolchain now, subsequent PRs won't be encumbered with changes in generated files.Hermetic tools guarantee identical output across all developer environments and CI, eliminating version drift and the manual setup steps (esp.
dda inv install-protoc).Describe how you validated your changes
Locally, on Linux and Windows (hence a few tweaked path constructions):
dda inv protobuf.generateproduces output identical to the committed generated files,dda inv security-agent.generate-cws-protoalso regenerates the CWS API proto files successfully.In CI, this is covered by existing jobs (now fetching Bazel's "progressive" cache):
protobuf_test,security_go_generate_check.Additional Notes
Reviewers will notice two categories of changes in the generated files:
protocnow reads// protoc v7.34.1; this is intentional: the hermetic toolchain guarantees consistency everywhere,api.pb.goshrinks from 171 KB to 129 KB):protoc-gen-gov1.36.6 indeed switched the raw file descriptor fromvar []byte{0x0a, ...}toconst string = "\x0a..."+unsafe.Slice(unsafe.StringData(...)), a source-level compaction with identical runtime semantics, see: https://go-review.googlesource.com/c/protobuf/+/657895.