|
| 1 | +--- |
| 2 | +layout: blog |
| 3 | +title: "Publishing a Kubernetes SIG's Images to registry.k8s.io" |
| 4 | +draft: true |
| 5 | +slug: publishing-images-to-registry-k8s-io |
| 6 | +author: > |
| 7 | + [Kahiro Okina](https://github.com/kahirokunn) (Craftsman Software, Inc.) |
| 8 | +--- |
| 9 | + |
| 10 | +For many Kubernetes SIG projects, shipping container images eventually |
| 11 | +becomes necessary, and `registry.k8s.io` is the official channel for them. I recently published a |
| 12 | +SIG project's first images there. No single step is hard, but the steps live |
| 13 | +in four repositories and have to happen in a particular order, which I mostly |
| 14 | +learned by tripping over them. |
| 15 | + |
| 16 | +This post is the guide I wish I had at the start. The example project is |
| 17 | +`cluster-inventory-api` from |
| 18 | +[SIG Multicluster](https://github.com/kubernetes/community/tree/master/sig-multicluster), |
| 19 | +but nothing in the procedure is specific to that SIG. |
| 20 | + |
| 21 | +## What Cluster Inventory API publishes |
| 22 | + |
| 23 | +Cluster Inventory API helps applications and tools work with multiple |
| 24 | +Kubernetes clusters. It publishes the |
| 25 | +[`secretreader`](https://github.com/kubernetes-sigs/cluster-inventory-api/tree/main/plugins/secretreader/cmd/plugin) |
| 26 | +and |
| 27 | +[`kubeconfig-secretreader`](https://github.com/kubernetes-sigs/cluster-inventory-api/tree/main/plugins/kubeconfig-secretreader/cmd/plugin) |
| 28 | +access-provider plugins as OCI images. Consumers mount these images as |
| 29 | +[image volumes](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/). |
| 30 | + |
| 31 | +## My first plan: ghcr.io |
| 32 | + |
| 33 | +I first tried a common GitHub release pattern: using GitHub Actions to publish |
| 34 | +images to `ghcr.io` on a tag push |
| 35 | +([cluster-inventory-api#40](https://github.com/kubernetes-sigs/cluster-inventory-api/pull/40)). |
| 36 | +The workflow succeeded, but Kubernetes GitHub organizations keep GHCR packages |
| 37 | +private, so GHCR cannot be used for public distribution. |
| 38 | + |
| 39 | +{{< figure src="ghcr-path-blocked.svg" class="text-center" width="660" alt="The blocked GHCR path: a tag push triggers GitHub Actions, which pushes the image to ghcr.io, where it cannot be made public." >}} |
| 40 | + |
| 41 | +As described in the |
| 42 | +[artifacts documentation](https://github.com/kubernetes/k8s.io/tree/main/artifacts#staging-buckets), |
| 43 | +official images take a different route: |
| 44 | +[Prow](https://docs.prow.k8s.io/) (the Kubernetes project's CI/CD system) |
| 45 | +picks up a tag push and runs Google Cloud Build on Kubernetes-owned |
| 46 | +infrastructure to push the image to a staging registry, and the image promoter |
| 47 | +then copies it to `registry.k8s.io`. |
| 48 | + |
| 49 | +{{< figure src="official-publishing-path.svg" class="text-center" width="820" alt="The official publishing path: a tag push triggers a Prow postsubmit job, which runs Cloud Build and pushes to a staging registry. The image promoter then copies the image to registry.k8s.io." >}} |
| 50 | + |
| 51 | +For this project, the images that finally shipped through that route were: |
| 52 | + |
| 53 | +```none |
| 54 | +registry.k8s.io/cluster-inventory-api/secretreader:v0.1.3 |
| 55 | +registry.k8s.io/cluster-inventory-api/kubeconfig-secretreader:v0.1.3 |
| 56 | +``` |
| 57 | + |
| 58 | +## The first-time setup, step by step |
| 59 | + |
| 60 | +Besides the image-owning repository, this touches three infrastructure |
| 61 | +repositories: [`kubernetes/k8s.io`](https://github.com/kubernetes/k8s.io), |
| 62 | +[`kubernetes/test-infra`](https://github.com/kubernetes/test-infra), and |
| 63 | +[`kubernetes/org`](https://github.com/kubernetes/org). The pieces depend on |
| 64 | +each other like this: |
| 65 | + |
| 66 | +{{< figure src="dependency-overview.svg" alt="A dependency graph for first-time registry.k8s.io image publishing. The image-owning repository provides cloudbuild.yaml and make release-staging, and a signed release tag triggers the kubernetes/test-infra image-pushing postsubmit job. In kubernetes/k8s.io, a staging Google Group must exist before the staging registry can be created, and that registry is the job's push target. The postsubmit job pushes the staging image. That staging image, together with image promoter config in kubernetes/k8s.io and OWNERS validation through kubernetes/org membership, promotes the image to registry.k8s.io." >}} |
| 67 | + |
| 68 | +### Before you start: choose registry paths, tag policy, and owners |
| 69 | + |
| 70 | +Reading `registry.k8s.io/cluster-inventory-api/secretreader:v0.1.3` from the |
| 71 | +example above: `<project>` is `cluster-inventory-api`, `<image>` is |
| 72 | +`secretreader`, and `v<version>` is `v0.1.3`. For your project, decide: |
| 73 | + |
| 74 | +- `<project>`, which also fixes the staging path |
| 75 | + `us-central1-docker.pkg.dev/k8s-staging-images/<project>`. |
| 76 | +- `<image>` for each image you ship. |
| 77 | +- The tag policy behind `<version>`. |
| 78 | +- Which SIG owns the project, and who reviews and approves. |
| 79 | +- Who goes in the promotion `OWNERS` file. |
| 80 | +- The staging access group name. |
| 81 | + |
| 82 | +### 1. Make the image-owning repository build images |
| 83 | + |
| 84 | +Set up the repository so that a tag push can build and push a staging image. |
| 85 | +You need: |
| 86 | + |
| 87 | +- a `RELEASE.md` documenting the release steps, |
| 88 | +- a `cloudbuild.yaml` (the `test-infra` job in step 4 invokes this to build |
| 89 | + and push the image), |
| 90 | +- a Dockerfile and/or Make target to build the image, |
| 91 | +- a release target that pushes to the staging registry (for example |
| 92 | + `make release-staging`). |
| 93 | + |
| 94 | +References: |
| 95 | +[cluster-inventory-api#53](https://github.com/kubernetes-sigs/cluster-inventory-api/pull/53) |
| 96 | +(moving to the Prow/Cloud Build approach) and |
| 97 | +[cluster-inventory-api#57](https://github.com/kubernetes-sigs/cluster-inventory-api/pull/57) |
| 98 | +(passing the staging repository explicitly to `kpromo`). |
| 99 | + |
| 100 | +### 2. Add a Google Group for staging artifacts |
| 101 | + |
| 102 | +Create the Google Group that will get push access to the staging registry, in |
| 103 | +your SIG's group configuration under `groups/` in `kubernetes/k8s.io` |
| 104 | +([kubernetes/k8s.io#9385](https://github.com/kubernetes/k8s.io/pull/9385)), |
| 105 | +and get approval from your SIG leads or chairs. Keep the group-name suffix |
| 106 | +within the 18-character limit |
| 107 | +([kubernetes/k8s.io#9402](https://github.com/kubernetes/k8s.io/pull/9402)). |
| 108 | + |
| 109 | +### 3. Add a staging registry in kubernetes/k8s.io |
| 110 | + |
| 111 | +Add one entry to the `registries` map in |
| 112 | +`infra/gcp/terraform/k8s-staging-images/registries.tf`, mapping `<project>` to |
| 113 | +the group from step 2. The module gives that group writer access and makes the |
| 114 | +repository publicly readable. Reference: |
| 115 | +[kubernetes/k8s.io#9347](https://github.com/kubernetes/k8s.io/pull/9347). |
| 116 | + |
| 117 | +### 4. Add an image-pushing postsubmit job in kubernetes/test-infra |
| 118 | + |
| 119 | +Add a job under `config/jobs/image-pushing/` that runs the image-owning |
| 120 | +repository's `cloudbuild.yaml` on a tag push and pushes to the staging |
| 121 | +registry. Reference: |
| 122 | +[kubernetes/test-infra#36821](https://github.com/kubernetes/test-infra/pull/36821). |
| 123 | + |
| 124 | +### 5. Push a release tag to build a staging image |
| 125 | + |
| 126 | +With everything above in place, push a |
| 127 | +[signed tag](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-tags) |
| 128 | +from the image-owning repository: |
| 129 | + |
| 130 | +```bash |
| 131 | +git tag -s v<version> |
| 132 | +git push origin v<version> |
| 133 | +gh release create v<version> --draft --generate-notes --verify-tag |
| 134 | +``` |
| 135 | + |
| 136 | +Then verify the staging image: |
| 137 | + |
| 138 | +```bash |
| 139 | +docker manifest inspect us-central1-docker.pkg.dev/k8s-staging-images/<project>/<image>:v<version> |
| 140 | +``` |
| 141 | + |
| 142 | +Tag events are not processed retroactively: tags created before the release |
| 143 | +pipeline existed will not produce a staging image. |
| 144 | + |
| 145 | +### 6. Add the image promoter configuration in kubernetes/k8s.io |
| 146 | + |
| 147 | +Open a `kubernetes/k8s.io` PR that adds the promotion configuration for this |
| 148 | +project |
| 149 | +([kubernetes/k8s.io#9499](https://github.com/kubernetes/k8s.io/pull/9499)): |
| 150 | + |
| 151 | +- `registry.k8s.io/images/k8s-staging-<project>/OWNERS`, |
| 152 | +- `registry.k8s.io/images/k8s-staging-<project>/images.yaml` (the promotion |
| 153 | + target), |
| 154 | +- `registry.k8s.io/manifests/k8s-staging-<project>/promoter-manifest.yaml`. |
| 155 | + |
| 156 | +For the first promotion, include the digest and tag entries for the staging |
| 157 | +images in `images.yaml`, and get `/lgtm` from a SIG lead. For later releases, |
| 158 | +[`kpromo`](https://github.com/kubernetes-sigs/promo-tools) generates this PR |
| 159 | +for you (see the routine release steps below). If you are curious how the |
| 160 | +promotion machinery works, see |
| 161 | +[The Invisible Rewrite: Modernizing the Kubernetes Image Promoter](https://kubernetes.io/blog/2026/03/17/image-promoter-rewrite/). |
| 162 | + |
| 163 | +If anyone you plan to list in `OWNERS` is not yet a Kubernetes organization |
| 164 | +member, submit a membership request first |
| 165 | +([kubernetes/org#6385](https://github.com/kubernetes/org/pull/6385), |
| 166 | +[kubernetes/org#6386](https://github.com/kubernetes/org/pull/6386)). |
| 167 | + |
| 168 | +### 7. Verify the release and publish it |
| 169 | + |
| 170 | +Once the promotion PR merges, run the project's release verification and |
| 171 | +confirm the production image is available: |
| 172 | + |
| 173 | +```bash |
| 174 | +docker manifest inspect registry.k8s.io/<project>/<image>:v<version> |
| 175 | +``` |
| 176 | + |
| 177 | +When that works, publish or update the GitHub release and announce it in the |
| 178 | +related issues and Slack channels. |
| 179 | + |
| 180 | +### Where to ask for help |
| 181 | + |
| 182 | +Several of these steps depend on other people: reviewers, approvers, and SIG |
| 183 | +leads. Expect to wait on reviews between steps rather than finishing in one |
| 184 | +sitting. On the [Kubernetes Slack](https://slack.k8s.io/), these channels line |
| 185 | +up with the work: |
| 186 | + |
| 187 | +| Channel | Use it for | |
| 188 | +| --- | --- | |
| 189 | +| `#github-management` | Repository access, GHCR questions, and Kubernetes organization membership | |
| 190 | +| `#sig-k8s-infra` | The staging Google Group and staging registry | |
| 191 | + |
| 192 | +## After the first time, it is much lighter |
| 193 | + |
| 194 | +Routine releases only touch the image-owning repository and one promotion PR: |
| 195 | + |
| 196 | +1. Push a signed tag, create a draft GitHub release, and confirm the |
| 197 | + postsubmit pushed the staging image, as in step 5 of the first-time setup. |
| 198 | +2. Create the promotion PR with `kpromo pr`, naming the Artifact Registry |
| 199 | + staging repository explicitly with `--staging-repo`: |
| 200 | + |
| 201 | + ```bash |
| 202 | + kpromo pr \ |
| 203 | + --fork <your-github-username> \ |
| 204 | + --project <project> \ |
| 205 | + --tag v<version> \ |
| 206 | + --staging-repo us-central1-docker.pkg.dev/k8s-staging-images/<project> |
| 207 | + ``` |
| 208 | + |
| 209 | +3. Once the promotion PR is reviewed and merged, finish as in step 7 of the |
| 210 | + first-time setup. |
| 211 | + |
| 212 | +## Acknowledgments |
| 213 | + |
| 214 | +Thanks to [Mike Ng](https://github.com/mikeshng) and |
| 215 | +[Laura Lorenz](https://github.com/lauralorenz) for attending meetings on my |
| 216 | +behalf, connecting me with the right people, and coordinating the work across |
| 217 | +SIG Multicluster; [Jian Qiu](https://github.com/qiujian16) for reviewing the |
| 218 | +implementation; |
| 219 | +[Stephen Kitt](https://github.com/skitt) for reviewing the release process and |
| 220 | +clarifying the publishing rules; and |
| 221 | +[Arnaud M.](https://github.com/ameukam) for reviewing the `kubernetes/k8s.io` |
| 222 | +pull requests and guiding the infrastructure and promotion changes. |
| 223 | + |
| 224 | +## References |
| 225 | + |
| 226 | +- [cluster-inventory-api releases](https://github.com/kubernetes-sigs/cluster-inventory-api/releases) |
| 227 | +- [Using Plugin OCI Images](https://github.com/kubernetes-sigs/cluster-inventory-api/blob/main/docs/plugin-images.md) |
| 228 | +- [Image volumes](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/) |
| 229 | +- [registry.k8s.io: faster, cheaper and Generally Available (GA)](https://kubernetes.io/blog/2022/11/28/registry-k8s-io-faster-cheaper-ga/) |
| 230 | +- [Publishing official artifact images (`kubernetes/k8s.io/artifacts`)](https://github.com/kubernetes/k8s.io/tree/main/artifacts#staging-buckets) |
| 231 | +- [`kpromo` (promo-tools)](https://github.com/kubernetes-sigs/promo-tools) |
| 232 | +- [Kubernetes Slack](https://slack.k8s.io/) |
0 commit comments