feat(build): migrate content ingestion to hugo modules by TineoC · Pull Request #647 · kubernetes/contributor-site

TineoC · 2026-02-05T06:30:43Z

Summary

Migrates the external content ingestion workflow from the custom hack/gen-content.sh script to Hugo Modules. This change modernizes the build process and leverages Hugo's native support for mounting content from external Git repositories.

Content freshness is handled entirely within Netlify's build pipeline: production builds run hugo mod get -u before building, and a daily scheduled deploy ensures content stays up to date even when no PRs merge. This eliminates the need for external automation (Prow / GitHub Actions).

Fixes

Fixes #93

/sig contribex
/cc @SayakMukhopadhyay @reylejano

lmktfy

Thanks for the PR.

I like the ambition, a lot, but I have concerns about the detail.

lmktfy · 2026-02-14T13:14:16Z

 # Docker buildx related settings for multi-arch images
 DOCKER_BUILDX ?= docker buildx

+# Reuse host Go module cache in container so modules aren't re-downloaded each run (default: $HOME/go/pkg/mod)


Does this have any security implications?

The cache has to be read-write for development. I mitigated this risk by reducing the privileges in the development container in the following commits.

lmktfy · 2026-02-14T13:15:10Z

 	--mount type=tmpfs,destination=/tmp,tmpfs-mode=01777 \
 	--mount type=bind,source=$(CURDIR)/hugo.yaml,target=/src/hugo.yaml,readonly
+# Writable mount for container-render output (Hugo writes to /out -> host public/)
+CONTAINER_RENDER_MOUNT	:= --mount type=bind,source=$(CURDIR)/public,target=/out


Could this make local previewing break for people using rootless containers (eg Podman in its default configuration)?

Thanks, good catch

(base) ➜ contributor-site git:(fix/content-ingestion) make container-server CONTAINER_ENGINE=podman IMAGE_REGISTRY=loal # no build lock to allow for read-only mounts Resolving "loal/k8s-contrib-site-hugo" using unqualified-search registries (/etc/containers/registries.conf.d/999-podman-machine.conf) Trying to pull docker.io/loal/k8s-contrib-site-hugo:v0.133.0-01a72e174420... Error: unable to copy from source docker://loal/k8s-contrib-site-hugo:v0.133.0-01a72e174420: initializing source docker://loal/k8s-contrib-site-hugo:v0.133.0-01a72e174420: reading manifest v0.133.0-01a72e174420 in docker.io/loal/k8s-contrib-site-hugo: requested access to the resource is denied podman build . -t loal/k8s-contrib-site-hugo:v0.133.0-01a72e174420 --label git_commit=666b40a --build-arg HUGO_VERSION=0.133.0 STEP 1/14: FROM alpine:latest STEP 2/14: ARG HUGO_VERSION --> Using cache 750fa2c5150b6c02fb021adf61b43dbc6cf09b6da450a2150cacb53d3f44ab01 --> 750fa2c5150b STEP 3/14: ARG TARGETARCH --> Using cache 7a0d9196102135bc758d3e03fdc1e0aafa403404f06a9f021f27e256ddeb1dad --> 7a0d91961021 STEP 4/14: ARG GO_VERSION=1.22.5 --> Using cache a5d780517d59410d47d4d83a45e37669999ffe0970072db1dc817ef98d76aa78 --> a5d780517d59 STEP 5/14: RUN apk add --no-cache bash build-base curl git grep gcompat libc6-compat rsync sed npm --> Using cache 74b38bbdbc935741f18d92564d3b9cc69aba211dba3176935ea456f4b2863cef --> 74b38bbdbc93 STEP 6/14: RUN curl -sSfL "https://go.dev/dl/go${GO_VERSION}.linux-${TARGETARCH}.tar.gz" -o /tmp/go.tgz && tar -xz -C /usr/local -f /tmp/go.tgz && rm /tmp/go.tgz --> Using cache 3736d031bd425146561a5f535e932382f15a2058c66b2cd8f03e0d3e233a57eb --> 3736d031bd42 STEP 7/14: ENV PATH="/usr/local/go/bin:${PATH}" --> Using cache aaed545b42ed81c14c05ee7019a885347b09771ccc58052b246c81f7efcb8da2 --> aaed545b42ed STEP 8/14: WORKDIR /src --> Using cache 4b3ff067b3fcacd7a50f299122d7654f6fe999c7b7afaf0a3f6be227233ac87c --> 4b3ff067b3fc STEP 9/14: COPY package*.json ./ --> Using cache 97bc89e07e9a90a3e6196d4c04635edd6a54763487e903653b09f73233accea6 --> 97bc89e07e9a STEP 10/14: RUN npm ci --ignore-scripts --> Using cache 9240380c24a038df1db6f32fd95283793cb02e661739c9cf74f536aa1753963f --> 9240380c24a0 STEP 11/14: RUN mkdir -p /usr/local/src && cd /usr/local/src && curl -L https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_linux-${TARGETARCH}.tar.gz | tar -xz && mv hugo /usr/local/bin/hugo && addgroup -Sg 1000 hugo && adduser -Sg hugo -u 1000 -h /src hugo --> Using cache 2f079a3fdc68021f602c80751af9c7a350f147ed24a5199738a1456dc15768a6 --> 2f079a3fdc68 STEP 12/14: USER hugo:hugo --> Using cache 2e8653e93b0eb9adbfdedea7299f3d89b744db5d7bded95a01dae157ad0c19ba --> 2e8653e93b0e STEP 13/14: EXPOSE 1313 --> Using cache dda6752227cc99270146c8b13c4fb9f663098f8dccdb126746593958bec59720 --> dda6752227cc STEP 14/14: LABEL "git_commit"="666b40a" COMMIT loal/k8s-contrib-site-hugo:v0.133.0-01a72e174420 --> 0f941871ebd1 Successfully tagged localhost/loal/k8s-contrib-site-hugo:v0.133.0-01a72e174420 0f941871ebd12db15ba4c9faabbf90c49b6621eb37c8857471f4c33856b10336 podman run --rm -it --security-opt=no-new-privileges --cap-drop=ALL --userns=keep-id -e GOMODCACHE=/tmp/gomod --read-only --mount type=bind,source=/Users/tineoc/Documents/Code/cncf/sig-contribex/contributor-site/.git,target=/src/.git,readonly,relabel=shared --mount type=bind,source=/Users/tineoc/Documents/Code/cncf/sig-contribex/contributor-site/go.mod,target=/src/go.mod,relabel=shared --mount type=bind,source=/Users/tineoc/Documents/Code/cncf/sig-contribex/contributor-site/go.sum,target=/src/go.sum,relabel=shared --mount type=bind,source=/Users/tineoc/go/pkg/mod,target=/tmp/gomod,relabel=shared --mount type=bind,source=/Users/tineoc/Documents/Code/cncf/sig-contribex/contributor-site/assets,target=/src/assets,readonly,relabel=shared --mount type=bind,source=/Users/tineoc/Documents/Code/cncf/sig-contribex/contributor-site/content,target=/src/content,readonly,relabel=shared --mount type=bind,source=/Users/tineoc/Documents/Code/cncf/sig-contribex/contributor-site/layouts,target=/src/layouts,readonly,relabel=shared --mount type=bind,source=/Users/tineoc/Documents/Code/cncf/sig-contribex/contributor-site/static,target=/src/static,readonly,relabel=shared --mount type=tmpfs,destination=/tmp,tmpfs-mode=01777 --mount type=bind,source=/Users/tineoc/Documents/Code/cncf/sig-contribex/contributor-site/hugo.yaml,target=/src/hugo.yaml,readonly,relabel=shared -p 1313:1313 loal/k8s-contrib-site-hugo:v0.133.0-01a72e174420 \ bash -c 'cd /src && hugo mod get && \ hugo server \ --environment preview \ --logLevel info \ --noBuildLock \ --bind 0.0.0.0 \ --buildDrafts \ --buildFuture \ --disableFastRender \ --ignoreCache \ --destination /tmp/hugo \ --cleanDestinationDir' Watching for changes in /src/{assets,content,layouts,node_modules,package.json,static} Watching for config changes in /src/hugo.yaml, /src/node_modules/docsy/hugo.yaml, /src/go.mod Start building sites … hugo v0.133.0-c9777473d1369f812d727a6c07dc57ad7be7bf62+extended linux/arm64 BuildDate=2024-08-17T19:57:41Z VendorInfo=gohugoio INFO static: removing all files from destination that don't exist in static dirs INFO static: syncing static files to / duration 14.88494ms INFO build: step process substep collect files 326 files_total 326 pages_total 240 resources_total 86 duration 56.02497ms INFO build: step process duration 57.29976ms INFO build: step assemble duration 9.640696ms INFO build: step render substep pages site en outputFormat html duration 1.128635893s INFO build: step render substep pages site en outputFormat rss duration 30.644128ms INFO build: step render pages 279 content 239 duration 1.159478437s INFO build: step render deferred count 0 duration 1µs INFO build: step postProcess duration 8.333µs INFO build: duration 1.226780226s | EN -------------------+------ Pages | 275 Paginator pages | 5 Non-page files | 85 Static files | 36 Processed images | 5 Aliases | 16 Cleaned | 0 Built in 1242 ms Environment: "preview" Serving pages from disk Web Server is available at http://localhost:1313/ (bind address 0.0.0.0)

lmktfy · 2026-02-14T13:16:11Z

 	$(MAKE) container-server

-container-server: ## Run Hugo locally within a container, available at http://localhost:1313/
+container-server: container-image ## Run Hugo locally within a container, available at http://localhost:1313/


We should ideally fetch the image rather than build it locally.

lmktfy · 2026-02-14T13:17:52Z

    build-base \
    curl \
    git \
+    go \


I would add a version pin or otherwise try to ensure it is close to the version that Netlify uses.

lmktfy · 2026-02-14T13:18:30Z

Under what circumstances will these values change?

With these changes... when someone manually runs hugo mod get. This isn't ideal. But right now, we run the latest changes always from every project, so we also have some security/stability risks because we also run the risk of breaking the site every time there's new content in every other repo and page.

I would like to know what you guys like most:

Ensuring scheduled workflows to update content (weekly, daily) to open a PR so we could have traceability of which new content broke the contributor website, and if, by any reason, any of our external sources broke, we could have better guidance for remediation.

Only update external sources modules on PR merged (the downside is that new content is dependent on how often we merge to master)

For example something like .github/workflows/update-modules.yml

1 name: Update Hugo Modules 2 on: 3 schedule: 4 - cron: '0 0 * * *' # Daily at midnight 5 workflow_dispatch: # Allow manual trigger 6 7 jobs: 8 update: 9 runs-on: ubuntu-latest 10 steps: 11 - uses: actions/checkout@v4 12 - name: Setup Hugo 13 uses: peaceiris/actions-hugo@v3 14 with: 15 hugo-version: 'latest' 16 extended: true 17 - name: Update Modules 18 run: | 19 hugo mod get -u 20 hugo mod tidy 21 - name: Create Pull Request 22 uses: peter-evans/create-pull-request@v6 23 with: 24 commit-message: "chore: update hugo modules" 25 title: "Update external content modules" 26 body: "This PR updates the Hugo modules to pull the latest content from upstream repositories." 27 branch: "automation/update-modules"

I would have automated builds, and notify maintainers if no utomated builds have succeeded in the last 36 hours.

we also have some security/stability risks because we also run the risk of breaking the site every time there's new content in every other repo and page.

If such a change breaks the build, the existing site still gets served.

What if we just run https://github.com/kubernetes/contributor-site/pull/647/changes#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R75 make module-download in dev and make that only the prod environment is the one making sure it is running with the latest go modules and in dev and feature branches, keeping the ones tracked by git in the go.mod file?

@lmktfy @jberkus

TineoC · 2026-03-20T02:19:33Z

/hold

k8s-ci-robot · 2026-05-02T07:53:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: TineoC

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [TineoC]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…ote revert, mount cleanup - CLA.md: local override with frontmatter + include-remote (source file has none) - community/code-of-conduct: revert to include-remote (include shortcode can't resolve module-mounted pages) - hugo.yaml: remove CLA.md module mount (local file takes precedence) - Adapt all related files for Hugo module mount compatibility

…emote degradation

…ilures Module mount targets were relative to the content component, whose root is `contentDir: content/en`. Targeting `content/en/<path>` therefore resolved to content/en/en/<path>, publishing every mounted page under /en/ and leaving all 35 existing URLs empty -- while the build still exited 0. Retarget all 37 mounts to `content/<path>`. Add layouts/partials/verify-mounts.html, which fails the build when a URL in the new `expectedPages` site param stops resolving. Hugo silently ignores a mount whose upstream source was renamed or deleted: no warning at any log level, and neither --panicOnWarning nor `hugo mod vendor` detects it. A template assertion is the only mechanism that sets a non-zero exit code. Preserve every URL the site already serves. Upstream renames that the site never adopted are absorbed by file mounts, each paired with an excludeFiles entry on its parent directory mount, since a file mount does not suppress the same file arriving via its parent. This removes all 15 redirect rules the migration would otherwise have introduced. /docs/comms/youtube/ keeps serving youtube-guidelines.md rather than silently switching to the directory README. Restore behaviour lost with hack/gen-content.sh: - page-title.html synthesises a title from the filename when upstream markdown has no front matter. Hugo does not do this, so such pages rendered an empty <h1>, <title>, and breadcrumb. - render-heading.html suppresses body H1s at parse time, replacing a regex over already-rendered HTML. - render-link.html expands relative .md links that point outside mounted content into upstream repository URLs instead of emitting dead relative links. - OWNERS and localised README-*.md files are excluded from directory mounts; the latter were publishing 11 translated cheatsheets as English pages. - Restore /community/mentoring/, which had no mount and whose redirect pointed at an unrelated document. Drop the Go bootstrap. `install-go` was declared .PHONY with no rule, so make reported "Nothing to be done" and the download rule was orphaned -- it never ran. Netlify supplies Go on PATH and honours GO_VERSION natively, caching both the toolchain and the module cache between builds, so the apparatus is redundant as well as dead. GO_VERSION stays in netlify.toml, where it is doing the real work. Also: `hugo mod download` is not a Hugo subcommand, which broke modules-download and container-server; use `hugo mod get`. Remove a duplicate .nvmrc rule that emitted an override warning on every build. Move `hugo mod get -u` from preview-build to production-build so previews build from the pin and predict what merging does, while production stays current.

resources.GetRemote returns the raw file, so .Content is markdown source. Passing it to safeHTML emitted it verbatim: /CLA/ shipped with literal [text](url) link syntax on the page and no rendered anchors at all. Run it through markdownify instead. Strip the leading H1 from the fetched document first. The layout already renders the page title, and markdownify output does not pass through layouts/_default/_markup/render-heading.html, so the body heading would otherwise duplicate it.

…CLA/ Section pages mounted from upstream README.md rendered an empty <h1>. The filename fallback was only wired into content.html, but _index.md pages are served by the docs, community and resources list templates, which emitted a bare {{ .Title }}. Ten pages were affected, among them /resources/github-management/, /community/processes/ and /docs/comms/slack-config/. Stop stripping the leading H1 in include-remote. Its only caller, content/en/CLA.md, is served by Docsy's _default/single.html, which is just {{ .Content }} and renders no title of its own. Note that render-heading.html still suppresses the H1 site-wide, so /CLA/ continues to render without one, matching current behaviour. Scoping the hook to module-mounted paths was tried so that page could keep its heading, and reverted: file mounts land outside the directory targets (/docs/contributor-cheatsheet/, /resources/rename/, /resources/release/) and KEP pages have a separate ingestion path, so the scoped rule left 574 pages with duplicate H1s against 0 for the unscoped one. Giving /CLA/ a heading needs a _default/single.html that renders the title, which is a wider change than this migration should carry.

hack/gen-content.sh wrote synced content directly into content/en/, so the build targets deleted those paths beforehand and .gitignore files kept the output untracked. Hugo Modules mount content virtually and write nothing into content/en/, leaving both mechanisms with nothing to act on. The purge was not merely inert. production-build ran, on every deploy: find content/en/docs/guide -maxdepth 1 -not -name ".gitignore" -exec rm -rf {} \; which would delete any genuine local content added under those paths. clean-all now matches clean; it is kept as an alias so existing muscle memory and any external references keep working. Removing the .gitignore files empties content/en/docs/guide, content/en/docs/orientation and content/en/resources/release, so those directories disappear from the tree. Module mounts do not require the target directory to exist locally: a full build still produces 981 pages with all 35 URLs in expectedPages resolving, /docs/guide/, /docs/orientation/ and /resources/release/ among them.

modules-download is documented as "no update", but bare `hugo mod get` upgrades every module to latest upstream. Since container-server depends on this target, starting the dev container silently rewrote go.mod and go.sum in the developer's working tree -- the opposite of the target's stated purpose, and it defeats the pin that governs local and preview builds. `go mod download` fetches exactly what go.mod specifies and changes nothing. It also populates $(HOME)/go/pkg/mod, which is the directory container-server mounts as GOMODCACHE; `hugo mod` subcommands use Hugo's own cache directory instead, so they were filling the wrong cache. Verified: after `make modules-download`, `git diff go.mod go.sum` is empty and the pinned k8s.io/community revision is present in $(HOME)/go/pkg/mod. modules-update keeps `hugo mod get -u`, which is where upgrading belongs.

The image is built from Alpine with Hugo but no Go toolchain, which was fine while the site had no module imports. This branch adds them, and Hugo shells out to `go` to resolve modules unless they are vendored, so container-server and container-render both failed: Error: command error: failed to load modules: failed to download modules: binary with name "go" not found in PATH Install Go from go.dev, pinned by a GO_VERSION build arg the Makefile reads out of netlify.toml, mirroring how HUGO_VERSION is already handled. That keeps the container on the same toolchain Netlify builds with, as requested in review. Both container-image and the multi-arch docker-push build pass the arg. Verified: `go version` inside the image reports go1.26.4, matching netlify.toml, and the container dev server builds 1055 pages and serves /, /docs/guide/, /community/mentoring/ and /docs/comms/youtube/ with the host's go.mod and go.sum left untouched.

The comments added while working through this migration recorded the investigation as much as the outcome: alternatives that were tried and reverted, page counts from verification runs, and restatements of what the next few lines plainly do. Keep the parts a reader cannot infer from the code -- that Hugo does not derive .Title from a filename, that mount targets are relative to the content component, that a file mount does not suppress its parent directory mount, that bare `hugo mod get` upgrades. Drop the rest. Rendered output is byte-identical to the previous build: 981 pages, no /en/ leak, no duplicate H1s, all 35 expectedPages resolving.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 5, 2026

k8s-ci-robot requested review from SayakMukhopadhyay and reylejano February 5, 2026 06:30

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Feb 5, 2026

TineoC changed the title ~~fix: correct contentDir mapping for root homepage~~ feature: use go modules for content ingestion Feb 5, 2026

TineoC changed the title ~~feature: use go modules for content ingestion~~ feat(build): migrate content ingestion to hugo modules Feb 5, 2026

TineoC mentioned this pull request Feb 13, 2026

Render community groups info using content adapter #654

Merged

lmktfy reviewed Feb 14, 2026

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 25, 2026

TineoC force-pushed the fix/content-ingestion branch from a50e8af to 782f34b Compare March 18, 2026 00:42

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 18, 2026

TineoC force-pushed the fix/content-ingestion branch from 782f34b to 3198581 Compare March 18, 2026 00:46

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Mar 20, 2026

TineoC force-pushed the fix/content-ingestion branch from 3198581 to 05ca5a1 Compare May 2, 2026 07:53

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels May 2, 2026

TineoC force-pushed the fix/content-ingestion branch 2 times, most recently from 0280259 to ae44faa Compare May 2, 2026 08:27

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 2, 2026

TineoC force-pushed the fix/content-ingestion branch from ae44faa to 08b0289 Compare May 2, 2026 08:28

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 2, 2026

TineoC force-pushed the fix/content-ingestion branch 4 times, most recently from 4356001 to f9ec0d8 Compare May 2, 2026 08:47

kubernetes-prow Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 10, 2026

TineoC added 2 commits July 10, 2026 09:36

chore: remove adapt-content.sh, update Makefile deps

eeb4ad4

fix: use include shortcode for module-mounted CoC, graceful include-r…

6a59ebf

…emote degradation

kubernetes-prow Bot added the language/en Issues or PRs related to English language label Aug 5, 2026

TineoC added 2 commits August 4, 2026 21:35

kubernetes-prow Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 5, 2026

TineoC added 4 commits August 4, 2026 21:49

kubernetes-prow Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 5, 2026

Uh oh!

Conversation

TineoC commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Fixes

Uh oh!

lmktfy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TineoC commented Mar 20, 2026

Uh oh!

k8s-ci-robot commented May 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TineoC commented Feb 5, 2026 •

edited

Loading