Contributing to NVIDIA AI Cluster Runtime (AICR)

We welcome contributions from developers of all backgrounds and experience levels.

Code of Conduct

This project adopts the Contributor Covenant v2.1. Please be respectful and professional in all interactions. See CODE_OF_CONDUCT.md for details.

Getting Started

Before contributing:

Read the README.md to understand the project
Check existing issues to avoid duplicates
Review the security policy for security-related contributions
Set up your development environment following DEVELOPMENT.md
If using coding assistants, review AGENTS.md for project rules and workflows

How to Contribute

Reporting Bugs

Use the bug report template
Describe the issue clearly with steps to reproduce
Include system information (OS, Go version, Kubernetes version)
Attach logs or screenshots if applicable
Check if the issue already exists before creating a new one

Suggesting Enhancements

Use the feature request template
Clearly describe the proposed feature and its use case
Explain how it benefits the project and users
Provide examples or mockups if applicable

Improving Documentation

Fix typos, clarify instructions, or add examples
Update README.md for user-facing changes
Update API documentation when endpoints change
Ensure code comments are accurate and helpful

Contributing Code

Fix bugs, add features, or improve performance
Follow the development workflow in DEVELOPMENT.md
Ensure all tests pass and code meets quality standards
Write tests for new functionality

Go dependencies (vendor)

This project vendors Go dependencies. After changing go.mod or go.sum, run make tidy (which runs go mod vendor) and commit go.mod, go.sum, and the vendor/ directory. CI will fail if vendor/ is out of sync.

Adding Validation Constraints

AICR uses a validator framework to check cluster state against requirements. To add new validation constraints:

Quick Start:

# Generate all necessary files
make generate-validator ARGS="--constraint Deployment.my-app.version --phase deployment --description 'Validates my-app version'"

This creates three files with TODOs guiding implementation:

Helper functions with validation logic
Unit tests with table-driven test cases
Integration test with automatic registration

Next Steps:

Implement the TODOs in generated files
Add comprehensive test cases
Run make test - registration validation ensures completeness
Submit PR - CI enforces all requirements

See docs/contributor/validator.md for complete guide with examples, architecture overview, and troubleshooting.

Adding a Component

AICR components are declarative — add an entry to recipes/registry.yaml with Helm or Kustomize settings, create a values.yaml, and optionally add a health check. No Go code needed.

Validate your component:

make build
make component-test COMPONENT=my-component

This auto-detects the right test tier, creates a Kind cluster, deploys the component, and runs its health check. See tools/component-test/README.md for details.

Design Principles

These principles guide all design decisions in AICR. When faced with trade-offs, these principles take precedence.

Local Development Equals CI

The same tools, same versions, and same validation run locally and in CI.

What: Tool versions are centralized in .settings.yaml. make tools-setup (first-time install), make tools-update (upgrade to current pins), make tools-check (verify), and GitHub Actions all use this single source of truth. make qualify runs the exact same checks as CI.

Why: "Works on my machine" is not acceptable. If a contributor can run make qualify locally and it passes, CI will pass. This eliminates surprise failures and reduces feedback loops. Note that this only holds when your local toolchain matches .settings.yaml — run make tools-update after a git pull that touches .settings.yaml, or whenever make tools-check shows a ⚠ for a lint-sensitive tool. A behind-CI golangci-lint will silently miss lint findings that CI catches.

Adoption Comes from Idiomatic Experience

The system integrates into how users already work. We provide validated configuration, not a new operational model.

What: AICR outputs standard formats (Helm values, Kubernetes manifests) that work with existing tools (kubectl, Argo CD, Flux). Users don't need to learn "the AICR way" of deploying.

Why: If adoption requires retraining users on a new workflow, our design has failed. Value comes from correctness, not from lock-in.

Correctness Must Be Reproducible

Given the same inputs, the same system version must always produce the same result (e.g. recipe, bundle artifacts).

What: No hidden state, no implicit defaults, no non-deterministic behavior. A recipe/bundle/image digest generated using the same version of aicr today must be identical to one generated tomorrow.

Why: Reproducibility is a prerequisite for debugging, validation, and trust. If users can't reproduce a result, they can't trust it.

Metadata Is Separate from Consumption

Validated configuration exists independent of how it is rendered, packaged, or deployed.

What: Recipes define what is correct. Bundlers and deployers determine how to deliver it (Helm, Argo CD, raw manifests). The recipe doesn't change based on the deployment mechanism.

Why: This prevents tight coupling of correctness to a specific tool, workflow, or delivery mechanism. Users can adopt new deployment tools without re-validating their configurations.

Recipe Specialization Requires Explicit Intent

More specific recipes are never matched unless explicitly requested. Generic intent cannot silently resolve to specialized configurations.

What: If a user requests a "training" recipe, they get the training configuration. The system never silently upgrades to a more specific variant (e.g., "training-distributed-horovod") without explicit opt-in.

Why: This prevents accidental misconfiguration and preserves user control. Surprises in infrastructure configuration are dangerous.

Trust Requires Verifiable Provenance

Trust is established through evidence, not assertions. Every released artifact carries verifiable proof of origin and build process.

What: Releases include SLSA Build Level 3 provenance for container images (via the reusable attestation workflow, since v0.17.0) and SLSA Build Provenance v1 for CLI binaries, plus SBOM attestations and Sigstore signatures. Users can verify exactly which commit, workflow, and build produced any artifact.

Why: This underpins supply-chain security, compliance, and confidence. "Trust us" is not a security model.

Pull Request Process

Before Submitting

Ensure all checks pass:
```
make qualify
```
Update documentation if needed:
- README.md for user-facing changes
- DEVELOPMENT.md for developer workflow changes
- Code comments and godoc for API changes
Sign and sign off every commit: all contributors use git commit -s -S — -s adds the DCO sign-off, -S cryptographically signs the commit. See Developer Certificate of Origin for one-time setup and what it certifies.

Creating the Pull Request

Push your branch and open a PR against main
Fill out the PR template completely:
- Summary: Brief description of changes
- Type of Change: Bug fix, feature, breaking change, etc.
- Testing: What testing was performed
- Checklist: Verify all items
Do not use the issue priority labels P0, P1, or P2 on PRs. They are reserved for issues and are automatically removed from pull requests by automation.

Review Process

Automated Checks run via GitHub Actions — the same gate make qualify runs locally. See Full Qualification and the Make Targets Reference in the development guide.
Maintainer Review covers:
- Correctness and functionality
- Code style and Go idioms
- Test coverage and quality
- Documentation completeness
Address Feedback by pushing new commits (signed and signed off, same as every commit):
```
git commit -s -S -m "address review: improve error handling"
git push origin your-branch
```
Append commits rather than amending or rebasing while the PR is open for review. The exceptions are narrow: a catch-up rebase onto main that the merge gate requires when your branch falls behind, a missing signature or sign-off, a wrong base branch, or a committed secret. Say on the PR whenever you do any of them. A force-push outdates every inline comment and drops the anchors reviewers left, which makes "was this addressed?" a manual diff. There is no need to tidy the history first: pull requests merge by squash, so the branch's commits become one commit on main regardless. Keep the PR in draft while you are still reshaping it — draft PRs do not page reviewers, and that is the phase where rewriting history is free.
Merge: Once approved and CI passes, a maintainer will merge

AI-Assisted Contributions Policy

We welcome the use of AI tools (e.g., GitHub Copilot, ChatGPT, Claude) to help you write code, brainstorm, or refactor. However, we maintain a strict human-in-the-loop policy for all submissions:

Full accountability: By submitting a PR, you (the human author) accept full responsibility for the code — its correctness, security, maintainability, and license compliance. "The AI wrote it" is not an acceptable explanation for bugs or security flaws.
Understand what you submit: Do not submit AI-generated code you do not fully understand. Reviewers expect you to explain and defend every line of code in your PR.

Issue and PR Lifecycle

Automated bots manage the lifecycle of issues and pull requests:

When	Action
On open	`needs-triage` label added to issues
14 days of PR inactivity	PR author receives a reminder comment (once per PR)
30 days of PR inactivity	PR marked `lifecycle/stale`
30 days after being marked stale	Stale PR auto-closed
90 days of issue inactivity	Issue marked `lifecycle/stale`
30 days after being marked stale	Stale issue auto-closed
90 days of inactivity while closed	Issue/PR thread locked

These are inactivity windows, not days since opening: any comment or update restarts the applicable counter — including the day-14 reminder itself, which is a comment and so pushes the stale mark out.

The day-14 reminder is the exception: it is posted at most once per pull request. The workflow skips any PR that already carries a reminder, so replying and later going quiet again does not produce a second nudge — the next automated action on that PR is the day-30 stale mark.

Each bot runs on a daily schedule, so an action lands on its next scheduled run rather than the moment a threshold is crossed — expect up to ~24 hours of lag.

To stay out of the stale process entirely: Add the lifecycle/frozen label. Exempt items are never marked lifecycle/stale in the first place, not merely spared from closing. good first issue also exempts issues, and do-not-merge exempts pull requests.

Claiming an Issue

To pick up an unassigned issue, comment /assign and a bot assigns it to you (or /assign @user to assign someone else). Issues use a single-owner model: if one is already assigned, the bot refuses and asks the current assignee to release it first with /unassign. Comment /unassign to release an issue assigned to you — it only ever removes your own claim. GitHub only allows assigning users with triage/write access or prior activity in the repository; the bot comments if it cannot assign a requested user.

After Merging

# Update your local repository
git checkout main
git pull upstream main

# Delete your feature branch
git branch -d your-branch
git push origin --delete your-branch

Developer Certificate of Origin

Every commit — from every contributor — must be both signed off and cryptographically signed:

git commit -s -S -m "Your commit message"

-s (lowercase) adds a Signed-off-by line, certifying the Developer Certificate of Origin 1.1 below.
-S (uppercase) attaches a GPG or SSH signature, proving the commit came from you.

The two are independent — use both, every time. A branch ruleset enforces Require signed commits on every branch, so a push containing an unsigned (-S-less) commit is rejected before review. The Signed-off-by line looks like:

Signed-off-by: Jane Developer <jane@example.com>

One-Time Setup

# Identity used in the Signed-off-by line (must match your signing key)
git config user.name "Your Name"
git config user.email "your.email@example.com"

# Sign every commit by default, so you only need -s going forward
git config commit.gpgsign true

You also need a signing key registered with GitHub. Follow GitHub's guide to generate a GPG or SSH signing key and add it to your account, then point git at it (git config user.signingkey <key>).

Forgot to Sign or Sign Off?

Fix the most recent commit and re-push:

git commit --amend -s -S --no-edit
git push --force-with-lease --force-if-includes origin your-branch

For an entire branch, re-sign every commit at once:

git rebase --exec 'git commit --amend -s -S --no-edit' origin/main
git push --force-with-lease --force-if-includes origin your-branch

--force-if-includes checks your local branch reflog, so in a fresh clone it can reject the push even when nothing is wrong — the clone's only reflog entry is the clone itself, which is not a valid rewrite base. It fails safe. Fetching does not help, because what is missing is a local reflog entry rather than remote data; push once with a pinned lease instead. Read the remote's current tip first and check it is the commit you meant to replace, then pass that recorded value explicitly:

git ls-remote origin your-branch          # note the SHA, and confirm it is yours
git push --force-with-lease=your-branch:<that-sha> origin your-branch

Do not inline the lookup into the push (--force-with-lease=your-branch:$(git ls-remote …)). That re-reads the remote at push time, so a commit someone else pushed in the meantime becomes the expected value and is silently overwritten — the same failure the pinned lease exists to prevent.

If the PR is already under review, say on the PR that you force-pushed and name the old and new SHA, since re-signing rewrites every commit and outdates the inline comments.

What You're Certifying

By signing off, you certify the Developer Certificate of Origin 1.1:

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

Tips for Contributors

First-Time Contributors

Recommended starting points:

Start with issues labeled good first issue
Read existing code in the package you're modifying before writing
Run make tools-check to verify your environment; run make tools-update if any tool is behind
Study the Design Principles section

Good first contributions:

Documentation improvements (typos, clarifications)
Adding test cases to existing tests
Improving error messages with better context

Writing Good Commit Messages

Short summary (50 chars or less)

More detailed explanation if needed. Wrap at 72 characters.
Explain the problem being solved and why this approach was chosen.

- Bullet points are fine
- Use present tense ("Add feature" not "Added feature")
- Reference issues: "Fixes #123" or "Related to #456"

Signed-off-by: Your Name <your@email.com>

Where this text ends up. Pull requests merge by squash, and the repo composes the merged commit from the PR title with an empty body, so a branch commit's body is discarded at merge — only the title and trailers reach main. Write the body for your reviewers, and put anything that needs to outlive the PR in the PR title and description.

PR titles are linted. Because the title becomes the whole commit message on main and cannot be corrected after merge, CI checks it against Conventional Commits format:

type: subject            # scope optional
type(scope): subject
type!: subject           # "!" marks a breaking change
type(scope)!: subject

Valid types: build, chore, ci, docs, feat, fix, perf, refactor, revert, style, test. A malformed title fails the check; editing the title re-runs it automatically, so no new commit is needed. Titles over 70 characters produce a warning but do not block: dependency-bot titles embed pseudo-versions that cannot be shortened. Titles of 70 characters or fewer pass without a warning.

Code Style

Follow existing patterns in the codebase
Use pkg/errors for error handling (not fmt.Errorf)
Always check ctx.Done() in loops and long operations
Write table-driven tests for multiple test cases
Use functional options for configuration

Getting Help

GitHub Issues: Create an issue with the "question" label
Existing Issues: Search for similar questions first
Recent PRs: Look at merged PRs for examples

Additional Resources

DEVELOPMENT.md - Development setup, architecture, and tooling
README.md - Project overview and quick start
docs/README.md - System overview and glossary
docs/contributor/index.md - Architecture documentation

FilesExpand file tree

CONTRIBUTING.md

Latest commit

History