Skip to content

Commit 3829b0d

Browse files
shiptuxshiptux
authored
Add official Debian packaging support (flagos-ai#355)
* Add official Debian packaging support This commit adds comprehensive Debian packaging infrastructure to enable building .deb packages for both NVIDIA and MetaX backends. Directory structure follows Debian UpstreamGuide best practices by using packaging/debian/ instead of top-level /debian to avoid conflicts with distribution maintainers and allow future expansion (e.g., RPM support). Changes: - Add packaging/debian/ with hand-crafted packaging metadata - Support multi-backend builds via build profiles (nvidia-only/metax-only) - Add containerized build scripts for reproducible builds - Add GitHub Actions workflow for automated package building - Support automated releases when tags are pushed The packaging uses debhelper 13 and supports: - NVIDIA backend: Requires CUDA toolkit, built in NVIDIA CUDA container - MetaX backend: Uses public MetaX APT repository, no auth required Build artifacts: - libflagcx-nvidia / libflagcx-nvidia-dev (NVIDIA packages) - libflagcx-metax / libflagcx-metax-dev (MetaX packages) Manual packaging chosen over automated tools (CMake cpack, debmake) to ensure Debian Policy compliance and support complex build requirements. References: - Debian UpstreamGuide: https://wiki.debian.org/UpstreamGuide - Neovim BUILD.md: https://github.com/neovim/neovim/blob/master/BUILD.md - Miniflux packaging structure: https://github.com/miniflux/v2/tree/main/packaging Signoff: Cheng Li <shiptux@gmail.com> * feat(ci): add Nexus package upload and unify Debian builds Introduce a new `upload-nexus.yml` workflow to automatically upload built Debian packages to a Nexus APT repository. This workflow triggers on new `v*` tags or can be manually dispatched, downloading artifacts from the `build-deb.yml` run. Refactor `build-deb.yml` by consolidating the `build-nvidia` and `build-metax` jobs into a single `build-packages` job using a matrix strategy. The `release` job for GitHub releases has been removed, as package distribution is now handled via Nexus. Signed-off-by: Cheng Li <shiptux@gmail.com> * ci(build-deb): update CI runner and ignore package artifacts The `.github/workflows/build-deb.yml` workflow has been updated to use the `h20` runner, likely a custom or self-hosted environment, instead of `ubuntu-22.04`. Additionally, the `debian-packages` directory, which contains the generated Debian package build output, has been added to `.gitignore` to prevent it from being tracked. * refactor(packaging): unify Debian package build with single Dockerfile This commit refactors the Debian packaging system to use a single, unified Dockerfile for building packages across different hardware backends (NVIDIA, MetaX). Key changes include: - Replaced `Dockerfile.nvidia-deb` and `Dockerfile.metax-deb` with a single `Dockerfile.deb` to reduce duplication and improve maintainability. - The `build-flagcx.sh` script now dynamically configures the unified Dockerfile using build arguments (`BASE_IMAGE`, `VENDOR`). - `dpkg-buildpackage` leverages Debian build profiles (`pkg.flagcx.${VENDOR}-only`) to ensure backend-specific packages are built correctly within the unified process. - Updated CI workflows (`build-deb.yml`, `upload-nexus.yml`): - Reduced artifact retention for build packages to 1 day. - Updated GitHub Action runner and `action-download-artifact` version. - Refactored the Nexus upload logic for better generalization, logging, and error handling. - Added a new `test-nexus-upload.sh` script for local verification of Nexus APT repository uploads. - Updated `packaging/README.md` to reflect the new build instructions, architecture, and installation steps. - Updated maintainer information and project URLs in `changelog`, `control`, and `copyright` files. Signed-off-by: Cheng Li <shiptux@gmail.com> * chore(build): relocate lintian checks to local build script Move `lintian` package validation from the GitHub Actions workflow to the `build-flagcx.sh` script. This change ensures `lintian` checks are performed consistently during local package builds and in CI, as the build script now handles the validation directly. The `build-deb.yml` workflow has been updated to remove the redundant `lintian` step. `packaging/README.md` has been updated to document the new behavior, including instructions for installing `lintian` and clarifying that its checks are non-fatal. Signed-off-by: Cheng Li <shiptux@gmail.com> * ci(build-deb): optimize workflow triggers to include source code changes Add 'flagcx/**' to both push and pull_request path filters to ensure builds are triggered when FlagCX source code changes, not just when packaging files are modified. This prevents: - Missing builds when source code is updated on main branch - Unnecessary builds when only documentation is changed in PRs Both push and pull_request now use consistent path filtering: - flagcx/** (source code) - packaging/debian/** (packaging files) - .github/workflows/build-deb.yml (workflow itself) Signed-off-by: Cheng Li <shiptux@gmail.com> * ci(build-deb): trigger workflow on version tags --------- Signed-off-by: Cheng Li <shiptux@gmail.com>
1 parent 3fbc9c9 commit 3829b0d

12 files changed

Lines changed: 904 additions & 0 deletions

File tree

‎.github/workflows/build-deb.yml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
name: Build Debian Packages
2+
3+
on:
4+
push:
5+
tags:
6+
- 'v*'
7+
pull_request:
8+
branches: [ main ]
9+
paths:
10+
- 'flagcx/**'
11+
- 'packaging/debian/**'
12+
- '.github/workflows/build-deb.yml'
13+
workflow_dispatch:
14+
15+
jobs:
16+
build-packages:
17+
runs-on: h20
18+
19+
strategy:
20+
fail-fast: false
21+
matrix:
22+
backend: [nvidia, metax]
23+
24+
steps:
25+
- name: Checkout repository
26+
uses: actions/checkout@v4
27+
with:
28+
submodules: recursive
29+
30+
- name: Set up Docker Buildx
31+
uses: docker/setup-buildx-action@v3
32+
33+
- name: Build ${{ matrix.backend }} packages
34+
run: ./packaging/debian/build-helpers/build-flagcx.sh ${{ matrix.backend }}
35+
36+
- name: Upload ${{ matrix.backend }} packages
37+
uses: actions/upload-artifact@v4
38+
with:
39+
name: flagcx-${{ matrix.backend }}-packages
40+
path: debian-packages/${{ matrix.backend }}/*.deb
41+
retention-days: 1

‎.github/workflows/upload-nexus.yml‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
name: Upload to Nexus Repository
2+
3+
on:
4+
push:
5+
tags:
6+
- 'v*'
7+
workflow_dispatch:
8+
inputs:
9+
run_id:
10+
description: 'Build workflow run ID (optional, uses latest if not specified)'
11+
required: false
12+
13+
jobs:
14+
upload:
15+
runs-on: h20
16+
17+
steps:
18+
- name: Download build artifacts
19+
uses: dawidd6/action-download-artifact@v12
20+
with:
21+
github_token: ${{ secrets.GITHUB_TOKEN }}
22+
workflow: build-deb.yml
23+
run_id: ${{ github.event.inputs.run_id }}
24+
workflow_conclusion: success
25+
path: packages/
26+
27+
- name: List downloaded packages
28+
run: |
29+
echo "Downloaded packages:"
30+
find packages/ -name "*.deb" -exec ls -lh {} \;
31+
32+
- name: Upload packages to Nexus APT repository
33+
env:
34+
NEXUS_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
35+
NEXUS_PASSWORD: ${{ secrets.CONTAINER_REGISTRY }}
36+
NEXUS_REPO_URL: https://resource.flagos.net/repository/flagos-apt-hosted
37+
run: |
38+
set -e
39+
40+
# Function to upload a deb package to Nexus APT hosted repository
41+
upload_deb() {
42+
local deb_file="$1"
43+
local backend="$2"
44+
local filename=$(basename "$deb_file")
45+
46+
# Extract package metadata for logging
47+
local package=$(dpkg-deb -f "$deb_file" Package)
48+
local version=$(dpkg-deb -f "$deb_file" Version)
49+
local arch=$(dpkg-deb -f "$deb_file" Architecture)
50+
51+
echo "Uploading: $filename"
52+
echo " Package: $package"
53+
echo " Version: $version"
54+
echo " Architecture: $arch"
55+
echo " Backend: $backend"
56+
57+
# Upload to Nexus APT hosted repository
58+
# Nexus APT hosted repositories accept uploads at the root level
59+
local upload_url="${NEXUS_REPO_URL}/${filename}"
60+
61+
if curl -f -u "${NEXUS_USERNAME}:${NEXUS_PASSWORD}" \
62+
--upload-file "$deb_file" \
63+
"$upload_url"; then
64+
echo "✓ Successfully uploaded $filename"
65+
else
66+
echo "✗ Failed to upload $filename"
67+
return 1
68+
fi
69+
}
70+
71+
# Upload packages for all vendors
72+
for vendor in nvidia metax; do
73+
vendor_dir="packages/flagcx-${vendor}-packages"
74+
75+
if [ -d "$vendor_dir" ]; then
76+
echo ""
77+
echo "=== Uploading ${vendor} packages ==="
78+
79+
uploaded=0
80+
for deb in "$vendor_dir"/*.deb; do
81+
[ -f "$deb" ] || continue
82+
if upload_deb "$deb" "$vendor"; then
83+
((uploaded++))
84+
fi
85+
done
86+
87+
if [ $uploaded -gt 0 ]; then
88+
echo "✓ Uploaded $uploaded ${vendor} package(s)"
89+
else
90+
echo "⚠ No ${vendor} packages found to upload"
91+
fi
92+
else
93+
echo "⚠ Skipping ${vendor}: directory not found"
94+
fi
95+
done
96+
97+
- name: Summary
98+
run: |
99+
echo ""
100+
echo "✅ Upload completed!"
101+
echo ""
102+
echo "Packages uploaded to Nexus APT repository:"
103+
find packages/ -name "*.deb" -exec basename {} \;
104+
echo ""
105+
echo "To use the repository, add to /etc/apt/sources.list.d/flagcx.list:"
106+
echo "deb https://resource.flagos.net/repository/flagos-apt-hosted/ flagos-apt-hosted main"

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
build
33
plugin/*/build
44
test/*/build
5+
debian-packages
56

67
# Ignore compiled Python files and shared object files
78
plugin/*/*.so

‎packaging/README.md‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
# FlagCX Packaging
2+
3+
This directory contains packaging configurations for various Linux distributions.
4+
5+
## Directory Structure
6+
7+
```
8+
packaging/
9+
├── debian/ # Debian/Ubuntu packaging
10+
│ ├── control # Package metadata (with build profiles)
11+
│ ├── rules # Build rules
12+
│ ├── changelog # Version history
13+
│ ├── copyright # License information
14+
│ └── build-helpers/ # Build scripts and Dockerfiles
15+
│ ├── build-flagcx.sh # Unified build script
16+
│ ├── Dockerfile.deb # Unified build configuration
17+
│ └── test-nexus-upload.sh # Local Nexus upload test script
18+
└── rpm/ # Future: RPM packaging for RHEL/Fedora/etc.
19+
```
20+
21+
## Why `packaging/` Instead of Top-Level `/debian`?
22+
23+
Following [Debian UpstreamGuide](https://wiki.debian.org/UpstreamGuide) recommendations:
24+
25+
> Upstream projects should NOT include a top-level `/debian` directory.
26+
> Use `contrib/debian/` or `packaging/debian/` instead.
27+
28+
**Benefits:**
29+
- Avoids conflicts with distribution maintainers' packaging
30+
- Clearly indicates upstream-maintained packaging
31+
- Allows multi-format support (Debian + RPM + others)
32+
- Industry standard (see [Miniflux](https://github.com/miniflux/v2/tree/main/packaging), etc.)
33+
34+
## Building Debian Packages
35+
36+
Use the unified build script to build packages for any vendor/backend:
37+
38+
### Usage
39+
40+
```bash
41+
./packaging/debian/build-helpers/build-flagcx.sh <vendor> [base_image_version]
42+
```
43+
44+
**Parameters:**
45+
- `<vendor>` - Hardware vendor/backend (e.g., `nvidia`, `metax`)
46+
- `[base_image_version]` - Optional base image version tag (default: `latest`)
47+
48+
**Output:** `debian-packages/<vendor>/*.deb`
49+
50+
### Examples
51+
52+
**Build for NVIDIA:**
53+
```bash
54+
./packaging/debian/build-helpers/build-flagcx.sh nvidia
55+
# Output: debian-packages/nvidia/*.deb
56+
```
57+
58+
**Build for MetaX:**
59+
```bash
60+
./packaging/debian/build-helpers/build-flagcx.sh metax
61+
# Output: debian-packages/metax/*.deb
62+
```
63+
64+
**Specify custom base image version:**
65+
```bash
66+
./packaging/debian/build-helpers/build-flagcx.sh nvidia v1.2.3
67+
./packaging/debian/build-helpers/build-flagcx.sh metax latest
68+
```
69+
70+
### Base Images
71+
72+
The build script uses upstream base images from `harbor.baai.ac.cn/flagbase/`:
73+
- NVIDIA: `flagbase-nvidia:<version>`
74+
- MetaX: `flagbase-metax:<version>`
75+
76+
To add support for a new vendor, ensure a corresponding base image exists at:
77+
`harbor.baai.ac.cn/flagbase/flagbase-<vendor>:<version>`
78+
79+
### Quality Checks
80+
81+
The build script automatically runs `lintian` to validate the generated packages if available:
82+
83+
```bash
84+
# Install lintian (optional but recommended)
85+
sudo apt-get install lintian
86+
87+
# Build packages - lintian runs automatically
88+
./packaging/debian/build-helpers/build-flagcx.sh nvidia
89+
```
90+
91+
Lintian checks are non-fatal and won't stop the build if issues are found.
92+
93+
## Installation
94+
95+
Install packages for your hardware vendor:
96+
97+
```bash
98+
# General syntax
99+
sudo dpkg -i debian-packages/<vendor>/*.deb
100+
101+
# Example: NVIDIA
102+
sudo dpkg -i debian-packages/nvidia/*.deb
103+
104+
# Example: MetaX
105+
sudo dpkg -i debian-packages/metax/*.deb
106+
```
107+
108+
## CI/CD
109+
110+
Automated builds are triggered by:
111+
- Push to `main` branch (when packaging files change)
112+
- Pull requests to `main`
113+
- Manual workflow dispatch
114+
115+
See `.github/workflows/build-deb.yml` for details.
116+
117+
### Publishing to Nexus APT Repository
118+
119+
Packages are uploaded to Nexus when:
120+
- A version tag is pushed: `git tag v1.0.0 && git push origin v1.0.0`
121+
- Manual workflow dispatch via GitHub Actions
122+
123+
After upload, users can install packages from the APT repository:
124+
125+
```bash
126+
# Add the FlagOS APT repository
127+
echo "deb https://resource.flagos.net/repository/flagos-apt-hosted/ flagos-apt-hosted main" | \
128+
sudo tee /etc/apt/sources.list.d/flagcx.list
129+
130+
# Update package list
131+
sudo apt-get update
132+
133+
# Install packages for your vendor
134+
sudo apt-get install libflagcx-<vendor> # Runtime library
135+
sudo apt-get install libflagcx-<vendor>-dev # Development files
136+
137+
# Examples:
138+
sudo apt-get install libflagcx-nvidia libflagcx-nvidia-dev
139+
sudo apt-get install libflagcx-metax libflagcx-metax-dev
140+
```
141+
142+
## Architecture
143+
144+
The build process uses a **unified multi-stage Dockerfile** with build profiles:
145+
146+
### Build Profiles Support
147+
148+
The `debian/control` file defines build profiles to support multiple backends:
149+
- `pkg.flagcx.nvidia-only` - Build only NVIDIA packages
150+
- `pkg.flagcx.metax-only` - Build only MetaX packages
151+
152+
### Unified Dockerfile
153+
154+
A single `Dockerfile.deb` builds packages for all backends using build arguments:
155+
- `BASE_IMAGE` - Upstream base image (e.g., `flagbase-nvidia`, `flagbase-metax`)
156+
- `BASE_IMAGE_VERSION` - Image version tag (default: `latest`)
157+
- `VENDOR` - Backend vendor name (used for build profile selection)
158+
159+
### Build Stages
160+
161+
1. **Builder stage**: Based on upstream flagbase images
162+
- Contains all necessary build dependencies (CUDA/NCCL or MACA SDK)
163+
- Installs Debian packaging tools (`debhelper`, `dpkg-dev`, etc.)
164+
- Runs `dpkg-buildpackage` with `DEB_BUILD_PROFILES=pkg.flagcx.${VENDOR}-only`
165+
- Only builds packages for the specified vendor
166+
167+
2. **Output stage**: Minimal Alpine image
168+
- Only contains the built `.deb` files
169+
- Used to extract packages to the host
170+
171+
This approach ensures:
172+
- ✓ Reproducible builds using official base images
173+
- ✓ Single Dockerfile for all backends (DRY principle)
174+
- ✓ Backend selection via build profiles
175+
- ✓ No custom Docker images to maintain
176+
- ✓ Clean separation of build environment and outputs
177+
178+
## Future Plans
179+
180+
- [ ] Add RPM packaging in `packaging/rpm/`
181+
- [ ] Add Arch Linux packaging
182+
- [x] Add APT repository hosting (Nexus)

‎packaging/debian/build-helpers/Dockerfile.deb‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Multi-stage Dockerfile to build Debian packages for FlagCX
2+
# Supports multiple backends via build arguments
3+
4+
ARG BASE_IMAGE
5+
ARG BASE_IMAGE_VERSION=latest
6+
7+
FROM ${BASE_IMAGE}:${BASE_IMAGE_VERSION} as builder
8+
9+
ARG VENDOR
10+
ENV DEBIAN_FRONTEND=noninteractive
11+
12+
# Install Debian packaging tools and build dependencies
13+
RUN apt-get update && apt-get install -y \
14+
debhelper \
15+
devscripts \
16+
dpkg-dev \
17+
fakeroot \
18+
lsb-release \
19+
chrpath \
20+
patchelf \
21+
nlohmann-json3-dev \
22+
&& apt-get clean \
23+
&& rm -rf /var/lib/apt/lists/*
24+
25+
# Copy FlagCX source
26+
COPY . /workspace/FlagCX
27+
WORKDIR /workspace/FlagCX
28+
29+
# Set backend environment
30+
ENV FLAGCX_BUILD_BACKEND=${VENDOR}
31+
32+
# Copy packaging/debian to debian/ for dpkg-buildpackage
33+
RUN if [ -d "packaging/debian" ]; then \
34+
cp -r packaging/debian debian; \
35+
fi
36+
37+
# Build Debian packages with backend-specific profile
38+
RUN DEB_BUILD_PROFILES="pkg.flagcx.${VENDOR}-only" \
39+
dpkg-buildpackage -us -uc -b -Pnocheck,pkg.flagcx.${VENDOR}-only || \
40+
{ echo "Build failed, checking logs..."; \
41+
find /workspace -name "*.log" -exec echo "=== {} ===" \; -exec cat {} \; 2>/dev/null || true; \
42+
exit 1; }
43+
44+
# Collect built .deb files
45+
RUN mkdir -p /output && \
46+
find /workspace -maxdepth 1 -name "*.deb" -exec cp {} /output/ \; && \
47+
ls -lh /output/
48+
49+
# Output stage: minimal image with only the .deb files
50+
FROM alpine:latest as output
51+
COPY --from=builder /output/*.deb /output/

0 commit comments

Comments
 (0)