Skip to content

Commit 3a25a2e

Browse files
Michael-A-KuykendallMichael-A-Kuykendall
authored
chore: consolidate test suite (19 files → 3) + SSE regression test + doc updates (#209)
* chore: rewrite test suite from 19 files to 3, ~2500 lines to ~660 Wipe all existing tests and write fresh ones matching current v2.2.x architecture. Previous test suite had accumulated barnacles across 3+ architecture rewrites (llama -> universal -> airframe). New structure: tests/core.rs 27 tests — CLI, registry, templates, discovery, safetensors, OpenAI/Ollama serde, versions, multi-part content, error format tests/handlers.rs 5 tests — HTTP server health, /v1/models, /v1/chat/completions, /api/tags, concurrency tests/compile_checks.rs 1 test — include_str template file existence Removed (19 files, ~1840 lines): tests/regression/ — 7 issue files + README + index tests/regression_tests.rs — duplicate coverage, stale tests tests/openai_api_real_tests.rs — merged into core + handlers tests/anthropic_api_integration_test.rs — thin translation layer tests/cli_integration_tests.rs — spawned real binary for flag edge cases tests/api_error_handling_test.rs — JSON shape tested in handlers tests/integration_tests.rs — merged into handlers + core tests/template_compilation_regression_test.rs — merged into compile_checks tests/workflow_tests.rs — workflow engine is #[allow(dead_code)] tests/safetensors_integration.rs — tested test helper No subprocess spawning. No cargo build in tests. No Python/MLX. All 646 tests pass locally (0 failures), format clean. Signed-off-by: Michael A. Kuykendall <michaelallenkuykendall@gmail.com> * chore: add Issue #53 SSE regression test, update docs and scripts Adds test_sse_streaming_chunk_format verifying ChatCompletionChunk serialization (initial, token, final chunk shapes). Updates docs/REGRESSION_TESTING.md to reflect the consolidated 3-file structure with full historical issue-to-test mapping table. Updates RELEASE_GATES_CHECKLIST.md, RELEASE_PROCESS.md, and both shell scripts to remove references to deleted regression tests. All 647 tests pass locally (0 failures). Signed-off-by: Michael A. Kuykendall <michaelallenkuykendall@gmail.com> * style: format test_sse_streaming_chunk_format assertion Signed-off-by: Michael A. Kuykendall <michaelallenkuykendall@gmail.com> --------- Signed-off-by: Michael A. Kuykendall <michaelallenkuykendall@gmail.com>
1 parent 85cdc07 commit 3a25a2e

26 files changed

Lines changed: 1185 additions & 3465 deletions

‎RELEASE_GATES_CHECKLIST.md‎

Lines changed: 126 additions & 125 deletions
Original file line numberDiff line numberDiff line change
@@ -1,125 +1,126 @@
1-
# Release Gates & Regression Tests — v2.0
2-
3-
**Date:** May 20, 2026
4-
**Branch:** `main`
5-
**Engine:** Airframe (wgpu/WebGPU — no CUDA, no llama.cpp)
6-
**Status:** Active
7-
8-
---
9-
10-
## Release Gate Pipeline
11-
12-
All 7 gates defined in `.github/workflows/release.yml` under the `preflight` job.
13-
Every gate must pass or the entire release stops.
14-
15-
| Gate | Name | Covers |
16-
|------|------|--------|
17-
| 1/7 | Core Build | `cargo build --features airframe,huggingface` — GPU engine via submodule |
18-
| ~~2~~| ~~CUDA Timeout~~ | **Removed v2.0** — Airframe uses wgpu; no CUDA dependency |
19-
| 3/7 | Template Packaging | Docker templates included in crates.io package (Issue #60 regression) |
20-
| 4/7 | Binary Size | Constitutional 20 MB limit on `shimmy` binary |
21-
| 5/7 | Test Suite | `cargo test --lib --no-default-features --features huggingface` |
22-
| 5.1/7 | Airframe Compile Check | `cargo check --features airframe` — integration builds without errors |
23-
| 5.5/7 | Issue Regression | Per-issue regression tests (see list below) |
24-
| 6/7 | Documentation | `cargo doc --no-deps --features huggingface` |
25-
| 7/7 | crates.io Dry-Run | `cargo publish --dry-run --features huggingface` validates crates.io package |
26-
27-
---
28-
29-
## Regression Test Files
30-
31-
### Active Test Files
32-
| File | Status |
33-
|------|--------|
34-
| `tests/regression_tests.rs` | ✅ Active — main issue regression suite |
35-
| `tests/release_gate_integration.rs` | ✅ Active — validates gate system itself |
36-
| `tests/template_compilation_regression_test.rs` | ✅ Active — template file inclusion |
37-
| `tests/compilation_regression_test.rs` | ✅ Active — compilation sanity |
38-
| `tests/apple_silicon_detection_test.rs` | ✅ Active — GPU detection (Issue #87) |
39-
| `tests/integration_tests.rs` | ✅ Active — API surface integration |
40-
| `tests/cli_integration_tests.rs` | ✅ Active — CLI surface |
41-
| `tests/gpu_backend_tests.rs` | ✅ Active — GPU backend selection |
42-
43-
### Gate 5.5 Issue Regression Tests (in `regression_tests.rs`)
44-
| Test | Issue |
45-
|------|-------|
46-
| `test_issue_111_gpu_metrics_endpoint` | #111 — GPU metrics endpoint |
47-
| `test_issue_112_safetensors_engine_selection` | #112 — SafeTensors engine selection |
48-
| `test_issue_113_openai_api_frontend_compatibility` | #113 — OAI `/v1/models` endpoint |
49-
| `test_issue_113_ollama_api_tags_response_structure` | #113 — Ollama `/api/tags` endpoint |
50-
| `test_issue_114_mlx_distribution_features` | #114 — MLX distribution feature flags |
51-
| `test_issue_191_multi_part_content_array_deserialization` | #191 — Multi-part content array (422 fix) |
52-
| `test_qwen_model_template_detection` | #13 — Qwen ChatML template |
53-
| `test_custom_model_directory_environment_variables` | #12 — Custom model directories |
54-
55-
---
56-
57-
## Test Execution Commands
58-
59-
```bash
60-
# Gate 5 — Core lib tests
61-
cargo test --lib --no-default-features --features huggingface -- --test-threads=1
62-
63-
# Gate 5.1 — Airframe compile check
64-
cargo check --features airframe
65-
66-
# Gate 5.5 — Issue regression tests
67-
cargo test --test regression_tests --no-default-features --features huggingface
68-
69-
# Gate validation system tests
70-
cargo test --test release_gate_integration
71-
72-
# Full regression pass
73-
cargo test --no-default-features --features huggingface
74-
```
75-
76-
---
77-
78-
## Pre-Release Checklist
79-
80-
### Code Quality
81-
- [ ] `cargo fmt -- --check`
82-
- [ ] `cargo clippy --no-default-features --features huggingface` (no warnings)
83-
- [ ] `cargo deny check` (license check)
84-
- [ ] No compilation warnings on `--features airframe,huggingface`
85-
86-
### Documentation
87-
- [ ] `CHANGELOG.md` updated with release version entry
88-
- [ ] `README.md` reflects current feature set
89-
- [ ] New features documented
90-
91-
### Version Management
92-
- [ ] `Cargo.toml` version bumped
93-
- [ ] Git tag created (format: `vX.Y.Z`)
94-
- [ ] Release notes prepared
95-
96-
### Workflow Validation
97-
- [ ] `release.yml` `preflight` job passes on a test branch
98-
- [ ] `ci.yml` `test` job passes on current branch
99-
100-
---
101-
102-
## Feature Flag Reference
103-
104-
| Feature | Purpose | crates.io safe |
105-
|---------|---------|----------------|
106-
| `huggingface` | Model downloading, tokenizer — **default** | ✅ Yes |
107-
| `airframe` | Airframe GPU engine via submodule path dep | ❌ No (path dep) |
108-
| `full` | `huggingface + airframe` convenience alias | ❌ No |
109-
| `apple` | Apple Silicon convenience alias | platform-only |
110-
111-
**crates.io publish** uses `--no-default-features --features huggingface`.
112-
**GitHub Release binaries** use `--features airframe,huggingface` (built in CI matrix).
113-
114-
---
115-
116-
## What's Gone (v2.0)
117-
118-
| Removed | Why |
119-
|---------|-----|
120-
| CUDA Gate (Gate 2) | Airframe uses wgpu; zero CUDA dependency |
121-
| `shimmy-llama-cpp-2` dependency | Replaced by Airframe GPU engine |
122-
| `--features llama` build path | No llama.cpp in this codebase |
123-
| MLX-specific release artifacts | Not part of v2.0 shipping target |
124-
| Binary size inflation from llama.cpp | Airframe is pure Rust; binary stays well under 20 MB |
125-
1+
# Release Gates & Regression Tests — v2.2
2+
3+
**Date:** June 30, 2026
4+
**Branch:** `main`
5+
**Engine:** Airframe (wgpu/WebGPU — no CUDA, no llama.cpp)
6+
**Status:** Active
7+
8+
---
9+
10+
## Release Gate Pipeline
11+
12+
All 7 gates defined in `.github/workflows/release.yml` under the `preflight` job.
13+
Every gate must pass or the entire release stops.
14+
15+
| Gate | Name | Covers |
16+
|------|------|--------|
17+
| 1/7 | Core Build | `cargo build --features airframe,huggingface` — GPU engine via submodule |
18+
| ~~2~~| ~~CUDA Timeout~~ | **Removed v2.0** — Airframe uses wgpu; no CUDA dependency |
19+
| 3/7 | Template Packaging | Docker templates included in crates.io package (Issue #60 regression) |
20+
| 4/7 | Binary Size | Constitutional 20 MB limit on `shimmy` binary |
21+
| 5/7 | Test Suite | `cargo test --lib --no-default-features --features huggingface` |
22+
| 5.1/7 | Airframe Compile Check | `cargo check --features airframe` — integration builds without errors |
23+
| 5.5/7 | Regression Tests | `cargo test --test core --test handlers --test compile_checks` |
24+
| 6/7 | Documentation | `cargo doc --no-deps --features huggingface` |
25+
| 7/7 | crates.io Dry-Run | `cargo publish --dry-run --features huggingface` validates crates.io package |
26+
27+
---
28+
29+
## Regression Test Files (v2.2+)
30+
31+
### Active Test Files
32+
| File | Status | Tests |
33+
|------|--------|-------|
34+
| `tests/core.rs` | ✅ Active — CLI, registry, templates, serde, SSE, discovery | 28 |
35+
| `tests/handlers.rs` | ✅ Active — HTTP endpoints (health, models, chat, tags, concurrency) | 5 |
36+
| `tests/compile_checks.rs` | ✅ Active — template file inclusion via include_str! | 1 |
37+
38+
### Retired Test Files
39+
The following files were consumed into the consolidated suite above:
40+
41+
| File | Absorbed By |
42+
|------|-------------|
43+
| `tests/regression_tests.rs` | `tests/core.rs` + `tests/handlers.rs` |
44+
| `tests/release_gate_integration.rs` | `cargo test --lib` (CI already passes) |
45+
| `tests/template_compilation_regression_test.rs` | `tests/compile_checks.rs` |
46+
| `tests/compilation_regression_test.rs` | `tests/compile_checks.rs` + `cargo check` |
47+
| `tests/apple_silicon_detection_test.rs` | — (MLX is a stub) |
48+
| `tests/integration_tests.rs` | `tests/handlers.rs` |
49+
| `tests/cli_integration_tests.rs` | `tests/core.rs` |
50+
| `tests/gpu_backend_tests.rs` | — (Airframe handles GPU) |
51+
| `tests/anthropic_api_integration_test.rs` | — (Anthropic layer removed) |
52+
| `tests/api_error_handling_test.rs` | `tests/core.rs` + `tests/handlers.rs` |
53+
| `tests/openai_api_real_tests.rs` | `tests/core.rs` + `tests/handlers.rs` |
54+
| `tests/safetensors_integration.rs` | `tests/core.rs` |
55+
| `tests/workflow_tests.rs` | — (dead_code module) |
56+
| `tests/regression/` (all 7 files) | `tests/core.rs` |
57+
58+
---
59+
60+
## Test Execution Commands
61+
62+
```bash
63+
# Gate 5 — Core lib tests
64+
cargo test --lib --no-default-features --features huggingface -- --test-threads=1
65+
66+
# Gate 5.1 — Airframe compile check
67+
cargo check --features airframe
68+
69+
# Gate 5.5 — Regression tests
70+
cargo test --test core --test handlers --test compile_checks --features airframe,huggingface
71+
72+
# Full regression pass
73+
cargo test --features airframe,huggingface
74+
```
75+
76+
---
77+
78+
## Pre-Release Checklist
79+
80+
### Code Quality
81+
- [ ] `cargo fmt -- --check`
82+
- [ ] `cargo clippy --no-default-features --features huggingface` (no warnings)
83+
- [ ] `cargo deny check` (license check)
84+
- [ ] No compilation warnings on `--features airframe,huggingface`
85+
86+
### Documentation
87+
- [ ] `CHANGELOG.md` updated with release version entry
88+
- [ ] `README.md` reflects current feature set
89+
- [ ] New features documented
90+
91+
### Version Management
92+
- [ ] `Cargo.toml` version bumped
93+
- [ ] Git tag created (format: `vX.Y.Z`)
94+
- [ ] Release notes prepared
95+
96+
### Workflow Validation
97+
- [ ] `release.yml` `preflight` job passes on a test branch
98+
- [ ] `ci.yml` `test` job passes on current branch
99+
100+
---
101+
102+
## Feature Flag Reference
103+
104+
| Feature | Purpose | crates.io safe |
105+
|---------|---------|----------------|
106+
| `huggingface` | Model downloading, tokenizer — **default** | ✅ Yes |
107+
| `airframe` | Airframe GPU engine via submodule path dep | ❌ No (path dep) |
108+
| `full` | `huggingface + airframe` convenience alias | ❌ No |
109+
| `apple` | Apple Silicon convenience alias | platform-only |
110+
111+
**crates.io publish** uses `--no-default-features --features huggingface`.
112+
**GitHub Release binaries** use `--features airframe,huggingface` (built in CI matrix).
113+
114+
---
115+
116+
## What's Gone (v2.0 → v2.2)
117+
118+
| Removed | Why |
119+
|---------|-----|
120+
| CUDA Gate (Gate 2) | Airframe uses wgpu; zero CUDA dependency |
121+
| `shimmy-llama-cpp-2` dependency | Replaced by Airframe GPU engine |
122+
| `--features llama` build path | No llama.cpp in this codebase |
123+
| MLX-specific release artifacts | Not part of v2.0 shipping target |
124+
| Binary size inflation from llama.cpp | Airframe is pure Rust; binary stays well under 20 MB |
125+
| ~2900 lines of tests in 19 files | Consolidated to ~660 lines across 3 files |
126+
| Per-issue-file regression structure | Replaced by architecturally-organized test files |

‎RELEASE_PROCESS.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ This document describes the release process for Shimmy v2.0+ (Airframe GPU engin
1616
| 4/7 — Binary Size | `shimmy` binary stays under 20 MB constitutional limit |
1717
| 5/7 — Test Suite | `cargo test --lib --no-default-features --features huggingface` passes |
1818
| 5.1/7 — Airframe Compile | `cargo check --features airframe` — GPU engine integration compiles cleanly |
19-
| 5.5/7 — Issue Regression | Per-issue tests in `tests/regression_tests.rs` all pass |
19+
| 5.5/7 — Issue Regression | `cargo test --test core --test handlers --test compile_checks` all pass (see `docs/REGRESSION_TESTING.md` for issue-to-test mapping) |
2020
| 6/7 — Documentation | `cargo doc` builds without errors |
2121
| 7/7 — crates.io Dry-Run | `cargo publish --dry-run` validates the crates.io-safe feature set |
2222

@@ -37,8 +37,8 @@ cargo test --no-default-features --features huggingface
3737
# Airframe compile check
3838
cargo check --features airframe
3939

40-
# Issue regression tests
41-
cargo test --test regression_tests --no-default-features --features huggingface
40+
# Regression tests (consolidated suite)
41+
cargo test --test core --test handlers --test compile_checks --features airframe,huggingface
4242
```
4343

4444
### 2. Dry-run on the test branch (exact CI environment)
@@ -84,8 +84,8 @@ git branch -d test-release-vX.Y.Z
8484
- The Airframe submodule at `airframe/` has a compilation error
8585
- `cargo check --features airframe` locally to see the error
8686

87-
### Gate 5.5 (Issue Regression) fails
88-
- A specific issue regression test failed; the gate output names the test and issue number
87+
### Gate 5.5 (Regression Tests) fails
88+
- A specific regression test failed; check `docs/REGRESSION_TESTING.md` for the issue mapping
8989
- Fix the regression before releasing
9090

9191
### Gate 7 (crates.io Dry-Run) fails

0 commit comments

Comments
 (0)