Skip to content

Commit 3f1d21c

Browse files
AlexStocksOmX
AlexStocks
and
OmX
authored
feat(runtime-baseline): add strict single-case threshold checker (#392)
* feat(runtime-baseline): add frozen threshold module + verifier (#390) Establish the threshold-freezing mechanism that serves as the #351 quantitative acceptance gate. This PR delivers the methodology plus placeholder constants, not fabricated numbers: - thresholds.rs: #351 acceptance threshold constants as placeholder sentinels (f64::NAN / usize::MAX / u64::MAX / i64::MAX, TODO(#351)) + all_frozen() - verify.rs: Outcome comparator returning Publishable / NonPublishable / Fail. While thresholds are still placeholders, any case is NonPublishable — this is the hard guarantee that no placeholder value is mistaken for a pass. (Design spec forbids freezing Windows / hosted-runner numbers.) - schema/outcome.schema.json: outcome structure expected by the verifier - docs/performance/storage-runtime-baseline.md: methodology + the Linux-only backfill plan Real values are filled on a fixed Linux/WSL host per a versioned cases.yaml (Baseline Results PR); only then does the verifier flip to Publishable. The observer wiring (Tasks 6-7) and the real-number matrix are separate Linux-host follow-ups. Refs #350 #378 #351. * fix(runtime-baseline): make single-case thresholds fail closed Replace incomplete run-level publishability with a strict single-case checker. Validate frozen policy values and case-specific inputs before evaluating every declared threshold, while leaving manifest conservation, repetition, CV, and provenance to the later controller. Constraint: A single-case Pass is not a publishable baseline-run decision. Confidence: High; boundary tests cover each threshold and isolated mutations prove the resume-drain and PING checks are observable. Scope-risk: Limited to runtime-baseline threshold inputs, schema, fixture assembly, license configuration, and baseline documentation. Tested: WSL cargo test -p runtime-baseline --all-targets; WSL cargo clippy -p runtime-baseline --all-targets -- -D warnings -D clippy::unwrap_used; cargo fmt --all -- --check; SkyWalking Eyes header check; git diff --check. Not-tested: Fixed-host runtime baseline matrix remains deferred until real thresholds and the controller verifier exist. Co-authored-by: OmX <omx@oh-my-codex.dev> --------- Co-authored-by: OmX <omx@oh-my-codex.dev>
1 parent 9102a6d commit 3f1d21c

7 files changed

Lines changed: 1126 additions & 2 deletions

File tree

‎.licenserc.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ header:
3939
4040
paths-ignore:
4141
- '**/*.md'
42-
- '*.json'
42+
- '**/*.json'
4343
- 'LICENSE'
4444
- 'NOTICE'
4545
- 'Cargo.lock'

‎docs/performance/storage-runtime-baseline.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Storage Runtime Baseline Harness
2+
3+
本目录与 `tools/runtime-baseline` 一起，提供 kiwi storage 执行链路的性能与背压基线
4+
（对应 epic #347 → #350 → #351 → #352）。
5+
6+
## 设计约束（必须守）
7+
8+
- **冻结阈值只能来自固定 Linux/WSL 主机。** 设计规格明确禁止用 Windows / GitHub
9+
hosted runner 的跑分数字冻结阈值——矩阵的可重现性依赖固定的内核、文件系统与
10+
CPU 拓扑。当前 hosted CI 只编译 harness 并执行代码检查；完整矩阵未来只能通过
11+
`workflow_dispatch` 在固定环境运行，或由维护者在规定的固定 Linux/WSL 主机执行。
12+
- **不在基线未稳前伪造数字。** `tools/runtime-baseline/src/thresholds.rs` 中的常量
13+
当前为占位哨兵值（`f64::NAN` / `usize::MAX` / `u64::MAX` / `i64::MAX`），并标注
14+
`TODO(#351)`。只要任一仍为占位值，`thresholds::all_frozen()` 返回 `false`，
15+
`verify_outcome` 会据此返回 `PolicyUnavailable`，杜绝「用占位值假装通过」。
16+
17+
## 本次 PR 的准备工作（Refs #390）
18+
19+
建立 #351 量化验收门禁的**阈值冻结机制**本身：
20+
21+
| 文件 | 作用 |
22+
|------|------|
23+
| `tools/runtime-baseline/src/thresholds.rs` | #351 验收所需的冻结基线阈值常量 + `all_frozen()` 哨兵检测 |
24+
| `tools/runtime-baseline/src/verify.rs` | 严格的单 case threshold checker；返回 `Pass` / `PolicyUnavailable` / `Fail` |
25+
| `tools/runtime-baseline/schema/outcome.schema.json` | 单 case threshold input 的 JSON Schema 2020-12 合同 |
26+
| 本文件 | 方法论与回填流程说明 |
27+
28+
单 case 的 `Pass` 只证明该 case 满足冻结阈值，不代表完整 baseline run 可发布。最终发布
29+
结论还必须由 controller verifier 核对 manifest case 集守恒、重复轮次、CV、来源身份和
30+
完整 run outcome。占位状态下 checker 只返回 `PolicyUnavailable`。
31+
32+
## 为什么 observer 接线与真实数字是 follow-up
33+
34+
- **Wiring & Smoke（Tasks 6–12）**：把 `baseline` observer 状态机接入 `runtime` crate 的
35+
`manager` / `message` / `storage_server` 生产路径，并实现 harness 运行时、控制面、Python
36+
控制器与 `cases.yaml`。这部分改动进入 `src/common/runtime/`，会拉入 `storage` → `rocksdb`，
37+
必须在能编译/运行 RocksDB 的环境验证——因此作为独立 PR 在 Linux 主机推进。
38+
- **Baseline Results（真实数字冻结）**：在固定 Linux/WSL 主机按版本化 `cases.yaml` 跑完整
39+
矩阵后，把真实数值回填进 `thresholds.rs`，再由完整 verifier 聚合全部 case、稳定性和
40+
provenance 证据。只有原始结果、报告和真实阈值满足 #390 Acceptance 后才关闭 #390。
41+
42+
## 未来回填流程
43+
44+
1. 固定 Linux/WSL 主机，按版本化 `cases.yaml` 跑完整矩阵（基础吞吐 + value 变体 + 单因素扫描
45+
+ offered-load 4 档 + lifecycle 3 类；性能 case 重复 5 次、lifecycle case 重复 3 次，
46+
CV<=5%）。
47+
2. 将实测值回填 `thresholds.rs`（替换 `TODO(#351)` 占位）。
48+
3. 由后续 `run_baseline.py verify` 对每个 case 执行 threshold check，并核对完整 case 集、
49+
重复轮次、CV 和来源身份后产生 run 级发布结论。
50+
4. 提交原始 JSON/CSV + 环境清单（内核、CPU、文件系统）+ 摘要；冻结 #351 量化验收阈值。
51+
5. 满足 #390 Acceptance 后关闭 #390，并在 #350 / #351 中引用结果。
52+
53+
## 本地验证（不依赖 RocksDB）
54+
55+
```bash
56+
cargo check -p runtime-baseline
57+
cargo test -p runtime-baseline
58+
cargo clippy -p runtime-baseline --all-targets -- -D warnings -D clippy::unwrap_used
59+
```

‎tools/runtime-baseline/schema/outcome.schema.json‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
{
2+
"$schema": "https://json-schema.org/draft/2020-12/schema",
3+
"$id": "https://arana-db.github.io/kiwi/schemas/runtime-baseline/threshold-outcome.schema.json",
4+
"title": "RuntimeBaselineThresholdOutcome",
5+
"description": "Strict input for one case threshold check. A passing case is not a publishable baseline run.",
6+
"type": "object",
7+
"additionalProperties": false,
8+
"required": [
9+
"case_id",
10+
"qps",
11+
"p99_ms",
12+
"max_latency_ms",
13+
"error_rate",
14+
"queued_plus_running_max",
15+
"cpu_change_ratio",
16+
"rss_change_ratio",
17+
"threads_delta",
18+
"case"
19+
],
20+
"properties": {
21+
"case_id": {
22+
"type": "string",
23+
"minLength": 1,
24+
"pattern": ".*\\S.*"
25+
},
26+
"qps": {
27+
"type": "number",
28+
"minimum": 0
29+
},
30+
"p99_ms": {
31+
"type": "number",
32+
"minimum": 0
33+
},
34+
"max_latency_ms": {
35+
"type": "number",
36+
"minimum": 0
37+
},
38+
"error_rate": {
39+
"type": "number",
40+
"minimum": 0,
41+
"maximum": 1
42+
},
43+
"queued_plus_running_max": {
44+
"type": "integer",
45+
"minimum": 0
46+
},
47+
"cpu_change_ratio": {
48+
"type": "number",
49+
"minimum": -1
50+
},
51+
"rss_change_ratio": {
52+
"type": "number",
53+
"minimum": -1
54+
},
55+
"threads_delta": {
56+
"type": "integer"
57+
},
58+
"case": {
59+
"oneOf": [
60+
{
61+
"type": "object",
62+
"additionalProperties": false,
63+
"required": [
64+
"kind",
65+
"anchor_qps",
66+
"anchor_p99_ms",
67+
"anchor_max_latency_ms"
68+
],
69+
"properties": {
70+
"kind": {
71+
"const": "normal_load"
72+
},
73+
"anchor_qps": {
74+
"type": "number",
75+
"exclusiveMinimum": 0
76+
},
77+
"anchor_p99_ms": {
78+
"type": "number",
79+
"exclusiveMinimum": 0
80+
},
81+
"anchor_max_latency_ms": {
82+
"type": "number",
83+
"exclusiveMinimum": 0
84+
}
85+
}
86+
},
87+
{
88+
"type": "object",
89+
"additionalProperties": false,
90+
"required": [
91+
"kind",
92+
"error_kinds"
93+
],
94+
"properties": {
95+
"kind": {
96+
"const": "overload_120pct"
97+
},
98+
"error_kinds": {
99+
"type": "array",
100+
"uniqueItems": true,
101+
"items": {
102+
"type": "string",
103+
"minLength": 1,
104+
"pattern": "^\\S(?:.*\\S)?$"
105+
}
106+
}
107+
}
108+
},
109+
{
110+
"type": "object",
111+
"additionalProperties": false,
112+
"required": [
113+
"kind",
114+
"pause_ms",
115+
"resume_drain_ms"
116+
],
117+
"properties": {
118+
"kind": {
119+
"const": "storage_gate"
120+
},
121+
"pause_ms": {
122+
"type": "integer",
123+
"minimum": 0
124+
},
125+
"resume_drain_ms": {
126+
"type": "integer",
127+
"minimum": 0
128+
}
129+
}
130+
},
131+
{
132+
"type": "object",
133+
"additionalProperties": false,
134+
"required": [
135+
"kind",
136+
"drain_ms"
137+
],
138+
"properties": {
139+
"kind": {
140+
"const": "shutdown"
141+
},
142+
"drain_ms": {
143+
"type": "integer",
144+
"minimum": 0
145+
}
146+
}
147+
},
148+
{
149+
"type": "object",
150+
"additionalProperties": false,
151+
"required": [
152+
"kind",
153+
"new_conn_ms",
154+
"ping_ms"
155+
],
156+
"properties": {
157+
"kind": {
158+
"const": "slow_storage"
159+
},
160+
"new_conn_ms": {
161+
"type": "integer",
162+
"minimum": 0
163+
},
164+
"ping_ms": {
165+
"type": "integer",
166+
"minimum": 0
167+
}
168+
}
169+
}
170+
]
171+
}
172+
}
173+
}

‎tools/runtime-baseline/src/lib.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,5 @@
1818
pub mod cli;
1919
pub mod schema;
2020
pub mod startup;
21+
pub mod thresholds;
22+
pub mod verify;

0 commit comments

Comments
 (0)