Skip to content

Commit 24ce43b

Browse files
committed
Merge remote-tracking branch 'origin/main' into HEAD
# Conflicts: # src/resp/src/encode.rs # src/storage/src/redis_strings.rs
2 parents 36b8e12 + 0c4795e commit 24ce43b

60 files changed

Lines changed: 2987 additions & 1442 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/ci.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -223,6 +223,10 @@ jobs:
223223
max_attempts: 2
224224
command: bash -o pipefail -c "make test 2>&1 | tee cargo-test.log"
225225

226+
- name: Test storage stats with fault injection
227+
if: matrix.os == 'ubuntu-latest'
228+
run: cargo test -p storage --features test-fault-injection --test storage_stats_test
229+
226230
- name: Test runtime baseline tool
227231
run: cargo test -p runtime-baseline --all-targets
228232

.planning/KANBAN.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
# Kiwi Kanban
22

3-
> 更新日期:2026-07-30
3+
> 更新日期:2026-07-31
44
>
55
> 当前里程碑:M1 Redis 8.8.1 Cache OFF compatibility foundation
66
>
7-
> 当前 task:PR `#388` implementation(`feat/rocksdb-build-accel-and-prd`);已验证远端 Base `0f8d96238860a5c29a5582e461e4cdeb974431b3`、Head `b47cb1eebe098e2d4d2d784020dc283a8d026d28`
7+
> 当前 task:PR `#404` integration(远端 `codex/fix-resp-parser-limits`,本地 v4);最新基线 `main` at `ed49ab4c3c362ba77111dbcd9791f93ebfce74a4`
88
>
99
> 当前运行模式:Cache OFF
1010
>
@@ -14,7 +14,7 @@
1414

1515
| ID | 工作项 | Requirement / Gate 关系 | 当前状态 |
1616
|---|---|---|---|
17-
| `PR388-001` | C/C++ sccache 构建接入、跨平台回归探针、PRD/用户故事及 review 一致性修复 | `REQ-STABILITY-003``REQ-WORK-001``REQ-WORK-003``REQ-WORK-005`;跨平台构建修复只维护 G6 CI 的可执行性与证据条件,不构成 Gate 通过证据 | 本地针对性验证通过并形成提交;远端发布状态和新 Head CI 结果以 PR/checks/threads 实时查询为准 |
17+
| `RESP-LIMITS-001` | 在已合并 #406 的零声明驱动分配合同上,完成 #404 的有界 RESP 解析、未认证 buffer、parser 历史副本和 optional pipeline 背压 | `REQ-COMPAT-002``REQ-COMPAT-006``REQ-STABILITY-002``REQ-STABILITY-003``REQ-WORK-003` | #402#403#405#406 已合并;v4 已解决最新 base 的 parser/pipeline 冲突,并通过 Windows/WSL 最终验证;当前发布、CI 与 review 状态从 PR #404 实时查询 |
1818

1919
## Ready for a separate implementation task
2020

@@ -37,6 +37,7 @@
3737
| ID | 工作项 | Evidence | 状态 |
3838
|---|---|---|---|
3939
| `M1-001-T1` | Redis 8.8.1 exact compatibility manifest | PR `#372`;final Head `6a692bc195f96327296296977a100af301deaf01`;merge commit `9e91707d774ad367d682e23677dcef79ecb14338` | 已合并到 `main` |
40+
| `PR388-001` | C/C++ sccache 构建接入、跨平台回归探针、PRD/用户故事及 review 一致性修复 | PR `#388`;final Head `1ee8c916a55d03d02a250ed95af83712fa14a742`;2026-07-30 merged | 已合并到 `main` |
4041

4142
## Frozen unaccepted drafts
4243

@@ -89,7 +90,7 @@
8990

9091
## WIP 与授权规则
9192

92-
- 同一时间只允许一个 implementation card 处于 In Progress;当前为 `PR388-001`,planning/docs task 不得隐式持有 implementation card。
93+
- 同一时间只允许一个 implementation card 处于 In Progress;当前为 `RESP-LIMITS-001`,planning/docs task 不得隐式持有 implementation card。
9394
- 规划批准不授权修改 source、tests、build scripts 或 CI,也不授权 stage/commit/push 实现文件。
9495
- 从 planning 转 implementation 必须创建新 Codex task、TaskId、worktree、dirty allowlist 和 recovery checkpoint;不能只把 recovery mode 从 `planning` 改成 `implementation`
9596
- 冻结草稿不得覆盖、续写、清理或回退;后续实施只读参考并重新审计。

.planning/STATE.md

Lines changed: 60 additions & 27 deletions
Large diffs are not rendered by default.

CHANGELOG.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -20,15 +20,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2020
- Recovery management for degraded storage states
2121
- Comprehensive health monitoring and statistics
2222
- Backpressure handling to prevent overload
23-
- Architecture documentation in `docs/ARCHITECTURE.md`
23+
- [Architecture documentation](docs/architecture/)
2424
- Runtime configuration options for thread pool sizing
2525
- Independent health checks for network and storage runtimes
2626

2727
### Changed
2828

2929
- **BREAKING CHANGE**: Default port changed from `9221` to `7379` for Redis compatibility
3030
- The configuration system has been migrated to Redis-style format (similar to `redis.conf`)
31-
- Configuration file moved from INI format to Redis-style format at `src/conf/kiwi.conf`
31+
- Configuration file moved from INI format to Redis-style format; generate it with `kiwi --sample-config > kiwi.conf` (`kiwi --full-sample-config > kiwi.conf` for all keys), then start Kiwi with `kiwi --config kiwi.conf`
3232
- Default port is now `7379` (a Redis-compatible variant of the standard `6379` port)
3333
- **BREAKING CHANGE**: Storage initialization now requires `RuntimeManager::initialize_storage_components()`
3434
- Must be called after `RuntimeManager::start()` and before starting storage server
@@ -50,11 +50,11 @@ If you are upgrading from a previous version that used port `9221`:
5050
- Old format: `config.ini` with `port=9221`
5151
- New format: `kiwi.conf` with `port 7379`
5252

53-
4. **Custom port configuration**: If you need to use a different port, update the `port` setting in `src/conf/kiwi.conf`:
54-
```conf
55-
# Accept connections on the specified port, default is 7379.
56-
# port 7379
57-
port YOUR_CUSTOM_PORT
53+
4. **Custom port configuration**: If you need to use a different port, redirect the generated configuration to a file, update its `port` setting, and start Kiwi with that file:
54+
```bash
55+
kiwi --sample-config > kiwi.conf
56+
# Edit kiwi.conf and set: port YOUR_CUSTOM_PORT
57+
kiwi --config kiwi.conf
5858
```
5959

6060
5. **Docker/Container deployments**: Update any port mappings from `9221` to `7379`
@@ -102,7 +102,7 @@ If you have custom code that integrates with Kiwi:
102102
- `channel_buffer_size` - Message channel buffer size
103103
- `request_timeout` - Default request timeout
104104

105-
See `docs/ARCHITECTURE.md` for detailed architecture documentation.
105+
See the [architecture documentation](docs/architecture/) for details.
106106

107107
### Fixed
108108

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ fmt-check:
5151
@cargo fmt --manifest-path ./Cargo.toml --all -- --check
5252

5353
lint:
54-
@cargo clippy --manifest-path ./Cargo.toml --all-features --workspace -- -D warnings -D clippy::unwrap_used
54+
@cargo clippy --manifest-path ./Cargo.toml --all-features --workspace --all-targets -- -D warnings -D clippy::unwrap_used
5555

5656
standalone:
5757
@$(DEV) build

README.md

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -78,13 +78,19 @@ the project in addition to `protoc`. See the platform-specific commands in the
7878
# Install rustup; rust-toolchain.toml selects Rust 1.97.1 stable in this repo
7979
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
8080

81-
# protobuf compiler (macOS)
82-
brew install protobuf
81+
# macOS: Xcode CLT + protobuf compiler + CMake (required to build RocksDB from source)
82+
xcode-select --install
83+
brew install protobuf cmake
8384

84-
# protobuf compiler (Linux)
85-
apt install protobuf-compiler
85+
# Debian/Ubuntu Linux: native C/C++ toolchain + protobuf compiler
86+
sudo apt install clang cmake libclang-dev llvm-dev pkg-config protobuf-compiler
8687
```
8788

89+
On Windows, use the same official Protobuf 27.1 release as CI. Download and
90+
extract `protoc-27.1-win64.zip`, add its `bin` directory to `PATH`, and run
91+
`protoc --version`. See the [development guide](docs/development.md#prerequisites)
92+
for the PowerShell commands.
93+
8894
Dated nightly toolchains are reserved for specialized checks such as
8995
Sanitizers. They do not define the normal development or release baseline.
9096

@@ -130,8 +136,13 @@ See [docs/cluster.md](docs/cluster.md) for the manual step-by-step procedure and
130136
| [System stability gate](docs/quality/system-stability-gate.md) | Required evidence before deferred hot-tier work can be reconsidered |
131137
| [Deferred native ABI contract](docs/architecture/redis-hot-tier-native-abi.md) | Future interface design; not an implementation authorization |
132138
| [Combined distribution licensing](docs/architecture/combined-distribution-licensing.md) | Future Redis-derived library and source-distribution obligations |
133-
| `kiwi --sample-config` | Generate a default config file |
134-
| `kiwi --full-sample-config` | Generate a config with all available keys |
139+
| [Product requirements (PRD)](docs/prd.md) | Goals, scope, and Redis 8.8.1 compatibility target |
140+
| [Engineering quality gates](docs/quality/quality-gates.md) | Code, test, and release quality gates |
141+
| [Personas and user stories](docs/personas-and-user-stories.md) | Target users and usage scenarios |
142+
| [Documentation index](docs/INDEX.md) | Map of the whole `docs/` tree and suggested reading order |
143+
| [Design plans & specs](docs/superpowers/) | Dated design records (plans + specs) |
144+
| `kiwi --sample-config > kiwi.conf` | Write the default configuration to `kiwi.conf` |
145+
| `kiwi --full-sample-config > kiwi.conf` | Write all available configuration keys to `kiwi.conf` |
135146

136147
## Dependencies
137148

README_CN.md

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,8 +137,13 @@ redis-cli -p 7379 get foo # "bar"
137137
| [系统稳定性门禁](docs/quality/system-stability-gate.md) | 重新评估延期热层工作前的 required 证据 |
138138
| [延期的 Native ABI 合同](docs/architecture/redis-hot-tier-native-abi.md) | 未来接口设计,不构成实现授权 |
139139
| [组合发行许可证设计](docs/architecture/combined-distribution-licensing.md) | 未来 Redis 派生库和源码发行义务 |
140-
| `kiwi --sample-config` | 生成默认配置文件 |
141-
| `kiwi --full-sample-config` | 生成包含所有配置项的完整配置文件 |
140+
| [产品需求文档 (PRD)](docs/prd.md) | 目标、范围与 Redis 8.8.1 兼容基线 |
141+
| [工程质量门禁](docs/quality/quality-gates.md) | 代码、测试与发布质量门禁 |
142+
| [人物角色与用户故事](docs/personas-and-user-stories.md) | 目标用户与使用场景 |
143+
| [文档索引](docs/INDEX.md) | 整个 `docs/` 树的地图与建议阅读顺序 |
144+
| [设计计划与规格](docs/superpowers/) | 带日期的设计记录(plans + specs) |
145+
| `kiwi --sample-config > kiwi.conf` | 将默认配置写入 `kiwi.conf` |
146+
| `kiwi --full-sample-config > kiwi.conf` | 将包含所有配置项的完整配置写入 `kiwi.conf` |
142147

143148
## 依赖说明
144149

docs/INDEX.md

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Documentation Index
2+
3+
A map of the Kiwi documentation tree and a suggested reading order. If you are not
4+
sure where to start, follow the **Newcomer** path below.
5+
6+
## Top-level entry points
7+
8+
| Document | Audience | What it covers |
9+
|----------|----------|----------------|
10+
| [`README.md`](../README.md) / [`README_CN.md`](../README_CN.md) | Everyone | Project intro, features, quickstart, toolchain, and documentation entry points |
11+
| [`CLAUDE.md`](../CLAUDE.md) (a.k.a. `AGENTS.md`) | Contributors / AI agents | Dev commands, crate layout, code style, behavioral guidelines |
12+
| [`CONTRIBUTING.md`](../CONTRIBUTING.md) | Contributors | How to contribute |
13+
| [`CHANGELOG.md`](../CHANGELOG.md) | Everyone | Notable changes (history) |
14+
15+
## Planning & requirements (source of truth for roadmap)
16+
17+
The `.planning/` directory holds the north star, requirements, milestones, current
18+
state, and Kanban. These files are referenced from the README and the PRD and are
19+
the authoritative tracking surface:
20+
21+
- `.planning/PROJECT.md` — constitution / north star
22+
- `.planning/REQUIREMENTS.md` — acceptance requirements (`REQ-*`)
23+
- `.planning/ROADMAP.md` — milestones (`M0``M10`)
24+
- `.planning/STATE.md` — current state
25+
- `.planning/KANBAN.md` — work board
26+
- `.planning/DECISIONS.md` / `.planning/OPEN_QUESTIONS.md` / `.planning/REFERENCES.md`
27+
28+
## Developer docs
29+
30+
| Document | What it covers |
31+
|----------|----------------|
32+
| [`docs/development.md`](development.md) | Dev environment, build optimization, sccache, lint rules, testing (incl. Python integration) |
33+
| [`docs/cluster.md`](cluster.md) | Raft cluster quickstart and write-path verification |
34+
| [`docs/key-encoding.md`](key-encoding.md) | Key/value encoding internals (Chinese, with diagrams) |
35+
36+
## Product & compatibility
37+
38+
| Document | What it covers |
39+
|----------|----------------|
40+
| [`docs/prd.md`](prd.md) | Product requirements; goals, scope, Redis 8.8.1 compatibility target |
41+
| [`docs/personas-and-user-stories.md`](personas-and-user-stories.md) | Target users and usage scenarios |
42+
| [`docs/compatibility/redis-8.8.1.md`](compatibility/redis-8.8.1.md) | Exact Oracle, raw RESP, TCL, and client-test boundaries |
43+
| [`docs/compatibility/redisraft-public-profile.md`](compatibility/redisraft-public-profile.md) | RedisRaft public compatibility profile |
44+
45+
## Architecture
46+
47+
| Document | What it covers |
48+
|----------|----------------|
49+
| [`docs/architecture/redis-8.8.1-system-boundaries.md`](architecture/redis-8.8.1-system-boundaries.md) | Cache OFF request, storage, and consensus boundaries |
50+
| [`docs/architecture/redis-hot-tier-native-abi.md`](architecture/redis-hot-tier-native-abi.md) | Deferred native ABI contract (future design, not an implementation authorization) |
51+
| [`docs/architecture/combined-distribution-licensing.md`](architecture/combined-distribution-licensing.md) | Future Redis-derived library and source-distribution licensing obligations |
52+
53+
## Quality & stability
54+
55+
| Document | What it covers |
56+
|----------|----------------|
57+
| [`docs/quality/system-stability-gate.md`](quality/system-stability-gate.md) | Required evidence (G1–G7) before deferred hot-tier work is reconsidered |
58+
| [`docs/quality/quality-gates.md`](quality/quality-gates.md) | Code, test, and release quality gates |
59+
| [`docs/performance/storage-runtime-baseline.md`](performance/storage-runtime-baseline.md) | Storage-runtime performance baseline |
60+
61+
## Design history (`docs/superpowers/`)
62+
63+
`docs/superpowers/plans/` and `docs/superpowers/specs/` contain **dated design
64+
records** (filename prefix `YYYY-MM-DD-`). They are historical decision logs, not
65+
a second roadmap. Follow related specs and plans by topic and by their explicit
66+
cross-references; dates do not define a one-to-one pairing. For the current
67+
authoritative plan, prefer `.planning/ROADMAP.md` and the gate documents above.
68+
69+
## Suggested reading order
70+
71+
- **Newcomer**: `README.md``docs/development.md``docs/cluster.md``docs/key-encoding.md`
72+
- **Contributor**: `CLAUDE.md``docs/development.md``docs/quality/quality-gates.md`
73+
- **Architecture / compatibility**: `docs/prd.md``docs/compatibility/redis-8.8.1.md``docs/architecture/redis-8.8.1-system-boundaries.md`
74+
- **Stability & quality**: `docs/quality/system-stability-gate.md``docs/quality/quality-gates.md``docs/performance/storage-runtime-baseline.md`
75+
- **Design history**: `docs/superpowers/plans/` + `docs/superpowers/specs/` (follow by topic and explicit cross-references)

docs/development.md

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -172,17 +172,24 @@ cargo test --package storage # tests for a specific crate
172172
cargo test test_redis_mset # run a single test by name
173173
```
174174

175-
Python integration tests require a running Kiwi server:
175+
The Python integration runner starts an isolated Kiwi server on `127.0.0.1:6379`,
176+
runs the test suite against it, and stops the server afterward:
176177

177178
```bash
178-
# Terminal 1
179-
cargo run --bin kiwi
179+
# Build the debug binary used by the runner and install the test dependencies
180+
cargo build --bin kiwi
181+
make -C tests install-deps
180182

181-
# Terminal 2
182-
pip install redis pytest
183-
pytest tests/python/ -v
183+
# Start the isolated server and run the Python integration suite
184+
./tests/run_python_integration.sh
184185
```
185186

187+
`make -C tests install-deps` installs the version-constrained dependencies from
188+
`tests/python/requirements.txt`. The runner sets `KIWI_TEST_REQUIRE_SERVER=1` and
189+
`KIWI_TEST_ISOLATED_SERVER=1` so tests cannot silently pass by skipping server-
190+
dependent cases. See [tests/README.md](../tests/README.md) and
191+
[tests/NEW_TESTS_GUIDE.md](../tests/NEW_TESTS_GUIDE.md) for details.
192+
186193
Storage tests use `tempfile::tempdir()` for isolated RocksDB instances.
187194

188195
## Gotchas

docs/key-encoding.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,13 @@
77
## String 结构的存储
88

99

10-
![img.png](images/01-string编码.png)
10+
![String 编码结构](images/01-string编码.png)
1111

1212
## Hash 结构的存储
1313

1414
Hash 类型数据结构由两部分构成,元数据(meta_key,meta_value)和普通数据(data_key,data_value)。每个 Hash 类型数据对应一条元数据,每个 field 对应一条普通数据。具体格式如下图所示。
1515

16-
![img.png](images/01-hash编码.png)
16+
![Hash 编码结构](images/01-hash编码.png)
1717

1818
元数据中的 key 由前缀保留字段,编码后的 user key 以及后缀保留字段构成,value 中记录了 hash 中元素个数,最新版本号,保留字段,数据写入时间以及数据过期时间。version 字段使用 u64 little-endian 编码,并用于实现秒删功能。
1919

@@ -29,13 +29,13 @@ Hash 类型数据结构由两部分构成,元数据(meta_key,meta_value)
2929

3030
set 结构与 hash 类型的存储格式基本相同,也是由元数据和普通数据两部分构成。不同的是,set 类型由于没有 value字段,所以其 data value 中只需要记录保留字段和数据写入时间即可。具体格式如下所示:
3131

32-
![img.png](images/01-set编码.png)
32+
![Set 编码结构](images/01-set编码.png)
3333

3434
## List 结构的存储
3535

3636
list 由两部分构成,元数据(meta_key, meta_value), 和普通数据(data_key, data_value)。 元数据中存储的主要是 list 链表的一些信息, 比如说当前 list 链表结点的的数量以及当前 list 链表的版本号和过期时间(用做秒删功能), 还有当前 list 链表的左右边界, 普通数据实际上就是指的 list 中每一个结点中的数据,作为具体最后 RocksDB 落盘的 KV 格式,具体格式如下所示:
3737

38-
![img.png](images/01-list编码.png)
38+
![List 编码结构](images/01-list编码.png)
3939

4040
元数据中记录了一个 list 的信息,包括 list 元素个数,左右边界 Index,最新 version以及数据写入时间和过期时间。普通数据的 key 拼接了 list key 和 Index,value 记录用户写入数据以及写入时间。
4141

@@ -47,7 +47,7 @@ list 由两部分构成,元数据(meta_key, meta_value), 和普通数据(data_
4747

4848
zset 由两部分构成,元数据(meta_key, meta_value) 和普通数据(data_key, data_value)。元数据中存储的主要是 zset 集合的一些信息,比如当前 zset 集合中 member 的数量以及当前 zset 集合的版本号和过期时间(用于秒删功能),而普通数据就是指 zset 中每个 member 以及对应的 score。由于 zset 这种数据结构比较特殊,需要按照 member 排序,也需要按照 score 排序,所以我们会按不同格式存储两份普通数据,在这里称为 member to score 和 score to member。作为最终 RocksDB 落盘的 KV 格式,具体如下:
4949

50-
![img.png](images/01-zset编码.png)
50+
![ZSet 编码结构](images/01-zset编码.png)
5151

5252
Meta KV 记录的是一个 zset 的元信息,包括集合元素个数,最新版本号,数据写入时间以及数据过期时间。对 zset 类型数据的读写删除操作都需要先获取元数据。
5353

0 commit comments

Comments
 (0)