docs(storage): document Bulk write semantics, retries, and data-loss cases

Update the storage best-practice docs to reflect the BulkIndexer-based
remote write path: flush triggers (5 MB / 1 s / shutdown), two-tier
retry policy (whole-batch 429/5xx with exponential backoff; per-item
failures dropped via OnFailure), the three data-loss scenarios, and
the four problem classes this design addresses (TLS handshake CPU,
RTT ceiling, transient jitter, capture/backend decoupling).

Author: hao022

docs/best-practice/storage_en.md

@@ -389,7 +389,9 @@ curl -k -u elastic:123456 -X GET "http://localhost:9200/huatuo_bamai/_count?pret
 
 ### System Architecture
 
-The HUATUO Storage module runs on each node. It writes kernel events captured by the Tracer concurrently to the local directory and to Elasticsearch or OpenSearch. Both storage backends share the same `[Storage.ES]` configuration interface and are differentiated by address.
+The HUATUO Storage module runs on each node. It writes kernel events captured by the Tracer to the local directory and to Elasticsearch or OpenSearch. Both backends share the same `[Storage.ES]` configuration interface and are differentiated by address.
+
+The remote write path uses the ES/OpenSearch **Bulk API** (`_bulk`): events are queued in an in-memory buffer and submitted in batches by background workers based on size and time thresholds, with transport-layer retries on transient failures.
 
 ```mermaid
 graph TB
@@ -401,7 +403,7 @@ graph TB
     subgraph huatuo["HUATUO Agent (node-level)"]
         T["Tracer Layer"]
         L["Local Directory\nhuatuo-local/"]
-        S["Storage Module\n(concurrent write)"]
+        S["Storage Module\nBulkIndexer Buffer"]
     end
 
     subgraph backends["Storage Backends"]
@@ -412,41 +414,80 @@ graph TB
     kernel --> T
     T --> L
     T --> S
-    S -->|Index API| ES
-    S -->|Index API| OS
+    S -->|Bulk API + auto retry| ES
+    S -->|Bulk API + auto retry| OS
 ```
 
 ### Write Flow
 
-After the Tracer captures a kernel event, the Storage module writes it concurrently to the local directory and the remote storage backend. The two write paths execute in parallel — the local directory retains a copy while the remote backend provides durable storage and query capabilities.
+`Save` returns immediately after the event is buffered. Background workers flush the buffer to the remote backend when **any** of the following triggers fire: byte threshold, time threshold, or process shutdown. The local directory write is synchronous and independent of the remote Bulk path.
 
 ```mermaid
 sequenceDiagram
     participant T as Tracer Layer
     participant L as Local Directory (huatuo-local/)
-    participant S as Storage Module
+    participant S as Storage Module (BulkIndexer)
     participant B as ES / OpenSearch
 
     T->>S: Kernel event captured, serialized to JSON
-    par concurrent write
+    par Local path (sync)
         S->>L: Write to local file
-    and
-        S->>B: Write to remote storage (Index API)
-        B-->>S: Write acknowledged (200 OK)
+    and Remote path (async batch)
+        S->>S: Enqueue into bulk buffer, return immediately
+        Note over S: Flush on 5 MB / 1 s / shutdown
+        S->>B: POST /_bulk (multiple records)
+        B-->>S: 200 OK + per-item results
+        Note over S: Failed items reported via OnFailure callback
     end
 ```
 
-### Storage Pipeline
+### Bulk Write Mechanism
 
-From kernel event to storage backend, the process involves three stages: capture, serialization, and concurrent write. The local directory and remote backend are written to in parallel without blocking each other.
+#### Buffering and Flush Triggers
 
-```mermaid
-flowchart LR
-    A([Kernel Event]) --> B["Tracer Capture\nSerialize to JSON"]
-    B --> C["Storage Module\n(concurrent write)"]
-    C --> D["Write to Local Directory\nhuatuo-local/"]
-    C --> E["Write to ES / OpenSearch\nIndex API"]
-```
+| Parameter         | Value               | Meaning                                         |
+|-------------------|---------------------|-------------------------------------------------|
+| `FlushBytes`      | 5 MB                | Flush when accumulated bytes reach the threshold |
+| `FlushInterval`   | 1 s                 | Force-flush 1 second after the previous flush    |
+| `NumWorkers`      | 4                   | Concurrent workers submitting Bulk requests      |
+| Process shutdown  | `Close(ctx)`        | SIGTERM/SIGINT triggers a 10 s bounded drain     |
+
+#### Two-Tier Retry Policy
+
+Bulk failures are split into two layers with different retry semantics:
+
+| Layer                | Trigger                                                                       | Behavior                                                                                                  | Retried? |
+|----------------------|-------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|----------|
+| **Whole-batch retry** | Transport error (connect / timeout / TLS)<br>HTTP status: `429 / 502 / 503 / 504` | Client retries with exponential backoff: 100 ms → 200 ms → 400 ms → 800 ms, up to **3 attempts**            | ✅ auto |
+| **Whole-batch reject**| HTTP status: `400 / 401 / 403 / 404 / 413`, etc.                              | Not retried; all records in the batch are dropped, an error is logged via `OnError`                        | ❌ drop |
+| **Per-item failure**  | 200 OK with per-item error: version conflict, mapping error, document too large| Not retried; only the failed item is dropped, `OnFailure` logs `index/id/status/type/reason`               | ❌ drop |
+| **Per-item success**  | 200 OK with per-item success                                                  | Considered durably indexed                                                                                 | —        |
+
+**Why this design**: 429/5xx and transport errors signal transient remote unavailability where retries are effective; 4xx (except 429) and per-item errors are client-side semantic issues (data shape, permissions) where retries would only amplify the failure — they should be surfaced via logs for human investigation.
+
+#### Data-Loss Scenarios
+
+In all three scenarios below, `Save` returns `nil` but the event never reaches the index:
+
+1. **Abnormal process exit**: `SIGKILL` or host power loss drops whatever is still buffered in the BulkIndexer (the local directory still keeps a copy).
+   - Mitigation: SIGTERM/SIGINT trigger graceful shutdown; `Close` force-flushes the buffer with a 10 s deadline.
+2. **Whole-batch permanent rejection**: 4xx (non-429) errors discard every record in the batch. Common causes: disabled index, expired credentials, document exceeding the cluster's `http.max_content_length`.
+   - Diagnosis: `OnError` log includes ES's `type` and `reason`.
+3. **Permanent per-item failure**: mapping conflict, version conflict, malformed document.
+   - Diagnosis: `OnFailure` log identifies the record by `index/id`.
+
+> **The local directory is always a fallback**: even if remote writes are lost, events remain available in `huatuo-local/` as the eventual-consistency safety net.
+
+#### Problems This Solves
+
+Replacing per-event Index API calls with a buffered BulkIndexer + auto-retry addresses four classes of problems:
+
+| Problem                                  | Old approach bottleneck                                 | Bulk approach improvement                                            |
+|------------------------------------------|---------------------------------------------------------|----------------------------------------------------------------------|
+| **TLS handshake CPU cost**               | One HTTPS handshake per event saturated CPU under FIPS/RSA-PSS | Many events share one connection and one handshake; TLS PSK tickets cached |
+| **Remote RTT throughput ceiling**        | One round-trip per event capped node-level write rate    | One Bulk request carries up to 5 MB; throughput scales with batch size |
+| **Transient remote jitter / 429 throttle** | A single failure dropped the event with no retry         | Client-level retry absorbs short-lived faults                         |
+| **Decoupling tracer layer from backend** | Slow remote backed pressure into capture, delaying tracing | Async buffer decouples capture from network — capture is no longer blocked on remote latency |
 
 ---
 

docs/best-practice/storage_zh.md

@@ -387,6 +387,8 @@ curl -k -u elastic:123456 \
 
 HUATUO Storage 模块部署在节点上，将采集到的内核事件同时写入本地目录和 Elasticsearch 或 OpenSearch。两种存储后端共用同一套 `[Storage.ES]` 配置接口，通过地址区分。
 
+写入远端时使用 ES/OpenSearch 的 **Bulk API**（`_bulk`）：事件先进入节点内的批量缓冲，由后台 worker 按"大小或时间"的阈值聚合后一次提交多条记录，并在传输层失败时按策略自动重试。
+
 ```mermaid
 graph TB
     subgraph kernel["Linux 内核"]
@@ -397,7 +399,7 @@ graph TB
     subgraph huatuo["HUATUO Agent（节点级）"]
         T["采集层"]
         L["本地目录\nhuatuo-local/"]
-        S["Storage 模块\n同步写入"]
+        S["Storage 模块\nBulkIndexer 缓冲"]
     end
 
     subgraph backends["存储后端"]
@@ -408,41 +410,80 @@ graph TB
     kernel --> T
     T --> L
     T --> S
-    S -->|Index API| ES
-    S -->|Index API| OS
+    S -->|Bulk API + 自动重试| ES
+    S -->|Bulk API + 自动重试| OS
 ```
 
 ### 数据写入流程
 
-HUATUO 采集到内核事件后，Storage 模块将事件同时写入本地目录和远端存储后端。两路写入并发执行，本地目录保留副本，远端存储提供持久化与查询能力。
+采集层调用 `Save` 后立即返回，事件落入 BulkIndexer 缓冲；后台 worker 在满足"字节阈值 / 时间阈值 / 进程退出"任一条件时将批次提交至远端。本地目录写入是同步落盘，与远端 Bulk 路径相互独立。
 
 ```mermaid
 sequenceDiagram
     participant T as 采集层
     participant L as 本地目录（huatuo-local/）
-    participant S as Storage 模块
+    participant S as Storage 模块（BulkIndexer）
     participant B as ES / OpenSearch
 
     T->>S: 采集到内核事件，序列化为 JSON
-    par 同时写入
+    par 本地路径（同步）
         S->>L: 写入本地文件
-    and
-        S->>B: 写入远端存储（Index API）
-        B-->>S: 写入确认（200 OK）
+    and 远端路径（异步批量）
+        S->>S: 加入 Bulk 缓冲，立即返回
+        Note over S: 满足 5 MB / 1 s / 退出 任一条件
+        S->>B: POST /_bulk（多条记录）
+        B-->>S: 200 OK + per-item 结果
+        Note over S: 失败项通过 OnFailure 回调记录日志
     end
 ```
 
-### 存储写入流程
+### Bulk 批量写入机制
 
-从内核事件产生到写入存储后端，经过采集、序列化、同步写入三个阶段。本地目录与远端存储并发写入，互不阻塞。
+#### 缓冲与刷新
 
-```mermaid
-flowchart LR
-    A([内核事件触发]) --> B["采集\n序列化为 JSON"]
-    B --> C["Storage 模块\n同步写入"]
-    C --> D["写入本地目录\nhuatuo-local/"]
-    C --> E["写入 ES / OpenSearch\nIndex API"]
-```
+| 参数              | 值                  | 含义                            |
+|-------------------|---------------------|---------------------------------|
+| `FlushBytes`      | 5 MB                | 缓冲累计达到该字节数立即刷新     |
+| `FlushInterval`   | 1 s                 | 距上次刷新满 1 秒后强制刷新      |
+| `NumWorkers`      | 4                   | 并发提交 Bulk 请求的后台 goroutine 数 |
+| 进程退出          | `Close(ctx)`        | SIGTERM/SIGINT 触发，限时 10 s 内排空缓冲 |
+
+#### 两级重试策略
+
+Bulk 请求的失败语义分为两层，重试范围不同：
+
+| 层级           | 触发条件                                            | 处理方式                                                                                              | 是否重试 |
+|----------------|-----------------------------------------------------|-------------------------------------------------------------------------------------------------------|----------|
+| **整批失败**   | 传输错误（连接失败、超时、TLS）<br>HTTP 状态：`429 / 502 / 503 / 504` | 客户端按指数退避自动重试：100 ms → 200 ms → 400 ms → 800 ms，最多 **3 次**                              | ✅ 自动 |
+| **整批拒绝**   | HTTP 状态：`400 / 401 / 403 / 404 / 413` 等          | 不重试，整批所有记录全部丢弃，并通过 `OnError` 写错误日志                                              | ❌ 丢弃 |
+| **单条失败**   | 200 OK 但 per-item 失败：版本冲突、字段映射错误、文档过大 | 不重试，仅该单条丢弃，通过 `OnFailure` 回调记录 `index/id/status/type/reason`                          | ❌ 丢弃 |
+| **单条成功**   | 200 OK 且 per-item 成功                              | 视为已落库                                                                                            | —        |
+
+**为什么这样设计**：429/5xx 与传输错误是远端短暂不可用的信号，重试有效；4xx（除 429）与 per-item 错误是客户端语义问题（数据格式、权限），重试只会放大错误，应交给开发与运维侧排查日志后修正。
+
+#### 数据丢失场景
+
+下列三种情况下，调用方调用 `Save` 时返回 `nil`，但事件最终未进入索引：
+
+1. **进程异常退出**：`SIGKILL` 或宿主机断电时，BulkIndexer 内存缓冲尚未刷新的部分直接丢失（仅本地目录保留副本）。
+   - 缓解：SIGTERM/SIGINT 走优雅退出路径，shutdown 时调用 `Close` 强制 flush，最长等待 10 秒。
+2. **整批被永久拒绝**：4xx（非 429）类错误一次性丢弃整批所有记录。常见诱因：索引被禁用、密码失效、单条文档超过集群 `http.max_content_length`。
+   - 排查：`OnError` 错误日志包含 ES 返回的 `type` 与 `reason`。
+3. **单条永久失败**：mapping 冲突、版本冲突、文档语法错误。
+   - 排查：`OnFailure` 错误日志按 `index/id` 定位失败记录。
+
+> **本地目录始终保留副本**：即使远端写入丢失，事件仍可从 `huatuo-local/` 中找回，作为最终一致性的兜底。
+
+#### 解决的问题
+
+将"逐事件 Index API"换成"BulkIndexer 批量 + 自动重试"主要解决以下四类问题：
+
+| 问题                              | 旧方案瓶颈                                  | Bulk 方案的改进                                       |
+|-----------------------------------|---------------------------------------------|-------------------------------------------------------|
+| **TLS 握手 CPU 开销**             | 每事件一次 HTTPS，握手在 FIPS / RSA-PSS 下占满 CPU | 多条事件复用单连接 + 单次握手；TLS PSK ticket 缓存复用 |
+| **远端 RTT 与吞吐**               | 每事件一次往返，节点级写入受 RTT 限制        | 单次 Bulk 请求最多 5 MB，吞吐随批大小线性提升          |
+| **远端短暂抖动 / 限流（429）**    | 单次失败立即丢弃，无重试                    | 客户端层面自动重试，吸收瞬态故障                       |
+| **采集层对存储后端解耦**          | 远端慢会回压采集，影响内核事件采集时延       | 异步缓冲将采集与远端写入解耦，采集路径不被网络阻塞     |
 
 ---