feat(observability): establish Prometheus metrics, structured tracing, and health endpoints

[ruojieranyishen]

Summary

Kiwi currently lacks production-grade observability. The existing MetricsCollector uses custom in-memory structs with JSON serialization — it cannot be scraped by Prometheus/Grafana, and the HealthCheckEndpoints HTTP server is a stub. Logging uses env_logger with unstructured text output. There is no distributed tracing.

This issue proposes a phased approach to establish a proper observability stack, informed by patterns from TiKV, Databend, and RisingWave.

Current State

Dimension	Status	Gap
Metrics	Custom RuntimeMetrics struct (in-memory, JSON)	No Prometheus scrape endpoint; custom percentile calculation; no standard naming
Logging	log crate + env_logger	Unstructured text; no JSON format; no log level hot-reload; env_logger unsuitable for production
Tracing	None	Cannot trace request flow across net → channel → storage
Health	HealthCheckEndpoints exists but HTTP server is a stub	Not functional
Raft Metrics	gRPC RaftMetricsService proto defined	Not integrated into standard metrics

Proposed Stack

Concern	Crate	Rationale
Metrics	prometheus v0.13	Production-proven by TiKV at scale; native Grafana support; prometheus-static-metric for hot-path optimization
Logging	tracing + tracing-subscriber	Tokio-native; structured JSON output; env-filter for runtime level control; ecosystem momentum over slog
Log bridging	tracing-log	Allows gradual migration — existing log::info!() calls work without changes
Tracing	Defer to Phase 4	Per-request metrics labels sufficient initially; add opentelemetry later for cross-node tracing
Process metrics	procfs (Linux)	Collect CPU, memory, FDs, threads — similar to Databend's ProcessCollector

Metrics Design

Naming Convention

Following TiKV's pattern: kiwi_{subsystem}_{metric}_{unit}

Key Metrics

1. Command Layer

kiwi_cmd_request_duration_seconds{cmd="GET"}          # Histogram
kiwi_cmd_requests_total{cmd="SET",status="ok"}        # Counter

2. Storage Layer (RocksDB, per CF)

kiwi_rocksdb_write_duration_seconds{cf="default"}     # Histogram
kiwi_rocksdb_compaction_duration_seconds{cf="default"} # Histogram
kiwi_rocksdb_write_stall_total                        # Counter
kiwi_rocksdb_block_cache_hit_ratio                    # Gauge
kiwi_rocksdb_total_sst_files{cf="default"}            # Gauge

3. Raft Layer (waterfall latency breakdown, inspired by TiKV)

kiwi_raft_propose_duration_seconds                     # Histogram
kiwi_raft_apply_duration_seconds                       # Histogram
kiwi_raft_log_entries_pending                          # Gauge
kiwi_raft_leader_changes_total                         # Counter
kiwi_raft_propose_stage_seconds{stage="batch_wait"}    # Histogram (per stage)

4. Dual Runtime Channel

kiwi_channel_messages_total{direction="to_storage"}    # Counter
kiwi_channel_buffer_utilization_ratio                  # Gauge
kiwi_channel_request_duration_seconds                  # Histogram

5. Connection Layer

kiwi_net_active_connections                            # Gauge
kiwi_net_bytes_received_total                          # Counter

Histogram Buckets

Following TiKV's proven bucket strategy:

// Latency: 10μs to ~11 minutes
exponential_buckets(0.00001, 2.0, 26)

// Byte sizes: 1KB to ~1TB
exponential_buckets(1024.0, 2.0, 31)

Per-Module Registration Pattern

Each crate owns its metrics.rs with LazyLock statics (Rust 2024 edition):

use prometheus::{register_histogram_vec, HistogramVec};
use std::sync::LazyLock;

pub static CMD_DURATION: LazyLock<HistogramVec> = LazyLock::new(|| {
    register_histogram_vec!(
        "kiwi_cmd_request_duration_seconds",
        "Command request latency",
        &["cmd"],
        exponential_buckets(0.00001, 2.0, 26).unwrap()
    ).unwrap()
});

Phased Implementation

Phase 1 — Core Metrics + Prometheus Endpoint (2-3 days)

• Add prometheus dependency to workspace Cargo.toml
• Create src/common/metrics/ crate with global registry and /metrics HTTP handler
• Define command-layer metrics in src/cmd/src/metrics.rs — record request duration and count
• Define storage-layer metrics in src/storage/src/metrics.rs — RocksDB latency, write stall, cache hit ratio
• Expose /metrics HTTP endpoint (on raft_addr or dedicated port)
• Ship initial Grafana dashboard JSON

Phase 2 — Structured Logging (1-2 days)

• Replace env_logger with tracing-subscriber (JSON format for production, pretty for dev)
• Bridge existing log crate calls via tracing-log
• Add #[instrument] spans to critical paths: Cmd::do_cmd, storage operations
• Support RUST_LOG env filter for runtime log level control

Phase 3 — Raft + Channel Metrics + Health (1-2 days)

• Raft propose/apply latency histograms with waterfall stage breakdown
• Channel backpressure and throughput metrics
• Functional /health endpoint (liveness + readiness probes)
• Prometheus alerting rules (write stall, high error rate, leader loss)

Phase 4 — Optional Enhancements

• OpenTelemetry distributed tracing for cross-node request tracking
• Per-thread CPU/IO metrics via procfs
• Slow score / slow trend detection (multi-window moving average, inspired by TiKV)
• RocksDB TablePropertiesCollector integration for granular CF stats

Architectural Notes

• Per-crate metrics.rs: Each workspace crate (cmd, storage, raft, net) owns its metrics definitions. This maps cleanly to Kiwi's existing layout.
• Metrics exposure: Standard Prometheus pull model via HTTP /metrics endpoint. No push gateway needed.
• Grafana dashboards: Ship JSON files in metrics/grafana/ directory, following TiKV's pattern.
• Existing code preservation: The current MetricsCollector in src/common/runtime/metrics.rs can coexist initially. New Prometheus-based metrics are additive. The custom health check structs can be reused as the backing data for the /health endpoint.
• No breaking changes: Phase 1-3 are purely additive. Existing log crate calls continue to work via tracing-log bridge.

References

• TiKV observability architecture — prometheus crate, per-module metrics.rs, waterfall latency, Grafana dashboards
• Databend observability — prometheus-client crate, tracing + OTel, GlobalRegistry pattern
• RisingWave observability — tracing + opentelemetry, mixtrics wrapper

Labels

proposal, P2, observability