Skip to content

Commit cbb1708

Browse files
authored
Advanced filtering (#4)
* fix: Windows CI build fixed - removed Windows runner dependency, use cross-compilation from Ubuntu * fix: add Content-MD5 headers for MinIO batch deletion compatibility - Fixed MissingContentMD5 errors in batch deletion operations - Added MD5 hash computation and base64 encoding for delete requests - Updated delete_objects_recursive and delete_all_versions in rm.rs - Updated delete_all_objects and delete_all_versions in bucket.rs - MinIO requires Content-MD5 headers for batch operations - AWS SDK Rust doesn't add these headers automatically - Successfully tested: cleared 10 buckets with 102 objects using only obsctl commands - Resolves phantom deletion success and batch operation failures - Maintains compatibility with AWS S3 and other S3-compatible services * feat(cli): implement cross-platform broken pipe handling Add comprehensive panic hook to detect broken pipe errors across all platforms. Supports Unix EPIPE, Linux ESHUTDOWN, Windows ERROR_NO_DATA error codes. Graceful exit with output flushing prevents ugly panic messages. Maintains professional CLI behavior when piping to head/tail commands.
1 parent b0886ca commit cbb1708

119 files changed

Lines changed: 38521 additions & 1125 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.docker/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ RUN apt-get update && apt-get install -y \
88
curl \
99
ca-certificates
1010

11-
RUN rustup component add rustfmt clippy
11+
RUN rustup component add rustfmt clippy
1212
RUN cargo install cargo-audit
1313

1414
WORKDIR /usr/src/app

.docker/grafana/README.md

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# obsctl Grafana Dashboards
2+
3+
This directory contains Grafana dashboards for monitoring the obsctl S3 CLI tool and its observability stack.
4+
5+
## Available Dashboards
6+
7+
### 1. obsctl OTEL Collector Overview (`obsctl-overview`)
8+
**Primary monitoring dashboard for the OpenTelemetry Collector**
9+
10+
- **OTEL Collector Status**: Real-time health status (UP/DOWN)
11+
- **OTEL Collector Uptime**: How long the collector has been running
12+
- **Exporter Queue Size**: Current queue size for telemetry export
13+
- **Queue Usage %**: Percentage of queue capacity being used
14+
- **Memory Usage**: RSS memory and heap allocation trends
15+
- **CPU Usage**: Collector CPU utilization over time
16+
- **Queue Metrics**: Queue size vs capacity trending
17+
18+
### 2. obsctl Distributed Tracing (`obsctl-traces`)
19+
**Jaeger traces and distributed tracing visualization**
20+
21+
- **Recent obsctl Operations**: Table view of recent traces from obsctl operations
22+
- **OTEL Collector Health**: Health monitoring for trace collection
23+
- **Telemetry Queue Activity**: Queue activity for trace processing
24+
- **Direct link to Jaeger UI**: Click the "Jaeger UI" link to view detailed traces
25+
26+
### 3. obsctl System Monitoring (`obsctl-system`)
27+
**Infrastructure and system-level monitoring**
28+
29+
- **Service Status**: Health status for OTEL Collector and Prometheus
30+
- **OTEL Uptime**: Collector uptime tracking
31+
- **Queue Size**: Current telemetry queue size
32+
- **Memory Usage**: Detailed memory consumption metrics
33+
- **CPU Usage**: System CPU utilization
34+
- **Prometheus TSDB Activity**: Database activity and metrics ingestion rates
35+
- **Direct links**: Quick access to Prometheus UI and MinIO Console
36+
37+
## Dashboard Access
38+
39+
All dashboards are automatically provisioned and available at:
40+
- **Grafana**: http://localhost:3000 (admin/admin)
41+
42+
## Related Services
43+
44+
- **Jaeger Traces**: http://localhost:16686
45+
- **Prometheus Metrics**: http://localhost:9090
46+
- **MinIO Console**: http://localhost:9001 (minioadmin/minioadmin123)
47+
48+
## Understanding the Data
49+
50+
### Telemetry Flow
51+
1. **obsctl** operations generate OpenTelemetry traces
52+
2. **OTEL Collector** receives and processes traces via port 4317
53+
3. **Jaeger** stores and displays distributed traces
54+
4. **Prometheus** collects metrics about the collector itself
55+
5. **Grafana** visualizes both metrics and provides trace access
56+
57+
### Key Metrics to Monitor
58+
59+
- **Queue Size**: Should remain low; high values indicate backpressure
60+
- **Memory Usage**: Monitor for memory leaks or excessive consumption
61+
- **CPU Usage**: Track collector performance impact
62+
- **Service Status**: Ensure all components are healthy (UP)
63+
64+
### Generating Test Data
65+
66+
Run integration tests to generate telemetry data:
67+
```bash
68+
# Generate comprehensive telemetry data
69+
tests/integration/run_tests.sh observability --verbose
70+
71+
# Generate specific test patterns
72+
tests/integration/run_tests.sh performance
73+
tests/integration/run_tests.sh concurrent
74+
```
75+
76+
## Troubleshooting
77+
78+
### No Data in Dashboards
79+
1. Verify all services are running: `docker compose ps`
80+
2. Check OTEL Collector logs: `docker compose logs otel-collector`
81+
3. Run observability tests to generate data
82+
4. Ensure obsctl is built with OTEL features: `cargo build --features otel`
83+
84+
### Missing Traces in Jaeger
85+
1. Check that obsctl operations are using OTEL endpoint: `http://localhost:4317`
86+
2. Verify service name in traces: `obsctl-integration-test`
87+
3. Check OTEL Collector configuration for Jaeger export
88+
89+
### Dashboard Errors
90+
1. Restart Grafana: `docker compose restart grafana`
91+
2. Check datasource connections in Grafana UI
92+
3. Verify Prometheus is scraping metrics: http://localhost:9090/targets
93+
94+
## Customization
95+
96+
Dashboards are provisioned from JSON files in `/var/lib/grafana/dashboards`.
97+
To modify:
98+
1. Edit the JSON files in `.docker/grafana/dashboards/`
99+
2. Restart Grafana: `docker compose restart grafana`
100+
3. Changes will be automatically loaded
101+
102+
## Performance Notes
103+
104+
- Dashboards refresh every 5 seconds by default
105+
- Historical data retention follows Prometheus configuration
106+
- Jaeger traces are stored in memory (ephemeral)
107+
- For production use, configure persistent storage for both Prometheus and Jaeger

0 commit comments

Comments
 (0)