Skip to content

Commit c6d2f12

Browse files
kevinelliottkevinelliott
committed
improvements
1 parent f7202cc commit c6d2f12

14 files changed

Lines changed: 3003 additions & 68 deletions

‎.github/workflows/ci.yml‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [ main, develop ]
6+
pull_request:
7+
branches: [ main, develop ]
8+
9+
jobs:
10+
test:
11+
runs-on: ubuntu-latest
12+
strategy:
13+
matrix:
14+
go-version: ['1.21', '1.22', '1.23']
15+
16+
steps:
17+
- uses: actions/checkout@v4
18+
19+
- name: Set up Go
20+
uses: actions/setup-go@v5
21+
with:
22+
go-version: ${{ matrix.go-version }}
23+
24+
- name: Cache Go modules
25+
uses: actions/cache@v4
26+
with:
27+
path: ~/go/pkg/mod
28+
key: ${{ runner.os }}-go-${{ matrix.go-version }}-${{ hashFiles('**/go.sum') }}
29+
restore-keys: |
30+
${{ runner.os }}-go-${{ matrix.go-version }}-
31+
32+
- name: Download dependencies
33+
run: go mod download
34+
35+
- name: Verify dependencies
36+
run: go mod verify
37+
38+
- name: Run go vet
39+
run: go vet ./...
40+
41+
- name: Install staticcheck
42+
run: go install honnef.co/go/tools/cmd/staticcheck@latest
43+
44+
- name: Run staticcheck
45+
run: staticcheck ./...
46+
47+
- name: Run tests
48+
run: go test -v -race -coverprofile=coverage.out -covermode=atomic ./...
49+
50+
- name: Upload coverage to Codecov
51+
uses: codecov/codecov-action@v4
52+
with:
53+
file: ./coverage.out
54+
flags: unittests
55+
name: codecov-${{ matrix.go-version }}
56+
env:
57+
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
58+
59+
- name: Build
60+
run: go build -v ./...
61+
62+
build-artifacts:
63+
runs-on: ubuntu-latest
64+
needs: test
65+
66+
steps:
67+
- uses: actions/checkout@v4
68+
69+
- name: Set up Go
70+
uses: actions/setup-go@v5
71+
with:
72+
go-version: '1.21'
73+
74+
- name: Build for multiple platforms
75+
run: |
76+
# Linux AMD64
77+
GOOS=linux GOARCH=amd64 go build -o postgresql-archiver-linux-amd64
78+
79+
# Linux ARM64
80+
GOOS=linux GOARCH=arm64 go build -o postgresql-archiver-linux-arm64
81+
82+
# macOS AMD64
83+
GOOS=darwin GOARCH=amd64 go build -o postgresql-archiver-darwin-amd64
84+
85+
# macOS ARM64 (Apple Silicon)
86+
GOOS=darwin GOARCH=arm64 go build -o postgresql-archiver-darwin-arm64
87+
88+
# Windows AMD64
89+
GOOS=windows GOARCH=amd64 go build -o postgresql-archiver-windows-amd64.exe
90+
91+
- name: Upload artifacts
92+
uses: actions/upload-artifact@v4
93+
with:
94+
name: binaries
95+
path: postgresql-archiver-*
96+
retention-days: 7
97+
98+
lint:
99+
runs-on: ubuntu-latest
100+
101+
steps:
102+
- uses: actions/checkout@v4
103+
104+
- name: Set up Go
105+
uses: actions/setup-go@v5
106+
with:
107+
go-version: '1.21'
108+
109+
- name: golangci-lint
110+
uses: golangci/golangci-lint-action@v6
111+
with:
112+
version: latest
113+
args: --timeout=5m

‎README.md‎

Lines changed: 136 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,47 @@
11
# PostgreSQL Archiver
22

3+
[![CI](https://github.com/airframesio/postgresql-archiver/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/airframesio/postgresql-archiver/actions/workflows/ci.yml)
4+
[![Go Report Card](https://goreportcard.com/badge/github.com/airframesio/postgresql-archiver)](https://goreportcard.com/report/github.com/airframesio/postgresql-archiver)
5+
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
6+
[![Go Version](https://img.shields.io/github/go-mod/go-version/airframesio/postgresql-archiver)](go.mod)
7+
38
A high-performance CLI tool for archiving PostgreSQL partitioned table data to S3-compatible object storage.
49

5-
![PostgreSQL Archiver Screenshot](screenshot.png)
10+
## Screenshots
11+
12+
### Terminal UI (TUI)
13+
![PostgreSQL Archiver TUI Screenshot](screenshot-tui.png)
14+
15+
### Web-based Cache Viewer
16+
![PostgreSQL Archiver Web UI Screenshot](screenshot-web.png)
617

718
## Features
819

920
- 🚀 **Parallel Processing** - Archive multiple partitions concurrently with configurable workers
1021
- 📊 **Beautiful Progress UI** - Real-time progress tracking with dual progress bars
11-
- 💾 **Smart Caching** - Caches row counts for faster subsequent runs
12-
- ⚡ **Compression** - Uses Zstandard compression with multi-core support
13-
- 🔄 **Resume Support** - Skips already archived files based on object storage
22+
- 🌐 **Embedded Cache Viewer** - Beautiful web interface for monitoring cache and progress:
23+
- Real-time task monitoring with progress bar
24+
- Shows archiver status (running/idle) with PID tracking
25+
- Live statistics: total partitions, sizes, compression ratios
26+
- Sortable table with S3 upload status indicators
27+
- Smooth animations highlight data changes
28+
- Error tracking with timestamps
29+
- Customizable refresh rate (1s to 60s)
30+
- 💾 **Intelligent Caching** - Advanced caching system for maximum efficiency:
31+
- Caches row counts for 24 hours (refreshed daily)
32+
- Caches file metadata permanently (size, MD5, compression ratio)
33+
- Tracks errors with timestamps
34+
- Skip extraction/compression entirely when cached metadata matches S3
35+
- 🔐 **Data Integrity** - Comprehensive file integrity verification:
36+
- Size comparison (both compressed and uncompressed)
37+
- MD5 hash verification for single-part uploads
38+
- Multipart ETag verification for large files (>100MB)
39+
- Automatic multipart upload for files >100MB
40+
- ⚡ **Smart Compression** - Uses Zstandard compression with multi-core support
41+
- 🔄 **Intelligent Resume** - Three-level skip detection:
42+
1. Fast skip using cached metadata (no extraction needed)
43+
2. Skip if S3 file matches after local processing
44+
3. Re-upload if size or hash differs
1445
- 🎯 **Flexible Partition Support** - Handles multiple partition naming formats:
1546
- `table_YYYYMMDD` (e.g., `messages_20240315`)
1647
- `table_pYYYYMMDD` (e.g., `messages_p20240315`)
@@ -72,6 +103,7 @@ Usage:
72103
postgresql-archiver [flags]
73104
74105
Flags:
106+
--cache-viewer start embedded cache viewer web server
75107
--config string config file (default is $HOME/.postgresql-archiver.yaml)
76108
--db-host string PostgreSQL host (default "localhost")
77109
--db-name string PostgreSQL database name
@@ -90,6 +122,7 @@ Flags:
90122
--skip-count skip counting rows (faster startup, no progress bars)
91123
--start-date string start date (YYYY-MM-DD)
92124
--table string base table name (required)
125+
--viewer-port int port for cache viewer web server (default 8080)
93126
--workers int number of parallel workers (default 4)
94127
```
95128

@@ -125,6 +158,8 @@ export ARCHIVE_S3_ACCESS_KEY=your_key
125158
export ARCHIVE_S3_SECRET_KEY=your_secret
126159
export ARCHIVE_TABLE=flights
127160
export ARCHIVE_WORKERS=8
161+
export ARCHIVE_CACHE_VIEWER=true
162+
export ARCHIVE_VIEWER_PORT=8080
128163
```
129164

130165
### Configuration File
@@ -150,6 +185,8 @@ table: flights
150185
workers: 8
151186
start_date: "2024-01-01"
152187
end_date: "2024-12-31"
188+
cache_viewer: false # Enable embedded cache viewer
189+
viewer_port: 8080 # Port for cache viewer web server
153190
```
154191
155192
## 📁 Output Structure
@@ -179,6 +216,33 @@ my-bucket/
179216

180217
## 🎨 Features in Detail
181218

219+
### Cache Viewer Web Interface
220+
221+
The archiver includes an embedded web server for monitoring cache and progress:
222+
223+
```bash
224+
# Start archiver with embedded cache viewer
225+
postgresql-archiver --cache-viewer --viewer-port 8080 [other options]
226+
227+
# Or run standalone cache viewer
228+
postgresql-archiver cache-viewer --port 8080
229+
```
230+
231+
Features:
232+
- **Real-time Updates**: Auto-refresh with customizable intervals (1s to 60s)
233+
- **Visual Change Detection**: Smooth animations highlight updated cells and stats
234+
- **S3 Upload Status**: Shows which files are uploaded vs only processed locally
235+
- **Comprehensive Metrics**: Shows both compressed and uncompressed sizes
236+
- **Compression Ratios**: Visual display of space savings
237+
- **Error Tracking**: Displays last error and timestamp for failed partitions
238+
- **Smart Rendering**: No page flashing - only updates changed values
239+
- **Sortable Columns**: Click any column header to sort (default: partition name)
240+
- **File Counts**: Shows total partitions, processed, uploaded, and errors
241+
- **Process Monitoring**: Checks if archiver is currently running via PID
242+
- **Task Progress**: Displays current archiving task and progress when active
243+
244+
Access the viewer at `http://localhost:8080` (or your configured port).
245+
182246
### Interactive Progress Display
183247

184248
The tool features a beautiful terminal UI with:
@@ -256,9 +320,76 @@ This will:
256320
- Connect to the database
257321
- Discover partitions
258322
- Extract and compress data
259-
- Calculate file sizes
323+
- Calculate file sizes and MD5 hashes
260324
- Skip the actual upload
261325

326+
## 💾 Caching System
327+
328+
The archiver uses an intelligent two-tier caching system to maximize performance:
329+
330+
### Row Count Cache
331+
- Caches partition row counts for 24 hours
332+
- Speeds up progress bar initialization
333+
- Always recounts today's partition for accuracy
334+
- Cache location: `~/.postgresql-archiver/cache/{table}_metadata.json`
335+
336+
### File Metadata Cache
337+
- Caches compressed/uncompressed sizes, MD5 hash, and S3 upload status
338+
- Tracks whether files have been successfully uploaded to S3
339+
- Enables fast skipping without extraction/compression on subsequent runs
340+
- Validates against S3 metadata before skipping
341+
- Preserves all metadata when updating row counts
342+
- Stores error messages with timestamps for failed uploads
343+
- File metadata is kept permanently (only row counts expire after 24 hours)
344+
345+
### Cache Efficiency
346+
On subsequent runs with cached metadata:
347+
1. Check cached size/MD5 against S3 (milliseconds)
348+
2. Skip extraction and compression if match found
349+
3. Result: 100-1000x faster for already-processed partitions
350+
351+
## 📊 Process Monitoring
352+
353+
The archiver provides real-time monitoring capabilities:
354+
355+
### PID Tracking
356+
- Creates PID file at `~/.postgresql-archiver/archiver.pid` when running
357+
- Allows external tools to check if archiver is active
358+
- Automatically cleaned up on exit
359+
360+
### Task Progress File
361+
- Writes current task details to `~/.postgresql-archiver/current_task.json`
362+
- Includes:
363+
- Current operation (connecting, counting, extracting, uploading)
364+
- Progress percentage
365+
- Total and completed partitions
366+
- Start time and last update time
367+
- Updated in real-time during processing
368+
369+
### Web API Endpoints
370+
The cache viewer provides REST API endpoints:
371+
- `/api/cache` - Returns all cached metadata
372+
- `/api/status` - Returns archiver running status and current task
373+
374+
## 🔐 Data Integrity Verification
375+
376+
The archiver ensures data integrity through multiple verification methods:
377+
378+
### Single-Part Uploads (files <100MB)
379+
- Calculates MD5 hash of compressed data
380+
- Compares with S3 ETag (which is MD5 for single-part uploads)
381+
- Only skips if both size and MD5 match exactly
382+
383+
### Multipart Uploads (files ≥100MB)
384+
- Automatically uses multipart upload for large files
385+
- Calculates multipart ETag using S3's algorithm
386+
- Verifies size and multipart ETag match before skipping
387+
388+
### Verification Process
389+
1. **First Run**: Extract → Compress → Calculate MD5 → Upload → Cache metadata
390+
2. **Subsequent Runs with Cache**: Check cache → Compare with S3 → Skip if match
391+
3. **Subsequent Runs without Cache**: Extract → Compress → Calculate MD5 → Compare with S3 → Skip or upload
392+
262393
## 🔍 Examples
263394

264395
### Archive Last 30 Days

0 commit comments

Comments
 (0)