Skip to content

CLAUDE.md

Latest commit

 

History

History
357 lines (274 loc) · 13.1 KB

CLAUDE.md

File metadata and controls

357 lines (274 loc) · 13.1 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

memwatch is a cross-platform job-level memory profiler for macOS and Linux. It tracks total memory, per-process memory, and peak memory for any command and all its child processes. Unlike top or htop, it monitors entire process trees and provides clean summaries suitable for automation and benchmarking.

Build and Development Commands

Building

cargo build --release

Binary location: target/release/memwatch

Key Dependencies:

  • regex 1.10: Process filtering with include/exclude patterns

Running Tests

cargo test

Development Build

cargo build

Running the Tool

# Basic usage
cargo run -- run -- <command> [args...]

# Example with sampling interval
cargo run -- run -i 200 -- cargo test --release

# JSON output
cargo run -- run --json -- <command>

# CSV export (per-process peak RSS)
cargo run -- run --csv processes.csv -- <command>

# Timeline export (time-series data)
cargo run -- run --timeline timeline.csv -- <command>

# Combined exports
cargo run -- run --csv procs.csv --timeline time.csv -- <command>

# Process filtering
cargo run -- run --exclude 'cargo|rustc' -- cargo test
cargo run -- run --include 'dis_cq' -- mpirun -n 8 cargo test
cargo run -- run --include 'test' --exclude 'cargo' -- cargo test

Architecture

Core Abstraction: ProcessInspector Trait

The project is built around a platform-agnostic abstraction for process inspection:

trait ProcessInspector {
    fn snapshot_all(&self) -> Result<Vec<ProcessSample>>;
}

This trait has OS-specific implementations:

  • Linux: /proc-based implementation (direct file reading, no external commands)
  • macOS: ps-based implementation using ps -axo pid,ppid,rss,command

Module Structure

Current structure:

src/
  cli.rs             # CLI argument parsing (clap)
  sampler.rs         # Sampling loop and process tree logic
  inspector/
      mod.rs         # ProcessInspector trait definition
      linux.rs       # Linux /proc implementation
      macos.rs       # macOS ps implementation
  reporter.rs        # Summary formatting and JSON output
  csv_writer.rs      # CSV export (per-process and timeline)
  types.rs           # Shared structs (ProcessSample, JobSnapshot, TimelinePoint, etc.)
  main.rs            # Binary entry point
workloads/
  mpi_distributed_compute.rs  # MPI workload example (requires OpenMPI)
  README.md                   # Example documentation

Key Data Types

ProcessSample: Single snapshot of one process

  • pid: i32
  • ppid: i32
  • rss_kib: u64 (always in KiB internally)
  • command: String

ProcessStats: Per-process statistics across job lifetime

  • pid: i32
  • ppid: i32
  • command: String
  • max_rss_kib: u64 - Peak RSS for this process
  • first_seen: DateTime<Utc> - When process first appeared
  • last_seen: DateTime<Utc> - When process last seen
  • peak_time: DateTime<Utc> - When process hit its peak RSS

TimelinePoint: Time-series data point for timeline exports

  • timestamp: DateTime<Utc>
  • elapsed_seconds: f64
  • total_rss_kib: u64
  • process_count: usize - Reflects all processes (unfiltered)

FilterConfig: Process filtering configuration

  • exclude_pattern: Option<String> - Regex pattern to exclude from display
  • include_pattern: Option<String> - Regex pattern to include in display
  • Included in JobProfile when filters are active

Job Metrics:

  • max_total_rss_kib: Peak sum of RSS across all job processes at any single moment
  • Per-PID peak RSS tracking with timestamps
  • Duration and sample count
  • Optional timeline data (only when --timeline is used)

Important: Per-process peaks may occur at different times. The sum of per-process peaks may exceed max_total_rss_kib.

Process Tree Detection

Both platforms use the same algorithm:

  1. Build a PID → (PPID, RSS, command) map
  2. A process belongs to the job if:
    • pid == root_pid, OR
    • Following PPID chain reaches root_pid

Platform-Specific Implementation Notes

Linux Backend

  • Use /proc/<pid>/status for RSS (VmRSS) or /proc/<pid>/statm
  • Parse /proc/<pid>/stat for PPID
  • Read /proc/<pid>/cmdline for command
  • No external commands required

macOS Backend

  • Execute ps -axo pid,ppid,rss,command once per interval
  • RSS from macOS ps is already in KiB
  • Parse output into ProcessSample records
  • v2 may use libproc APIs for better performance

Cross-Platform Requirements

CRITICAL: CLI, output format, and semantics must be identical between Linux and macOS. Only internals for process discovery differ.

All memory units are stored as KiB internally. Human-readable conversion (MiB, GiB) happens only at presentation layer.

Output Formats

Human-Readable (default)

  • Compact, table-formatted output with ANSI colors
  • Color scheme:
    • Cyan/Bold: Section headers (MEMORY SUMMARY, PER-PROCESS PEAKS)
    • Green: Memory values
    • Yellow: Timestamps
    • Dimmed: PIDs and process counts
    • Colors auto-disable when piping to files (ColorChoice::Auto)
  • Peak timestamps: Shows when each process hit its peak RSS (elapsed seconds from job start)
  • Process table: Aligned columns with headers (PID, MEMORY, TIME, COMMAND)
  • Process groups table: Aggregated view by command name
  • Filters out defunct/zombie processes and 0 RSS processes
  • Helpful error messages for quick-exit processes

Dependencies: Uses termcolor crate for cross-platform ANSI color support

JSON (--json flag)

Structured output with:

  • Command array
  • Start/end timestamps (ISO 8601)
  • Duration in seconds
  • max_total_rss_kib
  • processes array with per-PID metrics
  • Optional timeline array (when --timeline is used)

CSV Exports

Per-Process CSV (--csv )

Exports peak memory per process:

  • Headers: pid,ppid,command,max_rss_kib,max_rss_mib,first_seen,last_seen
  • One row per process
  • Filters out 0 RSS processes

Timeline CSV (--timeline )

Exports time-series data:

  • Headers: timestamp,elapsed_seconds,total_rss_kib,total_rss_mib,process_count
  • One row per sample
  • Perfect for plotting memory over time

CLI Structure

memwatch run [OPTIONS] -- <command> [args...]

Options:
  -i, --interval <ms>      Sampling interval in milliseconds (default: 500)
      --json               Output JSON instead of human-readable text
      --quiet              Suppress output (useful with --json)
      --csv <FILE>         Export per-process peak RSS to CSV file
      --timeline <FILE>    Export time-series memory data to CSV file
      --silent             Suppress command output (hide stdout/stderr from profiled command)
      --exclude <PATTERN>  Exclude processes matching regex from output
      --include <PATTERN>  Only include processes matching regex in output

Process Filtering

Behavior:

  • --exclude: Processes matching this regex pattern are hidden from output
  • --include: Only processes matching this regex are shown in output
  • Both flags can be combined: include is applied first, then exclude
  • Total RSS always reflects all processes (filtering only affects display)
  • Filter metadata is tracked and shown in all output formats

Filtering Logic:

  • Applied in types.rs::apply_filter() after job completion
  • Invalid regex patterns produce clear error messages (validated before filtering)
  • All processes filtered out triggers special warning message
  • Filters apply to human-readable, JSON, and CSV outputs consistently

Filter Metadata (in JobProfile):

  • filter: Option<FilterConfig> - Present when filters are active
  • filtered_process_count: Option<usize> - Number of processes excluded
  • filtered_total_rss_kib: Option<u64> - Total RSS of excluded processes

Key Implementation Considerations

Sampling Loop

  • Configurable interval (default 500ms)
  • Immediate first sample after process spawn to catch quick processes
  • Handle process churn (processes appearing/disappearing between samples)
  • Update running maxima, don't store all samples in memory (except timeline if enabled)
  • Timeline tracking: Optional, only when --timeline flag is used
  • Continue while at least one job process is alive
  • Filter defunct/zombie processes on macOS (contains <defunct> or (name))

Error Handling

  • Command fails to start: clear error message, non-zero exit
  • Quick-exit commands: Show helpful message with suggestion to use shorter interval
  • 0 RSS processes: Filtered from output, with explanation when all processes have 0 RSS
  • Invalid regex patterns: Validated before filtering, produces clear error with pattern shown
  • All processes filtered out: Special warning message showing filter criteria and suggestions
  • SIGINT forwarding: forward to root PID and children, then cleanup
  • Permission errors: skip with warning
  • CSV/Timeline export errors: Clear context with file path in error message

Testing Strategy

  • Unit tests for process-tree detection with mocked snapshots
  • Unit tests for unit conversion (KiB → MiB/GiB)
  • Integration tests with simple commands (e.g., sleep, small memory scripts)

Platform Support

Supported in v1: macOS, Linux Not supported: Windows (abstraction designed for future addition)

If built on Windows, should fail compilation with clear cfg error or exit immediately with unsupported message.

Implementation Phases

  1. CLI parsing + subprocess spawning: Parse args, spawn command, track timing
  2. Linux sampling: Implement /proc-based inspector
  3. macOS sampling: Implement ps-based inspector
  4. Output formatting: Human-readable summary + JSON
  5. Polish: Error handling, unit tests, integration tests

Memory Units

Always use KiB as the base unit internally:

  • 1 KiB = 1024 bytes
  • 1 MiB = 1024 KiB
  • 1 GiB = 1024 MiB

Convert to human-readable IEC units only for display.

Examples

The workloads/ directory contains demonstration programs showing memwatch's capabilities for different workload types.

MPI Distributed Computation Example

File: workloads/mpi_distributed_compute.rs

A realistic MPI example inspired by distributed zero-knowledge proof systems (specifically zeroasset2). Demonstrates memwatch's value for tracking memory across MPI process trees.

Prerequisites:

  • OpenMPI or MPICH installed
  • On macOS: brew install open-mpi
  • The mpi crate (0.8) is a dev dependency
  • Uses patched libffi-sys from git to fix ARM64 build issues

Building:

# Build the example
cargo build --release --example mpi_distributed_compute

# Run with 4 MPI processes
mpirun -n 4 target/release/workloads/mpi_distributed_compute

# Profile with memwatch
memwatch run -- mpirun -n 4 target/release/workloads/mpi_distributed_compute

What it demonstrates:

  • Process tree tracking: All MPI ranks tracked as single job
  • Memory scaling: Per-process memory inversely proportional to process count
  • Realistic workload: Simulates distributed polynomial computations with field elements
  • Short-lived execution: Captures peak memory in ~5-10 second runs
  • Memory phases: Initialization → Computation → Communication → Cleanup

Key features:

  • Distributed arrays (each rank allocates n/np elements)
  • Working buffers (2x allocation for simulated FFT workspace)
  • Communication spikes (temporary buffers during MPI collectives)
  • Configurable problem size via --size argument

See workloads/README.md for detailed documentation.

Code Quality Best Practices

Always write clean code and use best practices. Key principles followed in this codebase:

Performance

  • Avoid unnecessary cloning: Take ownership with Vec<T> instead of borrowing &[T] when consumption is needed
  • Use HashMap::remove() instead of get() when extracting values to avoid clones
  • Count filtered items instead of storing them when only statistics are needed

Maintainability

  • Extract magic numbers to named constants in semantic modules (e.g., memory::KIB_PER_MIB)
  • Eliminate code duplication by extracting helper functions or adding methods to structs
  • Use explicit error messages with expect() when documenting invariants
  • Simplify conditional logic by extracting condition checks and using pattern binding

Testing

  • Add comprehensive tests for edge cases (empty input, all filtered, invalid patterns)
  • Test error paths with clear assertions on error messages
  • Document behavior through tests (each test name describes the scenario)

Recent Refactorings (v1.1)

Two major refactoring commits improved code quality after filtering feature completion:

Commit 1 - Performance & Maintainability (0289a45):

  • Eliminated 180K+ unnecessary allocations per typical job by removing clones in apply_filter() and sample_job_tree()
  • Added types::memory module with named constants
  • Added FilterConfig::display_patterns() and to_csv_comment() methods
  • Extracted write_filter_comment() helper in CSV writer
  • Simplified into_profile() conditional logic

Commit 2 - Test Coverage (bb48bff):

  • Added 10 comprehensive tests for apply_filter() covering edge cases, pattern matching, and error handling