Enhanced Historical Functionality for All Storage Types and Complex Multi-Branch Testing (#81)

Author: zhangfengcdt

README.md

@@ -226,8 +226,11 @@ The following features are for Prolly tree library for Version 0.2.1:
 - [X] add usage examples for git-prolly use cases
 - [X] add usage examples for AI agent memory use cases
 - [X] support rocksdb as storage backend
+- [X] add agent memory system api support
+
+The following features are for Prolly tree library for future versions:
 - [ ] support IPDL as storage backend
-- [X] generic storage backend support for VersionedKvStore
+
 
 ## Contributing
 

doc/git.md docs/git.mddoc/git.md renamed to docs/git.md

@@ -320,6 +320,118 @@ git-prolly show HEAD --keys-only
 #           user:456
 ```
 
+#### `git-prolly history <key>`
+Show commit history for a specific key, tracking all changes made to that key over time.
+
+**Usage:**
+```bash
+git-prolly history <key> [--format=<format>] [--limit=<number>]
+```
+
+**Options:**
+- `--format=compact`: Show concise one-line format (default)
+- `--format=detailed`: Show detailed commit information
+- `--format=json`: Output in JSON format
+- `--limit=<number>`: Maximum number of commits to show
+
+**Examples:**
+```bash
+# Basic history
+git-prolly history user:123
+# Output: History for key 'user:123':
+#           2024-01-15 10:30:00 f1e2d3c4 Update user profile
+#           2024-01-15 09:15:00 a1b2c3d4 Add new user
+
+# Detailed history
+git-prolly history user:123 --format=detailed
+# Output: Detailed History for key 'user:123':
+#         ═══════════════════════════════════════
+#         Commit: f1e2d3c4b5a6789012345678901234567890abcd
+#         Date: 2024-01-15 10:30:00 UTC
+#         Author: Developer
+#         Message: Update user profile
+#         
+#         Commit: a1b2c3d4e5f6789012345678901234567890abcd
+#         Date: 2024-01-15 09:15:00 UTC
+#         Author: Developer
+#         Message: Add new user
+
+# Limited results
+git-prolly history user:123 --limit=5
+# Output: History for key 'user:123' (showing 5 most recent):
+#           2024-01-15 10:30:00 f1e2d3c4 Update user profile
+#           2024-01-15 09:15:00 a1b2c3d4 Add new user
+
+# JSON output
+git-prolly history user:123 --format=json
+# Output: {
+#           "key": "user:123",
+#           "history": [
+#             {
+#               "commit": "f1e2d3c4b5a6789012345678901234567890abcd",
+#               "timestamp": 1705315800,
+#               "author": "Developer",
+#               "message": "Update user profile"
+#             }
+#           ]
+#         }
+```
+
+#### `git-prolly keys-at <reference>`
+Show all keys that existed at a specific commit or branch reference.
+
+**Usage:**
+```bash
+git-prolly keys-at <reference> [--values] [--format=<format>]
+```
+
+**Options:**
+- `--values`: Show values as well as keys
+- `--format=list`: Show as a simple list (default)
+- `--format=json`: Output in JSON format
+
+**Examples:**
+```bash
+# List keys at HEAD
+git-prolly keys-at HEAD
+# Output: Keys at HEAD:
+#           config:theme
+#           user:123
+#           user:456
+
+# List keys with values
+git-prolly keys-at HEAD --values
+# Output: Keys at HEAD:
+#           config:theme = "dark"
+#           user:123 = "John Doe"
+#           user:456 = "Jane Smith"
+
+# Keys at specific commit
+git-prolly keys-at a1b2c3d4
+# Output: Keys at a1b2c3d4:
+#           config:theme
+#           user:123
+
+# Keys at branch
+git-prolly keys-at feature/new-users
+# Output: Keys at feature/new-users:
+#           config:theme
+#           user:123
+#           user:456
+#           user:789
+
+# JSON output with values
+git-prolly keys-at HEAD --values --format=json
+# Output: {
+#           "reference": "HEAD",
+#           "keys": [
+#             {"key": "config:theme", "value": "dark"},
+#             {"key": "user:123", "value": "John Doe"},
+#             {"key": "user:456", "value": "Jane Smith"}
+#           ]
+#         }
+```
+
 ### Advanced Operations
 
 #### `git-prolly revert <commit>`

doc/sql.md docs/sql.mddoc/sql.md renamed to docs/sql.md

@@ -60,6 +60,7 @@ Options:
   -o, --format <FORMAT>  Output format (table, json, csv)
   -i, --interactive      Start interactive SQL shell
       --verbose          Show detailed error messages
+  -b, --branch <BRANCH>  Execute against specific branch or commit (SELECT queries only, requires clean status)
   -h, --help             Print help
 ```
 
@@ -109,6 +110,38 @@ In interactive mode:
 - Type `help` for available commands
 - Type `exit` or `quit` to leave the shell
 
+### 4. Historical Data Querying
+
+Query data from specific branches or commits using the `-b` parameter:
+
+```bash
+# Query data from main branch
+git prolly sql -b main "SELECT * FROM users"
+
+# Query data from a specific commit
+git prolly sql -b a1b2c3d4 "SELECT COUNT(*) FROM products"
+
+# Query data from a feature branch
+git prolly sql -b feature/new-schema "SELECT * FROM categories"
+```
+
+**Important Requirements:**
+- Only `SELECT` statements are allowed when using `-b` parameter
+- Your working directory must have clean status (no uncommitted staging changes)
+- The branch/commit will be temporarily checked out and restored after execution
+
+**Example with staging changes:**
+```bash
+# This will be blocked if you have uncommitted changes
+git prolly set user:123 "John Doe"  # Creates staging changes
+git prolly sql -b main "SELECT * FROM users"
+# Error: Cannot use -b/--branch parameter with uncommitted staging changes
+
+# Commit your changes first
+git prolly commit -m "Add new user"
+git prolly sql -b main "SELECT * FROM users"  # Now works
+```
+
 ## SQL Operations
 
 ### Supported SQL Features
@@ -329,9 +362,66 @@ prolly-sql> exit
 Goodbye!
 ```
 
+### Interactive Mode with Historical Data
+
+Use interactive mode to explore historical data:
+
+```bash
+# Start interactive mode against a specific branch
+git prolly sql -b feature/analytics -i
+```
+
+```
+🌟 ProllyTree SQL Interactive Shell
+====================================
+Executing against branch/commit: feature/analytics
+⚠️  Only SELECT statements are allowed in this mode
+Type 'exit' or 'quit' to exit
+Type 'help' for available commands
+
+prolly-sql> SELECT COUNT(*) FROM new_analytics_table;
+│ COUNT(*) │
+├──────────┤
+│ I64(150) │
+
+prolly-sql> SELECT * FROM products WHERE price > 1000;
+│ id     │ name                │ price     │
+├────────┼─────────────────────┼───────────┤
+│ I64(1) │ Str("Gaming PC")    │ I64(1500) │
+│ I64(2) │ Str("MacBook Pro")  │ I64(2000) │
+
+prolly-sql> INSERT INTO products VALUES (3, 'iPad', 800);
+Error: Only SELECT statements are allowed when using -b/--branch parameter
+       Historical commits/branches are read-only for data integrity
+
+prolly-sql> exit
+Goodbye!
+Restored to original branch: main
+```
+
 ## Advanced Examples
 
-### 1. Complex Data Analysis
+### 1. Historical Data Analysis
+
+Compare data across different points in time:
+
+```bash
+# Query current data
+git prolly sql "SELECT COUNT(*) as current_users FROM users"
+
+# Query data from last week's commit
+git prolly sql -b 7d1a2b3c "SELECT COUNT(*) as users_last_week FROM users"
+
+# Compare product prices between branches
+git prolly sql -b main "SELECT name, price FROM products WHERE category = 'Electronics'"
+git prolly sql -b feature/price-update "SELECT name, price FROM products WHERE category = 'Electronics'"
+
+# Analyze data growth over time
+git prolly sql -b v1.0 "SELECT COUNT(*) as v1_orders FROM orders"
+git prolly sql -b v2.0 "SELECT COUNT(*) as v2_orders FROM orders"
+```
+
+### 2. Complex Data Analysis
 
 ```sql
 -- Create sales data
@@ -461,10 +551,34 @@ git checkout main
 # The new tables don't exist on main branch
 git prolly sql "SELECT * FROM categories"  # Error: table not found
 
+# Query the new schema without switching branches
+git prolly sql -b feature/new-schema "SELECT * FROM categories"
+
 # Merge when ready
 git merge feature/new-schema
 ```
 
+### Cross-Branch Data Comparison
+
+Compare data between branches without switching contexts:
+
+```bash
+# Compare user counts between branches
+echo "Main branch users:"
+git prolly sql -b main "SELECT COUNT(*) FROM users"
+
+echo "Feature branch users:"
+git prolly sql -b feature/user-management "SELECT COUNT(*) FROM users"
+
+# Generate reports from different branches
+git prolly sql -b production -o json "SELECT * FROM daily_metrics WHERE date = '2024-01-15'" > prod_metrics.json
+git prolly sql -b staging -o json "SELECT * FROM daily_metrics WHERE date = '2024-01-15'" > staging_metrics.json
+
+# Compare table schemas between versions
+git prolly sql -b v1.0 "SELECT name FROM sqlite_master WHERE type='table'"
+git prolly sql -b v2.0 "SELECT name FROM sqlite_master WHERE type='table'"
+```
+
 ## Best Practices
 
 ### 1. Schema Design
@@ -497,6 +611,23 @@ git checkout -b migration-test
 # If successful, merge to main
 ```
 
+### 5. Historical Data Querying
+
+- **Commit changes before using `-b`**: Always commit your staging changes before querying historical data
+- **Use for read-only analysis**: The `-b` parameter is perfect for generating reports without affecting current work
+- **Branch-specific schemas**: Use `-b` to query data from branches with different table structures
+- **Performance**: Historical queries access committed data, so they may be slower than current branch queries
+
+```bash
+# Good practice: commit first
+git prolly commit -m "Save current work"
+git prolly sql -b production "SELECT * FROM metrics"
+
+# Avoid: Don't leave uncommitted changes
+git prolly set user:new "data"  # Uncommitted change
+git prolly sql -b main "SELECT * FROM users"  # Will be blocked
+```
+
 ## Troubleshooting
 
 ### Common Issues
@@ -515,6 +646,21 @@ git checkout -b migration-test
    - Check SQL syntax - the parser is strict about formatting
    - Ensure column names match exactly (case-sensitive)
 
+4. **"Cannot use -b/--branch parameter with uncommitted staging changes"**
+   - Check staging status with `git prolly status`
+   - Commit your changes first: `git prolly commit -m "Save changes"`
+   - Or discard changes if not needed
+
+5. **"Only SELECT statements are allowed when using -b/--branch parameter"**
+   - Historical data is read-only for safety
+   - Use regular `git prolly sql` (without `-b`) for data modifications
+   - Switch to the target branch if you need to make changes there
+
+6. **"Failed to checkout branch/commit"**
+   - Verify the branch/commit exists: `git branch -a` or `git log --oneline`
+   - Check branch name spelling (case-sensitive)
+   - Ensure you have access to the specified commit
+
 ### Performance Tips
 
 1. **Large Result Sets**: Use LIMIT to restrict output