Implement Nats K/V Streamlining into OPA memory

.cursor/rules/golang-master.mdc

@@ -0,0 +1,31 @@
+---
+alwaysApply: true
+---
+  You are an expert AI programming assistant specializing in building APIs with Go, using the standard library's net/http package and the new ServeMux introduced in Go 1.22.
+
+  Always use the latest stable version of Go (1.22 or newer) and be familiar with RESTful API design principles, best practices, and Go idioms.
+
+  - Follow the user's requirements carefully & to the letter.
+  - First think step-by-step - describe your plan for the API structure, endpoints, and data flow in pseudocode, written out in great detail.
+  - Confirm the plan, then write code!
+  - Write correct, up-to-date, bug-free, fully functional, secure, and efficient Go code for APIs.
+  - Use the standard library's net/http package for API development:
+    - Utilize the new ServeMux introduced in Go 1.22 for routing
+    - Implement proper handling of different HTTP methods (GET, POST, PUT, DELETE, etc.)
+    - Use method handlers with appropriate signatures (e.g., func(w http.ResponseWriter, r *http.Request))
+    - Leverage new features like wildcard matching and regex support in routes
+  - Implement proper error handling, including custom error types when beneficial.
+  - Use appropriate status codes and format JSON responses correctly.
+  - Implement input validation for API endpoints.
+  - Utilize Go's built-in concurrency features when beneficial for API performance.
+  - Follow RESTful API design principles and best practices.
+  - Include necessary imports, package declarations, and any required setup code.
+  - Implement proper logging using the standard library's log package or a simple custom logger.
+  - Consider implementing middleware for cross-cutting concerns (e.g., logging, authentication).
+  - Implement rate limiting and authentication/authorization when appropriate, using standard library features or simple custom implementations.
+  - Leave NO todos, placeholders, or missing pieces in the API implementation.
+  - Be concise in explanations, but provide brief comments for complex logic or Go-specific idioms.
+  - If unsure about a best practice or implementation detail, say so instead of guessing.
+  - Offer suggestions for testing the API endpoints using Go's testing package.
+
+  Always prioritize security, scalability, and maintainability in your API designs and implementations. Leverage the power and simplicity of Go's standard library to create efficient and idiomatic APIs.

.cursor/rules/project-master.mdc

@@ -0,0 +1,125 @@
+---
+alwaysApply: true
+---
+# Cursor Rules — NATS Plugin for OPA
+
+You are editing **github.com/permitio/opa-nats**. The library implements an **LRU cache backed by NATS Key-Value (JetStream)** that *selectively* overrides OPA's runtime store for paths matched by regexes. It is **not** a full OPA storage implementation.
+
+## Core Principles
+
+1. **Do not change the contract**: This is a cache + path router around an embedded `storage.Store`. Only `Read`/`Write` are overridden; everything else must go to the embedded store.
+2. **Regex decides routing**: `handled_paths_regex` is the single source of truth. If it’s empty, nothing is routed to NATS.
+3. **NATS is source of truth for handled paths**: Reads fall back to NATS on cache miss. Writes go to NATS, then update the LRU cache.
+4. **Keep the cache simple**: No hidden global state, no additional layers unless explicitly justified.
+5. **Production-grade (practical) engineering**: clear errors, timeouts, reconnect logic, metrics/status surface, and structured logs.
+
+## Go / Tooling
+
+- **Go version**: match the `go.mod` (bump only with CI + docs update).
+- **Linting**: keep `golangci-lint` clean; fix or justify with inline comments.
+- **Static analysis**: `go vet`, `staticcheck` where possible.
+- **Error handling**: wrap with `%w`; give context (`fmt.Errorf("connect NATS: %w", err)`).
+- **Context**: every external I/O (NATS ops, watchers) must accept `context.Context`.
+- **Generics**: use only when it simplifies code meaningfully.
+
+## Concurrency & Lifecycles
+
+- Watchers/goroutines must:
+  - Respect `ctx.Done()`.
+  - Return on shutdown.
+  - Avoid leaks (document ownership & stop semantics).
+- Reconnect logic:
+  - Honor `max_reconnect_attempts` and `reconnect_wait`.
+  - Surface status to OPA status API.
+
+## Configuration Rules
+
+- Keep config in `natsstore.Config`. Add fields only when needed, document them in README and code comments.
+- Validate config on startup (URLs, TTLs, bucket, auth mode exclusivity).
+- If both `credentials` and `token`/`username` are provided, error out with a helpful message.
+
+## Path Handling
+
+- **Never** route non-matching paths to NATS.
+- Path mapping: OPA `["data","schemas","user"]` → NATS key `data.schemas.user`. Any change here must be explicitly documented and migration noted.
+- Regex must be compiled once and reused.
+
+## Logging & Monitoring
+
+- Structured logging only (key/value). No `Printf`.
+- Respect `log_level` in config.
+- Status API should expose:
+  - Cache size & capacity.
+  - NATS connection state.
+  - Trigger/watch count.
+  - Policy count (from embedded store).
+
+## Testing
+
+- **Unit tests** for:
+  - Regex routing (hit/miss).
+  - Cache TTL behavior.
+  - NATS read/write fallbacks.
+  - Reconnect logic.
+- **Integration tests** (with JetStream) behind a build tag (e.g. `integration`).
+- **Race detector** (`-race`) must pass.
+- Maintain/raise coverage; do not merge coverage regressions without rationale.
+
+## CI
+
+- Do not break `.github/workflows/pr_checks.yml`.
+- Add new tooling steps only if they’re reproducible locally.
+
+## Commits & PRs
+
+- Use **Conventional Commits**:
+  - `feat:`, `fix:`, `refactor:`, `docs:`, `test:`, `ci:`, `chore:`
+- Each PR must:
+  - Explain what path(s) it touches (NATS vs embedded store).
+  - List any config or status API changes.
+  - Include tests or justify why not.
+
+## Docs
+
+- README is the single entry point for users. If you add/rename:
+  - Config fields
+  - Status metrics
+  - Path mapping logic
+  then **update README, examples, and comments**.
+- Avoid marketing-y fluff and filler words.
+
+## Security
+
+- Never log secrets (token, username, password, creds path contents).
+- TLS flags must be honored; `tls_insecure` defaults to `false`.
+- Encourage secure defaults in examples.
+
+## Troubleshooting & Errors
+
+- Emit actionable errors:
+  - Connection failures: include URL and selected auth mode (not the secret).
+  - Bucket not found: suggest permission or creation options.
+- Prefer retryable errors where appropriate; otherwise fail fast with context.
+
+## Style
+
+- Keep files short and focused. Split out packages only when it clarifies ownership/boundaries.
+- Prefer small, cohesive interfaces.
+- Name things after what they *do* (e.g., `pathRouter`, `kvBackedCache`).
+
+## Banned Phrases (do not write these)
+
+- dynamic world
+- evolving landscape
+- the world of
+- digital age
+- ever-evolving
+- digital landscape
+- robust
+- myriad
+
+## When You’re Unsure
+
+- Default to the simplest change that preserves public behavior.
+- Add a small test that demonstrates the intent.
+- Leave a short comment pointing back to the README section that explains the behavior.

.github/workflows/pr_checks.yml

@@ -0,0 +1,213 @@
+name: PR Checks
+
+on:
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  lint:
+    name: Lint
+    runs-on: ubuntu-latest
+    services:
+      nats:
+        image: nats:2.10.22-alpine
+        ports:
+          - 4222:4222
+          - 8222:8222
+        options: >-
+          --name nats-ci-lint-${{ github.run_id }}
+          --health-cmd "wget --no-verbose --tries=1 --spider http://localhost:8222/healthz"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v4
+        with:
+          go-version: '1.22'
+      - name: Setup NATS Jetstream for Lint
+        run: |
+          # Setup NATS with JetStream for linting
+          docker cp ${{ github.workspace }}/.github/workflows/utils/nats-jetstream.conf nats-ci-lint-${{ github.run_id }}:/tmp/nats-jetstream.conf
+          docker exec nats-ci-lint-${{ github.run_id }} sh -c 'cat /tmp/nats-jetstream.conf > /etc/nats/nats-server.conf'
+          docker restart nats-ci-lint-${{ github.run_id }}
+
+          # Wait for NATS to be ready
+          timeout=30
+          start_time=$(date +%s)
+          while true; do
+            if curl --fail --silent http://localhost:8222/healthz; then
+              echo "NATS is ready for linting!"
+              break
+            fi
+            current_time=$(date +%s)
+            elapsed=$((current_time - start_time))
+            if [ $elapsed -ge $timeout ]; then
+              echo "Timeout waiting for NATS"
+              exit 1
+            fi
+            sleep 1
+          done
+      - name: golangci-lint
+        uses: golangci/golangci-lint-action@v8
+        with:
+          version: v2.3.1
+          args: --timeout=10m
+      - uses: pre-commit/action@v3.0.1
+
+  test:
+    name: Test
+    runs-on: ubuntu-latest
+    services:
+      nats:
+        image: nats:2.10.22-alpine
+        ports:
+          - 4222:4222
+          - 8222:8222
+        options: >-
+          --name nats-ci-${{ github.run_id }}
+          --health-cmd "wget --no-verbose --tries=1 --spider http://localhost:8222/healthz"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        # we can't have health check becasue at the moment the nats server
+        # is not configured with monitoring.
+        # this is due to nats not having environment variables configuration available,
+        # and githuhb actions not allowing to edit the CMD of the container.
+        # see https://github.com/nats-io/nats-docker/issues/110
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.23'
+          cache: true
+      - name: Setup NATS Jetstream
+        run: |
+          # Print current entrypoint command of the container
+          echo "Current container entrypoint command:"
+          docker inspect nats-ci-${{ github.run_id }} --format='{{.Config.Entrypoint}} {{.Config.Cmd}}' || echo "Failed to get entrypoint info"
+
+          # Find and print the full path of the entrypoint file
+          echo "Discovering entrypoint file path:"
+          docker exec nats-ci-${{ github.run_id }} find / -name "docker-entrypoint.sh" 2>/dev/null || echo "docker-entrypoint.sh not found"
+          docker exec nats-ci-${{ github.run_id }} which docker-entrypoint.sh 2>/dev/null || echo "docker-entrypoint.sh not in PATH"
+
+          # Check what's in common entrypoint locations
+          echo "Checking common entrypoint locations:"
+          docker exec nats-ci-${{ github.run_id }} ls -la /usr/local/bin/ | grep -E "(entrypoint|docker)" || echo "No entrypoint files in /usr/local/bin/"
+          docker exec nats-ci-${{ github.run_id }} ls -la /docker-entrypoint.sh 2>/dev/null || echo "No /docker-entrypoint.sh"
+          docker exec nats-ci-${{ github.run_id }} ls -la /entrypoint.sh 2>/dev/null || echo "No /entrypoint.sh"
+
+          # Show current NATS configuration
+          echo "Current NATS configuration:"
+          docker exec nats-ci-${{ github.run_id }} cat /etc/nats/nats-server.conf || echo "No config file found"
+
+          # Mount the JetStream configuration from utils directory
+          echo "Mounting NATS JetStream configuration"
+          docker cp ${{ github.workspace }}/.github/workflows/utils/nats-jetstream.conf nats-ci-${{ github.run_id }}:/tmp/nats-jetstream.conf
+
+          # Replace the content of the existing config file with our JetStream config
+          echo "Replacing NATS configuration content with JetStream config"
+          docker exec nats-ci-${{ github.run_id }} sh -c 'cat /tmp/nats-jetstream.conf > /etc/nats/nats-server.conf'
+
+          # Show updated configuration
+          echo "Updated NATS configuration:"
+          docker exec nats-ci-${{ github.run_id }} cat /etc/nats/nats-server.conf
+
+          # Restart the container to pick up the new configuration
+          echo "Restarting NATS container with JetStream configuration..."
+          docker restart nats-ci-${{ github.run_id }}
+
+          # Print initial NATS logs for debugging
+          echo "Initial NATS container logs after restart:"
+          docker logs nats-ci-${{ github.run_id }}
+          echo "Container status:"
+          docker ps -a | grep nats-ci-${{ github.run_id }}
+
+          # ensure the nats server is ready by running curl to the health endpoint
+          # in while loop until success or timeout of 10 seconds of failures
+          timeout=10
+          start_time=$(date +%s)
+          while true; do
+            if curl --fail --silent --show-error http://localhost:8222/healthz?js-enabled-only=true; then
+              echo "NATS Jetstream is ready!"
+              echo "Verifying JetStream configuration:"
+              jsz_response=$(curl -s http://localhost:8222/jsz)
+              echo "$jsz_response" | jq '.' || echo "JetStream info not available"
+
+              # Check if JetStream is disabled
+              if echo "$jsz_response" | jq -e '.disabled == true' > /dev/null 2>&1; then
+                echo "ERROR: JetStream is disabled in the server configuration!"
+                echo "JetStream response: $jsz_response"
+                exit 1
+              elif echo "$jsz_response" | jq -e '.disabled == false' > /dev/null 2>&1; then
+                echo "SUCCESS: JetStream is enabled and configured properly"
+              else
+                echo "WARNING: Could not determine JetStream disabled status from response"
+                echo "JetStream response: $jsz_response"
+              fi
+
+              echo "Final NATS container logs:"
+              docker logs nats-ci-${{ github.run_id }}
+              break
+            fi
+            current_time=$(date +%s)
+            elapsed=$((current_time - start_time))
+            if [ $elapsed -ge $timeout ]; then
+              echo "Timeout reached after ${timeout} seconds"
+              echo "Container logs:"
+              docker logs nats-ci-${{ github.run_id }}
+              echo "Container status:"
+              docker ps -a | grep nats-ci-${{ github.run_id }}
+              exit 1
+            fi
+            sleep 1
+          done
+      - name: Run tests
+        run: go test -race -coverprofile=coverage.txt -covermode=atomic ./...
+      - name: Upload coverage reports
+        uses: codecov/codecov-action@v4
+        with:
+          file: ./coverage.txt
+          fail_ci_if_error: false
+
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.23'
+          cache: true
+      - name: Verify dependencies
+        run: go mod verify
+      - name: Build
+        run: go build -v ./...
+      - name: Check formatting
+        run: |
+          gofmt_output=$(gofmt -l -d .)
+          if [ -n "$gofmt_output" ]; then
+            echo "Code is not properly formatted:"
+            echo "$gofmt_output"
+            exit 1
+          fi
+
+  security:
+    name: Security Check
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.23'
+          cache: true
+      - name: Install govulncheck
+        run: go install golang.org/x/vuln/cmd/govulncheck@latest
+      - name: Run govulncheck
+        run: govulncheck ./...
+        continue-on-error: true  # Make this check informational rather than blocking
+      - name: Report vulnerabilities
+        run: |
+          echo "::warning ::Security vulnerabilities were found. Please review the govulncheck output above."
+        if: ${{ failure() }}