feat(sources): add DynamoDB table provider (#141)

* feat(sources): add DynamoDB table provider

Add Amazon DynamoDB as a read-write data source so users can query and
mutate DynamoDB tables through Skardi's SQL/pipeline surface alongside
the existing backends. DynamoDB is a common operational store, and
pairing it with Skardi's federation lets users join live items against
files and other databases without an ETL hop.

Items map to rows and top-level attributes to columns; binary filters
push down as DynamoDB FilterExpressions, and INSERT/UPDATE/DELETE are
supported (UPDATE/DELETE resolve matching keys via a filtered scan,
since DynamoDB cannot mutate by predicate). No FTS/KNN, matching the
backend's native capabilities. Wired through the DataSourceType dispatch,
config/validation, CLI, and job-destination resolver (non-transactional).
Ships unit + ignore-gated integration tests (DynamoDB Local), a CI
service + seed step, and a runnable docs page verified end-to-end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sources): key-aware DynamoDB reads (GetItem/Query vs Scan)

Route DynamoDB reads to the cheapest physical access pattern the query's
predicates allow instead of always issuing a full Scan. A predicate that
pins the full primary key by equality uses GetItem; one that pins the
partition key (with an optional sort-key condition) uses Query; anything
else falls back to Scan. Because a filtered Scan still reads (and bills)
the entire table, this turns key lookups from O(table) into O(1)/O(partition).

The choice is driven only by the key portion of the predicates; all filters
stay Inexact so DataFusion re-applies every predicate after the fetch — a
misroute can only cost a fallback, never a wrong row. An operator whitelist
keeps NotEq/range predicates off the key (illegal in a KeyConditionExpression).

The key schema is now read authoritatively via DescribeTable at registration,
which also auto-detects the sort key; the partition_key/sort_key options
become a fallback for when DescribeTable is unavailable (e.g. restricted IAM).

Adds classifier unit tests and an ignore-gated composite-key integration test
covering the Query and composite GetItem paths against DynamoDB Local. Updates
the docs page with the read-planning model and the now-optional key options.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(server): cap aws_config logging at WARN to avoid credential leakage

aws_config logs the AWS access key id in plaintext at INFO while
resolving credentials. Cap the crate at WARN in the tracing filter
unless RUST_LOG explicitly mentions aws_config, keeping
credential-resolution debugging available as an opt-in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sources): address DynamoDB provider review (safety, correctness, efficiency)

Resolves the deep-review findings on the DynamoDB provider:

Safety / correctness
- DELETE/UPDATE now reject any WHERE predicate that isn't a fully pushable
  comparison instead of silently dropping it — `DELETE ... WHERE a=1 OR b=2`
  no longer wipes the whole table.
- Enforce access_mode: a read_only source rejects INSERT/UPDATE/DELETE at plan
  time (threaded through register_dynamodb_tables + provider; wired in server
  config and CLI).
- Plain INSERT (Append) is a conditional PutItem (attribute_not_exists on the
  partition key) — a duplicate key errors rather than silently replacing the
  whole item. INSERT OVERWRITE/REPLACE opt into upsert.
- Numeric coercion is type-strict: fractional N no longer truncates into Int64
  columns and strings are never coerced into numeric columns (both -> NULL),
  so Inexact pushdown stays a superset. N now infers as Float64 by default.

Robustness
- Schema inference samples several items and merges attributes; new `columns`
  option declares an explicit typed schema (needed for empty tables).

Efficiency
- Scan/Query stream one page at a time (bounded memory) with a
  ProjectionExpression and a request-level limit; page items are moved, not
  cloned. count(*) uses Scan Select=COUNT.
- DELETE/UPDATE route to Query when the partition key is pinned (residual
  predicate applied server-side) instead of always scanning; DELETE removes
  keys via BatchWriteItem, upsert INSERT batches via BatchWriteItem.

Cleanup
- Lift is_pushable_binary_filter to providers/mod.rs (shared with mongo);
  derive(Clone) replaces manual clone_op/clone_filter; single shared paginator
  (fetch_page) replaces three hand-rolled loops; drop the RwLock<Option> scan
  batch and its unreachable "consumed" arm.
- docs/jobs.md: list DynamoDB and InfluxDB as non-transactional destinations.

cargo fmt, cargo check --workspace --all-targets, and the skardi (26 dynamodb)
+ skardi-server unit suites are green. Live DynamoDB integration tests remain
#[ignore]-gated (run in CI against dynamodb-local).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(jobs): also list DynamoDB/InfluxDB in the error-type table

Follow-up to 90c6de6: the non_transactional_destination row in the
error-type reference still named only Redis/MongoDB/SeekDB.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(sources): expand DynamoDB provider unit coverage

Add 21 network-free tests covering previously-untested pure helpers and
plan-time write guards: schema shaping, item⇄RecordBatch conversion,
projection/key-condition expression builders, operator/normalization
helpers, value/count builders, key extraction, and the read-only /
key-immutability DML guards.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* amend fixes

* test(sources): cover DynamoDB attribute merge, NULL handling, and plan shapes

Extract merge_sampled_attributes() from schema inference and add unit
tests for first-seen-type-wins merging, explicit NULL round-tripping to
Arrow nulls, update-expression placeholder namespacing, and the count
row shape advertised by the count/insert execution plans.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: cover DynamoDB wiring across server, CLI, and jobs crates

Codecov flagged the provider-wiring diff at 17.85% patch coverage. Cover
the uncovered arms without needing a live DynamoDB endpoint:

- config.rs: validate/register paths for the Dynamodb source type
  (read_write allowed, missing connection string, registration failure
  mapping), plus the updated UnsupportedWriteMode message
- cli/main.rs: register_source dynamodb arm (missing connection_string,
  missing options)
- server/main.rs: extract build_env_filter() and test the aws_config
  log-cap behavior for default, custom, opt-in, and invalid RUST_LOG
- pipeline_handlers.rs: /data_source reports dynamodb as a url source
- gui.rs: data_source_type_str covers Dynamodb
- executor.rs: dynamodb destination rejected as non-transactional
  (shared helper with the influxdb case)
- data_source_type.rs: dynamodb variant serde/as_str roundtrip

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Ubuntu <boyang@claude-dev.yv012fvwp44enno1egfnxmb5fh.xx.internal.cloudapp.net>

Author: 3 people

.github/workflows/ci.yml

@@ -106,6 +106,13 @@ jobs:
           --health-timeout=10s
           --health-retries=20
 
+      # DynamoDB Local has no in-image shell tooling for a healthcheck, so
+      # readiness is polled with the AWS CLI in a dedicated step below.
+      dynamodb:
+        image: amazon/dynamodb-local:2.5.2
+        ports:
+          - 8000:8000
+
     env:
       MYSQL_USER: skardi_user
       MYSQL_PASSWORD: skardi_pass
@@ -117,6 +124,11 @@ jobs:
       SEEKDB_PASSWORD: ""
       INFLUXDB_URL: "http://127.0.0.1:8181"
       INFLUXDB_DATABASE: "metrics"
+      # DynamoDB Local ignores credential values but the AWS SDK requires them
+      # to be present. The endpoint is supplied via the context/test config.
+      AWS_ACCESS_KEY_ID: dummy
+      AWS_SECRET_ACCESS_KEY: dummy
+      AWS_DEFAULT_REGION: us-east-1
 
     steps:
       - name: Checkout code
@@ -448,6 +460,35 @@ jobs:
           mem,host=host3,region=us-east used_percent=31.5 1700000000
           EOF
 
+      - name: Seed DynamoDB data
+        env:
+          EP: http://127.0.0.1:8000
+        run: |
+          # Wait for DynamoDB Local to accept connections.
+          for i in $(seq 1 30); do
+            if aws dynamodb list-tables --endpoint-url "$EP" >/dev/null 2>&1; then
+              echo "DynamoDB Local is ready"
+              break
+            fi
+            echo "Waiting for DynamoDB Local... (attempt $i/30)"
+            sleep 2
+          done
+
+          aws dynamodb create-table --endpoint-url "$EP" \
+            --table-name products \
+            --attribute-definitions AttributeName=product_id,AttributeType=S \
+            --key-schema AttributeName=product_id,KeyType=HASH \
+            --billing-mode PAY_PER_REQUEST
+          aws dynamodb wait table-exists --endpoint-url "$EP" --table-name products
+
+          put() { aws dynamodb put-item --endpoint-url "$EP" --table-name products --item "$1"; }
+          put '{"product_id":{"S":"PROD001"},"name":{"S":"Laptop"},"category":{"S":"Electronics"},"price":{"N":"999.99"},"in_stock":{"BOOL":true}}'
+          put '{"product_id":{"S":"PROD002"},"name":{"S":"Keyboard"},"category":{"S":"Electronics"},"price":{"N":"79.99"},"in_stock":{"BOOL":true}}'
+          put '{"product_id":{"S":"PROD003"},"name":{"S":"Monitor"},"category":{"S":"Electronics"},"price":{"N":"299.99"},"in_stock":{"BOOL":false}}'
+          put '{"product_id":{"S":"PROD004"},"name":{"S":"Mouse"},"category":{"S":"Electronics"},"price":{"N":"29.99"},"in_stock":{"BOOL":true}}'
+          # NULL-bearing row: no `category` attribute, exercising NULL handling.
+          put '{"product_id":{"S":"PROD005"},"name":{"S":"Desk Chair"},"price":{"N":"199.99"},"in_stock":{"BOOL":true}}'
+
       - name: Execute Integration tests
         run: cargo llvm-cov --no-report nextest --all-features -- --ignored