feat(sources): add ClickHouse table provider

[abbccdda]

Summary

Adds ClickHouse as a read-only data source, backed by the clickhouse feature of datafusion-table-providers. Filters, projections, and LIMITs are unparsed back to ClickHouse SQL and executed server-side, so scans stream only the rows a query actually needs — and ClickHouse tables federate with any other Skardi source in a single query.

Details

• Two registration modes, mirroring the other SQL providers:
  • Table mode (default): one ClickHouse table registered under the source name, with an optional database option.
  • Catalog mode: every table across the server's non-system databases registered as name.<database>.<table>, filtered by an optional allowed_schemas allow-list.
• Credentials come from environment variables via user_env / pass_env options and are never embedded in the connection URL.
• Read-only by design: ClickHouse mutations (ALTER TABLE ... UPDATE/DELETE) are asynchronous background rewrites and a mid-stream INSERT failure leaves partial parts visible, which doesn't match the semantics Skardi's write path promises. access_mode: read_write and job destinations are rejected for ClickHouse sources.
• CI: a clickhouse/clickhouse-server:24.8 service container with seeded fixture tables (including a NULL-column and an empty table for schema-inference coverage); integration tests are #[ignore]-gated and run via the existing --ignored pass.
• Robust catalog registration (review follow-up): best-effort per-table assembly (one broken view or Kafka-engine table no longer aborts startup), with stream-like engines and materialized-view .inner* tables filtered up front, and introspection batched into a single system.tables query.
• Validation at the provider boundary (review follow-up): unknown options, mode-mismatched options, empty allowed_schemas, URL-embedded credentials, query strings, and access_mode: read_write are all hard errors in register_clickhouse_tables itself, so the CLI and public API enforce the same contract as server config validation — and secrets can't reach logs or the data-sources API.
• Docs: quick-start guide, table- and catalog-mode demo contexts, and federated demo pipelines (ClickHouse × CSV join) under docs/clickhouse/, plus a README source-matrix row.

Test plan

• Unit tests: option parsing, URL validation, and DataSourceType roundtrip (cargo nextest run)
• #[ignore]-gated integration tests against the CI ClickHouse service container: registration, scans, filter/limit pushdown, NULL handling, empty-table schema inference, catalog mode
• submit_rejects_clickhouse_destination_as_non_transactional covers the job-destination rejection

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.96864% with 50 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/server/src/config.rs	70.27%	22 Missing ⚠️
crates/skardi/src/sources/providers/clickhouse.rs	97.07%	19 Missing ⚠️
crates/skardi/src/sources/providers/mod.rs	80.43%	9 Missing ⚠️

📢 Thoughts on this report? Let us know!

[abbccdda]

Review: ClickHouse table provider

Overall a solid, well-tested addition that follows the house patterns (env-based credentials, eager schema fetch, read-only enforced at config/executor/provider layers, the count(*) wrapper mirroring influxdb.rs). Findings are posted as inline comments, ordered by production impact (#1–#5). I verified the upstream claims against the vendored datafusion-table-providers 0.10.1 source.

If prioritizing: #1 (best-effort catalog) before merge; #2/#3 are small changes with security/correctness payoff; #4/#5 can be fast-follows.

Minor

• Doc inaccuracy: clickhouse.rs:256 (and the README) say schema is inferred via DESCRIBE TABLE; upstream actually uses SELECT * … LIMIT 0 through the ArrowStream format plus an engine lookup in system.tables. Worth correcting since the mechanism determines which failures occur (e.g. the Kafka-engine case in #1).
• No timeout around per-table schema fetch: build_clickhouse_table_provider isn't wrapped in retry_with_timeout, so an endpoint that hangs after listing succeeds stalls startup indefinitely. Matches mysql (inherited gap), so not a blocker — but catalog mode multiplies exposure by table count.
• No config-validation tests: the DynamoDB PR added validate_accepts_dynamodb_read_write / catalog-option tests in config.rs; ClickHouse has none — read_write rejection rides the generic UnsupportedWriteMode path untested for this type, and a test for #3 (database in catalog mode) would have caught that gap.
• Type-coverage gap in fixtures: CI tables only use UInt32/String/Float64/Bool/DateTime/Nullable(String). The types most likely to break ClickHouse→Arrow mapping — LowCardinality(String) (dictionary encoding can differ between the LIMIT-0 schema fetch and real data batches, and upstream derives the stream schema from the first batch), DateTime64, Decimal, Enum8, UUID, Array — are untested. One kitchen-sink types fixture would harden this a lot. A catalog-mode test including a VIEW would also cover the view branch in upstream get_schema.

What checks out

• Read-only stance enforced consistently: config validation, JobExecutor destination rejection (with test), no write path in the provider.
• Option validation fires before any network call (tested offline-safe).
• Upstream behaviors the docs rely on verified against vendored 0.10.1 source: schemas() excludes system/information_schema, and nonexistent allowed_schemas entries contribute nothing rather than erroring.

Verification (run locally against clickhouse/clickhouse-server:24.8)

• Unit + integration tests: cargo test -p skardi clickhouse -- --include-ignored → 20 passed, 0 failed (12 unit + all 8 live integration tests: register/scan, filter pushdown, count(*) empty projection, NULL handling, empty-table schema inference, numeric aggregation, catalog mode, plus the executor destination-rejection test).
• README demo, table mode: all 5 pipelines (query_user_by_id, list_all_users, products_by_category, user_order_summary, federated_stock_value) return exactly the documented rows. One cosmetic nit: federated_stock_value returns stock_value: 16999.390000000003 where the README shows 16999.39 — consider a ROUND(..., 2) in the pipeline or noting float formatting.
• README demo, catalog mode: both catalog_demo pipelines return the documented rows; catalog registered 4 tables.
• Finding #1 repro: CREATE TABLE mydb.kafka_events (...) ENGINE = Kafka → catalog-mode server exits 1 at startup with Code: 620 Direct select is not allowed.
• Finding #2 repro: context with connection_string: http://skardi_user:skardi_pass@localhost:8123 and no user_env/pass_env → embedded creds ignored, 3 retries of default: Authentication failed, registration fails; warning + full connection string appear in server logs.