Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 23 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Skardi lets AI agents and applications query files, databases, data lakes, and v
- **CLI for local agents & queries** — Run SQL against local files, remote object stores (S3, GCS, Azure), databases, and datalake formats — ideal for local AI agents like [OpenClaw](https://github.com/openclaw/openclaw)
- **Declarative pipelines** — Define SQL queries in YAML, get REST APIs automatically
- **Automatic parameter inference** — Request parameters, types, and response schemas are inferred from your SQL
- **Multi-source federation** — JOIN across CSV, Parquet, PostgreSQL, MySQL, SQLite, MongoDB, Iceberg, and Lance in a single query
- **Multi-source federation** — JOIN across CSV, Parquet, PostgreSQL, MySQL, SQLite, MongoDB, Redis, Iceberg, and Lance in a single query
- **Full CRUD** — SELECT, INSERT, UPDATE, and DELETE operations on supported databases
- **Vector search** — Native KNN similarity search via Lance integration
- **S3 support** — Read CSV, Parquet, and Lance files directly from S3
Expand All @@ -54,6 +54,7 @@ Skardi lets AI agents and applications query files, databases, data lakes, and v
- [MySQL](#mysql)
- [SQLite](#sqlite)
- [MongoDB](#mongodb)
- [Redis](#redis)
- [Apache Iceberg](#apache-iceberg)
- [Lance (Vector Search)](#lance-vector-search)
- [S3 Remote Files](#s3-remote-files)
Expand Down Expand Up @@ -133,7 +134,7 @@ skardi query --ctx ./ctx.yaml --schema -t products
| Local files | CSV, Parquet, JSON/NDJSON, Lance |
| Remote stores | S3, GCS, Azure Blob, HTTP/HTTPS, OSS, COS |
| Datalake formats | Lance, Iceberg |
| Databases | PostgreSQL, MySQL, SQLite, MongoDB |
| Databases | PostgreSQL, MySQL, SQLite, MongoDB, Redis |

**Context file resolution** (when `--ctx` is omitted): checks `SKARDICONFIG` env var, then `~/.skardi/config/ctx.yaml`. If no context file is found, the query runs without pre-registered tables (you can still query files directly by path).

Expand Down Expand Up @@ -206,7 +207,7 @@ data_sources:

### Access Mode

By default, all data sources are **read-only** — only `SELECT` queries are allowed. To enable write operations (`INSERT`, `UPDATE`, `DELETE`), set `access_mode: read_write` on the data source. Only `postgres`, `mysql`, and `sqlite` sources support `read_write` mode; setting it on other types will produce an error at startup.
By default, all data sources are **read-only** — only `SELECT` queries are allowed. To enable write operations (`INSERT`, `UPDATE`, `DELETE`), set `access_mode: read_write` on the data source. Only `postgres`, `mysql`, `sqlite`, `mongo`, and `redis` sources support `read_write` mode; setting it on other types will produce an error at startup.

```yaml
data_sources:
Expand Down Expand Up @@ -412,6 +413,24 @@ export MONGO_PASS="mypassword"

For detailed setup, CRUD examples, and federated queries, see [demo/mongo/MONGO_DEMO.md](demo/mongo/MONGO_DEMO.md).

### Redis

Full CRUD support with point lookups (O(1) via direct key construction), full scans, and federated queries. Redis hashes map directly to SQL rows.

```yaml
- name: "products"
type: "redis"
connection_string: "redis://localhost:6379"
options:
key_space: "mydb"
table: "products"
key_column: "product_id"
```

Redis keys follow the pattern `{key_space}:{table}:{key_column_value}`, where `key_column` is extracted from the key suffix and exposed as a SQL column. For initially empty tables, use the `columns` option to declare the schema upfront so INSERT operations work immediately.

For detailed setup, CRUD examples, and federated queries, see [demo/redis/REDIS_DEMO.md](demo/redis/REDIS_DEMO.md).

### Apache Iceberg

Query Iceberg tables with support for schema evolution, partition pruning, and time travel.
Expand Down Expand Up @@ -612,6 +631,7 @@ The [demo/](demo/) directory contains complete working examples:
| [demo/mysql/](demo/mysql/) | MySQL CRUD and federated query examples |
| [demo/sqlite/](demo/sqlite/) | SQLite CRUD and federated query examples |
| [demo/mongo/](demo/mongo/) | MongoDB CRUD and federated query examples |
| [demo/redis/](demo/redis/) | Redis CRUD and federated query examples |
| [demo/iceberg/](demo/iceberg/) | Apache Iceberg integration examples |
| [demo/lance/](demo/lance/) | Lance vector search examples |
| [demo/onnx_predict/](demo/onnx_predict/) | ONNX model inference in SQL |
Expand Down
50 changes: 47 additions & 3 deletions crates/server/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ pub enum DataSourceType {
Sqlite,
Iceberg,
Mongo,
Redis,
Lance,
}

Expand Down Expand Up @@ -145,7 +146,7 @@ pub enum ConfigError {
#[error("S3 object store registration failed: {name} - {error}")]
S3ObjectStoreRegistrationFailed { name: String, error: String },

#[error("Data source '{name}' has access_mode 'read_write' but type '{source_type:?}' does not support write operations. Only 'postgres', 'mysql', 'sqlite', and 'mongo' sources support read_write mode.")]
#[error("Data source '{name}' has access_mode 'read_write' but type '{source_type:?}' does not support write operations. Only 'postgres', 'mysql', 'sqlite', 'mongo', and 'redis' sources support read_write mode.")]
UnsupportedWriteMode {
name: String,
source_type: DataSourceType,
Expand Down Expand Up @@ -435,6 +436,7 @@ const WRITABLE_SOURCE_TYPES: &[DataSourceType] = &[
DataSourceType::Mysql,
DataSourceType::Sqlite,
DataSourceType::Mongo,
DataSourceType::Redis,
];

/// Validate data source configurations
Expand Down Expand Up @@ -473,7 +475,13 @@ fn validate_data_sources(data_sources: &[DataSource]) -> Result<()> {
// Validate S3 configuration for S3 paths
s3_storage.validate_configuration(source)?;
}
(DataSourceType::Postgres | DataSourceType::Mysql | DataSourceType::Mongo, false) => {
(
DataSourceType::Postgres
| DataSourceType::Mysql
| DataSourceType::Mongo
| DataSourceType::Redis,
false,
) => {
// For database connections, ensure connection string is provided
if source.connection_string.is_none() {
return Err(ConfigError::MissingConnectionString {
Expand Down Expand Up @@ -637,7 +645,13 @@ async fn register_data_source(
.setup_object_store(session_ctx, &source.name, s3_path)
.await?;
}
(DataSourceType::Postgres | DataSourceType::Mysql | DataSourceType::Mongo, _) => {
(
DataSourceType::Postgres
| DataSourceType::Mysql
| DataSourceType::Mongo
| DataSourceType::Redis,
_,
) => {
// Database sources don't need file path validation
}
(DataSourceType::Iceberg, _) => {
Expand Down Expand Up @@ -879,6 +893,36 @@ async fn register_data_source(
}
})?;
}
DataSourceType::Redis => {
tracing::info!("Registering Redis table: {}", source.name);

let connection_string = source.connection_string.as_ref().ok_or_else(|| {
ConfigError::MissingConnectionString {
name: source.name.clone(),
}
})?;

tracing::debug!(
"Connection string for {}: {} (options: {:?})",
source.name,
connection_string,
source.options
);

source::redis::datasource::register_redis_tables(
session_ctx,
&source.name,
connection_string,
source.options.as_ref(),
)
.map_err(|e| {
tracing::error!("Redis registration failed for '{}': {:?}", source.name, e);
ConfigError::DataSourceRegistrationFailed {
name: source.name.clone(),
error: format!("{:?}", e),
}
})?;
}
DataSourceType::Lance => {
tracing::info!(
"Registering Lance dataset: {} at {:?}",
Expand Down
11 changes: 9 additions & 2 deletions crates/server/src/handlers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -390,6 +390,7 @@ pub async fn get_data_sources(
DataSourceType::Mongo => "mongo",
DataSourceType::Sqlite => "sqlite",
DataSourceType::Lance => "lance",
DataSourceType::Redis => "redis",
};

// Determine path or URL based on source type
Expand All @@ -399,11 +400,17 @@ pub async fn get_data_sources(
| DataSourceType::Lance
| DataSourceType::Sqlite
| DataSourceType::Iceberg => Some(data_source.path.to_string_lossy().to_string()),
DataSourceType::Postgres | DataSourceType::Mysql | DataSourceType::Mongo => None,
DataSourceType::Postgres
| DataSourceType::Mysql
| DataSourceType::Mongo
| DataSourceType::Redis => None,
};

let url = match data_source.source_type {
DataSourceType::Postgres | DataSourceType::Mysql | DataSourceType::Mongo => {
DataSourceType::Postgres
| DataSourceType::Mysql
| DataSourceType::Mongo
| DataSourceType::Redis => {
// For database sources, return the connection string as-is
// (credentials are not stored in connection strings, only in env vars)
data_source.connection_string.clone()
Expand Down
Loading
Loading