Skip to content

Commit 73f76aa

Browse files
authored
Quent Task Instrumentation (#809)
Defines a task model based on the Sirius' execution semantics, and adds instrumentation to capture information about task events. Also add basic analyzers that [quent](github.com/rapidsai/quent) can use to generate query timelines for tasks. Notes - This PR instruments the task lifecycle, not the new scan split lifecycle. - The new upstream scan path is covered because `sirius_gpu_scan_operator::execute()` still runs under `gpu_pipeline_task`. - Split production, footer reads, and row-group pruning happen before task creation and remain outside the task FSM.
1 parent e326e58 commit 73f76aa

48 files changed

Lines changed: 4102 additions & 105 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CMakeLists.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -263,6 +263,7 @@ set(EXTENSION_SOURCES
263263
src/parallel/task_executor.cpp
264264
src/pipeline/gpu_pipeline_executor.cpp
265265
src/pipeline/gpu_pipeline_task.cpp
266+
src/pipeline/sirius_pipeline_itask.cpp
266267
src/pipeline/task_scheduler.cpp
267268
src/pipeline/sirius_meta_pipeline.cpp
268269
src/pipeline/sirius_pipeline.cpp

docs/super-sirius/configuration.md

Lines changed: 68 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -212,7 +212,74 @@ sirius:
212212
| `output_directory` | string | `telemetry_data` | Directory for Quent ndjson files. |
213213
| `engine_name` | string | `siriusDB` | Engine name reported in engine-level telemetry. |
214214

215-
Per-query labels are configured separately with `CALL sirius_set_query_label(...)` SQL function or the `query_label` named parameter on `gpu_execution(...)`.
215+
Per-query labels are configured separately from YAML. They can be set with the
216+
`sirius_set_query_label` SQL function or inline with the `query_label` named
217+
parameter on `gpu_execution(...)`:
218+
219+
```sql
220+
-- Applies to the next Sirius query, including transparent plain-SQL execution.
221+
CALL sirius_set_query_label('tpch_q1_iter1');
222+
SELECT *
223+
FROM lineitem
224+
WHERE l_orderkey < 100;
225+
226+
-- Inline label for an explicit gpu_execution call.
227+
CALL gpu_execution(
228+
'SELECT * FROM lineitem WHERE l_orderkey < 100',
229+
query_label = 'tpch_q1_iter1'
230+
);
231+
```
232+
233+
`sirius_set_query_label` is consumed once by the next Sirius query. For explicit
234+
`gpu_execution(...)`, an inline `query_label` parameter takes precedence over a
235+
pending label set with `sirius_set_query_label`.
236+
237+
### Generating Query Telemetry
238+
239+
To generate telemetry from Sirius queries, update the Sirius config file used by
240+
the query run and enable Quent export:
241+
242+
```yaml
243+
sirius:
244+
telemetry:
245+
enable_quent: true
246+
output_directory: telemetry_data
247+
engine_name: siriusDB
248+
```
249+
250+
Load that config through the normal config resolution path, usually by setting
251+
`SIRIUS_CONFIG_FILE=/path/to/sirius.yaml`. Any Sirius query run with
252+
`enable_quent: true` writes Quent ndjson files into `output_directory`.
253+
254+
For TPC-H Parquet runs, the helper script runs queries and labels each
255+
`(query, iteration)` pair before executing it:
256+
257+
```bash
258+
pixi run -- ./test/tpch_performance/run_tpch_parquet_and_generate_telemetry.sh \
259+
--iterations 1 \
260+
--parquet-dir /data/tpch/sf100/p16/zstd-8/ \
261+
100
262+
```
263+
264+
Pass `--config <path>` to use a full Sirius config. If no query numbers are
265+
provided, the script runs all 22 TPC-H queries.
266+
267+
Start the Quent analyzer server over the same telemetry directory to view the
268+
captured telemetry:
269+
270+
```bash
271+
pixi run quent
272+
```
273+
274+
The `quent` Pixi task defaults to `telemetry_data`, and runs the telemetry
275+
server with the UI enabled. If the config uses a different `output_directory`,
276+
pass that path as the task argument:
277+
278+
```bash
279+
pixi run quent /path/to/telemetry_data
280+
```
281+
282+
Then open `http://localhost:8080` and select the captured Sirius engine/query.
216283

217284
## Thread Pool Configuration
218285

pixi.toml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,11 @@ go-cgo = ">=1.25"
5252

5353
[tasks]
5454

55+
[tasks.quent]
56+
description = "Run the Quent telemetry UI server."
57+
cmd = "echo 'Quent UI will be served at: http://localhost:8080' && cargo run --manifest-path rust/Cargo.toml -p sirius-telemetry-server --features ui -- --output-dir {{ output_dir }}"
58+
args = [{ arg = "output_dir", default = "telemetry_data" }]
59+
5560
# CUDA version features: use `pixi run -e cuda12` for CUDA 12.x systems
5661
[feature.cuda13.system-requirements]
5762
cuda = "13"

0 commit comments

Comments
 (0)