Findings

What's Already There

async-profiler is already installed on all nodes by packer:

Installed to /usr/local/async-profiler by packer/base/install/install_async_profiler.sh (v4.3)
Kernel parameters set at install time: perf_event_paranoid=1, kptr_restrict=0
A flamegraph script already exists at packer/cassandra/bin/flamegraph:
```
sudo /usr/local/async-profiler/bin/asprof -f $OUTFILE $@ $(cassandra-pid)
# Outputs HTML to /mnt/db1/cassandra/artifacts/flame-{hostname}-{timestamp}.html
```
This is a manual one-shot script, NOT a SystemD service, and it saves HTML to disk rather than sending to Pyroscope.

Pyroscope is fully deployed:

Server: K8s Deployment on control plane (port 4040), S3 backend
Continuous eBPF profiling: Grafana Alloy DaemonSet on all nodes (CPU profiles only, via eBPF)
Java agent: Cassandra, Stress jobs, and Spark/EMR (CPU, alloc, lock, wall — via JFR)
Constants: Constants.K8s.PYROSCOPE_PORT = 4040, Constants.PyroscopeJavaAgent

Pyroscope HTTP Ingest API — Pyroscope accepts profiles via:

POST /ingest?name=<app>&spyName=async-profiler&format=collapsed — collapsed stacks (async-profiler native output)
POST /ingest?name=<app>&format=jfr — JFR format (async-profiler also supports -o jfr)
This means async-profiler → push to Pyroscope is already technically possible with a script

The Gap

The existing flamegraph script:

Only profiles Cassandra (hardcoded via cassandra-pid)
Saves output as HTML to disk — not sent to Pyroscope
Is manually invoked, not a discoverable SystemD service
Has no duration control, no labeling for Pyroscope

What "Common Flamegraph One-Shot Services" Should Be

The phrase "common" suggests a reusable, parameterizable pattern — not just a Cassandra-specific script.

The design should cover:

Cassandra (JVM process — use async-profiler in JFR mode for best JVM support)
ClickHouse (native C++ — async-profiler with eBPF/native mode)
Future databases too

SystemD one-shot service is the right mechanism. The pattern:

[Unit]
Description=Flamegraph profiler for <service>

[Service]
Type=oneshot
ExecStart=/usr/local/bin/flamegraph-to-pyroscope <process> <duration>

Trigger with: sudo systemctl start flamegraph-cassandra

Proposed Design

1. New shell script: /usr/local/bin/flamegraph-to-pyroscope

Wraps asprof and pushes collapsed stacks to Pyroscope HTTP API:

asprof -d 30 -o collapsed /path/to/output.txt <PID>
curl -X POST "http://$PYROSCOPE_ADDR/ingest?name=<app>&spyName=asyncprofiler&format=collapsed" \
     -H "Content-Type: text/plain" --data-binary @/path/to/output.txt

The PYROSCOPE_ADDR can be sourced from /etc/default/cassandra (where PYROSCOPE_SERVER_ADDRESS is already set by SetupInstance).

2. Packer additions (probably under packer/cassandra/services/ and packer/cassandra/bin/):

flamegraph-cassandra.service — Type=oneshot, profiles Cassandra PID, sends to Pyroscope
flamegraph-to-pyroscope script installed to /usr/local/bin/
Install script that sets up the systemd service unit

3. SystemD template units (optional but elegant)

Could use a template: flamegraph@.service

sudo systemctl start flamegraph@cassandra
sudo systemctl start flamegraph@clickhouse

4. Kotlin integration (optional)

A new command or service method could trigger the one-shot service via SSH:

remoteOps.executeRemotely(host, "sudo systemctl start flamegraph@cassandra")

This would fit into the existing SystemDServiceManager pattern, possibly with a new FlamegraphService.

Key Design Decisions to Settle

Profile duration — hardcoded 30s? Configurable via environment file at /etc/default/flamegraph?
Service per database or template unit? — Template (flamegraph@cassandra.service) is cleaner for supporting multiple databases
Output format — Collapsed stacks are simplest; JFR gives more detail (alloc, lock, wall). For the one-shot case, collapsed CPU stacks is probably enough since Pyroscope Java agent already provides JFR continuously.
PID discovery for non-JVM — ClickHouse PID lookup needs a different helper than cassandra-pid
Pyroscope address source — already in /etc/default/cassandra as PYROSCOPE_SERVER_ADDRESS; probably extract to a shared /etc/default/easydblab or pass as a service environment variable

Files to Create/Modify

File	Action	Notes
`packer/cassandra/bin/flamegraph-to-pyroscope`	Create	Core script: asprof → POST to Pyroscope
`packer/cassandra/services/flamegraph@.service`	Create	SystemD template unit (or per-service files)
`packer/cassandra/install/install_flamegraph_service.sh`	Create	Installs script + service, `systemctl daemon-reload`
Possibly update `packer/cassandra/bin/flamegraph`	Update	Point to Pyroscope instead of HTML output

The Kotlin side (triggering via FlamegraphService) could come later if desired.

Create common flamegraph one shot services #159

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions