Skip to content

Commit c94b4e0

Browse files
mrproliumrproliu
authored
Align default BanyanDB stage segmentInterval to integer multiples (#13910)
### Improve the efficiency of BanyanDB hot → warm → cold lifecycle stage migration (default config) Each coarser lifecycle stage's `segmentInterval` should be an integer multiple of the immediately finer (preceding) stage's. BanyanDB segments are epoch-anchored, so when the coarser interval is a multiple of the finer one, every source segment nests entirely inside exactly one target segment and the tier migration stays on the cheap **whole-segment fast path** (byte-copy). A non-multiple interval makes source segments straddle target boundaries, forcing a row-by-row decode-and-re-bucket **slow path** that costs significantly more CPU, IO, and peak memory, and produces more small parts for the background merge loop to compact. It does not affect correctness — only migration efficiency. This aligns the shipped defaults that previously violated the rule: | Group | Field | Old → New | |-------|-------|-----------| | records | cold `segmentInterval` | 3 → 4 (multiple of warm=2) | | metricsMinute | cold `segmentInterval` | 5 → 6 (multiple of warm=3) | | metricsHour | warm `segmentInterval` | 7 → 10 (multiple of hot=5) | | metricsHour | cold `segmentInterval` | 15 → 20 (multiple of warm=10) | All adjusted values still satisfy `hot ≤ warm ≤ cold` and `segmentInterval ≤ that stage's ttl`. The other groups already comply and are unchanged.
1 parent 53baf8e commit c94b4e0

9 files changed

Lines changed: 44 additions & 42 deletions

File tree

‎.github/workflows/publish-docker-e2e-service.yaml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -51,14 +51,14 @@ jobs:
5151
restore-keys: |
5252
${{ runner.os }}-maven-
5353
- name: Log in to the Container registry
54-
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
54+
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
5555
with:
5656
registry: ${{ env.HUB }}
5757
username: ${{ github.actor }}
5858
password: ${{ secrets.GITHUB_TOKEN }}
5959
- name: Set up QEMU
60-
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392
60+
uses: docker/setup-qemu-action@06116385d9baf250c9f4dcb4858b16962ea869c3 # v4.1.0
6161
- name: Set up Docker Buildx
62-
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f
62+
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
6363
- name: Build and push images
6464
run: make -C test build.e2e-service docker.push-e2e-service

‎.github/workflows/publish-docker.yaml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,15 +65,15 @@ jobs:
6565
echo "TAG=${{ github.sha }}" >> $GITHUB_ENV
6666
fi
6767
- name: Log in to the Container registry
68-
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
68+
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
6969
with:
7070
registry: ${{ env.DOCKER_REGISTRY }}
7171
username: ${{ env.DOCKER_USERNAME }}
7272
password: ${{ env.DOCKER_PASSWORD }}
7373
- name: Set up QEMU
74-
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392
74+
uses: docker/setup-qemu-action@06116385d9baf250c9f4dcb4858b16962ea869c3 # v4.1.0
7575
- name: Set up Docker Buildx
76-
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f
76+
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
7777
- name: Build and push docker images based on Java 11
7878
env:
7979
SW_OAP_BASE_IMAGE: eclipse-temurin:11-jre

‎.github/workflows/skywalking.yaml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,7 @@ jobs:
122122
persist-credentials: false
123123
- name: Filter
124124
id: filter
125-
uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36
125+
uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
126126
with:
127127
list-files: 'shell'
128128
predicate-quantifier: 'every'
@@ -765,7 +765,7 @@ jobs:
765765
name: dist
766766
path: dist
767767
- name: Login to ghcr
768-
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
768+
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
769769
with:
770770
registry: ghcr.io
771771
username: ${{ github.repository_owner }}
@@ -854,7 +854,7 @@ jobs:
854854
find docker-images -name "*.tar" -exec docker load -i {} \;
855855
find docker-images -name "*.tar" -exec rm {} \;
856856
- name: Login to ghcr
857-
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
857+
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
858858
with:
859859
registry: ghcr.io
860860
username: ${{ github.repository_owner }}
@@ -926,7 +926,7 @@ jobs:
926926
find docker-images -name "*.tar" -exec docker load -i {} \;
927927
find docker-images -name "*.tar" -exec rm {} \;
928928
- name: Login to ghcr
929-
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
929+
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
930930
with:
931931
registry: ghcr.io
932932
username: ${{ github.repository_owner }}
@@ -1042,7 +1042,7 @@ jobs:
10421042
name: dist
10431043
path: dist
10441044
- name: Login to ghcr
1045-
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
1045+
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
10461046
with:
10471047
registry: ghcr.io
10481048
username: ${{ github.repository_owner }}

‎docs/en/banyandb/stages.md‎

Lines changed: 16 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -14,31 +14,32 @@ Each group of records/metrics can be automatically migrated and stored in differ
1414
If necessary, you also can jump the warm stage, and only use hot and cold stages. Then the data will be moved to the cold stage after the TTL of the hot stage.
1515

1616
## Configuration Guidelines
17-
The lifecycle stages configuration is under each group settings of the `bydb.yml` file, for example, the `metricsMin` group:
17+
The lifecycle stages configuration is under each group settings of the `bydb.yml` file, for example, the `metricsMinute` group:
1818

1919
```yaml
20-
metricsMin:
20+
metricsMinute:
2121
# The settings for the default `hot` stage.
22-
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_SHARD_NUM:2}
23-
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_SI_DAYS:1}
24-
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_TTL_DAYS:7}
25-
enableWarmStage: ${SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_WARM_STAGE:false}
26-
enableColdStage: ${SW_STORAGE_BANYANDB_GM_MINUTE_ENABLE_COLD_STAGE:false}
22+
shardNum: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_SHARD_NUM:2}
23+
segmentInterval: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_SI_DAYS:1}
24+
ttl: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_TTL_DAYS:7}
25+
enableWarmStage: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_ENABLE_WARM_STAGE:false}
26+
enableColdStage: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_ENABLE_COLD_STAGE:false}
2727
warm:
28-
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SHARD_NUM:2}
29-
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_SI_DAYS:3}
30-
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_TTL_DAYS:15}
31-
nodeSelector: ${SW_STORAGE_BANYANDB_GM_MINUTE_WARM_NODE_SELECTOR:"type=warm"}
28+
shardNum: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_WARM_SHARD_NUM:2}
29+
segmentInterval: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_WARM_SI_DAYS:3}
30+
ttl: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_WARM_TTL_DAYS:15}
31+
nodeSelector: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_WARM_NODE_SELECTOR:"type=warm"}
3232
cold:
33-
shardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SHARD_NUM:2}
34-
segmentInterval: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_SI_DAYS:5}
35-
ttl: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_TTL_DAYS:60}
36-
nodeSelector: ${SW_STORAGE_BANYANDB_GM_MINUTE_COLD_NODE_SELECTOR:"type=cold"}
33+
shardNum: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_SHARD_NUM:2}
34+
segmentInterval: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_SI_DAYS:6}
35+
ttl: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_TTL_DAYS:60}
36+
nodeSelector: ${SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_NODE_SELECTOR:"type=cold"}
3737
```
3838
3939
1. **shardNum**: The number of shards for the group.
4040
2. **segmentInterval**: The time interval in days for creating a new data segment.
4141
- According to the freshness of the data, the `segmentInterval` days should: `hot` < `warm` < `cold`.
42+
- Each coarser stage's `segmentInterval` should be an **integer multiple** of the immediately finer (preceding) stage's `segmentInterval` (e.g. `hot=5` → `warm=10` → `cold=20`). BanyanDB segments are epoch-anchored, so when the coarser interval is a multiple of the finer one each source segment nests entirely inside one target segment and the hot → warm → cold migration stays on the cheap whole-segment fast path. A non-multiple interval (e.g. `hot=5`, `warm=7`) makes source segments straddle target boundaries, forcing a row-by-row re-bucketing that costs significantly more CPU, IO, and memory. It does not affect correctness, only migration efficiency.
4243
3. **ttl**: The time-to-live for data within the group, in days.
4344
4. **enableWarmStage/enableColdStage**: Enable the warm/cold stage for the group.
4445
- The `hot` stage is always enabled by default.

‎docs/en/changes/changes.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -297,6 +297,7 @@
297297
* Bump up dependencies to clear CVE alerts on shipped OAP jars: log4j `2.25.3` → `2.25.4`, jackson `2.18.5` → `2.18.6`, kafka-clients `3.4.0` → `3.9.2`, postgresql `42.4.4` → `42.7.11`, commons-compress `1.21` → `1.26.2`.
298298
* Fix: continuous profiling policy validation now rejects a threshold / count of `0` to match the error messages and rover's `value >= threshold` trigger semantics (a `0` threshold would always trigger). CPU percent and HTTP error rate are tightened from `[0-100]` to `(0-100]`.
299299
* Fix wrong BanyanDB resource options in record data.
300+
* Align the default BanyanDB stage `segmentInterval` values so each coarser stage is an integer multiple of the finer one (`records` cold `3` → `4`, `metricsMinute` cold `5` → `6`, `metricsHour` warm `7` → `10` and cold `15` → `20`), keeping hot → warm → cold lifecycle migration on the cheap whole-segment fast path.
300301

301302
#### UI
302303
* Add Airflow layer dashboards and menu i18n under Workflow Scheduler in Horizon UI (SWIP-7).

‎docs/en/setup/backend/configuration-vocabulary.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -432,7 +432,7 @@ OAP will query the data from the "hot and warm" stage by default if the "warm" s
432432
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_RECORDS_WARM_NODE_SELECTOR | type=warm |
433433
| - | cold | - | The cold stage settings. | - | - |
434434
| - | - | shardNum | Shards Number for for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_SHARD_NUM | 1 |
435-
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_SI_DAYS | 3 |
435+
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_SI_DAYS | 4 |
436436
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_TTL_DAYS | 30 |
437437
| - | - | replicas | Replicas for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_REPLICAS | 0 |
438438
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_RECORDS_COLD_NODE_SELECTOR | type=cold |
@@ -527,7 +527,7 @@ OAP will query the data from the "hot and warm" stage by default if the "warm" s
527527
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_WARM_NODE_SELECTOR | type=warm |
528528
| - | cold | - | The cold stage settings. | - | - |
529529
| - | - | shardNum | Shards Number for for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_SHARD_NUM | 2 |
530-
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_SI_DAYS | 5 |
530+
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_SI_DAYS | 6 |
531531
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_TTL_DAYS | 60 |
532532
| - | - | replicas | Replicas for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_REPLICAS | 0 |
533533
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_METRICS_MINUTE_COLD_NODE_SELECTOR | type=cold |
@@ -539,13 +539,13 @@ OAP will query the data from the "hot and warm" stage by default if the "warm" s
539539
| - | enableColdStage | - | Activate cold stage for hour granularity metrics group. | SW_STORAGE_BANYANDB_METRICS_HOUR_ENABLE_COLD_STAGE | false |
540540
| - | warm | - | The warm stage settings. | - | - |
541541
| - | - | shardNum | Shards Number for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_SHARD_NUM | 1 |
542-
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_SI_DAYS | 7 |
542+
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_SI_DAYS | 10 |
543543
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_TTL_DAYS | 30 |
544544
| - | - | replicas | Replicas for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_REPLICAS | 0 |
545545
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_WARM_NODE_SELECTOR | type=warm |
546546
| - | cold | - | The cold stage settings. | - | - |
547547
| - | - | shardNum | Shards Number for for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_SHARD_NUM | 1 |
548-
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_SI_DAYS | 15 |
548+
| - | - | segmentInterval | Segment Interval Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_SI_DAYS | 20 |
549549
| - | - | ttl | TTL Days for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_TTL_DAYS | 120 |
550550
| - | - | replicas | Replicas for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_REPLICAS | 0 |
551551
| - | - | nodeSelector | Specifying target nodes for this stage. | SW_STORAGE_BANYANDB_METRICS_HOUR_COLD_NODE_SELECTOR | type=cold |

0 commit comments

Comments
 (0)