[Enhancement] Support native MAP/STRUCT target columns for Avro routine load by banmoy · Pull Request #74901 · StarRocks/starrocks

banmoy · 2026-06-16T09:55:01Z

Why I'm doing:

Routine Load with format="avro" (Confluent Schema Registry) aborts at the BE
scanner open() when a destination column is MAP or STRUCT, failing with:

Not support cast VARCHAR(1048576) to MAP<VARCHAR(65533), INT>.

AvroScanner::_construct_avro_types() only had a native branch for ARRAY;
MAP/STRUCT fell through to VARCHAR(MAX), and the subsequent
VARCHAR -> MAP/STRUCT cast does not exist, so the task fails before reading any
data. Scalars, ARRAY, and nullable unions already worked; only MAP/STRUCT
(and nesting that contains them) were broken.

What I'm doing:

Load avro MAP/STRUCT natively, with no JSON intermediary:

avro_scanner.cpp — replace the ARRAY-only switch in
_construct_avro_types with a recursive construct_avro_type() that builds
native ARRAY/MAP/STRUCT intermediate load types (mirrors
JsonUtils::construct_json_type). Avro map keys are always strings, so the
intermediate map key is forced to VARCHAR/CHAR; the cast stage converts it
to the declared key type.
Dispatch fix — _construct_column now dispatches the column writer on the
intermediate (_avro_types) type, which is what the source column was built
with, instead of the destination type. Otherwise a complex column whose
intermediate type differs from the destination (e.g. MAP<VARCHAR,VARCHAR> vs
MAP<VARCHAR,DATE>) is written by a writer chosen for the destination and
crashes on down_cast. Both call sites are fixed: the jsonpath path uses
_avro_types[i]; the no-jsonpath path stores the intermediate type via a new
slot-id -> type map.
formats/avro/nullable_column.cpp — native add_map_column /
add_struct_column, mirroring the avro-cpp MapColumnReader/StructColumnReader:
map key written with a length check + value recurses; struct fields matched by
name, missing fields become NULL. Inner unions in nested map values / struct
fields / array elements are unwrapped, and a partial nested append is rewound
(snapshot + resize) on error before deciding NULL vs. propagate.

The existing avro -> JSON path (TYPE_JSON columns) and scalar/array handling are
unchanged. The cast factory already supports recursive MAP->MAP /
STRUCT->STRUCT / ARRAY->ARRAY casts, so e.g. MAP<VARCHAR,DATE> works without
any JSON round-trip.

Tests

BE UT (AvroScannerTest, AvroAddNullableColumnTest): native MAP (with and
without jsonpaths), STRUCT (nullable-union field, null and non-null, both
avro_ignore_union_type_tag values), nested ARRAY<MAP>, MAP<ARRAY>,
ARRAY<STRUCT>, STRUCT-with-MAP, and a 4-level-deep
ARRAY<STRUCT<VARCHAR, MAP<VARCHAR, ARRAY<INT>>>>; the test_map_to_json
regression is preserved.
End-to-end: routine load from Kafka + Confluent Schema Registry into a
MAP<VARCHAR(65533), INT> column loads native maps (map['k'], map_size,
map_keys, map_values all work); previously it aborted at open().
python3 build-support/check_be_module_boundaries.py --mode full clean.

Docs

Updated the Avro → StarRocks complex-type mapping table in loading/RoutineLoad.md
(en/zh/ja): maps → MAP or JSON, record → STRUCT, or ... JSON.

Fixes #70928

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

No currently-working behavior changes: avro routine load into a MAP/STRUCT
destination column previously always failed at the BE scanner open()
(Not support cast VARCHAR to MAP), so this only enables a path that always
errored. Scalar / ARRAY / TYPE_JSON columns and the parse_json workaround
are unaffected.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
- This pr needs auto generate documentation
This is a backport pr

Bugfix cherry-pick branch check:

I have checked the version labels which the pr will be auto-backported to the target branch
- 4.1
- 4.0
- 3.5

CelerData-Reviewer · 2026-06-16T10:36:55Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 59ae97bb0d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-actions · 2026-06-16T10:42:58Z

🌎 Translation Required?

✅ All translation files are up to date.
Great job! No translation actions are required for this PR.

🕒 Last updated: Tue, 16 Jun 2026 14:17:37 GMT

…ne load Routine Load with format=avro aborted at BE scanner open() when a destination column was MAP or STRUCT, failing with "Not support cast VARCHAR(1048576) to MAP<...>". _construct_avro_types() only had a native ARRAY branch; MAP/STRUCT fell through to VARCHAR(MAX), and the subsequent VARCHAR->MAP/STRUCT cast does not exist. Build native MAP/STRUCT intermediate types and writers (mirroring the avro-cpp readers), and dispatch the column writer on the intermediate (_avro_types) type rather than the destination type so the source-column down_casts match. - avro_scanner: recursive construct_avro_type() for ARRAY/MAP/STRUCT; dispatch _construct_column on _avro_types in both the jsonpath and no-jsonpath paths (via a slot-id -> intermediate-type map). - formats/avro/nullable_column: native add_map_column/add_struct_column, union unwrapping for nested map values / struct fields / array elements, and snapshot/resize rewind on partial nested append. Avro map keys are always strings, so the intermediate map key is forced to VARCHAR/CHAR and the cast stage converts to the declared key type. The existing avro->JSON path (TYPE_JSON columns) is unchanged. Tests: native MAP/STRUCT, nested (ARRAY<MAP>, MAP<ARRAY>, ARRAY<STRUCT>, STRUCT-with-MAP, 4-level deep), nullable-union fields (null + value), both avro_ignore_union_type_tag values; regression test_map_to_json preserved. Docs: update the Avro -> StarRocks complex-type mapping table (maps -> MAP, record -> STRUCT) in loading/RoutineLoad.md (en/zh/ja). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: PengFei Li <lpengfei2016@gmail.com>

CelerData-Reviewer · 2026-06-16T14:12:33Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c8fbc9ddf0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-actions · 2026-06-16T15:22:51Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-06-16T15:22:54Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-06-16T15:34:11Z

[BE Incremental Coverage Report]

✅ pass : 128 / 152 (84.21%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/formats/avro/nullable_column.cpp	82	98	83.67%	[36, 53, 63, 71, 75, 76, 93, 102, 238, 244, 246, 248, 389, 390, 391, 392]
🔵	be/src/exec/file_scanner/avro_scanner.cpp	46	54	85.19%	[486, 491, 495, 519, 520, 529, 530, 540]

eshishki · 2026-06-16T18:46:53Z

do we by any chance overlap with #73821 ?

dirtysalt

LGTM! 🚀

banmoy · 2026-06-17T03:27:22Z

do we by any chance overlap with #73821 ?

@eshishki Yes, they have overlap. Functionally, this PR is indeed part of #73821, but there are the following differences:

[Enhancement] Add avrocpp-based Avro scanner for routine load #73821 requires users to modify properties, so existing jobs cannot use map/struct seamlessly — this affects user behavior.
[Enhancement] Add avrocpp-based Avro scanner for routine load #73821 is an architectural change to the Avro scanner with a fairly broad impact. Users need to be more careful when migrating from the old scanner to the new one, so we'd better not couple struct/map with it.

github-actions · 2026-06-23T03:29:36Z

@Mergifyio backport branch-4.1

mergify · 2026-06-23T03:29:44Z

backport branch-4.1

✅ Backports have been created

Details

#75187 [Enhancement] Support native MAP/STRUCT target columns for Avro routine load (backport #74901) has been created for branch branch-4.1 but encountered conflicts

Cherry-pick of 5e41562 has failed:

On branch mergify/bp/branch-4.1/pr-74901
Your branch is up to date with 'origin/branch-4.1'.

You are currently cherry-picking commit 5e41562d31.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   be/src/exec/file_scanner/avro_scanner.cpp
	modified:   be/src/exec/file_scanner/avro_scanner.h
	modified:   be/test/exec/file_scanner/avro_scanner_test.cpp
	modified:   docs/en/loading/RoutineLoad.md
	modified:   docs/ja/loading/RoutineLoad.md
	modified:   docs/zh/loading/RoutineLoad.md

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   be/src/formats/avro/nullable_column.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

…ne load (backport #74901) (#75192) Signed-off-by: PengFei Li <lpengfei2016@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ne load (StarRocks#74901) Signed-off-by: PengFei Li <lpengfei2016@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Oliver Layer <o.layer@celonis.com>

…ibility test_struct_with_boolean depends on native STRUCT target support (PR StarRocks#74901, main only), so it cannot cherry-pick to release branches that lack it. Replace it with test_array_with_boolean, which drives the same add_nullable_column TYPE_BOOLEAN dispatch (the fixed line) and whose ARRAY path is present on all release branches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GN8hHLLGz7o25mvSJhwqXr

github-actions Bot added the behavior_changed label Jun 16, 2026

mergify Bot assigned banmoy Jun 16, 2026

banmoy force-pushed the avro-map-struct-native branch from aa6f59a to d5b2127 Compare June 16, 2026 09:59

banmoy changed the title ~~[BugFix] Support native MAP/STRUCT target columns for Avro routine load~~ [Enhancement] Support native MAP/STRUCT target columns for Avro routine load Jun 16, 2026

github-actions Bot requested review from dirtysalt and gengjun-git June 16, 2026 09:59

banmoy force-pushed the avro-map-struct-native branch from b89345c to 2ea05e4 Compare June 16, 2026 10:00

github-actions Bot added the 4.1 label Jun 16, 2026

banmoy marked this pull request as ready for review June 16, 2026 10:15

banmoy force-pushed the avro-map-struct-native branch from 2ea05e4 to 59ae97b Compare June 16, 2026 10:34

github-actions Bot added documentation Documentation changes and removed behavior_changed labels Jun 16, 2026

banmoy marked this pull request as draft June 16, 2026 10:35

banmoy marked this pull request as ready for review June 16, 2026 10:36

banmoy marked this pull request as draft June 16, 2026 10:36

banmoy marked this pull request as ready for review June 16, 2026 10:40

chatgpt-codex-connector Bot reviewed Jun 16, 2026

View reviewed changes

Comment thread be/src/formats/avro/nullable_column.cpp

banmoy force-pushed the avro-map-struct-native branch from 59ae97b to c8fbc9d Compare June 16, 2026 14:11

chatgpt-codex-connector Bot reviewed Jun 16, 2026

View reviewed changes

Comment thread be/src/formats/avro/nullable_column.cpp

dirtysalt approved these changes Jun 17, 2026

View reviewed changes

wyb approved these changes Jun 23, 2026

View reviewed changes

wyb merged commit 5e41562 into StarRocks:main Jun 23, 2026
78 checks passed

github-actions Bot removed the 4.1 label Jun 23, 2026

mergify Bot mentioned this pull request Jun 23, 2026

[Enhancement] Support native MAP/STRUCT target columns for Avro routine load (backport #74901) #75187

Closed

23 tasks

banmoy mentioned this pull request Jun 23, 2026

[Enhancement] Support native MAP/STRUCT target columns for Avro routine load (backport #74901) #75192

Merged

23 tasks

github-actions Bot added the 4.1-merged label Jun 23, 2026

eshishki mentioned this pull request Jun 24, 2026

[Enhancement] Add avrocpp-based Avro scanner for routine load #73821

Open

23 tasks

banmoy mentioned this pull request Jul 8, 2026

[BugFix] Fix ASAN crash for BOOLEAN nested in Avro complex-type columns #76041

Merged

18 tasks

This was referenced Jul 9, 2026

[BugFix] Fix ASAN crash for BOOLEAN nested in Avro complex-type columns (backport #76041) #76064

Merged

[BugFix] Fix ASAN crash for BOOLEAN nested in Avro complex-type columns (backport #76041) #76065

Closed

banmoy mentioned this pull request Jul 9, 2026

[BugFix] Fix ASAN crash for BOOLEAN nested in Avro complex-type columns (backport #76041) #76066

Closed

18 tasks

Uh oh!

Conversation

banmoy commented Jun 16, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I'm doing:

What I'm doing:

Tests

Docs

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

CelerData-Reviewer commented Jun 16, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

github-actions Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌎 Translation Required?

Uh oh!

CelerData-Reviewer commented Jun 16, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

github-actions Bot commented Jun 16, 2026

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jun 16, 2026

[FE Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jun 16, 2026

[BE Incremental Coverage Report]

file detail

Uh oh!

eshishki commented Jun 16, 2026

Uh oh!

dirtysalt left a comment

Choose a reason for hiding this comment

Uh oh!

banmoy commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

mergify Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

banmoy commented Jun 16, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jun 16, 2026 •

edited

Loading

banmoy commented Jun 17, 2026 •

edited

Loading

mergify Bot commented Jun 23, 2026 •

edited

Loading