[Enhancement] Speed up JSON string parsing by zhangyifan27 · Pull Request #75533 · StarRocks/starrocks

zhangyifan27 · 2026-06-29T09:29:14Z

Why I'm doing:

Speed up JSON string parsing

What I'm doing:

Replace std::iswspace in parse_json_or_string with an inline ASCII
whitespace check: avoids UB when passing a (possibly negative) char to
iswspace and removes locale overhead.
Evidence: https://quick-bench.com/q/qp3OqCWLRKWA4heZS677TuLLOf8
Add a reusable velocypack Parser overload of parse_json_or_string
(make_reusable_parser) and use it in CAST(VARCHAR AS JSON) so the
Parser/Builder buffer is allocated once per column instead of once per
row (up to ~2.6x on small objects).
Add the BM_ParseJson benchmark (OneShot vs Reused) and fix build.sh
--module getopt to accept an argument.

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
- This pr needs auto generate documentation
This is a backport pr

Bugfix cherry-pick branch check:

I have checked the version labels which the pr will be auto-backported to the target branch
- 4.1
- 4.0
- 3.5

CelerData-Reviewer · 2026-06-29T09:30:13Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a571a60a88

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-29T09:34:19Z

+        }
+    }
+
+    *applied = true;


Validate the whole JSON before applying fast path

When the input has a constant path and malformed JSON after the requested value, such as {"k1":1,"bad":}, the on-demand parser can extract $.k1 without ever consuming the invalid tail; setting applied here then returns 1. The previous velocypack path fully parsed each row and would return NULL, or an error with allow_throw_exception, so invalid JSON can now produce non-null results.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-29T09:34:20Z

+            } else if (st.is_not_found()) {
+                result.append_null();


Preserve lookups for escaped JSON keys

For constant paths, extract_from_object uses simdjson raw-key lookup, which does not unescape object keys; valid rows like {"\u006b1":1} with path $.k1 are reported as not found here and return NULL. The old velocypack path unescaped keys while parsing and returned the value, so this fast path changes results for valid JSON that uses escaped member names.

Useful? React with 👍 / 👎.

eshishki · 2026-06-29T20:09:54Z

fyi #73279

- Replace std::iswspace in parse_json_or_string with an inline ASCII whitespace check: avoids UB when passing a (possibly negative) char to iswspace and removes locale overhead. Evidence: https://quick-bench.com/q/qp3OqCWLRKWA4heZS677TuLLOf8 - Add a reusable velocypack Parser overload of parse_json_or_string (make_reusable_parser) and use it in CAST(VARCHAR AS JSON) so the Parser/Builder buffer is allocated once per column instead of once per row (up to ~2.6x on small objects). - Add the BM_ParseJson benchmark (OneShot vs Reused) and fix build.sh --module getopt to accept an argument. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: zhangyifan27 <chinazhangyifan@163.com> Co-authored-by: Cursor <cursoragent@cursor.com>

CelerData-Reviewer · 2026-06-30T06:45:07Z

@codex review

zhangyifan27 · 2026-06-30T06:48:50Z

fyi #73279

Thanks for pointing this out! #73729 already provides a more comprehensive optimization for JSON parsing, so we've scoped this patch to a few targeted improvements around the VPack slow path instead.

chatgpt-codex-connector · 2026-06-30T06:49:50Z

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: e5ce37c27a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

CelerData-Reviewer · 2026-07-10T08:42:38Z

@codex review

github-actions · 2026-07-10T08:43:04Z

Module-risk briefing for Codex review

This is review focus, not a finding list. Validate against the diff.

High-confidence risks

Module: types
Changed files: be/src/types/json_value.cpp, be/src/types/json_value.h, be/src/exprs/cast_expr.cpp
Historical failure pattern: this module has repeatedly shipped bugs where a wrapper/inner-call split silently dropped or diverged in forwarded state (str_to_date wrapper hardcoding a field instead of forwarding it), and where reused/stateful parsing objects carried stale data across calls (non-reentrant std::gmtime static buffer corrupting concurrent callers).
Review focus: The PR introduces a json_parser reused across every row of a column (cast_expr.cpp's cast_to_json_fn) and a new parse_json_or_string(src, parser) overload that reuses arangodb::velocypack::Parser's internal Builder/Buffer (json_value.cpp). Confirm Parser::parse() fully resets ALL internal parser state — not just the builder capacity — before each reuse, so no bytes or exception state from one row's JSON leak into the next row's parsed value; also confirm the new single-arg parse_json_or_string(src) wrapper (now delegating to parse_json_or_string(src, nullptr)) is exactly equivalent to the removed one-shot path for existing callers.
Evidence: [BugFix] Fix str_to_date losing microsecond precision in BE runtime e… #70068, [BugFix] Replace non-thread-safe std::gmtime with gmtime_r #73094 (from 8 merged bugfixes)

chatgpt-codex-connector · 2026-07-10T08:48:11Z

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 7b3e69743a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-actions · 2026-07-10T11:23:32Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-07-10T11:23:35Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-07-10T11:25:19Z

[BE Incremental Coverage Report]

❌ fail : 2 / 3 (66.67%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	src/exprs/cast_expr.cpp	2	3	66.67%	[168]

Only allocate the reusable vpack Parser for string/variant casts, and add a UT that exercises CAST(VARIANT AS JSON). Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: zhangyifan27 <chinazhangyifan@163.com>

CelerData-Reviewer · 2026-07-15T06:43:08Z

@codex review

chatgpt-codex-connector · 2026-07-15T06:48:29Z

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 1c887f682e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mergify Bot assigned zhangyifan27 Jun 29, 2026

github-actions Bot requested review from stdpain and trueeyu June 29, 2026 09:31

chatgpt-codex-connector Bot reviewed Jun 29, 2026

View reviewed changes

zhangyifan27 force-pushed the optimize_parse_json branch from a571a60 to e5ce37c Compare June 30, 2026 06:44

zhangyifan27 changed the title ~~[Enhancement] Speed up JSON parsing and get_json extraction~~ [Enhancement] Speed up JSON string parsing Jun 30, 2026

Merge remote-tracking branch 'origin/main' into optimize_parse_json

7b3e697

[UT] Cover VARIANT to JSON cast reusable parser path

1c887f6

Only allocate the reusable vpack Parser for string/variant casts, and add a UT that exercises CAST(VARIANT AS JSON). Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: zhangyifan27 <chinazhangyifan@163.com>

Uh oh!

Conversation

zhangyifan27 commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

CelerData-Reviewer commented Jun 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

eshishki commented Jun 29, 2026

Uh oh!

CelerData-Reviewer commented Jun 30, 2026

Uh oh!

zhangyifan27 commented Jun 30, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 30, 2026

Uh oh!

CelerData-Reviewer commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Module-risk briefing for Codex review

High-confidence risks

Uh oh!

chatgpt-codex-connector Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jul 10, 2026

[FE Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jul 10, 2026

[BE Incremental Coverage Report]

file detail

Uh oh!

CelerData-Reviewer commented Jul 15, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhangyifan27 commented Jun 29, 2026 •

edited

Loading