YTDB-634: Lazy ExecutionStream

[sandrawar]

PR Title:

YTDB-634: Lazy ExecutionStream

Motivation:

MatchEdgeTraverser.executeTraversal() eagerly materializes all recursive results into an ArrayList before returning any of them:

List<Result> result = new ArrayList<>();
// ... recurse into all neighbors, drain everything ...
result.add(sub);                                                              
// ... only now return as stream
return ExecutionStream.resultIterator(result.iterator());   

This means the entire subtree is fully traversed and collected before the first result is yielded to the consumer. There is no backpressure — once recursion starts, it runs to completion. LIMIT cannot short-circuit the traversal, and peak memory is O(total_results).

This PR replaces the eager collection with LazyRecursiveTraversalStream — a stack-based DFS iterator that yields results one at a time. An explicit Deque replaces Java call-stack recursion: each frame tracks a node's self-result and its neighbor iteration state. The advance() method processes the stack iteratively — yield self, push neighbor frames, pop when exhausted — with O(1) call-stack depth regardless of traversal depth.

When the downstream consumer stops pulling (e.g., LimitExecutionStep is satisfied), unexplored subtrees are never expanded. For deep WHILE chains, the flat loop avoids StackOverflow risk that the previous recursive implementation had.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the recursive graph traversal mechanism in MatchEdgeTraverser to use a lazy, pull-based approach. This change addresses performance and memory issues caused by eager materialization of all traversal results into an ArrayList, and enables proper backpressure and LIMIT clause short-circuiting.

Highlights

• Introduced Lazy Traversal Stream: A new LazyRecursiveTraversalStream class was added to implement a stack-based Depth-First Search (DFS) iterator, replacing the previous eager collection logic for recursive graph traversals.
• Improved Performance and Memory Usage: The new pull-based approach eliminates O(total_results) memory allocation by yielding results one at a time, rather than collecting all of them upfront into an ArrayList.
• Enabled Backpressure and LIMIT Short-Circuiting: The on-demand nature of the stream allows downstream consumers (such as LIMIT clauses) to stop pulling results, preventing unnecessary expansion of unexplored subtrees and improving efficiency.
• Prevented Stack Overflow Errors: Java call-stack recursion was replaced with an explicit Deque<Frame> for DFS, ensuring O(1) call-stack depth regardless of traversal depth, which mitigates StackOverflowError risks for deep traversals.
• Enhanced Test Coverage: Comprehensive new tests were added to validate the lazy traversal's correctness for deep chains, LIMIT interactions, branching graphs, and maxDepth constraints.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new LazyRecursiveTraversalStream class to replace the eager ArrayList materialization in recursive graph traversals, particularly for WHILE and maxDepth conditions. This new stream uses an explicit stack to avoid stack depth issues in deep traversals and enables lazy evaluation, which is beneficial for LIMIT clauses. The MatchEdgeTraverser was refactored to utilize this new lazy stream. New test cases were added to validate the lazy traversal's behavior with deep chains, limits, branching graphs, and max depth. A high-severity issue was identified in LazyRecursiveTraversalStream.pushFrame where the CommandContext could be left in an inconsistent state if an exception occurs during frame processing; a try-catch block is suggested to restore the context.

[github-actions]

Coverage Gate Results

Thresholds: 85% line, 70% branch

Line Coverage: ✅ 94.3% (100/106 lines)

File	Coverage	Uncovered Lines
core/src/main/java/com/jetbrains/youtrackdb/internal/core/sql/executor/match/LazyRecursiveTraversalStream.java	✅ 94.3% (99/105)	126, 135, 218, 286-288
core/src/main/java/com/jetbrains/youtrackdb/internal/core/sql/executor/match/MatchEdgeTraverser.java	✅ 100.0% (1/1)	-

Branch Coverage: ✅ 84.3% (59/70 branches)

File	Coverage	Lines with Uncovered Branches
core/src/main/java/com/jetbrains/youtrackdb/internal/core/sql/executor/match/LazyRecursiveTraversalStream.java	✅ 84.3% (59/70)	125, 134, 217, 224, 257, 259, 265, 267-268, 275, 293

[github-actions]

Test Count Gate Results

✅ No baseline available yet — gate skipped (first run).

[github-actions]

JMH LDBC Benchmark Comparison

Base: 244d8c7275 (fork-point with develop) | Head: a2e4166e19
Summary: 🔴 8 regression(s) (>±5% threshold)

Single-Thread Results

Benchmark	Base ops/s	Base err	Head ops/s	Head err	Δ%
ic10_friendRecommendation	0.123	±3.2%	0.127	±4.4%	+3.7%
ic11_jobReferral	36.2	±1.4%	36.6	±2.0%	+1.2%
ic12_expertSearch	21.6	±1.8%	21.9	±1.8%	+1.4%
ic13_shortestPath	4,037	±2.9%	3,307	±4.0%	-18.1% 🔴
ic1_transitiveFriends	51.4	±1.3%	35.4	±1.4%	-31.1% 🔴
ic2_recentFriendMessages	205.8	±2.3%	206.8	±2.2%	+0.5%
ic3_friendsInCountries	0.149	±1.7%	0.152	±4.4%	+2.2%
ic4_newTopics	3.1	±7.8%	2.4	±6.5%	-23.4% 🔴
ic5_newGroups	0.083	±24.8%	0.081	±23.1%	-2.4%
ic6_tagCoOccurrence	3.5	±4.3%	3.5	±4.0%	+1.7%
ic7_recentLikers	52.8	±6.0%	54.4	±6.1%	+3.1%
ic8_recentReplies	842.7	±1.3%	858.5	±1.5%	+1.9%
ic9_recentFofMessages	1.2	±2.1%	1.3	±2.4%	+3.2%
is1_personProfile	45,082	±2.8%	45,120	±1.8%	+0.1%
is2_personPosts	494.8	±1.4%	507.8	±1.4%	+2.6%
is3_personFriends	13,106	±3.8%	13,091	±3.3%	-0.1%
is4_messageContent	62,273	±1.3%	63,338	±0.7%	+1.7%
is5_messageCreator	57,242	±1.6%	56,206	±2.9%	-1.8%
is6_messageForum	38,154	±0.9%	37,516	±2.2%	-1.7%
is7_messageReplies	2,338	±1.2%	2,200	±1.2%	-5.9% 🔴

Multi-Thread Results

Benchmark	Base ops/s	Base err	Head ops/s	Head err	Δ%
ic10_friendRecommendation	0.560	±3.5%	0.574	±3.8%	+2.5%
ic11_jobReferral	156.3	±2.4%	160.3	±3.2%	+2.6%
ic12_expertSearch	98.5	±1.5%	102.0	±1.1%	+3.6%
ic13_shortestPath	19,275	±3.8%	15,703	±3.7%	-18.5% 🔴
ic1_transitiveFriends	245.9	±1.5%	174.9	±1.4%	-28.9% 🔴
ic2_recentFriendMessages	938.3	±1.9%	930.5	±4.7%	-0.8%
ic3_friendsInCountries	0.636	±1.9%	0.664	±2.7%	+4.3%
ic4_newTopics	11.8	±4.3%	9.5	±2.8%	-19.7% 🔴
ic5_newGroups	0.312	±8.9%	0.318	±7.3%	+2.0%
ic6_tagCoOccurrence	15.7	±2.1%	16.3	±1.9%	+3.3%
ic7_recentLikers	238.1	±2.3%	243.1	±3.3%	+2.1%
ic8_recentReplies	4,131	±1.2%	4,177	±1.1%	+1.1%
ic9_recentFofMessages	6.2	±3.4%	6.3	±4.1%	+1.4%
is1_personProfile	174,840	±1.6%	174,690	±1.3%	-0.1%
is2_personPosts	2,361	±1.1%	2,416	±1.0%	+2.3%
is3_personFriends	59,011	±1.6%	58,593	±1.4%	-0.7%
is4_messageContent	234,748	±1.3%	233,657	±1.5%	-0.5%
is5_messageCreator	211,987	±1.5%	212,060	±2.1%	+0.0%
is6_messageForum	144,831	±0.9%	145,653	±1.4%	+0.6%
is7_messageReplies	10,998	±1.9%	10,394	±1.6%	-5.5% 🔴

Scalability (MT/ST ratio)

Benchmark	Base ratio	Head ratio	Δ%
ic10_friendRecommendation	4.56x	4.51x	-1.1%
ic11_jobReferral	4.32x	4.38x	+1.3%
ic12_expertSearch	4.56x	4.66x	+2.2%
ic13_shortestPath	4.77x	4.75x	-0.5%
ic1_transitiveFriends	4.79x	4.94x	+3.2%
ic2_recentFriendMessages	4.56x	4.50x	-1.3%
ic3_friendsInCountries	4.27x	4.36x	+2.1%
ic4_newTopics	3.80x	3.98x	+4.8%
ic5_newGroups	3.78x	3.95x	+4.4%
ic6_tagCoOccurrence	4.53x	4.60x	+1.5%
ic7_recentLikers	4.51x	4.47x	-1.0%
ic8_recentReplies	4.90x	4.87x	-0.7%
ic9_recentFofMessages	5.02x	4.93x	-1.7%
is1_personProfile	3.88x	3.87x	-0.2%
is2_personPosts	4.77x	4.76x	-0.3%
is3_personFriends	4.50x	4.48x	-0.6%
is4_messageContent	3.77x	3.69x	-2.1%
is5_messageCreator	3.70x	3.77x	+1.9%
is6_messageForum	3.80x	3.88x	+2.3%
is7_messageReplies	4.70x	4.72x	+0.4%

[andrii0lomakin]

Hi @sandrawar, please profile regressions using asyncprofiler on Hetzner CCX 33 node and find out what caused the regressions. Please bear in mind that IC4 is quite noisy, so we need to likely find a better approach to run it, but it also can be that there is an issue in implementation, so we need to profile results for sure.