[Feature] Add Lance connector optimizer operators and rules by beinan · Pull Request #74660 · StarRocks/starrocks

beinan · 2026-06-10T19:15:43Z

Why I'm doing this

The Lance connector needs FE optimizer support so that SELECT queries against
Lance tables can be planned and optimized through the standard StarRocks
cost-based optimizer pipeline.

What I'm doing

Add the optimizer-layer plumbing for Lance, following the same pattern as the
existing Kudu connector:

Operator types: LOGICAL_LANCE_SCAN and PHYSICAL_LANCE_SCAN in OperatorType
Operators: LogicalLanceScanOperator and PhysicalLanceScanOperator
Visitor hooks: visitLogicalLanceScan / visitPhysicalLanceScan in
OperatorVisitor and OptExpressionVisitor
Implementation rule: LanceScanImplementationRule + IMP_LANCE_LSCAN_TO_PSCAN
in RuleType
Rule wiring: Registered in RuleSet implementation rules list
Predicate pushdown: LOGICAL_LANCE_SCAN added to PushDownPredicateScanRule
Statistics: StatisticsCalculator computes stats via MetadataMgr
Plan printer: LogicalPlanPrinter prints "LANCE SCAN"
Relation transformer: RelationTransformer creates LogicalLanceScanOperator
for LANCE table type

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
This pr needs user documentation (for new or modified features or behaviors)

Related Issue

Fixes #74665

CelerData-Reviewer · 2026-06-10T19:16:19Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 78e577bb81

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-10T19:19:20Z

+        } else if (Table.TableType.LANCE.equals(node.getTable().getType())) {
+            scanOperator = new LogicalLanceScanOperator(node.getTable(), colRefToColumnMetaMapBuilder.build(),
+                columnMetaToColRefMap, Operator.DEFAULT_LIMIT, null);


Enable Lance tables before adding scan planning

This new Lance branch is unreachable for normal SQL queries because QueryAnalyzer only passes a TableRelation through when table.isSupported() is true, and LanceTable.isSupported() currently returns false; a SELECT from any Lance connector table is still rejected as Unsupported scan table type: LANCE before RelationTransformer can create this operator. The feature's optimizer plumbing therefore does not make Lance scans plannable unless the Lance table support gate is updated as part of the same change.

Useful? React with 👍 / 👎.

False positive: LanceTable.isSupported() is changed to true in PR #74661 which is part of this series. See #74665 for the full dependency chain.

chatgpt-codex-connector · 2026-06-10T19:19:20Z

+    public <R, C> R accept(OptExpressionVisitor<R, C> visitor, OptExpression optExpression, C context) {
+        return visitor.visitPhysicalLanceScan(optExpression, context);


Add a plan-fragment translator for Lance scans

Once a Lance scan reaches physical planning, this dispatch has no corresponding visitPhysicalLanceScan override in PlanFragmentBuilder.PhysicalPlanTranslator (unlike Kudu/Hive/Paimon/etc.). The base visitor routes visitPhysicalLanceScan to visitPhysicalScan, which calls the translator's overridden visit(...) and re-enters this accept(...) method, so any planned Lance scan recurses until a stack overflow instead of producing a ScanNode.

Useful? React with 👍 / 👎.

False positive: PlanFragmentBuilder.visitPhysicalLanceScan is added in PR #74661 which depends on this PR. See #74665 for the full dependency chain.

beinan · 2026-06-16T20:58:51Z

@github-actions run-checks

Add the FE optimizer plumbing for the Lance connector: - LogicalLanceScanOperator / PhysicalLanceScanOperator with visitor hooks - LOGICAL_LANCE_SCAN / PHYSICAL_LANCE_SCAN enum entries in OperatorType - LanceScanImplementationRule + IMP_LANCE_LSCAN_TO_PSCAN rule type - Wire into RuleSet, PushDownPredicateScanRule, StatisticsCalculator, LogicalPlanPrinter, and RelationTransformer This follows the same pattern used by the Kudu connector. Signed-off-by: Beinan Wang <soros.wang@yahoo.com> Co-Authored-By: Beinan Wang <beinanwang@microsoft.com> Signed-off-by: Beinan Wang <soros.wang@yahoo.com>

beinan · 2026-06-30T17:41:16Z

@github-actions run-checks

beinan · 2026-06-30T18:28:26Z

@github-actions run-checks

Signed-off-by: Beinan Wang <soros.wang@yahoo.com>

beinan · 2026-07-01T00:37:14Z

Hi @stephen-shelby @HangyuanLiu, gentle ping on this PR when you have a chance. It is the first (bottom) PR of the Lance connector stack (#74660 → #74661 → #74662 → #74663 → #74664) and only adds the optimizer operators/rules. CI is green except for a known flaky PublishVersionDaemonTest.testSharedNothingPublishWithThreadPool (unrelated to this change), which I have just re-triggered. Would appreciate your review to get the stack moving. Thanks!

Cover LogicalLanceScanOperator, PhysicalLanceScanOperator, and LanceScanImplementationRule to satisfy FE incremental coverage. Signed-off-by: Beinan Wang <soros.wang@yahoo.com> Co-Authored-By: Beinan Wang <beinanwang@microsoft.com>

github-actions · 2026-07-01T05:36:37Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-07-01T05:37:25Z

[FE Incremental Coverage Report]

✅ pass : 45 / 54 (83.33%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/sql/optimizer/statistics/StatisticsCalculator.java	0	8	00.00%	[814, 819, 824, 825, 826, 828, 829, 832]
🔵	com/starrocks/sql/optimizer/transformer/RelationTransformer.java	1	2	50.00%	[739]
🔵	com/starrocks/sql/optimizer/operator/physical/PhysicalLanceScanOperator.java	14	14	100.00%	[]
🔵	com/starrocks/sql/optimizer/rule/implementation/LanceScanImplementationRule.java	7	7	100.00%	[]
🔵	com/starrocks/sql/optimizer/operator/OperatorVisitor.java	2	2	100.00%	[]
🔵	com/starrocks/sql/optimizer/rule/RuleType.java	1	1	100.00%	[]
🔵	com/starrocks/sql/optimizer/LogicalPlanPrinter.java	1	1	100.00%	[]
🔵	com/starrocks/sql/optimizer/operator/OperatorType.java	2	2	100.00%	[]
🔵	com/starrocks/sql/optimizer/operator/logical/LogicalLanceScanOperator.java	16	16	100.00%	[]
🔵	com/starrocks/sql/optimizer/OptExpressionVisitor.java	1	1	100.00%	[]

github-actions · 2026-07-01T05:38:03Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

mergify Bot assigned beinan Jun 10, 2026

github-actions Bot requested review from HangyuanLiu and stephen-shelby June 10, 2026 19:17

chatgpt-codex-connector Bot reviewed Jun 10, 2026

View reviewed changes

beinan mentioned this pull request Jun 10, 2026

[Feature] Native Lance connector for StarRocks #74665

Open

1 task

beinan force-pushed the lance/pr2-fe-optimizer branch from 78e577b to e8767ae Compare June 12, 2026 17:14

beinan force-pushed the lance/pr2-fe-optimizer branch from e8767ae to 601ce03 Compare June 16, 2026 21:01

beinan force-pushed the lance/pr2-fe-optimizer branch from 601ce03 to 7de25df Compare June 24, 2026 20:46

ci: trigger re-run

bcd5d7b

Signed-off-by: Beinan Wang <soros.wang@yahoo.com>

[UT] Add unit tests for Lance scan operators and rule

75f0058

Cover LogicalLanceScanOperator, PhysicalLanceScanOperator, and LanceScanImplementationRule to satisfy FE incremental coverage. Signed-off-by: Beinan Wang <soros.wang@yahoo.com> Co-Authored-By: Beinan Wang <beinanwang@microsoft.com>

		public <R, C> R accept(OptExpressionVisitor<R, C> visitor, OptExpression optExpression, C context) {
		return visitor.visitPhysicalLanceScan(optExpression, context);

Uh oh!

Conversation

beinan commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I'm doing this

What I'm doing

Does this PR entail a change in behavior?

Related Issue

Uh oh!

CelerData-Reviewer commented Jun 10, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

beinan Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

beinan Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

beinan commented Jun 16, 2026

Uh oh!

beinan commented Jun 30, 2026

Uh oh!

beinan commented Jun 30, 2026

Uh oh!

beinan commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jul 1, 2026

[FE Incremental Coverage Report]

file detail

Uh oh!

github-actions Bot commented Jul 1, 2026

[BE Incremental Coverage Report]

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

beinan commented Jun 10, 2026 •

edited

Loading