[SPARK-58965][SQL][PYTHON] Add gcd and lcm math functions by Vivek1106-04 · Pull Request #58242 · apache/spark

Vivek1106-04 · 2026-08-24T10:57:57Z

What changes were proposed in this pull request?

This PR adds two new built-in math functions, gcd and lcm:

gcd(expr1, expr2) -- returns BIGINT
lcm(expr1, expr2) -- returns BIGINT

Both arguments are implicitly cast to BIGINT and the result is BIGINT, matching the existing
factorial expression, which likewise takes an integral argument and returns BIGINT.

Semantics:

Case	`gcd`	`lcm`
either argument is NULL	NULL	NULL
both arguments are 0	0	0
one argument is 0	`abs(other)`	0
negative arguments	result is non-negative	result is non-negative
result not representable as BIGINT	`ARITHMETIC_OVERFLOW` under ANSI mode, NULL otherwise	`ARITHMETIC_OVERFLOW` under ANSI mode, NULL otherwise

The greatest common divisor is computed with the Euclidean algorithm. The least common multiple
divides by the greatest common divisor before multiplying, so a representable result never
overflows on the way there — for example lcm(4611686018427387904, 2) returns 4611686018427387904
even though the naive product 4611686018427387904 * 2 would not fit.

Overflow arises in exactly two places, and in both PostgreSQL raises as well:

gcd where the result would be -Long.MinValue, which is not representable — that is, the
input pairs (0, x), (x, 0) and (x, x) for x = Long.MinValue.
lcm where abs(a) / gcd(a, b) * abs(b) exceeds Long.MaxValue.

Both are reported with the existing ARITHMETIC_OVERFLOW error condition, so no new error
condition is introduced. Overflow is gated on ANSI mode as it is elsewhere in Spark — conv in the
same file takes the same approach — raising under ANSI mode and returning NULL otherwise.

Both helpers live in MathUtils and are shared by the interpreted and codegen paths so the two
cannot diverge.

The functions are exposed through SQL, the Scala/Java functions API, PySpark, and Spark Connect.

Why are the changes needed?

Spark SQL currently has no way to compute either value. There is no expression for it, and no
combination of existing built-ins produces the result, so users have to fall back to a UDF — which
for PySpark means a Python round trip per row and no whole-stage codegen.

Both functions are standard in comparable engines:

PostgreSQL 13+: gcd(a, b), lcm(a, b) for integer, bigint and numeric
DuckDB: gcd(a, b), lcm(a, b) (aliases greatest_common_divisor, least_common_multiple)

Common uses include reducing fractions and ratios to lowest terms, aligning batch or partition
sizes, computing the repeat period of overlapping schedules, and normalizing denominators before
aggregation.

Does this PR introduce any user-facing change?

Yes. Two new built-in functions are available in SQL, the Scala/Java functions API, PySpark and
Spark Connect. Previously gcd and lcm were unresolved function names:

-- Before
spark-sql> SELECT gcd(24, 36);
[UNRESOLVED_ROUTINE] Cannot resolve routine `gcd` on search path
[`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`]. SQLSTATE: 42883

-- After
spark-sql> SELECT gcd(24, 36);
12
spark-sql> SELECT lcm(4, 6);
12

No existing behavior changes; the change is purely additive.

How was this patch tested?

New tests:

MathExpressionsSuite — unit tests for both expressions covering ordinary values, negative
inputs, zero, NULL, the boundary values around Long.MinValue / Long.MaxValue, and overflow in
both ANSI and non-ANSI mode, plus interpreted-vs-codegen consistency checks.
math.sql golden-file tests, regenerated for the ANSI and non-ANSI results and the analyzer
results.
PlanGenerationTestSuite function tests, with the regenerated Spark Connect plan and explain
golden files.
PySpark doctests for gcd and lcm.

Existing suites re-run and passing: MathExpressionsSuite, SQLQueryTestSuite (math.sql),
ExpressionsSchemaSuite, PlanGenerationTestSuite, ProtoToParsedPlanTestSuite.

dev/lint-scala and dev/lint-python both pass.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

Vivek1106-04 · 2026-09-04T00:31:29Z

@uros-b ! can you review this, added two new functions which are standard functions in other databases.

uros-b · 2026-09-04T10:21:24Z

Thank you for @Vivek1106-04! I'll take a look soon. Also, cc @bojana-db to help with reviewing this PR.

uros-b · 2026-09-04T16:43:57Z

   */
  def factorial(e: Column): Column = Column.fn("factorial", e)

+  /**


gcd/lcm sit between factorial (5443) and floor (5488). The section is strictly alphabetical (cos, cosh, cot, csc, exp, expm1, factorial, floor, greatest, hex, hypot, least, ln, log), so gcd belongs after the floor overloads / before greatest (~5526) and lcm after the last hypot overload / before least (~5696).

__init__.py and functions.rst place both correctly, so this file is the lone inconsistency.

Vivek1106-04 · 2026-09-06T09:29:37Z

I also double-checked the other places the functions are registered/exported: FunctionRegistry.scala is already alphabetical there (factorial, gcd, hypot, lcm), and the Python functions/builtin.py files aren't alphabetically ordered to begin with, so functions.scala was indeed the only inconsistency.

bojana-db

LGTM, left a few comments

bojana-db · 2026-09-07T14:45:08Z

+
+  override def dataType: DataType = LongType
+
+  // The result overflows for inputs whose divisor is -Long.MinValue, which is null in ANSI mode.


Suggested change

// The result overflows for inputs whose divisor is -Long.MinValue, which is null in ANSI mode.

// The result overflows for inputs whose divisor is -Long.MinValue, which is null in non-ANSI mode.

bojana-db · 2026-09-07T15:11:00Z

+    })
+  }
+
+  override def prettyName: String = "gcd"


This is redundant. Single word function names are correctly handled by the base class. (Same for lcm)

bojana-db · 2026-09-07T15:17:36Z

+    checkEvaluation(Gcd(Literal(24L), Literal(36L)), 12L)
+    checkEvaluation(Gcd(Literal(36L), Literal(24L)), 12L)
+    checkEvaluation(Gcd(Literal(17L), Literal(5L)), 1L)
+    // The result is never negative, whichever inputs are.


Suggested change

// The result is never negative, whichever inputs are.

// The result is non-negative regardless of the inputs' signs.

(Same for other occurrences.)

bojana-db

LGTM, thank you for the feature!

Left some suggestions for grammatically awkward comments.

bojana-db · 2026-09-08T09:51:17Z

+-- Gcd
+SELECT gcd(24, 36);
+SELECT gcd(17, 5);
+-- The result is never negative, whichever inputs are.


Suggested change

-- The result is never negative, whichever inputs are.

-- The result is non-negative regardless of the inputs' signs.

bojana-db · 2026-09-08T09:51:40Z

+-- Lcm
+SELECT lcm(4, 6);
+SELECT lcm(17, 5);
+-- The result is never negative, whichever inputs are.


Suggested change

-- The result is never negative, whichever inputs are.

-- The result is non-negative regardless of the inputs' signs.

uros-b · 2026-09-10T13:13:12Z

Thank you @Vivek1106-04 and @bojana-db!

uros-b · 2026-09-10T13:13:33Z

Let's loop in @Yicong-Huang and/or @HyukjinKwon for PySpark as well ^^

HyukjinKwon · 2026-09-10T23:13:18Z

cc @cloud-fan

uros-b · 2026-09-11T09:39:34Z

Also, please resolve conflicts @Vivek1106-04

Spark SQL has no expression, and no combination of existing built-ins, that computes the greatest common divisor or the least common multiple of two integers, so users have to fall back to a UDF. This adds `gcd(expr1, expr2)` and `lcm(expr1, expr2)`. Both arguments are implicitly cast to BIGINT and the result is BIGINT, matching the existing `factorial` expression. The result is never negative, `gcd(0, 0)` and `lcm(0, 0)` are 0, and a NULL argument yields NULL. The greatest common divisor is computed with the Euclidean algorithm. The least common multiple divides by it before multiplying, so a result that is representable does not overflow on the way there. Results that are genuinely unrepresentable raise ARITHMETIC_OVERFLOW under ANSI mode and return NULL otherwise, as elsewhere in Spark. Both helpers live in MathUtils and are shared by the interpreted and codegen paths. PostgreSQL 13+ and DuckDB provide the same two functions.

….scala The math_funcs section of functions.scala is ordered alphabetically. Place gcd after the floor overloads (before greatest) and lcm after the last hypot overload (before least). Pure code movement, no behavior change.

- Fix the Gcd nullable comment: overflow yields null in non-ANSI mode, not ANSI mode (ANSI raises). - Drop the redundant prettyName overrides on Gcd and Lcm; the base Expression.prettyName already lowercases the single-word node name. - Reword the sign-related comments in the gcd/lcm tests.

- Reword the sign-related comments in math.sql to match the wording already applied in MathExpressionsSuite. - Use the same "non-negative" wording in the public Scala and Python docs for gcd and lcm. - Collapse the Gcd and Lcm nullable comments to the fact they explain: overflow yields null in non-ANSI mode. - Drop the note that MathUtils.gcd/lcm are shared by the eval and codegen paths; that is what a shared helper is for.

The lcm scaladoc line exceeded the 98-column scalafmt limit for sql/api, failing the Scala linter check. Rewrap it to match the gcd scaladoc.

uros-b reviewed Sep 4, 2026

View reviewed changes

Vivek1106-04 requested a review from uros-b September 6, 2026 09:29

bojana-db reviewed Sep 7, 2026

View reviewed changes

Vivek1106-04 requested a review from bojana-db September 8, 2026 07:57

bojana-db approved these changes Sep 8, 2026

View reviewed changes

Vivek1106-04 requested a review from bojana-db September 10, 2026 04:10

uros-b approved these changes Sep 10, 2026

View reviewed changes

uros-b requested review from HyukjinKwon and Yicong-Huang and removed request for bojana-db September 10, 2026 13:14

uros-b requested a review from cloud-fan September 11, 2026 10:24

Vivek1106-04 added 5 commits September 12, 2026 15:19

[SPARK-58965][SQL] Fix scalafmt line length in lcm scaladoc

182c0be

The lcm scaladoc line exceeded the 98-column scalafmt limit for sql/api, failing the Scala linter check. Rewrap it to match the gcd scaladoc.

Vivek1106-04 force-pushed the SPARK-58965-gcd-lcm branch from c51ef70 to 182c0be Compare September 12, 2026 09:50


		override def dataType: DataType = LongType

		// The result overflows for inputs whose divisor is -Long.MinValue, which is null in ANSI mode.

	// The result is never negative, whichever inputs are.
	// The result is non-negative regardless of the inputs' signs.

	-- The result is never negative, whichever inputs are.
	-- The result is non-negative regardless of the inputs' signs.

Uh oh!

Conversation

Vivek1106-04 commented Aug 24, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Vivek1106-04 commented Sep 4, 2026

Uh oh!

uros-b commented Sep 4, 2026

Uh oh!

uros-b Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

Vivek1106-04 commented Sep 6, 2026

Uh oh!

bojana-db left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bojana-db Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

bojana-db Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

bojana-db Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

bojana-db left a comment

Choose a reason for hiding this comment

Uh oh!

bojana-db Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

bojana-db Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

uros-b commented Sep 10, 2026

Uh oh!

uros-b commented Sep 10, 2026

Uh oh!

HyukjinKwon commented Sep 10, 2026

Uh oh!

uros-b commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bojana-db left a comment •

edited

Loading