Skip to content

Commit b6887e8

Browse files
srielaucloud-fan
srielau
authored and
cloud-fan
committed
[SPARK-58176][DOC] Add SQL reference documentation for ASOF JOIN
## What changes were proposed in this pull request? Add Spark SQL Language Reference documentation for `ASOF JOIN`: - New page: `docs/sql-ref-syntax-qry-select-asof-join.md` (syntax, parameters, semantics, examples) - Cross-links from `JOIN`, `SELECT`, and SQL Syntax index pages - Migration guide note for Spark 4.3 (`spark.sql.join.asofJoin.enabled`, default `false`) Docs-only PR. The ASOF JOIN implementation and FVT golden-file tests landed separately via #57380 and #57306. ## Why are the changes needed? SPARK-58176 tracks user-facing documentation for the new `ASOF JOIN` SQL syntax. ## Does this PR introduce _any_ user-facing change? Yes — documentation only. ## How was this patch tested? Docs-only change. Examples follow the SQL reference spec and match the FVT golden-file scenarios in #57306. ## Was this patch authored or co-authored using generative AI tooling? Yes, with human review. Closes #57312 from srielau/SPARK-58176. Authored-by: Serge Rielau <serge@rielau.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent 31f58da commit b6887e8

5 files changed

Lines changed: 240 additions & 1 deletion

‎docs/sql-migration-guide.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ license: |
2424

2525
## Upgrading from Spark SQL 4.2 to 4.3
2626

27+
- Since Spark 4.3, [ASOF JOIN](sql-ref-syntax-qry-select-asof-join.html) is available as an opt-in SQL feature gated by `spark.sql.join.asofJoin.enabled` (default `false`). When disabled, `ASOF JOIN` fails at parse time with `UNSUPPORTED_FEATURE.ASOF_JOIN`.
2728
- Since Spark 4.3, zero-length files are skipped during Parquet schema inference instead of failing with a `FAILED_READ_FILE.CANNOT_READ_FILE_FOOTER` error.
2829
- Since Spark 4.3, the configuration key `spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled` has been renamed to `spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` to reflect that it now applies to storage-partitioned joins, aggregates, and windows. The old key continues to work as an alias.
2930
- Since Spark 4.3, the Spark Thrift Server rejects setting JVM system properties through the `set:system:` session configuration overlay (for example, in a JDBC connection string). To restore the previous behavior, set `spark.sql.legacy.hive.thriftServer.allowSettingSystemProperties` to `true`.

‎docs/sql-ref-syntax-qry-select-asof-join.md‎

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
---
2+
layout: global
3+
title: ASOF JOIN
4+
displayTitle: ASOF JOIN
5+
license: |
6+
Licensed to the Apache Software Foundation (ASF) under one or more
7+
contributor license agreements. See the NOTICE file distributed with
8+
this work for additional information regarding copyright ownership.
9+
The ASF licenses this file to You under the Apache License, Version 2.0
10+
(the "License"); you may not use this file except in compliance with
11+
the License. You may obtain a copy of the License at
12+
13+
http://www.apache.org/licenses/LICENSE-2.0
14+
15+
Unless required by applicable law or agreed to in writing, software
16+
distributed under the License is distributed on an "AS IS" BASIS,
17+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18+
See the License for the specific language governing permissions and
19+
limitations under the License.
20+
---
21+
22+
### Description
23+
24+
`ASOF JOIN` combines each row from the left relation with at most one row from the
25+
right relation. The match is the closest row on the right that satisfies a required
26+
`MATCH_CONDITION` comparison and an optional `ON` or `USING` filter.
27+
28+
`ASOF JOIN` is asymmetric: the left relation drives per-row lookups into the right,
29+
and reversing the operands changes the result.
30+
31+
`ASOF JOIN` is disabled by default. Set `spark.sql.join.asofJoin.enabled` to `true`
32+
to enable the syntax. When disabled, `ASOF JOIN` fails at parse time with
33+
`UNSUPPORTED_FEATURE.ASOF_JOIN`.
34+
35+
Only `INNER` (the default) and `LEFT OUTER` join types are supported.
36+
37+
### Syntax
38+
39+
```sql
40+
[ asof_join_type ] ASOF JOIN right_table_reference join_criteria
41+
42+
asof_join_type
43+
{ [ INNER ] |
44+
LEFT [ OUTER ] }
45+
46+
join_criteria
47+
MATCH_CONDITION ( expr1 comparison_operator expr2 )
48+
[ ON boolean_expression | USING ( column_name [, ...] ) ]
49+
50+
comparison_operator
51+
{ >= | > | <= | < }
52+
```
53+
54+
### Parameters
55+
56+
* **right_table_reference**
57+
58+
The table reference on the right of the join; searched once per left row.
59+
60+
* **asof_join_type**
61+
62+
The outer form of the `ASOF JOIN`.
63+
* `INNER` (default): drops left rows with no qualifying match.
64+
* `LEFT [ OUTER ]`: retains such rows with `NULL` right-side columns.
65+
66+
* **join_criteria**
67+
68+
A required `MATCH_CONDITION` clause and an optional `ON` or `USING` clause that
69+
filters the join input before the closest-match selection. When both `ON` and
70+
`USING` are omitted, every left row is matched against the entire right table on
71+
the `MATCH_CONDITION` alone.
72+
73+
* **MATCH_CONDITION ( expr1 comparison_operator expr2 )**
74+
75+
Specifies the ordered comparison that determines the closest match. One of
76+
`expr1` and `expr2` must reference only the left table; the other must
77+
reference only the right table.
78+
79+
* **expr1**, **expr2**
80+
81+
Scalar expressions of an orderable, mutually comparable type. Each must be
82+
deterministic; subqueries, aggregate functions, window functions, and
83+
generator functions are not permitted. `STRUCT` operands compare
84+
lexicographically field-by-field, enabling multi-column match conditions
85+
(see Examples).
86+
87+
* **comparison_operator**
88+
89+
One of `>=`, `>`, `<=`, `<`. Equality operators (`=` and `<>`) are not
90+
permitted.
91+
92+
* **ON boolean_expression**
93+
94+
An expression with a return type of `BOOLEAN` that specifies how rows from the
95+
two relations are matched before the closest-match selection. If the result is
96+
`true`, the rows are considered a match. See [JOIN](sql-ref-syntax-qry-select-join.html).
97+
98+
* **USING ( column_name [, ...] )**
99+
100+
Matches rows by comparing equality for the list of columns, which must exist in
101+
both relations. See [JOIN](sql-ref-syntax-qry-select-join.html).
102+
103+
### Notes
104+
105+
* **Direction of match.** Let *L* be the operand of `MATCH_CONDITION` that references
106+
the left table and *R* the operand that references the right table. The operator
107+
determines which row on the right is closest:
108+
* *L* `>=` *R* (equivalently *R* `<=` *L*): the right row with the largest *R* not
109+
greater than *L* (last-preceding).
110+
* *L* `>` *R*: the largest *R* strictly less than *L* (last strictly-preceding).
111+
* *L* `<=` *R*: the smallest *R* not less than *L* (first-following).
112+
* *L* `<` *R*: the smallest *R* strictly greater than *L* (first strictly-following).
113+
114+
* **NULL.** A `NULL` in either operand never satisfies the `MATCH_CONDITION`
115+
comparison. Left rows whose operand is `NULL` are dropped under `INNER ASOF` and
116+
retained with `NULL` right-side columns under `LEFT ASOF`.
117+
118+
* **Ties.** If multiple right rows share the same *R* value that is closest to *L*
119+
after applying `ON`/`USING`, the choice among tied rows is not deterministic. Add a
120+
tie-breaker either as an equi-key in `ON`/`USING` or as a trailing field of a
121+
`STRUCT`-valued `MATCH_CONDITION` (see Examples).
122+
123+
### Examples
124+
125+
```sql
126+
SET spark.sql.join.asofJoin.enabled=true;
127+
128+
CREATE OR REPLACE TEMP VIEW trades(trade_time, symbol, quantity) AS
129+
VALUES (TIMESTAMP '2026-06-29 10:00:05', 'AAPL', 100),
130+
(TIMESTAMP '2026-06-29 10:00:11', 'AAPL', 200),
131+
(TIMESTAMP '2026-06-29 10:00:12', 'MSFT', 50),
132+
(TIMESTAMP '2026-06-29 09:59:59', 'GOOG', 30);
133+
134+
CREATE OR REPLACE TEMP VIEW quotes(quote_time, symbol, bid_price) AS
135+
VALUES (TIMESTAMP '2026-06-29 10:00:00', 'AAPL', 180.10),
136+
(TIMESTAMP '2026-06-29 10:00:07', 'AAPL', 180.15),
137+
(TIMESTAMP '2026-06-29 10:00:10', 'AAPL', 180.20),
138+
(TIMESTAMP '2026-06-29 10:00:08', 'MSFT', 420.50);
139+
140+
-- Attach the most recent (last-preceding) quote to each trade.
141+
-- The GOOG trade has no matching quote and is dropped by INNER ASOF JOIN.
142+
SELECT t.trade_time, t.symbol, t.quantity, q.bid_price
143+
FROM trades t
144+
ASOF JOIN quotes q
145+
MATCH_CONDITION (t.trade_time >= q.quote_time)
146+
ON t.symbol = q.symbol;
147+
+-------------------+------+--------+---------+
148+
| trade_time|symbol|quantity|bid_price|
149+
+-------------------+------+--------+---------+
150+
|2026-06-29 10:00:05| AAPL| 100| 180.10|
151+
|2026-06-29 10:00:11| AAPL| 200| 180.20|
152+
|2026-06-29 10:00:12| MSFT| 50| 420.50|
153+
+-------------------+------+--------+---------+
154+
155+
-- Preserve unmatched left rows with LEFT ASOF JOIN.
156+
SELECT t.trade_time, t.symbol, q.bid_price
157+
FROM trades t
158+
LEFT ASOF JOIN quotes q
159+
MATCH_CONDITION (t.trade_time >= q.quote_time)
160+
ON t.symbol = q.symbol;
161+
+-------------------+------+---------+
162+
| trade_time|symbol|bid_price|
163+
+-------------------+------+---------+
164+
|2026-06-29 09:59:59| GOOG| NULL|
165+
|2026-06-29 10:00:05| AAPL| 180.10|
166+
|2026-06-29 10:00:11| AAPL| 180.20|
167+
|2026-06-29 10:00:12| MSFT| 420.50|
168+
+-------------------+------+---------+
169+
170+
-- USING is equivalent to ON symbol equality.
171+
SELECT trade_time, symbol, bid_price
172+
FROM trades
173+
ASOF JOIN quotes
174+
MATCH_CONDITION (trades.trade_time >= quotes.quote_time)
175+
USING (symbol);
176+
177+
-- Find the next scheduled maintenance window for each alert.
178+
-- A <= operator selects the first-following right row.
179+
CREATE OR REPLACE TEMP VIEW alerts(alert_time, host) AS
180+
VALUES (TIMESTAMP '2026-06-29 10:00:00', 'db-01');
181+
182+
CREATE OR REPLACE TEMP VIEW maintenance(window_start, host) AS
183+
VALUES (TIMESTAMP '2026-06-29 08:00:00', 'db-01'),
184+
(TIMESTAMP '2026-06-29 12:00:00', 'db-01');
185+
186+
SELECT a.alert_time, a.host, m.window_start
187+
FROM alerts a
188+
ASOF JOIN maintenance m
189+
MATCH_CONDITION (a.alert_time <= m.window_start)
190+
ON a.host = m.host;
191+
+-------------------+-----+-------------------+
192+
| alert_time| host| window_start|
193+
+-------------------+-----+-------------------+
194+
|2026-06-29 10:00:00|db-01|2026-06-29 12:00:00|
195+
+-------------------+-----+-------------------+
196+
197+
-- Multi-column MATCH_CONDITION using tuple comparison. Both sides carry a
198+
-- (time, seq) pair; ties on time are broken lexicographically by seq.
199+
CREATE OR REPLACE TEMP VIEW deploys(deploy_ts, seq, service, version) AS
200+
VALUES (TIMESTAMP '2026-06-29 10:00:00', 1, 'api', 'v1.0'),
201+
(TIMESTAMP '2026-06-29 10:00:00', 2, 'api', 'v1.1'),
202+
(TIMESTAMP '2026-06-29 10:05:00', 1, 'api', 'v1.2');
203+
204+
CREATE OR REPLACE TEMP VIEW requests(req_ts, seq, service) AS
205+
VALUES (TIMESTAMP '2026-06-29 10:00:00', 5, 'api'),
206+
(TIMESTAMP '2026-06-29 10:03:00', 1, 'api');
207+
208+
SELECT r.req_ts, r.seq, d.version
209+
FROM requests r
210+
ASOF JOIN deploys d
211+
MATCH_CONDITION ((r.req_ts, r.seq) >= (d.deploy_ts, d.seq))
212+
ON r.service = d.service;
213+
+-------------------+---+-------+
214+
| req_ts|seq|version|
215+
+-------------------+---+-------+
216+
|2026-06-29 10:00:00| 5| v1.1|
217+
|2026-06-29 10:03:00| 1| v1.1|
218+
+-------------------+---+-------+
219+
```
220+
221+
### Related Statements
222+
223+
* [SELECT](sql-ref-syntax-qry-select.html)
224+
* [JOIN](sql-ref-syntax-qry-select-join.html)

‎docs/sql-ref-syntax-qry-select-join.md‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,9 @@ A SQL join is used to combine rows from two relations based on join criteria. Th
2626
### Syntax
2727

2828
```sql
29-
relation { [ join_type ] JOIN [ LATERAL ] relation [ join_criteria | nearest_by_clause ] | NATURAL join_type JOIN [ LATERAL ] relation }
29+
relation { [ join_type ] JOIN [ LATERAL ] relation [ join_criteria | nearest_by_clause ]
30+
| [ asof_join_type ] ASOF JOIN relation join_criteria
31+
| NATURAL join_type JOIN [ LATERAL ] relation }
3032
```
3133

3234
### Parameters
@@ -53,6 +55,14 @@ relation { [ join_type ] JOIN [ LATERAL ] relation [ join_criteria | nearest_by_
5355

5456
Specifies an expression with a return type of boolean.
5557

58+
* **asof_join_type**
59+
60+
Specifies the outer form of an [ASOF JOIN](sql-ref-syntax-qry-select-asof-join.html). Only `INNER` (the default) and `LEFT OUTER` are supported.
61+
62+
* **join_criteria** (for `ASOF JOIN`)
63+
64+
A required `MATCH_CONDITION` clause and an optional `ON` or `USING` clause. See [ASOF JOIN](sql-ref-syntax-qry-select-asof-join.html).
65+
5666
* **nearest_by_clause**
5767

5868
Specifies a nearest-by top-K ranking join. For each row on the left (query side), returns up to `num_results` rows from the right (base side), ranked by `ranking_expression`. Only `INNER` (the default) and `LEFT OUTER` join types are supported with this clause.
@@ -259,5 +269,6 @@ SELECT * FROM employee ANTI JOIN department ON employee.deptno = department.dept
259269
### Related Statements
260270

261271
* [SELECT](sql-ref-syntax-qry-select.html)
272+
* [ASOF JOIN](sql-ref-syntax-qry-select-asof-join.html)
262273
* [Hints](sql-ref-syntax-qry-select-hints.html)
263274
* [LATERAL Subquery](sql-ref-syntax-qry-select-lateral-subquery.html)

‎docs/sql-ref-syntax-qry-select.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ SELECT [ hints , ... ] [ ALL | DISTINCT ] { [ [ named_expression | regex_column_
8989
Specifies a source of input for the query. It can be one of the following:
9090
* Table relation
9191
* [Join relation](sql-ref-syntax-qry-select-join.html)
92+
* [ASOF JOIN](sql-ref-syntax-qry-select-asof-join.html)
9293
* [Pivot relation](sql-ref-syntax-qry-select-pivot.html)
9394
* [Unpivot relation](sql-ref-syntax-qry-select-unpivot.html)
9495
* [Table-value function](sql-ref-syntax-qry-select-tvf.html)
@@ -205,6 +206,7 @@ SELECT [ hints , ... ] [ ALL | DISTINCT ] { [ [ named_expression | regex_column_
205206
* [Inline Table](sql-ref-syntax-qry-select-inline-table.html)
206207
* [File](sql-ref-syntax-qry-select-file.html)
207208
* [JOIN](sql-ref-syntax-qry-select-join.html)
209+
* [ASOF JOIN](sql-ref-syntax-qry-select-asof-join.html)
208210
* [LIKE Predicate](sql-ref-syntax-qry-select-like.html)
209211
* [Set Operators](sql-ref-syntax-qry-select-setops.html)
210212
* [TABLESAMPLE](sql-ref-syntax-qry-select-sampling.html)

‎docs/sql-ref-syntax.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ ability to generate logical and physical plan for a given query using
7373
* [Inline Table](sql-ref-syntax-qry-select-inline-table.html)
7474
* [File](sql-ref-syntax-qry-select-file.html)
7575
* [JOIN](sql-ref-syntax-qry-select-join.html)
76+
* [ASOF JOIN](sql-ref-syntax-qry-select-asof-join.html)
7677
* [LIKE Predicate](sql-ref-syntax-qry-select-like.html)
7778
* [LIMIT Clause](sql-ref-syntax-qry-select-limit.html)
7879
* [OFFSET Clause](sql-ref-syntax-qry-select-offset.html)

0 commit comments

Comments
 (0)