Skip to content

Commit d7bd595

Browse files
haiyangsun-dbHyukjinKwon
authored andcommitted
[SPARK-59435][UDF] Qualify UDF worker proto import paths
### What changes were proposed in this pull request? This PR relocates the UDF worker protobuf definitions beneath `udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/` and updates their intra-proto imports to use the corresponding qualified paths. This follows the namespaced source layouts used by other Spark protobuf definitions, including: - [Spark Connect common.proto](https://github.com/apache/spark/blob/master/sql/connect/common/src/main/protobuf/spark/connect/common.proto) - [configuration schema](https://github.com/apache/spark/blob/master/common/config/src/main/protobuf/org/apache/spark/config/config_schema.proto) - [status store types](https://github.com/apache/spark/blob/master/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto) - [streaming state message](https://github.com/apache/spark/blob/master/sql/core/src/main/protobuf/org/apache/spark/sql/execution/streaming/StateMessage.proto) ### Why are the changes needed? The UDF worker definitions previously used generic bare import names such as `common.proto` and `udf_message.proto`. Relocating them establishes stable, qualified import paths, avoiding ambiguity if Spark protobuf sources are later collected, indexed, or compiled alongside definitions with similarly named files. The message definitions, generated Java package, and wire format are unchanged. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - `protoc` compilation of the relocated UDF worker protobuf definitions. - `build/mvn -pl udf/worker/proto,udf/worker/grpc -am -DskipTests compile` - `build/mvn -pl udf/worker/grpc -am test` - UDF worker core: 59 tests passed. - UDF worker gRPC: 60 tests passed. ### Was this patch authored or co-authored using generative AI tooling? Yes. Closes #58738 from haiyangsun-db/SPARK-59435-qualify-udf-worker-proto-imports. Authored-by: Haiyang Sun <haiyang.sun@databricks.com> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> (cherry picked from commit f3b1883) Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
1 parent e5dcad6 commit d7bd595

5 files changed

Lines changed: 14 additions & 12 deletions

File tree

udf/worker/README.md

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -34,10 +34,11 @@ provisioning service or daemon).
3434
```
3535
udf/worker/
3636
├── proto/ -- protobuf message classes only (protobuf-java)
37-
│ worker_spec.proto -- UDFWorkerSpecification protobuf
38-
│ udf_message.proto -- UDF execution protocol messages (Init, UdfPayload, ...)
39-
│ udf_service.proto -- UdfWorker gRPC service (Execute, Manage)
40-
│ common.proto -- shared enums (UDFWorkerDataFormat, etc.)
37+
│ └── src/main/protobuf/org/apache/spark/udf/worker/
38+
│ worker_spec.proto -- UDFWorkerSpecification protobuf
39+
│ udf_message.proto -- UDF execution protocol messages (Init, UdfPayload, ...)
40+
│ udf_service.proto -- UdfWorker gRPC service (Execute, Manage)
41+
│ common.proto -- shared enums (UDFWorkerDataFormat, etc.)
4142
4243
├── core/ -- abstract interfaces
4344
│ WorkerDispatcher.scala -- creates sessions, manages worker lifecycle
@@ -56,7 +57,7 @@ udf/worker/
5657
GrpcWorkerChannel.scala -- owns a Netty gRPC channel and event loop
5758
GrpcWorkerSession.scala -- one UDF execution over an Execute stream
5859
UnixDomainSocketTransport.scala -- selects native epoll/kqueue transport
59-
(generated) -- UdfWorkerGrpc stubs from proto/udf_service.proto
60+
(generated) -- UdfWorkerGrpc stubs from the UDF worker service proto
6061
```
6162

6263
The `core/` package defines abstract interfaces that are independent of how
@@ -67,7 +68,7 @@ obtaining workers from a provisioning service or daemon.
6768

6869
The `grpc/` module owns the concrete direct dispatcher, channel and session
6970
implementations, native Unix-domain-socket transport selection, gRPC service-stub
70-
generation (from `proto/`'s `udf_service.proto`), and the gRPC runtime dependencies.
71+
generation (from the UDF worker service proto above), and the gRPC runtime dependencies.
7172
Keeping gRPC here means `proto/`, `core/`, and their consumers (`core`, `catalyst`,
7273
`sql/core`) carry no gRPC dependency on their classpath.
7374

@@ -81,9 +82,10 @@ Engine -> Worker: Init -> PayloadChunk* -> (DataRequest)* -> Finish (Cancel)?
8182
Worker -> Engine: InitResponse -> (DataResponse)* -> (ErrorResponse)? -> (FinishResponse | CancelResponse)
8283
```
8384

84-
See `udf/worker/proto/src/main/protobuf/udf_message.proto` for the complete
85-
message definitions, ordering invariants, and error contract, and
86-
`udf_service.proto` for the gRPC service.
85+
See `udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_message.proto`
86+
for the complete message definitions, ordering invariants, and error contract, and
87+
`udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_service.proto`
88+
for the gRPC service.
8789

8890
### Direct worker creation
8991

udf/worker/proto/src/main/protobuf/common.proto renamed to udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/common.proto

File renamed without changes.

udf/worker/proto/src/main/protobuf/udf_message.proto renamed to udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_message.proto

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717

1818
syntax = "proto3";
1919

20-
import "common.proto";
20+
import "org/apache/spark/udf/worker/common.proto";
2121

2222
package org.apache.spark.udf.worker;
2323

udf/worker/proto/src/main/protobuf/udf_service.proto renamed to udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_service.proto

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717

1818
syntax = "proto3";
1919

20-
import "udf_message.proto";
20+
import "org/apache/spark/udf/worker/udf_message.proto";
2121

2222
package org.apache.spark.udf.worker;
2323

udf/worker/proto/src/main/protobuf/worker_spec.proto renamed to udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/worker_spec.proto

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717

1818
syntax = "proto3";
1919

20-
import "common.proto";
20+
import "org/apache/spark/udf/worker/common.proto";
2121

2222
package org.apache.spark.udf.worker;
2323

0 commit comments

Comments
 (0)