Commit c1d8707
committed
[SPARK-58250][ML][CONNECT] Add OneVsRest size estimate
### What changes were proposed in this pull request?
This patch adds a model-owned `estimatedSize` implementation for `OneVsRestModel`. The estimate includes the model's parameter metadata, label metadata, and the estimates of its binary classification submodels.
### Why are the changes needed?
Spark Connect uses `Model.estimatedSize` to enforce ML model-cache limits. `OneVsRestModel` was the only concrete classification model that inherited the default graph traversal. Its copied submodels retain their parent estimators, so that traversal can include shared Spark session/context state rather than only the cached model's data. The explicit estimate follows the established classifier-model pattern and accounts only for state owned by the One-vs-Rest model.
### Does this PR introduce _any_ user-facing change?
Yes. Spark Connect cache accounting for One-vs-Rest models becomes more accurate, preventing shared runtime state from inflating the estimated model size. There is no public API change.
### How was this patch tested?
Added a regression test that verifies the estimate is the sum of One-vs-Rest parameter metadata, label metadata, and its contained models' estimates.
Ran:
```
JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 build/sbt 'mllib/testOnly *OneVsRestSuite'
```
All 17 tests passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57420 from zhengruifeng/SPARK-58181-one-vs-rest-estimated-size-dev3.
Authored-by: Ruifeng Zheng <ruifengz@apache.org>
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>1 parent ce39ba8 commit c1d8707
2 files changed
Lines changed: 24 additions & 0 deletions
File tree
- mllib/src
- main/scala/org/apache/spark/ml/classification
- test/scala/org/apache/spark/ml/classification
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
45 | 46 | | |
46 | 47 | | |
47 | 48 | | |
| |||
148 | 149 | | |
149 | 150 | | |
150 | 151 | | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
151 | 157 | | |
152 | 158 | | |
153 | 159 | | |
| |||
Lines changed: 18 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
71 | 89 | | |
72 | 90 | | |
73 | 91 | | |
| |||
0 commit comments