Make GP Sampler the standard Optuna Sampler when torch is installed

[thomasATbayer]

Use GPSampler as the Default Optuna Sampler When torch Is Available

Summary

MotherML currently hard-codes TPESampler as the default Optuna sampler in MotherTuner. The official Optuna documentation and benchmarks show that GPSampler is the better choice for budgets up to ~500 trials, which covers virtually all practical MotherML tuning runs (the default n_trials_optuna is 100). We should switch the default to GPSampler whenever torch (and scipy) is available, with a graceful fallback to TPESampler when it is not.

---

Evidence from the Optuna Documentation

Sampler Comparison Table

The official Optuna sampler reference contains an explicit recommended budget row:

Sampler	Recommended Budget (# trials)	Time Complexity (per trial)
TPESampler	100 – 1 000	$O(d \cdot n \log n)$
GPSampler	up to ~500	$O(n^3)$
CmaEsSampler	1 000 – 10 000	$O(d^3)$

Where $d$ = search-space dimension, $n$ = number of completed trials.

Key observation: GPSampler is explicitly recommended for budgets ≤ 500 trials — the exact regime MotherML operates in by default. The $O(n^3)$ cost is negligible when $n < 200$ and the evaluation cost of a CatBoost/RF model dominates by orders of magnitude.

Algorithm Selection Guide

The Efficient Optimization Algorithms tutorial from the Optuna developers summarises:

For limited parallel compute resources and no (or few) categorical hyperparameters:
TPE. GP-EI if search space is low-dimensional and continuous.

MotherML's CatBoost/RandomForest search spaces are exactly this: low-to-medium dimensional, mostly continuous/integer parameters (learning rate, depth, L2 reg, etc.), with only a handful of categorical choices (boosting type, grow policy). This is the ideal GPSampler regime.

---

Evidence from AutoSampler (OptunaHub)

The Optuna team's own AutoSampler — their recommended "just works" sampler — applies exactly this logic:

"Using GPSampler during the early stages of a search, as it has excellent sample efficiency but is only applicable to a small number of evaluations."

Benchmark results from the AutoSampler announcement blog post show:

• AutoSampler (GP-first) with < 100 trials finds a better solution than TPESampler with 1 000 trials on multiple benchmark functions.
• AutoSampler converges significantly faster and is more stable (narrower variance bands) than default TPESampler on Hartmann6 and McCourt11 benchmarks.

This is the core argument: GPSampler uses the trial budget dramatically more efficiently at small $n$, because it builds a principled probabilistic surrogate (Matérn-5/2 GP with ARD) that models parameter correlations globally, whereas TPE's kernel density estimates only become reliable after ~50–100 observations.

---

Why GPSampler Is Better in the Low-Budget Regime

Property	TPESampler	GPSampler
Surrogate model	Kernel Density Estimates (l/g split)	Gaussian Process (Matérn-5/2, ARD)
Models parameter correlations	Partially (multivariate mode)	Yes, natively
Sample efficiency at $n < 100$	Moderate	High
Sample efficiency at $n > 500$	Good	Degrades ($O(n^3)$)
Acquisition function	EI via ratio of densities	log-EI (numerically superior)
Handles integers + floats well	Yes	Yes (better than CMA-ES)
Requires torch	No	Yes (scipy + torch)

The GP's ability to reason about the global landscape from very few points is the fundamental advantage. TPE requires enough trials in each density bucket to estimate l(x) and g(x) reliably — this threshold is usually around 25–50 trials just for n_startup_trials, meaning effective Bayesian search only starts after that point.

---

Fit for MotherML's Search Spaces

MotherML's CatBoost hyperparameter spaces (e.g., learning_rate, depth, l2_leaf_reg, bagging_temperature) are:

• Primarily continuous and integer — GPSampler's strength.
• Low-to-medium dimensional (typically 5–12 parameters) — well within the GP's reliable range.
• Not heavily conditional/tree-structured — GPSampler handles this adequately (vs. TPE's strength in deep conditional spaces).

The few categorical dimensions (boosting_type, grow_policy) are handled by GPSampler's independent fallback sampler (random by default), identical to how TPE handles them.

---

Proposed Implementation

In src/mother/optimization/core.py, replace the hard-coded TPESampler default with a conditional:

def _default_sampler(
    seed: int, n_startup_trials: int, multivariate: bool
) -> optuna.samplers.BaseSampler:
    """Return GPSampler when torch is available, else fall back to TPESampler."""
    try:
        import torch  # noqa: F401
        import scipy  # noqa: F401
        module_logger.debug("torch available — using GPSampler as default")
        return optuna.samplers.GPSampler(
            seed=seed,
            n_startup_trials=n_startup_trials,
            deterministic_objective=False,
        )
    except ImportError:
        module_logger.debug("torch not available — falling back to TPESampler")
        return optuna.samplers.TPESampler(
            multivariate=multivariate,
            group=True,
            constant_liar=True,
            seed=seed,
            n_startup_trials=n_startup_trials,
        )

Then in MotherTuner.__init__:

if sampler is None:
    self.sampler = _default_sampler(
        seed=seed,
        n_startup_trials=n_startup_trials,
        multivariate=kwargs.get("multivariate", True),
    )
else:
    self.sampler = sampler

Users who need the old behaviour can still pass sampler=optuna.samplers.TPESampler(...) explicitly.

---

Considerations / Open Questions

1. torch is already a CatBoost dependency in many environments; the incremental cost of this switch is zero for most MotherML users.
2. GPSampler does not support pruning (NopPruner is effectively required). MotherML does not currently use Optuna pruners, so this is a non-issue.
3. GPSampler does not support batch/distributed optimization well. MotherML runs n_threads_optuna=1 by default; multi-threaded runs may want TPE with constant_liar=True. Could expose sampler selection as a config option tied to n_threads_optuna.
4. n_startup_trials: GPSampler defaults to 10 random warm-up trials; MotherML's current default is 12 — compatible.
5. Categorical parameters (boosting type, grow policy): GPSampler falls back to its independent_sampler (random) for categoricals. This is acceptable; TPE's handling of categoricals at small $n$ is not much better.

---

References

• Optuna Sampler Comparison Table
• Optuna — Which Sampler and Pruner Should be Used?
• Optuna GPSampler API Reference
• AutoSampler: Automatic Selection of Optimization Algorithms in Optuna (Ozaki, 2024)
• Unexpected Improvements to Expected Improvement for Bayesian Optimization — logEI acquisition used by GPSampler
• Ozaki et al., Hyperparameter Optimization Methods: Overview and Characteristics, IEICE Trans, 2020

[thomasATbayer]

A constant liar strategy has just been introduced to the GPsampler in Optuna meaning that parallelization would be fully supported: optuna/optuna#6512