AutoNHITS with prediction_intervals creates second optimization study

[willadamskeane]

What happened + What you expected to happen

When using AutoNHITS with prediction_intervals, I noticed it creates a new optimization study and reruns the entire hyperparameter search process a second time. Example:

model = AutoNHITS(h=24, num_samples=2, backend="optuna")
nf = NeuralForecast(models=[model], freq="H")
nf.fit(df=data, prediction_intervals=PredictionIntervals(n_windows=2))

The logs show "A new study created in memory" after the initial training completes, and it runs another complete optimization cycle.

Is this intended behavior? I would have expected it to run one hyperparameter optimization to find the best configuration, and then use the best model to generate prediction intervals.

Versions / Dependencies

Python 3.11, Neural Forecast 1.77

Reproduction script

import numpy as np
import pandas as pd
from neuralforecast import NeuralForecast
from neuralforecast.auto import AutoNHITS
from neuralforecast.utils import PredictionIntervals

# Create mock data
np.random.seed(42)
n_samples = 1000
dates = pd.date_range("2020-01-01", periods=n_samples, freq="h")
data = pd.DataFrame(
    {"ds": dates, "unique_id": "A", "y": np.random.normal(0, 1, n_samples)}
)

# Configure model
model = AutoNHITS(
    h=24,  # Forecast horizon
    num_samples=2,  # Number of trials
    backend="optuna",
)

# Create NeuralForecast wrapper
nf = NeuralForecast(models=[model], freq="H")

# Fit model - this will show sampling running twice
print("Starting model fit...")
nf.fit(df=data, prediction_intervals=PredictionIntervals(n_windows=2))
print("Finished model fit")

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[elephaint]

Thanks for raising this issue. It's a bit of quirk of the design and of how conformal scores are calculated. Have to think about if/how to address this.

Is this intended behavior? I would have expected it to run one hyperparameter optimization to find the best configuration, and then use the best model to generate prediction intervals.

I'd drop the conformal intervals from the optimization, optimize the model, and add the prediction intervals thereafter on the best settings, that should get you the desired result.