RNN-Direct / LSTM-Direct produce incorrect multi-step forecasts

[alireza-jafari]

What happened + What you expected to happen

When using Nixtla’s RNN-Direct or LSTM-Direct classes to forecast four steps at once, the resulting mean squared error is consistently identical across all horizons, which is far higher than expected and significantly worse than other models in the same framework. By contrast, non‐RNN direct‐forecasting models such as PatchTST(h=4) produce reasonable errors, and the recurrent versions (RNNRecurrent, LSTMRecurrent) also behave as expected. This discrepancy indicates that there is likely a bug in the multi‐step prediction logic for RNN‐based direct forecasting—when RNNs are tasked with predicting all horizons simultaneously, they appear to output implausibly high error values instead of matching the baseline performance of other models.

Versions / Dependencies

neuralforecast: 3.0.1

Reproduction script

import pandas as pd
from neuralforecast import NeuralForecast
from neuralforecast.models import LSTM

models = [LSTM(h=forecasting_horizon, # forecasting_horizon > 1
input_size=look_back_size,
max_steps=max_steps,
learning_rate=learning_rate,
loss=MSE(), random_seed=seed,
scaler_type=scaler_type,
batch_size=batch_size,
step_size=step_size,
recurrent = False,
)]
nf = NeuralForecast(models=models, freq=1, local_scaler_type=scaler_type)
nf.fit(df_train, verbose=True, val_size=0)
preds = nf.predict(df_test)

Issue Severity

High: It blocks me from completing my task.

[marcopeix]

Hello,

I tested the LSTM model with recurrent=False on the AirPassengers dataset, varying the horizon from 1 to 12, and I do not get an identical MSE for all horizons.

Y_train_df = AirPassengersPanel_train = AirPassengersPanel[AirPassengersPanel['ds'] < AirPassengersPanel['ds'].values[-12]].reset_index(drop=True)
Y_test_df = AirPassengersPanel_test = AirPassengersPanel[AirPassengersPanel['ds'] >= AirPassengersPanel['ds'].values[-12]].reset_index(drop=True)

horizons = range(1, 13)

for horizon in horizons:

    lstm = LSTM(
        h=horizon,
        input_size=24,
        recurrent=False,
        max_steps=200
    )
    nf = NeuralForecast(models=[lstm], freq="ME")
    nf.fit(df=Y_train_df)
    preds = nf.predict()
    test_df = pd.merge(Y_test_df, preds, 'left', ['unique_id', 'ds'])
    evaluation = evaluate(
        test_df,
        metrics=[mse],
        models=["LSTM"],
    )
    print(evaluation)

Otherwise, I think I'm missing the "bug" portion of this issue.

1. The fact that the error is higher than expected (what is the expected error) might indicate that LSTM or RNN is not suitable for your dataset.
2. Models within the same framework will perform differently, that's normal. PatchTST is a very solid model that tends to perform well.
3. Setting recurrent=True or False will change the performance of the model, since the predictions are generated differently.

Let me know if I'm missing something, or if you have reproducible code with some data.

[alireza-jafari]

Thank you for your quick response!
By “multi-step forecasting,” I mean predicting multiple future time-steps at once, where the step_size is 1. It means we are forecasting 4 time series with horizon = 4, the model outputs four series of forecasts: one step ahead (h₁), two steps ahead (h₂), three steps ahead (h₃), and four steps ahead (h₄).

I evaluated the following models—data scaled with scikit-learn’s StandardScaler:
models = [
NBEATS(input_size=look_back_size, h=horizon, max_steps=200, scaler_type=None),
PatchTST(input_size=look_back_size, h=horizon, max_steps=200, scaler_type=None),
LSTM(input_size=look_back_size, h=horizon, max_steps=500, recurrent=False, scaler_type=None),
LSTM(input_size=look_back_size, h=horizon, max_steps=500, recurrent=True, scaler_type=None)
]

The behavior of the RNN-based models is not normal. I tested it on AirPassengersDF, and please look at the results.

MSE on Prediction Length 1-4

Horizon	NBEATS	PatchTST	LSTM‑direct	LSTM‑recurr
h1	0.025	0.020	0.459	0.720
h2	0.031	0.024	0.396	0.727
h3	0.034	0.028	0.360	0.739
h4	0.039	0.024	0.325	0.758

We expected to see a pattern of rising error when the horizon lengthens.

[elephaint]

• You should always set a scaler when training an RNN. Try with robust.
• Train more steps, for recurrent easily 2000.

[alireza-jafari]

Thanks for the pointers!
I actually did scale the data before fitting, using sklearn.preprocessing.StandardScaler. To avoid double-scaling I kept scaler_type=None inside the model.
I pushed max_steps to 2000 as suggested. Convergence looks smoother, but the horizon-specific errors still don't make sense.
I implemented a very simple LSTM in TensorFlow or PyTorch (about 20 lines of code) and compared its results with neuralforecast’s LSTM. The behavior of the neuralforecast’s RNN-based models still seems off. However, other models work perfectly well.