arima-autotune

A suite of derivative-free optimization methods for Autotuning ARIMA models using solveBlackbox action.

Table of Contents

1. Overview
2. Packages and permissions
3. Parameter configuration
4. Experiments
5. Contributing
6. License
7. Security
8. Additional Resources

Overview

This project implements an automated framework for ARIMA model tuning using solveBlackbox action, with code written in Python and CASL, and provides the framework for the patent US12380369B1 as well a paper currently under review. The framework supports:

• Seasonal and nonseasonal ARIMA order selection.
• Subset model and full-model ARIMA specifications.
• Box-Cox transformations and differencing.
• Inclusion of an intercept term
• Optimization of fit statistics (e.g., AICc, RMSE) using the solveBlackbox action.
• Automated ARIMA model order selection using pmdarima and SAS Diagnose as benchmarks.
• Parallelized execution for efficient computation.

The framework is modular, allowing for easy configuration and extension. Unlike any existing method in literature, these methods can identify full as well as subset ARIMA models, and can identify other hyperparameters, such as the Box-Cox transformation parameter, and the choice of inclusion of an intercept term simultaneously. Multi-objective optimization methods with novel objective functions are used to regularize the validity and complexity of an ARIMA model.

Packages and permissions

Clone this repository for all latest updates. The following packages are required to run the notebooks of this repository:

• swat
• PyYAML
• numpy
• pandas
• scipy
• pmdarima
• matplotlib
• scikit_posthocs

Installation of all the above packages can be done via pip, using the following sample commands:

• pip install swat
• pip install scikit-posthocs
• pip install PyYAML

Running all the notebooks (apart from Analysis and visualization) requires a SAS Viya license with Visual Forecasting. You will have to enter a hostname, port, username and password to run these notebooks. For the DFO Experiments notebook specifically, the SAS Optimization actionset is required as well.

Parameter configuration

The following table provides details on the parameters that populate the file config.yaml, for daily_data.sashdat and weekly_data.sashdat. Notice that daily_data.sashdat is a subset of the M5 competition dataset filtered according to the by-variable values in M5subset.csv of the data folder, while weekly_data.sashdat is its accumulated weekly version.

Parameter	Values (Daily)	Values (Weekly)	Notes
inlib	CASUSERHDFS / CASUSER	CASUSERHDFS / CASUSER	The name of CASLIB to upload the dataset (CASUSERHDFS for DFO experiments, CASUSER for Auto-ARIMA).
datasets_path	"<filesDirectories>"	"<filesDirectories>"	Path to the dataset.
tabname	daily_data.sashdat	weekly_data.sashdat	Name of the dataset file.
outTablePrefix	M5_	M5_	Prefix for output tables.
byvarparms.enabled	true	true	Whether by-variable processing is enabled (always true).
byvarparms.name	id	id	Name of by-variable (multiple by variables not supported).
byvarparms.values	[]	[]	Indices of values of the byvariable to be considered (If list is not empty, then start_from will be ignored).
byvarparms.start_from	0	0	Number of values of by-variable to ignore from the start (ignored if values parameter is non-empty).
idVar	numeric_date	numeric_date	Identifier variable (e.g., date column).
interval	day	week	Time interval (e.g., week, month).
seasonality	7	52	Seasonal period (e.g., 52 for weekly data).
depVar	sales	sales	Dependent variable (e.g., value column).
lead	56	26	Length of forecast region.
back	56	26	Length of out-of-sample region.
holdout	56	26	Length of holdout region.
filter_flag	false	false	true if we want filtering of pareto front; false otherwise
maxRows	2000000	2000000	Maximum number of rows to fetch using table.Fetch (Make sure to encompass entire data set).
verbose	false	false	true if need to debug in log; false otherwise
maxDur_AA	45	45	Maximum number of seconds for auto-ARIMA to run for any combination of byvar, method, boxcox and subset.
sbbparms.nParallel	10	10	Number of parallel processes for the SBB algorithm.
sbbparms.maxGen	25	25	Maximum number of generations for the SBB algorithm.
sbbparms.popsize	150	150	Population size for the SBB algorithm.
sbbparms.paretoMax	25	25	Maximum number of Pareto-optimal solutions to retain.
sbbparms.seed	5	5	Random seed for reproducibility.
sbbparms.boxcoxlb	-5	-5	Lower bound for Box-Cox transformation parameter to search for.
sbbparms.boxcoxub	5	5	Upper bound for Box-Cox transformation parameter to search for.
sbbparms.ublist	[9, 3, 9, 1, 1, 1]	[9, 3, 9, 1, 1, 1]	Upper bounds for [p,d,q,P,D,Q] to search for.
sbbparms.tolerances	[2, 2, 0.05]	[2, 2, 0.05]	Tolerances for calculating MRV: [Tolerance for abs(acf) to exceed, to increase acfflag count, Tolerance for abs(pacf) to exceed, to increase pacfflag count, p-value for comparing significance of white noise probability]
sbbparms.insamp_fit_stat	AICc	AICc	Fit statistic for the in-sample region.
sbbparms.holdout_fit_stat	RMSE	RMSE	Fit statistic for the holdout region.
sbbparms.back_fit_stat	RMSE	RMSE	Fit statistic for the out-of-sample region.
sbbparms.iqr_mult	1.5	1.5	Multiple of Interquartile range outside of which, to drop points in Pareto front during filtering.
sbbparms.max_amv	0.1	0.1	Maximum value of MRV beyond which to drop points in Pareto front during filtering.

For running our methods on individual datasets like Airline, Electricity T151, NN5 T10, or Pedestrian T27, please refer to the parameter configurations in README.

Experiments

Here we provide general instructions on how to replicate our experiment results.

• For experiments on M5 daily or weekly:

1. Filter the M5 dataset as per the values of the by-variable id to obtain daily_data, and accumulate it over weeks to obtain weekly_data.
2. Modify the config file according to the parameter configurations mentioned in this page.
3. Run the DFO_experiments notebook. Output tables will be saved in results directory.
4. Run the Auto-ARIMA_experiments notebook (if you wish to compare with auto-ARIMA). Output tables will be saved in results directory.
5. Run the two files pertaining to SAS Diagnose, i.e. SASDiagnose_Holdout_RMSE and SASDiagnose_InSample_AICC with properly defined macros. Please note that unlike the ipynb notebooks, the results of the sas files will not be saved in the results directory by default. The sas_diag_tables folder contains the output tables obtained by running these files, for your reference.
6. Run the first few sections of Analysis_and_visualization to compare out-of-sample fit statistics and runtimes of all the tuning methods for the case (Box-cox = False, Subset = False). The resulting figures will be saved in images directory.
7. Run the last section of Analysis_and_visualization to compare out-of-sample fit statistics and runtimes of the variants of our DFO methods only, including cases where Box-cox transformation parameter and subset models are being selected. The resulting figures will again be saved in images directory.

• For experiments on individual datasets like Airline, Electricity T151, NN5 T10, or Pedestrian T27, modify the config file according to the parameter configurations in README, and follow the rest of the above points.

• The README file contains more details on the resulting tables and files in the results directory.

Contributing

We welcome contributions!

Please read CONTRIBUTING.md for details on how to submit contributions to this project.

License

See the LICENSE file for details.

Security

See the SECURITY file for details.

Additional Resources

• solveBlackbox Action Documentation
• SAS Developer Community