From Uniform to Learned Knots

Source code for the research paper:
"From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning" (under review)

This repository provides a full experiment framework for evaluating spline-based numerical encodings (B-splines, M-splines, I-splines, PLE) against standard baselines across regression and classification tasks. All experiments use neural network backbones (MLP, ResNet, FT-Transformer) and support CPU, Apple Metal (MPS), and CUDA GPUs.

---

Setup

Requirements

• Python 3.13+
• uv (recommended) or pip

Install

git clone <repo-url>
cd <repo-directory>
uv sync
source .venv/bin/activate

If uv sync fails, use the manual approach:

uv venv --python=3.13
source .venv/bin/activate
uv sync

Verify GPU detection

python -c "from src.device_utils import print_device_info; print_device_info()"

---

Models

Key	Architecture	Notes
mlp	Multi-Layer Perceptron	Feed-forward network with BatchNorm and dropout
resnet	ResNet	Residual blocks; better gradient flow on deep transforms
ftt	FT-Transformer	Feature Tokenizer + Transformer (attention-based)
mlp_learnable	MLP + Learnable Knots	Knot positions are gradient-optimised during training
resnet_learnable	ResNet + Learnable Knots	Same as above on ResNet backbone
ftt_learnable	FT-Transformer + Learnable Knots	Same as above on FT-Transformer backbone

All models support early stopping, configurable architectures, and automatic GPU selection.

---

Preprocessing Methods

Baselines

Key	Description
standard	StandardScaler — zero mean, unit variance
minmax	MinMaxScaler — range [0, 1]

Spline-Based Encodings

Each spline family comes in four knot-placement variants:

Variant suffix	Knot strategy
_uniform	Uniformly spaced knots
_quantile	Quantile-based knots
_cart	CART tree-adaptive knots
_lightgbm	LightGBM tree-adaptive knots

Family	Keys	Properties
B-spline	bspline_{uniform,quantile,cart,lightgbm}	General-purpose, flexible basis
M-spline	mspline_{uniform,quantile,cart,lightgbm}	Non-negative basis (integrates to give I-splines)
I-spline	ispline_{uniform,quantile,cart,lightgbm}	Monotone-increasing basis
PLE	ple	Piecewise Linear Encoding

Learnable Knots

mlp_learnable / resnet_learnable / ftt_learnable use preprocessing_method: none — the spline basis is built inside the model and knot positions are jointly optimised with network weights using a separate knot learning rate (lr_knots) and a spacing regulariser (lam_spacing).

---

Running Experiments

All experiments are run through a single entry point with CLI flags:

python -m src.main [--task {regression,classification}] [--learnable] [--config <yaml>] [--test]

Flag	Default	Description
--task	regression	Task type
--learnable	off	Use learnable-knots models
--config	—	Path to YAML config file
--test	off	Quick run on a single dataset

Standard experiments (CPU / MPS / single GPU)

# Regression — uses GPU automatically (CUDA or MPS) if available
python -m src.main --task regression

# Classification
python -m src.main --task classification

# Use a YAML config (recommended for reproducible runs)
python -m src.main --task regression --config config/regression.yaml
python -m src.main --task classification --config config/classification.yaml

# Quick smoke-test (single dataset)
python -m src.main --task regression --test

Parallel experiments on two CUDA GPUs

Regression and classification are independent — run them simultaneously, one model per GPU:

# Terminal 1 — GPU 0: regression
CUDA_VISIBLE_DEVICES=0 python -m src.main --task regression --config config/regression.yaml

# Terminal 2 — GPU 1: classification
CUDA_VISIBLE_DEVICES=1 python -m src.main --task classification --config config/classification.yaml

Or launch both inside a tmux session using the provided script:

bash src/start_experiments_standard.sh

Learnable knots experiments

# Learnable regression
python -m src.main --task regression --learnable

# Learnable classification
python -m src.main --task classification --learnable

# From YAML config
python -m src.main --task regression --learnable --config config/regression.yaml

# Parallel on two GPUs
CUDA_VISIBLE_DEVICES=0 python -m src.main --task regression --learnable
CUDA_VISIBLE_DEVICES=1 python -m src.main --task classification --learnable

Or via the tmux script:

bash src/start_experiments_learnable.sh

---

Configuration

Experiment parameters are controlled via YAML files in config/. Edit the file to change datasets, models, preprocessing methods, or hyperparameters:

File	Purpose
config/regression.yaml	Full regression benchmark
config/classification.yaml	Full classification benchmark

Tip: The models: list in each YAML can contain multiple entries. When running a single-GPU experiment, comment out all but one:

models:
  - mlp
  # - resnet
  # - ftt

Standard vs learnable models: Never place standard (mlp, resnet, ftt) and learnable (mlp_learnable, resnet_learnable, ftt_learnable) keys in the same models: list. When passing --learnable, swap the list to learnable variants. preprocessing_methods is ignored during learnable runs — the active spline family is set by learnable.spline_type, one family per run. See config/README.md for full details.

---

Results

Results are saved under results/{task}/{dataset}/:

File	Contents
CONSOLIDATED_detailed_results_*.csv	Per-fold / per-seed raw metrics
CONSOLIDATED_summary_results_*.csv	Mean ± std across folds and seeds
CONSOLIDATED_config_*.json	Full experiment configuration snapshot

---

Simulated Data & Ablation Experiments

The paper includes a set of controlled experiments on a small synthetic regression dataset (datasets/synthetic/simulated_data_regression.csv) to study the effect of basis resolution and knot placement strategy in isolation. These are driven by three standalone scripts rather than YAML configs. The device is auto-detected (MPS → CUDA → CPU) unless overridden with --device.

Baseline comparison

Runs standard, minmax, and PLE with adaptive mode (up to 50 bins) across multiple seeds:

python src/run_simulated_baseline.py

Resolution sweep

Runs all spline methods (B-spline, M-spline, I-spline — all four knot variants each) plus PLE and all three learnable families across basis/bin counts from 5 to 50 in steps of 5, with 5 seeds per configuration:

python src/run_simulated_resolution_sweep.py

Knot position illustration

Short single-seed run used to generate knot relocation visualisations. Pass --model-type learnable to enable per-epoch knot tracking:

python -m src.run_knot_illustration --model-type learnable
python -m src.run_knot_illustration --model-type learnable --lr-knots 2e-4 --knot-init uniform

Running baseline and sweep together

A convenience script launches both in parallel (tmux) or sequentially:

bash src/start_simulated_sweep.sh

---

Datasets

Regression (13): abalone, ca_housing, cpu_small, diamonds, fifa_wage, house8L, house_sales, parkinsons, protein, pulsar, sgemm_gpu, sulphur, wine_quality_reg

Classification (12): adult, air_quality, bank, churn, eeg_eye, fico, gamma_telescope, ipums, loan_status, loan_type, marketing, shuttle

All datasets are pre-processed and stored in datasets/regression/ and datasets/classification/ as CSV files with a Targets column.