Approximate GP implementation

[roussel-ryan]

This pull request introduces support for approximate Gaussian Process (GP) model building to the codebase, enabling scalable modeling for larger datasets. The main changes include the implementation of an ApproximateModelConstructor using variational GP models, updates to benchmarking and documentation, and refactoring of model training logic to support both exact and variational approaches.

Approximate GP Modeling Support:

• Added ApproximateModelConstructor class in xopt/generators/bayesian/models/approximate.py to construct scalable variational GP models for each outcome, leveraging SingleTaskVariationalGP and variational ELBO training.
• Introduced build_variational_gp utility method in base_model.py for creating and training variational GP models.
• Refactored model training logic in standard.py to allow flexible training of individual models with custom marginal log likelihoods, supporting both exact and variational approaches. [1] [2] [3]

Documentation and Examples:

• Added a new notebook example, approximate.ipynb, demonstrating benchmarking and usage of approximate GP models versus standard models, including scalability and performance comparisons.
• Updated documentation (index.md and mkdocs.yml) to reference approximate GP model building and the new example notebook. [1] [2]

Testing:

• Added tests to verify that ApproximateModelConstructor builds models using SingleTaskVariationalGP, and updated tests for model training and error handling. [1] [2] [3]

Underlying API Enhancements:

• Updated imports and type signatures to support variational GP models throughout the base model code. [1] [2] [3]

[codecov]

Codecov Report

❌ Patch coverage is 96.93878% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
xopt/generators/bayesian/models/standard.py	93.93%	0 Missing and 2 partials ⚠️
xopt/generators/bayesian/models/approximate.py	97.87%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[nikitakuklev]

General question - does caching hyperparameters work for approximate GPs? Seems ok since only kernel parameters should be saved, and they are valid even if inducing point is removed, but good to check.

Also a request - add parameter for inducing point count and allocator (the two available in botorch)

[roussel-ryan]

Looking at caching hyperparameters right now, will add the options you mentioned

[roussel-ryan]

Looks like there is a botorch bug with loading in a state dict for ApproximateGP models, will submit a PR for botorch when I get a chance tomorrow, for now I will disable use_cached_hyperparameters

[roussel-ryan]

Added a botorch issue meta-pytorch/botorch#3250

[roussel-ryan]

@nikitakuklev I've added the requested options, lmk what you think

[nikitakuklev]

See comments. Also, need to add to validate_gp_constructor and all

[roussel-ryan]

@nikitakuklev thanks for the comments, should be addressed now, lmk if there are more changes you'd like

[nikitakuklev]

Yvar is not supported (this is masked test issue only)

[roussel-ryan]

thanks for catching that