Use Botorch MultiTaskGP for transfer learning by AVHopp · Pull Request #549 · emdgroup/baybe

AVHopp · 2025-05-07T07:39:26Z

Replaces the custom IndexKernel construction with BoTorch's MultiTaskGP (which became possible due the added all_tasks argument).

AdrianSosic

Hi @Hrovatin, here the first batch of comments

Scienfitz · 2025-08-15T10:26:31Z

@Hrovatin would you consider abandoning this PR? I think if this topic is picked up again its better to start afresh (and only open a PR after investigations have concluded).

Hrovatin · 2025-08-23T16:51:50Z

@Scienfitz I would keep open as the main blocker for this was randomness in benchmarks. Since that may be solved now I would suggest running benchmarks again on the new HPC (need to confirm it is also reproducible there)

Scienfitz · 2025-09-09T07:50:50Z

@Hrovatin any update?

Hrovatin · 2025-09-09T08:16:53Z

No, I need to first set up testing on oneHPC to reproducibly benchmark - as that seems to be the only option to make fully reproducible. I will post update here once I have the results @Scienfitz

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Hrovatin · 2025-09-17T10:26:47Z

@AdrianSosic @Scienfitz @AVHopp Update on the comparison of MultiTask GP from botorch and current kernel:

The results are not identical, but very close, except for michaelewicz (but it seems that variation is likely not significant here as well)
A concern: When using botorch multitask gp the hartman tl benchmark always fails due to ooo (when using at 0.05 but not 0.01 source data). I have not yet figured out why. Before investigating this we should probably make a call if we are ok with accepting some deviation from current main (named benchmarks-reproducibility-beforeBug on the plot) or not as if we decide we need 100% reproducibility anyways it also does not make sense to investigate any other issues further.

AVHopp

First round of comments, but we should discuss some of the points (in particular the one regarding multiple active values) internally first.

AVHopp

Would be willing to approve - however, since this is technically my PR I can't

Hrovatin · 2025-10-02T07:21:29Z

Results after rebase:
Note:

Hartmann tl did not run for the new branch due to issues in local setup (shows only the main branch). But I tested that it runs successfully in actions
Reproducibility is in general not 100% (also when not using the tl code that was changed)

…happen

Co-authored-by: Alexander V. Hopp <alexander.hopp@merckgroup.com>

The active_dims argument can now be dropped due to #671

meta-pytorch/botorch#3085

Unfortunately, previous botorch version have an (unnecessary?) hard pin for gpytorch on version 1.14, causing troubles with other tests due to the following issue, which has only be fixed in 1.14.1: cornellius-gp/gpytorch#2633

Does not solve the problem since there is still a failing example

AdrianSosic · 2025-11-28T09:10:40Z

@copilot: Explain the reason for the CI failure

Copilot · 2025-11-28T09:10:53Z

@AdrianSosic I've opened a new pull request, #703, to work on those changes. Once the pull request is ready, I'll request review from you.

AdrianSosic · 2026-02-10T14:47:06Z

Closed in favor of #743. In particular, we don't switch to MultiTaskGP because we do not want to adopt BoTorch's latest decisions on how to handle incoming task data (see validate_task_values argument). Staying with SingleTaskGP keeps full flexibility (also for implementing other transfer mechanisms that do not involve IndexKernel) and allows us to keep our GP class as a slim / general purpose skeleton that implements the core logic of a GP.

AVHopp requested review from AdrianSosic and Scienfitz as code owners May 7, 2025 07:39

AVHopp assigned AVHopp and Hrovatin May 7, 2025

AVHopp marked this pull request as draft May 7, 2025 07:40

AVHopp mentioned this pull request May 7, 2025

Use Botorch MultiTaskGP for transfer learning #484

Closed

3 tasks

AVHopp changed the title ~~Tl benchmarking investigation~~ Use Botorch MultiTaskGP for transfer learning May 7, 2025

Hrovatin force-pushed the tl_benchmarking_investigation branch 2 times, most recently from 8fee382 to 88e1dfe Compare June 4, 2025 11:18

Hrovatin marked this pull request as ready for review June 5, 2025 10:39

AdrianSosic reviewed Jun 6, 2025

View reviewed changes

Hrovatin requested a review from AdrianSosic June 6, 2025 14:40

Copilot AI review requested due to automatic review settings September 12, 2025 11:22

Hrovatin force-pushed the tl_benchmarking_investigation branch from 8ce5fba to bee32aa Compare September 12, 2025 11:22

Copilot AI reviewed Sep 12, 2025

AVHopp commented Sep 22, 2025

View reviewed changes

Comment thread baybe/parameters/categorical.py

Comment thread baybe/surrogates/gaussian_process/core.py Outdated

Hrovatin force-pushed the tl_benchmarking_investigation branch from de81707 to 68a9c24 Compare September 25, 2025 07:13

AVHopp commented Sep 30, 2025

View reviewed changes

Hrovatin reviewed Oct 2, 2025

View reviewed changes

Comment thread .github/workflows/benchmark.yml Outdated

AdrianSosic force-pushed the tl_benchmarking_investigation branch 4 times, most recently from 5cfb366 to 7bb49d9 Compare October 6, 2025 08:50

AdrianSosic approved these changes Oct 6, 2025

View reviewed changes

Hrovatin and others added 20 commits November 27, 2025 09:22

Remove constraint to use single active task parameter value

80acc3f

Update tests and assert that multiple active values are recommended

706a984

Remove mypy errors

26da89a

Remove check that both tasks were recommended as this may not always …

e450a71

…happen

Update baybe/surrogates/gaussian_process/core.py

70bd19f

Co-authored-by: Alexander V. Hopp <alexander.hopp@merckgroup.com>

Update tests/test_transfer_learning.py

366fd49

Co-authored-by: Alexander V. Hopp <alexander.hopp@merckgroup.com>

Remove unnecessary comments

9affa1a

Clarify tests

bcc2a19

Reuse parent method for integer casting

0208e31

Add temporary _task_parameter property to SearchSpace class

7c842cd

Refactor GP fitting method

b8c6de5

Refactor transfer learning tests using parametrization/fixtures

2ec60a8

Update CHANGELOG.md

e8942f7

Use parametrization instead of request

e416675

Directly specify active_dims in kernel

0171ce3

Drop unnecessary arguments

4ec6cac

The active_dims argument can now be dropped due to #671

Pin botorch

036c261

meta-pytorch/botorch#3085

xfail tests on latest release instead of pinning version

0f6b063

Unfortunately, previous botorch version have an (unnecessary?) hard pin for gpytorch on version 1.14, causing troubles with other tests due to the following issue, which has only be fixed in 1.14.1: cornellius-gp/gpytorch#2633

Revert xfail

8f25dd7

Does not solve the problem since there is still a failing example

Set covariance matrix rank according to square root of number of tasks

ace40f7

AdrianSosic force-pushed the tl_benchmarking_investigation branch from 9f060b9 to ace40f7 Compare November 27, 2025 08:23

AdrianSosic added 2 commits November 27, 2025 15:23

Switch to new botorch logic until better solution is available

7a27863

Fix type in botorch translation

4f46223

Copilot AI mentioned this pull request Nov 28, 2025

Mark transfer learning tests as xfail due to BoTorch MultiTaskGP limitation #703

Closed

AdrianSosic force-pushed the tl_benchmarking_investigation branch from 4881367 to 4f46223 Compare November 28, 2025 10:24

AdrianSosic closed this Feb 10, 2026

AVHopp deleted the tl_benchmarking_investigation branch April 23, 2026 08:09

Uh oh!

Conversation

AVHopp commented May 7, 2025 • edited by AdrianSosic Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AdrianSosic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Scienfitz commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hrovatin commented Aug 23, 2025

Uh oh!

Scienfitz commented Sep 9, 2025

Uh oh!

Hrovatin commented Sep 9, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Hrovatin commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AVHopp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AVHopp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Hrovatin commented Oct 2, 2025

Uh oh!

Uh oh!

AdrianSosic commented Nov 28, 2025

Uh oh!

Copilot AI commented Nov 28, 2025

Uh oh!

AdrianSosic commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AVHopp commented May 7, 2025 •

edited by AdrianSosic

Loading

Scienfitz commented Aug 15, 2025 •

edited

Loading

Hrovatin commented Sep 17, 2025 •

edited

Loading