Add Hyperparameter Optimization (HPO) examples using Ray Tune and HyperNOs by MaxGhi8 · Pull Request #2070 · lululxvi/deepxde

MaxGhi8 · 2026-03-22T09:14:37Z

This PR introduces a comprehensive set of examples for hyperparameter optimization (HPO) in DeepXDE, leveraging Ray Tune and the HyperNOs (https://github.com/lu-group/HyperNOs) framework.

The examples cover the core functionalities of the library:

Operator Learning: 1D Poisson, 1D Advection, and 2D Advection (mapped from 1D time-dependent) using PI-DeepONet.
PINN Forward Problems: 1D Diffusion architecture optimization.
PINN Inverse Problems: Architecture optimization for parameter identification in 1D Diffusion.

These examples provide a unified, "supervised operator style" template for HPO that is easily extensible to other physics-informed learning tasks.

Copilot

Pull request overview

Adds new hyperparameter-optimization (HPO) example scripts intended to demonstrate integrating DeepXDE problems with HyperNOs + Ray Tune, and updates the top-level README to reference these capabilities.

Changes:

Update README.md formatting and add HPO mentions/links.
Add Ray Tune + HyperNOs HPO example scripts for operator learning (Poisson, Advection 1D/2D).
Add Ray Tune + HyperNOs HPO example scripts for PINN forward/inverse diffusion.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
`README.md`	Reformats algorithm list; adds HPO feature bullet and a new “Demos” link.
`examples/operator/poisson_1d_hpo.py`	New HPO example for Poisson operator learning via DeepONetCartesianProd.
`examples/operator/advection_hpo.py`	New HPO example for 1D advection PI-DeepONet-style operator learning.
`examples/operator/advection_2d_hpo.py`	New HPO example for 2D (mapped) advection PI-DeepONet-style operator learning.
`examples/pinn_forward/diffusion_1d_hpo.py`	New HPO example script for 1D diffusion forward problem.
`examples/pinn_inverse/diffusion_1d_inverse_hpo.py`	New HPO example script for 1D diffusion inverse problem.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-22T09:18:26Z

 - [Demos of forward problems](https://deepxde.readthedocs.io/en/latest/demos/pinn_forward.html)
 - [Demos of inverse problems](https://deepxde.readthedocs.io/en/latest/demos/pinn_inverse.html)
 - [Demos of operator learning](https://deepxde.readthedocs.io/en/latest/demos/operator.html)
+- [Demos of hyperparameter optimization](examples/README.md)


The new README link points to examples/README.md, but there is no examples/README.md file in the repository. This will render as a broken link on GitHub; either add that file or link to an existing docs page/path that contains the HPO examples.

Suggested change

- [Demos of hyperparameter optimization](examples/README.md)

- [Demos of hyperparameter optimization](https://deepxde.readthedocs.io/en/latest/demos/hpo.html)

Copilot · 2026-03-22T09:18:26Z

 - 4 **function spaces**: power series, Chebyshev polynomial, Gaussian random field (1D/2D).
 - **data-parallel training** on multiple GPUs.
 - different **optimizers**: Adam, L-BFGS, etc.
+- **hyperparameter optimization** using [HyperNOs](https://github.com/MaxGhi8/HyperNOs) and [Ray Tune](https://docs.ray.io/en/latest/tune/index.html).


The README claims DeepXDE supports hyperparameter optimization using HyperNOs + Ray Tune, but these are not project dependencies (and the PR description links a different HyperNOs org). Consider rephrasing this as an example integration and mentioning that hypernos/ray[tune] are optional extra installs, and align the HyperNOs URL with the one intended for this PR.

Suggested change

- **hyperparameter optimization** using [HyperNOs](https://github.com/MaxGhi8/HyperNOs) and [Ray Tune](https://docs.ray.io/en/latest/tune/index.html).

- example **hyperparameter optimization** integrations (via optional extra packages) using [HyperNOs](https://pypi.org/project/hypernos/) and [Ray Tune](https://docs.ray.io/en/latest/tune/index.html) (install `hypernos` and `ray[tune]` separately).

Copilot · 2026-03-22T09:18:27Z

+def solve_advection_1d():
+    # PDE: u_t + u_x = 0
+    def pde_fn(x, y):
+        dy_x = dde.grad.jacobian(y, x, j=0)
+        dy_t = dde.grad.jacobian(y, x, j=1)
+        return dy_t + dy_x
+


PDEOperatorCartesianProd operator-learning setups pass the sampled branch function values into the PDE callback (see the referenced advection_aligned_pideeponet.py, which defines pde(x, y, v)). Here pde_fn is defined as (x, y) only, which will raise a TypeError when DeepXDE calls it with the extra v argument. Update the signature to accept v (even if unused).

Copilot · 2026-03-22T09:18:27Z

+def solve_advection_2d():
+    # PDE: u_y + u_x = 0 (where y is time)
+    def pde_fn(x, y):
+        dy_x = dde.grad.jacobian(y, x, j=0)
+        dy_y = dde.grad.jacobian(y, x, j=1)
+        return dy_y + dy_x
+


Same issue as the 1D version: the referenced base example defines the PDE as pde(x, y, v) for operator learning. Here pde_fn is (x, y) only, so DeepXDE is likely to call it with an extra branch-function argument and fail at runtime. Adjust pde_fn to accept the additional v parameter.

Copilot · 2026-03-22T09:18:27Z

+    # For HPO with HyperNOs, we use a dummy branch of ones
+    # and dummy targets of ones to avoid division by zero in relative loss.
+    # The target shape must match (batch, num_points)
+    X_branch_train = np.ones((1, 10))  # 1 sample, 10 eval points
+    X_branch_test = np.ones((1, 10))
+
+    y_train = np.ones((1, num_train))
+    y_test = np.ones((1, num_test))
+


The current HPO objective is trained against an all-ones target (y_train/y_test). That makes the Ray Tune search largely uninformative (most configs can fit a constant field), and it no longer reflects a diffusion PINN example as described. Consider generating targets from the known analytic solution in examples/pinn_forward/diffusion_1d.py (or using a physics-residual-based objective) so the validation loss actually measures PDE solution quality.

Copilot · 2026-03-22T09:18:28Z

+    # Observation points as trunk inputs
+    X_trunk_train = np.vstack((np.linspace(-1, 1, num=50), np.full((50), 1))).T
+    X_trunk_test = X_trunk_train  # Simplified for HPO
+
+    num_pts = X_trunk_train.shape[0]
+
+    # Dummy branch and non-zero targets to avoid division by zero
+    X_branch_train = np.ones((1, 10))
+    X_branch_test = np.ones((1, 10))
+
+    y_train = np.ones((1, num_pts))
+    y_test = np.ones((1, num_pts))
+
+    class PDEData:
+        def __init__(self, X_branch, X_trunk, y, batch_size):
+            self.train_loader = DataLoader(
+                PINNInverseCartesianDataset(X_branch, X_trunk, y),
+                batch_size=batch_size,
+                shuffle=True,
+                collate_fn=deeponet_collate_fn,
+            )
+            self.val_loader = DataLoader(
+                PINNInverseCartesianDataset(X_branch, X_trunk, y),
+                batch_size=batch_size,
+                shuffle=False,
+                collate_fn=deeponet_collate_fn,
+            )
+            self.test_loader = self.val_loader
+
+    return PDEData(X_branch_train, X_trunk_train, y_train, config.get("batch_size", 1))


X_trunk_test, X_branch_test, and y_test are defined but unused, and val_loader is constructed from the same data as train_loader. If the intention is to do architecture selection, consider providing a distinct validation loader (even if small) to prevent tuning on training loss only.

Suggested change

# Observation points as trunk inputs

X_trunk_train = np.vstack((np.linspace(-1, 1, num=50), np.full((50), 1))).T

X_trunk_test = X_trunk_train # Simplified for HPO

num_pts = X_trunk_train.shape[0]

# Dummy branch and non-zero targets to avoid division by zero

X_branch_train = np.ones((1, 10))

X_branch_test = np.ones((1, 10))

y_train = np.ones((1, num_pts))

y_test = np.ones((1, num_pts))

class PDEData:

def __init__(self, X_branch, X_trunk, y, batch_size):

self.train_loader = DataLoader(

PINNInverseCartesianDataset(X_branch, X_trunk, y),

batch_size=batch_size,

shuffle=True,

collate_fn=deeponet_collate_fn,

)

self.val_loader = DataLoader(

PINNInverseCartesianDataset(X_branch, X_trunk, y),

batch_size=batch_size,

shuffle=False,

collate_fn=deeponet_collate_fn,

)

self.test_loader = self.val_loader

return PDEData(X_branch_train, X_trunk_train, y_train, config.get("batch_size", 1))

# Observation points as trunk inputs (train)

X_trunk_train = np.vstack((np.linspace(-1, 1, num=50), np.full((50), 1))).T

# Separate observation points for validation/test to avoid tuning on training loss only

X_trunk_test = np.vstack((np.linspace(-1, 1, num=20), np.full((20), 1))).T

num_pts_train = X_trunk_train.shape[0]

num_pts_test = X_trunk_test.shape[0]

# Dummy branch and non-zero targets to avoid division by zero

X_branch_train = np.ones((1, 10))

X_branch_test = np.ones((1, 10))

y_train = np.ones((1, num_pts_train))

y_test = np.ones((1, num_pts_test))

class PDEData:

def __init__(

self,

X_branch_train,

X_trunk_train,

y_train,

X_branch_val,

X_trunk_val,

y_val,

batch_size,

):

self.train_loader = DataLoader(

PINNInverseCartesianDataset(X_branch_train, X_trunk_train, y_train),

batch_size=batch_size,

shuffle=True,

collate_fn=deeponet_collate_fn,

)

self.val_loader = DataLoader(

PINNInverseCartesianDataset(X_branch_val, X_trunk_val, y_val),

batch_size=batch_size,

shuffle=False,

collate_fn=deeponet_collate_fn,

)

self.test_loader = DataLoader(

PINNInverseCartesianDataset(X_branch_val, X_trunk_val, y_val),

batch_size=batch_size,

shuffle=False,

collate_fn=deeponet_collate_fn,

)

return PDEData(

X_branch_train,

X_trunk_train,

y_train,

X_branch_test,

X_trunk_test,

y_test,

config.get("batch_size", 1),

)

Copilot · 2026-03-22T09:18:28Z

+    # Generate training and test data from DeepXDE operator
+    X_train, y_train, aux_train = pde_op.train_next_batch(config["training_samples"])
+    X_test, y_test, aux_test = pde_op.test()
+
+    # In PDEOperatorCartesianProd, y_train is often None (physics-informed).
+    # For this HPO demo, we'll use a dummy target if y is None,
+    # or you could solve the PDE to get ground truth.
+    if y_train is None:
+        y_train = np.ones((X_train[0].shape[0], X_train[1].shape[0], 1))
+    if y_test is None:
+        y_test = np.ones((X_test[0].shape[0], X_test[1].shape[0], 1))
+


For operator-learning via PDEOperatorCartesianProd, y_train is None because DeepXDE expects to train via the physics-informed residual. Replacing it with a constant dummy target means the HPO run is no longer optimizing for the Poisson operator (it’s just fitting a constant). Either compute true solution targets for the sampled forcings (or implement a residual-based loss inside the training loop) or make it explicit in the script/docs that this is only an integration template and not a meaningful Poisson HPO benchmark.

Copilot · 2026-03-22T09:18:28Z

+        [dim_trunk] + [config["network_width"]] * config["trunk_depth"] + [p]
+    )
+
+    model = dde.nn.DeepONetCartesianProd(
+        layer_sizes_branch,
+        layer_sizes_trunk,
+        "tanh",
+        "Glorot normal",
+    )
+
+    # Optional: Feature transform for periodicity as in the base example
+    def periodic(x):
+        xt, tt = x[:, :1], x[:, 1:]
+        xt = xt * 2 * np.pi
+        return torch.cat(
+            [torch.cos(xt), torch.sin(xt), torch.cos(2 * xt), torch.sin(2 * xt), tt], 1
+        )
+
+    # Note: Applying feature transform might change trunk input dim to 5
+    # If we apply it, we need to adjust layer_sizes_trunk[0]
+    layer_sizes_trunk_transformed = (
+        [5] + [config["network_width"]] * config["trunk_depth"] + [p]
+    )
+    model = dde.nn.DeepONetCartesianProd(
+        layer_sizes_branch,
+        layer_sizes_trunk_transformed,
+        "tanh",
+        "Glorot normal",
+    )
+    model.apply_feature_transform(periodic)
+


model_builder instantiates a DeepONetCartesianProd twice; the first model is immediately overwritten when applying the feature transform. This extra instantiation is unnecessary and can confuse readers. Consider constructing the transformed trunk layer sizes up-front and creating the model only once before calling apply_feature_transform.

Copilot · 2026-03-22T09:18:28Z

+    # In PI-DeepONet, we often use dummy targets if we are pure physics-informed,
+    # or initial condition values. In this HPO demo, we use ones to avoid division by zero.
+    if y_train is None:
+        y_train = np.ones((X_train[0].shape[0], X_train[1].shape[0], 1))
+    if y_test is None:
+        y_test = np.ones((X_test[0].shape[0], X_test[1].shape[0], 1))
+
+    if y_train.shape[-1] == 1:
+        y_train = y_train.squeeze(-1)
+    if y_test.shape[-1] == 1:
+        y_test = y_test.squeeze(-1)


When y_train/y_test are None (physics-informed operator setup), this example replaces them with constant ones. That means the HPO run is not optimizing the PI-DeepONet physics objective at all (it becomes a trivial supervised constant-fitting task). Either (1) implement a residual-based loss using the PDE definition, or (2) compute non-trivial targets from an analytic/numerical solution, or (3) clearly label this as a Ray/HyperNOs integration template rather than an advection PI-DeepONet HPO example.

Copilot · 2026-03-22T09:18:28Z

+    # Handle potentially missing targets in pure physics-informed setup.
+    # In this HPO demo, we use ones to avoid division by zero.
+    if y_train is None:
+        y_train = np.ones((X_train[0].shape[0], X_train[1].shape[0], 1))
+    if y_test is None:
+        y_test = np.ones((X_test[0].shape[0], X_test[1].shape[0], 1))
+
+    if y_train.shape[-1] == 1:
+        y_train = y_train.squeeze(-1)
+    if y_test.shape[-1] == 1:
+        y_test = y_test.squeeze(-1)
+


Same concern as the 1D advection HPO script: substituting y_train/y_test with constant ones turns this into a trivial supervised objective and does not reflect PI-DeepONet physics-informed training. Consider using a residual-based loss (preferred for PI-DeepONet) or generating meaningful solution targets, or explicitly documenting that this is only an integration/template example.

echen5503

Please provide docs for at least one of the examples. Follow the format as in #2059.

Additionally, the examples themselves seem dubious. Not every hyperparameter needs to be tuned at once, additionally it seems that each example is mostly boilerplate. Please make each example more unique and show different aspects of hyperparam tuning.

echen5503 · 2026-03-22T23:24:58Z

 DeepXDE is a library for scientific machine learning and physics-informed learning. DeepXDE includes the following algorithms:

 - physics-informed neural network (PINN)
-    - solving different problems


Don't randomly change white space, keep minimal changes.

MaxGhi8 · 2026-03-27T07:15:13Z

Thank you for the feedback. I have refactored the PR as requested:

Consolidated Example: Replaced redundant scripts with a single master example advection_2d_hpo.py.
Enhanced HPO: Expanded the search space to include architecture, optimizer dynamics, and activations.
Documentation: Added a dedicated .rst file in docs/demos/.
Clean Repo: Reverted unrelated changes to README.md.

echen5503 · 2026-03-27T16:50:13Z

Great, this looks much better now. Can you provide screenshots of the documentation and outputs of the code so that we can be sure it runs correctly?

MaxGhi8 · 2026-03-29T07:38:34Z

Running with runs_per_cpu=4.0 on a 12-core machine, the code successfully executed 3 parallel runs simultaneously, as shown in the attached screenshots of the source, start, and final output of the main python function added. I also attached a preview of the documentation. I only made a minor change in the documentation and ran the formatter.

echen5503 · 2026-03-30T15:38:45Z

@@ -0,0 +1,223 @@
+"""


Suggested change

"""

"""Backend Supported: pytorch

echen5503 · 2026-03-30T15:39:06Z

@@ -0,0 +1,223 @@
+"""
+Backend supported: pytorch


Suggested change

Backend supported: pytorch

echen5503 · 2026-03-30T15:39:20Z

@@ -0,0 +1,195 @@
+"""


same here. remove the enter

echen5503 · 2026-03-30T15:42:58Z

 - [Demos of forward problems](https://deepxde.readthedocs.io/en/latest/demos/pinn_forward.html)
 - [Demos of inverse problems](https://deepxde.readthedocs.io/en/latest/demos/pinn_inverse.html)
 - [Demos of operator learning](https://deepxde.readthedocs.io/en/latest/demos/operator.html)
+- [Demos of hyperparameter optimization](https://deepxde.readthedocs.io/en/latest/demos/operator/advection_2d_hpo.html)


It could be considered that this may not be a very extensible approach. If someone were to add more hyperparam optimization examples, then they would have to individually link them.

Hyperparameter optimization doesn't fit neatly inside of operator, inverse, or forward, maybe you could create a hyperparameter folder, where it would also be extensible to people wanting to add examples of learning rate annealing, optuna, etc.

echen5503 · 2026-03-30T15:44:24Z

@@ -0,0 +1,205 @@
+2D advection: Comprehensive HPO with HyperNOs
+=============================================
+


Can you put the approximate runtime here? Hyperparam optimization often takes a long time, so the user should be assured of the code's runtime.

echen5503 · 2026-03-30T15:54:51Z

In the future, please actually build the documentation and check it, because unexpected issues might arise in the production build.

This time, I did it for you; it looks good.

echen5503 · 2026-03-30T15:55:40Z

+
+    python examples/operator/advection_2d_hpo.py
+
+The script will launch the Ray Tune dashboard where you can monitor the progress of each trial in real-time. Once finished, it will print the best configuration and the corresponding relative loss.


let the user see the final code here, it is standard for all the other documentation examples.

MaxGhi8 · 2026-04-01T07:54:38Z

Hello, I have just pushed the latest updates. I followed your feedback and implemented all the suggested changes. Specifically, I have reorganized the files by moving the examples and documentation into dedicated hyperparameter directories. Thank you for the guidance!

echen5503

Looking much better. just a few small changes before I think it's good.

echen5503 · 2026-04-01T16:44:35Z

 -
    - `Diffusion reaction equation with aligned points using ZCS <https://github.com/lululxvi/deepxde/tree/master/examples/operator/diff_rec_aligned_zcs_pideeponet.py>`_
    - `Stokes flow with aligned points using ZCS <https://github.com/lululxvi/deepxde/tree/master/examples/operator/stokes_aligned_zcs_pideeponet.py>`_
+


what's the point of this?

echen5503 · 2026-04-01T16:45:25Z

+
+.. note::
+
+    This code takes about 5 minutes to run.


mention what computer you ran this on.

echen5503 · 2026-04-01T16:45:32Z

+   :maxdepth: 1
+
+   hyperparameter/advection_2d_hpo
+


remove this space, it looks weird

MaxGhi8 · 2026-04-02T07:46:41Z

I've implemented the requested changes.

echen5503 · 2026-04-02T17:07:45Z

Ok. Looks good now.

adding hypernos examples

6d416c7

Copilot AI review requested due to automatic review settings March 22, 2026 09:14

Copilot started reviewing on behalf of MaxGhi8 March 22, 2026 09:15 View session

Copilot AI reviewed Mar 22, 2026

View reviewed changes

echen5503 suggested changes Mar 22, 2026

View reviewed changes

MaxGhi8 force-pushed the master branch 4 times, most recently from d3b1642 to e822d5b Compare March 26, 2026 17:34

adding hypernos

ac3980a

MaxGhi8 force-pushed the master branch from e822d5b to ac3980a Compare March 26, 2026 19:01

Refine 2D advection HPO example and documentation

76d2dbb

echen5503 reviewed Mar 30, 2026

View reviewed changes

echen5503 suggested changes Mar 30, 2026

View reviewed changes

add hyperparameter folder in doc and examples

0d0a8d4

echen5503 suggested changes Apr 1, 2026

View reviewed changes

update documentation

e1fa4c1

	- [Demos of hyperparameter optimization](examples/README.md)
	- [Demos of hyperparameter optimization](https://deepxde.readthedocs.io/en/latest/demos/hpo.html)

	- hyperparameter optimization using [HyperNOs](https://github.com/MaxGhi8/HyperNOs) and [Ray Tune](https://docs.ray.io/en/latest/tune/index.html).
	- example hyperparameter optimization integrations (via optional extra packages) using [HyperNOs](https://pypi.org/project/hypernos/) and [Ray Tune](https://docs.ray.io/en/latest/tune/index.html) (install `hypernos` and `ray[tune]` separately).

		@@ -0,0 +1,205 @@
		2D advection: Comprehensive HPO with HyperNOs
		=============================================


		python examples/operator/advection_2d_hpo.py

		The script will launch the Ray Tune dashboard where you can monitor the progress of each trial in real-time. Once finished, it will print the best configuration and the corresponding relative loss. No newline at end of file

Conversation

MaxGhi8 commented Mar 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 22, 2026

Choose a reason for hiding this comment

Uh oh!

echen5503 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGhi8 commented Mar 27, 2026

Uh oh!

echen5503 commented Mar 27, 2026

Uh oh!

MaxGhi8 commented Mar 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

echen5503 commented Mar 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGhi8 commented Apr 1, 2026

Uh oh!

echen5503 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGhi8 commented Apr 2, 2026

Uh oh!

echen5503 commented Apr 2, 2026

Uh oh!

Reviewers

Assignees