Multiprocessing Deprecation Warning

[danineamati]

When running credit train -c config/gen_2/smoke/integration_test_hrrr.yml (or similar), I get the following deprecation warning:

/glade/work/dneamati/conda-envs/credit/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=35951) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
/glade/work/dneamati/conda-envs/credit/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=35951) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
/glade/work/dneamati/conda-envs/credit/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=35951) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
/glade/work/dneamati/conda-envs/credit/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=35951) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()

[danineamati]

It seems that this might be causing a clash with workers in preprocessing as well (though I see 16 processes in htop):

/glade/work/dneamati/conda-envs/credit/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=5837) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
/glade/work/dneamati/conda-envs/credit/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=5837) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
/glade/work/dneamati/conda-envs/credit/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=5837) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
INFO:root:Worker 0: Processing batch 1 of 5481.
INFO:root:Worker 0: Processing batch 1 of 5481.
INFO:root:Worker 0: Processing batch 2 of 5481.
INFO:root:Worker 0: Processing batch 2 of 5481.
INFO:root:Worker 0: Processing batch 3 of 5481.
INFO:root:Worker 0: Processing batch 3 of 5481.
INFO:root:Worker 0: Processing batch 4 of 5481.
INFO:root:Worker 0: Processing batch 4 of 5481.
INFO:root:Worker 0: Processing batch 5 of 5481.
INFO:root:Worker 0: Processing batch 5 of 5481.
INFO:root:Worker 0: Processing batch 6 of 5481.
INFO:root:Worker 0: Processing batch 6 of 5481.
INFO:root:Worker 0: Processing batch 7 of 5481.
INFO:root:Worker 0: Processing batch 7 of 5481.
INFO:root:Worker 0: Processing batch 8 of 5481.
INFO:root:Worker 0: Processing batch 8 of 5481.
INFO:root:Worker 0: Processing batch 9 of 5481.
INFO:root:Worker 0: Processing batch 9 of 5481.
INFO:root:Worker 0: Processing batch 10 of 5481.
INFO:root:Worker 0: Processing batch 10 of 5481.
INFO:root:Worker 0: Processing batch 11 of 5481.
INFO:root:Worker 0: Processing batch 11 of 5481.
INFO:root:Worker 0: Processing batch 12 of 5481.
INFO:root:Worker 0: Processing batch 12 of 5481.
INFO:root:Worker 0: Processing batch 13 of 5481.
INFO:root:Worker 0: Processing batch 13 of 5481.
INFO:root:Worker 0: Processing batch 14 of 5481.
INFO:root:Worker 0: Processing batch 14 of 5481.
INFO:root:Worker 0: Processing batch 15 of 5481.
INFO:root:Worker 0: Processing batch 15 of 5481.
INFO:root:Worker 0: Processing batch 16 of 5481.
INFO:root:Worker 0: Processing batch 16 of 5481.

[danineamati]

In case it becomes relevant later:

> python --version                                                
Python 3.12.13
> conda list | grep "torch"                                                                                       
segmentation-models-pytorch      0.5.0                                 pypi_0                pypi
torch                            2.11.0+cu126                          pypi_0                pypi
torch-geometric                  2.7.0                                 pypi_0                pypi
torch-harmonics                  0.8.0                                 pypi_0                pypi
torchmetrics                     1.9.0                                 pypi_0                pypi
torchvision                      0.26.0+cu126                          pypi_0                pypi

[djgagne]

Based on some searching, it looks like when a torch DataLoader object is created, it needs to have the keyword argument multiprocessing_context set to spawn instead of fork. This may get rid of the warning, but someone would need to test it. Add the argument to

miles-credit/credit/trainers/utils.py

Lines 485 to 491 in 595cf70

	return DataLoader(
	dataset,
	batch_sampler=sampler,
	num_workers=num_workers,
	prefetch_factor=prefetch_factor,
	pin_memory=True,
	)

where the DataLoader is created in load_dataloader.

[charlie-becker]

I tested this. Spawn did not resolve the warning, and made the pipeline a bit slower. I did set persistent_workers=True, which reduces this error ro only being shown on the first epoch instead of every epoch, but didn't fully resolve the issue. The persistent_worker flag was added in #407.

[kevinyang-cky]

While reviewing #440, I tested implementing the following, and the warnings are now gone.

miles-credit/credit/trainers/utils.py

Line 528 in d92640b

multiprocessing_context="spawn" if num_workers > 0 else None,