[BUG] CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered

[pfren1998]

Describe the bug
First of all, thank you for developing such a useful tool. We have been using rapids_singlecell for large-scale single-cell analysis, and it has been extremely helpful in accelerating our workflows.

I recently encountered a CUDA illegal memory access error when running rsc.pp.calculate_qc_metrics(adata) on our in-house dataset (adata.shape = (4,729,695, 18,082)). The error occurs when the function attempts to copy cells_per_gene from GPU to CPU via cp.asnumpy(), resulting in cudaErrorIllegalAddress: an illegal memory access was encountered. Could you help me understand what might be causing this issue? Is it related to dataset size, GPU memory limitations, a bug in the current version of rapids_singlecell/CuPy, or potential corruption in the input matrix? Any suggestions for debugging or possible workarounds would be greatly appreciated.

Thank you very much for your time and assistance.

Steps/Code to reproduce bug

import rmm
import cupy as cp
from rmm.allocators.cupy import rmm_cupy_allocator
rmm.reinitialize(managed_memory=True, pool_allocator=False)
cp.cuda.set_allocator(rmm_cupy_allocator)
import warnings
warnings.filterwarnings("ignore")
import anndata as ad
import scanpy as sc
import cupy as cp
import rapids_singlecell as rsc
adata = ad.read_zarr("sc_merged.zarr")
rsc.get.anndata_to_GPU(adata)
rsc.pp.calculate_qc_metrics(adata)

Traceback (most recent call last):
  File "<python-input-11>", line 1, in <module>
    rsc.pp.calculate_qc_metrics(adata)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/lustre1/zxzeng/pfren/software/miniconda3/envs/rsc/lib/python3.14/site-packages/rapids_singlecell/preprocessing/_qc.py", line 81, in calculate_qc_metrics
    adata.var[f"n_cells_by_{expr_type}"] = cp.asnumpy(cells_per_gene)
                                           ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/lustre1/zxzeng/pfren/software/miniconda3/envs/rsc/lib/python3.14/site-packages/cupy/__init__.py", line 844, in asnumpy
    return a.get(stream=stream, order=order, out=out, blocking=blocking)
           ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cupy/_core/core.pyx", line 1990, in cupy._core.core._ndarray_base.get
  File "cupy/_core/core.pyx", line 2095, in cupy._core.core._ndarray_base.get
  File "cupy/cuda/memory.pyx", line 695, in cupy.cuda.memory.MemoryPointer.copy_to_host_async
  File "cupy_backends/cuda/api/runtime.pyx", line 628, in cupy_backends.cuda.api.runtime.memcpyAsync
  File "cupy_backends/cuda/api/runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered

Environment details (please complete the following information):

• Environment location: [Bare-metal]
• Linux Distro/Architecture: [Ubuntu 22.04 amd64]
• GPU Model/Driver: [A100 and driver 535.247.01]
• CUDA: [12.2]
• Method of Rapids install: [conda, pip or from source]
  aiohappyeyeballs 2.6.2 pyhd8ed1ab_0 conda-forge
  aiohttp 3.13.5 pyhf64b827_0 conda-forge
  aiosignal 1.4.0 pyhd8ed1ab_0 conda-forge
  anndata 0.12.16 pypi_0 pypi
  anyio 4.13.0 pyhcf101f3_0 conda-forge
  aom 3.9.1 hac33072_0 conda-forge
  argon2-cffi 25.1.0 pyhd8ed1ab_0 conda-forge
  argon2-cffi-bindings 25.1.0 py314h5bd0f2a_2 conda-forge
  array-api-compat 1.14.0 pypi_0 pypi
  arrow 1.4.0 pyhcf101f3_0 conda-forge
  asttokens 3.0.1 pyhd8ed1ab_0 conda-forge
  async-lru 2.3.0 pyhcf101f3_0 conda-forge
  async-timeout 5.0.1 pyhcf101f3_2 conda-forge
  attrs 26.1.0 pyhcf101f3_0 conda-forge
  aws-c-auth 0.10.3 h3aafcba_1 conda-forge
  aws-c-cal 0.9.14 h8e43964_1 conda-forge
  aws-c-common 0.13.1 hb03c661_0 conda-forge
  aws-c-compression 0.3.2 h16e98cb_1 conda-forge
  aws-c-event-stream 0.7.1 h9be7a74_1 conda-forge
  aws-c-http 0.11.0 hcbcd92d_1 conda-forge
  aws-c-io 0.26.3 h955231c_3 conda-forge
  aws-c-mqtt 0.15.2 h8af55cf_3 conda-forge
  aws-c-s3 0.12.3 h00bea6e_2 conda-forge
  aws-c-sdkutils 0.2.4 h16e98cb_5 conda-forge
  aws-checksums 0.2.10 h16e98cb_1 conda-forge
  aws-crt-cpp 0.38.3 h7b0d4b4_2 conda-forge
  aws-sdk-cpp 1.11.747 h5a171d8_5 conda-forge
  azure-core-cpp 1.16.2 h206d751_0 conda-forge
  azure-identity-cpp 1.13.3 hed0cdb0_1 conda-forge
  azure-storage-blobs-cpp 12.17.0 hf824e48_1 conda-forge
  azure-storage-common-cpp 12.13.0 ha7a2c86_0 conda-forge
  azure-storage-files-datalake-cpp 12.15.0 h1e5b466_0 conda-forge
  babel 2.18.0 pyhcf101f3_1 conda-forge
  backports.zstd 1.5.0 py314h680f03e_0 conda-forge
  beautifulsoup4 4.14.3 pyha770c72_0 conda-forge
  bleach 6.3.0 pyhcf101f3_1 conda-forge
  bleach-with-css 6.3.0 hbca2aae_1 conda-forge
  blosc 1.21.6 he440d0b_1 conda-forge
  bokeh 3.9.0 pyhd8ed1ab_0 conda-forge
  branca 0.8.2 pyhd8ed1ab_0 conda-forge
  brotli 1.2.0 hed03a55_1 conda-forge
  brotli-bin 1.2.0 hb03c661_1 conda-forge
  brotli-python 1.2.0 py314h3de4e8d_1 conda-forge
  brunsli 0.1 hd1e3526_2 conda-forge
  bzip2 1.0.8 hda65f42_9 conda-forge
  c-ares 1.34.6 hb03c661_0 conda-forge
  c-blosc2 3.1.2 hc31b594_0 conda-forge
  ca-certificates 2026.5.20 hbd8a1cb_0 conda-forge
  cached-property 1.5.2 hd8ed1ab_1 conda-forge
  cached_property 1.5.2 pyha770c72_1 conda-forge
  cachetools 7.1.4 pyhd8ed1ab_0 conda-forge
  certifi 2026.5.20 pyhd8ed1ab_0 conda-forge
  cffi 2.0.0 py314h4a8dc5f_1 conda-forge
  charls 2.4.3 hecca717_0 conda-forge
  charset-normalizer 3.4.7 pyhd8ed1ab_0 conda-forge
  click 8.4.1 pyhc90fa1f_0 conda-forge
  cloudpickle 3.1.2 pyhcf101f3_1 conda-forge
  colorcet 3.2.1 pyhd8ed1ab_0 conda-forge
  comm 0.2.3 pyhe01879c_0 conda-forge
  contourpy 1.3.3 py314h97ea11e_4 conda-forge
  cpython 3.14.5 py314hd8ed1ab_100 conda-forge
  cucim 26.04.00 cuda12_py314_260408_ghgd9ab164_gn0_phc494990 rapidsai
  cuda-bindings 12.9.4 py314h421e08d_1 nvidia
  cuda-cccl_linux-64 12.9.27 0 nvidia
  cuda-core 0.4.2 cuda12_py314h6985919_0 nvidia
  cuda-crt-dev_linux-64 12.9.86 0 nvidia
  cuda-crt-tools 12.9.86 0 nvidia
  cuda-cudart 12.9.79 0 nvidia
  cuda-cudart-dev 12.9.79 0 nvidia
  cuda-cudart-dev_linux-64 12.9.79 0 nvidia
  cuda-cudart-static 12.9.79 0 nvidia
  cuda-cudart-static_linux-64 12.9.79 0 nvidia
  cuda-cudart_linux-64 12.9.79 0 nvidia
  cuda-nvcc-dev_linux-64 12.9.86 0 nvidia
  cuda-nvcc-impl 12.9.86 0 nvidia
  cuda-nvcc-tools 12.9.86 0 nvidia
  cuda-nvrtc 12.9.86 0 nvidia
  cuda-nvvm-dev_linux-64 12.9.86 0 nvidia
  cuda-nvvm-impl 12.9.86 0 nvidia
  cuda-nvvm-tools 12.9.86 0 nvidia
  cuda-pathfinder 1.5.5 pyhc364b38_0 conda-forge
  cuda-profiler-api 12.9.79 0 nvidia
  cuda-python 12.9.4 pyh15a92d1_1 nvidia
  cuda-version 12.9 3 nvidia
  cudf 26.04.00 cuda12_cp311_abi3_260408_f9c3cf19 rapidsai
  cudf-polars 26.04.00 cuda12_260408_f9c3cf19 rapidsai
  cudf_kafka 26.04.00 cuda12_cp311_abi3_260408_f9c3cf19 rapidsai
  cudnn 9.14.0.64 h321a1a8_0 nvidia
  cugraph 26.04.00 cuda12_cp311_abi3_260408_03ba9c2d rapidsai
  cuml 26.04.00 cuda12_cp311_abi3_260409_f223d89b rapidsai
  cupy 14.0.1 py314h3d8d815_0 conda-forge
  cupy-core 14.0.1 py314hf9e62a7_0 conda-forge
  cusparselt 0.8.1.1 h2b6041c_0 nvidia
  custreamz 26.04.00 260408_f9c3cf19 rapidsai
  cutensor 2.6.0.4 h2d75abc_0 conda-forge
  cuvs 26.04.00 cuda12_cp311_abi3_260408_ rapidsai
  cuxfilter 26.04.00 cuda12_260409_3cec658d rapidsai
  cycler 0.12.1 pyhcf101f3_2 conda-forge
  cyrus-sasl 2.1.28 hac629b4_1 conda-forge
  cytoolz 1.1.0 py314h5bd0f2a_2 conda-forge
  dask 2026.1.1 pyhcf101f3_0 conda-forge
  dask-core 2026.1.1 pyhcf101f3_1 conda-forge
  dask-cuda 26.04.00 py314_260408_157ffa32 rapidsai
  dask-cudf 26.04.00 260408_f9c3cf19 rapidsai
  datashader 0.19.1 pyhd8ed1ab_0 conda-forge
  dav1d 1.2.1 hd590300_0 conda-forge
  debugpy 1.8.20 py314h42812f9_0 conda-forge
  decorator 5.3.1 pyhd8ed1ab_0 conda-forge
  defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
  distributed 2026.1.1 pyhcf101f3_1 conda-forge
  distributed-ucxx 0.49.00 260408_8d47a9ff_h546327b rapidsai
  dlpack 0.8 h59595ed_3 conda-forge
  docrep 0.3.2 pypi_0 pypi
  donfig 0.8.1.post1 pypi_0 pypi
  exceptiongroup 1.3.1 pyhd8ed1ab_0 conda-forge
  executing 2.2.1 pyhd8ed1ab_0 conda-forge
  fast-array-utils 1.4.1 pypi_0 pypi
  filelock 3.29.0 pypi_0 pypi
  folium 0.20.0 pyhd8ed1ab_0 conda-forge
  fonttools 4.63.0 pyh7db6752_0 conda-forge
  fqdn 1.5.1 pyhd8ed1ab_1 conda-forge
  freetype 2.14.3 ha770c72_0 conda-forge
  freexl 2.0.0 h9dce30a_2 conda-forge
  frozenlist 1.8.0 pyhf298e5d_0 conda-forge
  fsspec 2026.4.0 pyhd8ed1ab_0 conda-forge
  gdown 6.1.0 pypi_0 pypi
  geopandas 1.1.3 pyhd8ed1ab_0 conda-forge
  geopandas-base 1.1.3 pyha770c72_0 conda-forge
  geos 3.14.1 h480dda7_0 conda-forge
  gflags 2.2.2 h5888daf_1005 conda-forge
  giflib 5.2.2 hd590300_0 conda-forge
  glog 0.7.1 hbabe93e_0 conda-forge
  google-crc32c 1.8.0 pypi_0 pypi
  h11 0.16.0 pyhcf101f3_1 conda-forge
  h2 4.3.0 pyhcf101f3_0 conda-forge
  h5py 3.16.0 pypi_0 pypi
  holoviews 1.22.1 pyhd8ed1ab_0 conda-forge
  hpack 4.1.0 pyhd8ed1ab_0 conda-forge
  httpcore 1.0.9 pyh29332c3_0 conda-forge
  httpx 0.28.1 pyhd8ed1ab_0 conda-forge
  hyperframe 6.1.0 pyhd8ed1ab_0 conda-forge
  icu 78.3 h33c6efd_0 conda-forge
  idna 3.17 pyhcf101f3_0 conda-forge
  imagecodecs 2026.5.10 py314hc63dc0c_1 conda-forge
  imageio 2.37.0 pyhfb79c49_0 conda-forge
  importlib-metadata 9.0.0 pyhcf101f3_0 conda-forge
  ipykernel 7.2.0 pyha191276_1 conda-forge
  ipython 9.14.0 pyh53cf698_0 conda-forge
  ipython_pygments_lexers 1.1.1 pyhd8ed1ab_0 conda-forge
  isoduration 20.11.0 pyhd8ed1ab_1 conda-forge
  jedi 0.19.2 pyhd8ed1ab_1 conda-forge
  jinja2 3.1.6 pyhcf101f3_1 conda-forge
  joblib 1.5.3 pyhd8ed1ab_0 conda-forge
  json-c 0.18 h6688a6e_0 conda-forge
  json5 0.14.0 pyhd8ed1ab_0 conda-forge
  jsonpointer 3.1.1 pyhcf101f3_0 conda-forge
  jsonschema 4.26.0 pyhcf101f3_0 conda-forge
  jsonschema-specifications 2025.9.1 pyhcf101f3_0 conda-forge
  jsonschema-with-format-nongpl 4.26.0 hcf101f3_0 conda-forge
  jupyter-lsp 2.3.1 pyhcf101f3_0 conda-forge
  jupyter-server-proxy 4.5.0 pyhcf101f3_1 conda-forge
  jupyter_client 8.8.0 pyhcf101f3_0 conda-forge
  jupyter_core 5.9.1 pyhc90fa1f_0 conda-forge
  jupyter_events 0.12.1 pyhcf101f3_0 conda-forge
  jupyter_server 2.19.0 pyhcf101f3_0 conda-forge
  jupyter_server_terminals 0.5.4 pyhcf101f3_0 conda-forge
  jupyterlab 4.5.7 pyhd8ed1ab_0 conda-forge
  jupyterlab_pygments 0.3.0 pyhd8ed1ab_2 conda-forge
  jupyterlab_server 2.28.0 pyhcf101f3_0 conda-forge
  jxrlib 1.1 hd590300_3 conda-forge
  keyutils 1.6.3 hb9d3cd8_0 conda-forge
  kiwisolver 1.5.0 py314h97ea11e_0 conda-forge
  krb5 1.22.2 ha1258a1_0 conda-forge
  lark 1.3.1 pyhd8ed1ab_0 conda-forge
  lazy-loader 0.5 pyhd8ed1ab_0 conda-forge
  lazy_loader 0.5 pyhd8ed1ab_0 conda-forge
  lcms2 2.19.1 h0c24ade_1 conda-forge
  ld_impl_linux-64 2.45.1 default_hbd61a6d_102 conda-forge
  legacy-api-wrap 1.5 pypi_0 pypi
  lerc 4.1.0 hdb68285_0 conda-forge
  libabseil 20260107.1 cxx17_h7b12aa8_0 conda-forge
  libaec 1.1.5 h088129d_0 conda-forge
  libarchive 3.8.7 gpl_hc2c16d8_101 conda-forge
  libarrow 24.0.0 h61d77b5_4_cpu conda-forge
  libarrow-acero 24.0.0 h635bf11_4_cpu conda-forge
  libarrow-compute 24.0.0 h53684a4_4_cpu conda-forge
  libarrow-dataset 24.0.0 h635bf11_4_cpu conda-forge
  libarrow-substrait 24.0.0 hb4dd7c2_4_cpu conda-forge
  libavif16 1.4.2 hcfa2d63_0 conda-forge
  libblas 3.11.0 8_h4a7cf45_openblas conda-forge
  libbrotlicommon 1.2.0 hb03c661_1 conda-forge
  libbrotlidec 1.2.0 hb03c661_1 conda-forge
  libbrotlienc 1.2.0 hb03c661_1 conda-forge
  libcap 2.78 hd0affe5_0 conda-forge
  libcblas 3.11.0 8_h0358290_openblas conda-forge
  libcrc32c 1.1.2 h9c3ff4c_0 conda-forge
  libcublas 12.9.2.10 h46aabaf_0 nvidia
  libcublas-dev 12.9.2.10 h46aabaf_0 nvidia
  libcucim 26.04.00 cuda12_260408_ghgd9ab164_gn0_phf3d4060 rapidsai
  libcudf 26.04.00 cuda12_260408_f9c3cf19 rapidsai
  libcudf_kafka 26.04.00 cuda12_260408_f9c3cf19 rapidsai
  libcudnn 9.14.0.64 hf94a4fa_0 nvidia
  libcudnn-dev 9.14.0.64 h321a1a8_0 nvidia
  libcufft 11.4.1.4 0 nvidia
  libcufile 1.14.1.1 4 nvidia
  libcufile-dev 1.14.1.1 4 nvidia
  libcugraph 26.04.00 cuda12_260408_03ba9c2d rapidsai
  libcugraph_etl 26.04.00 cuda12_260408_03ba9c2d rapidsai
  libcuml 26.04.00 cuda12_260409_f223d89b rapidsai
  libcurand 10.3.10.19 0 nvidia
  libcurand-dev 10.3.10.19 0 nvidia
  libcurl 8.20.0 hcf29cc6_0 conda-forge
  libcusolver 11.7.5.82 0 nvidia
  libcusolver-dev 11.7.5.82 0 nvidia
  libcusparse 12.5.10.65 0 nvidia
  libcusparse-dev 12.5.10.65 0 nvidia
  libcuvs 26.04.00 cuda12_260408_60b19a0f rapidsai
  libcuvs-headers 26.04.00 cuda12_260408_60b19a0f rapidsai
  libdeflate 1.25 h17f619e_0 conda-forge
  libedit 3.1.20250104 pl5321h7949ede_0 conda-forge
  libev 4.33 hd590300_2 conda-forge
  libevent 2.1.12 hf998b51_1 conda-forge
  libexpat 2.8.1 hecca717_0 conda-forge
  libffi 3.5.2 h3435931_0 conda-forge
  libfreetype 2.14.3 ha770c72_0 conda-forge
  libfreetype6 2.14.3 h73754d4_0 conda-forge
  libgcc 15.2.0 he0feb66_19 conda-forge
  libgcc-ng 15.2.0 h69a702a_19 conda-forge
  libgdal-core 3.13.0 h08c5dba_2 conda-forge
  libgfortran 15.2.0 h69a702a_19 conda-forge
  libgfortran5 15.2.0 h68bc16d_19 conda-forge
  libgomp 15.2.0 he0feb66_19 conda-forge
  libgoogle-cloud 3.5.0 h8d2ee43_1 conda-forge
  libgoogle-cloud-storage 3.5.0 hdbdcf42_1 conda-forge
  libgrpc 1.78.1 h1d1128b_0 conda-forge
  libhwy 1.4.0 h10be129_0 conda-forge
  libiconv 1.18 h3b78370_2 conda-forge
  libjpeg-turbo 3.1.4.1 hb03c661_0 conda-forge
  libjxl 0.11.2 h174a0a3_1 conda-forge
  libkml 1.3.0 haa4a5bd_1023 conda-forge
  libkvikio 26.04.00 cuda12_260408_5aba1d10 rapidsai
  liblapack 3.11.0 8_h47877c9_openblas conda-forge
  liblzma 5.8.3 hb03c661_0 conda-forge
  libmpdec 4.0.0 hb03c661_1 conda-forge
  libnghttp2 1.68.1 h877daf1_0 conda-forge
  libnl 3.11.0 hb9d3cd8_0 conda-forge
  libnsl 2.0.1 hb9d3cd8_1 conda-forge
  libntlm 1.8 hb9d3cd8_0 conda-forge
  libnuma 2.0.18 hb03c661_3 conda-forge
  libnvcomp 5.1.0.21 hb7e823c_0 conda-forge
  libnvcomp-dev 5.1.0.21 hb7e823c_0 conda-forge
  libnvimgcodec 0.7.0 h40eb5e6_2 conda-forge
  libnvimgcodec0 0.7.0 hb7e823c_2 conda-forge
  libnvjitlink 12.9.86 0 nvidia
  libnvjpeg 12.4.0.76 0 nvidia
  libnvjpeg2k0 0.9.1.47 hf2c0dc5_1 conda-forge
  libnvtiff 0.6.0.78 ha770c72_1 conda-forge
  libnvtiff0 0.6.0.78 h3ac0392_1 conda-forge
  libopenblas 0.3.33 pthreads_h94d23a6_0 conda-forge
  libopentelemetry-cpp 1.27.0 h9692893_0 conda-forge
  libopentelemetry-cpp-headers 1.27.0 ha770c72_0 conda-forge
  libparquet 24.0.0 h7376487_4_cpu conda-forge
  libpng 1.6.58 h421ea60_0 conda-forge
  libprotobuf 6.33.5 h6eeba95_1 conda-forge
  libraft 26.04.00 cuda12_260408_b01e3028 rapidsai
  libraft-headers 26.04.00 cuda12_260408_b01e3028 rapidsai
  libraft-headers-only 26.04.00 cuda12_260408_b01e3028 rapidsai
  librdkafka 2.14.1 he5e3081_0 conda-forge
  libre2-11 2025.11.05 h0dc7533_1 conda-forge
  librmm 26.04.00 cuda12_260408_48b36cc6 rapidsai
  librttopo 1.1.0 h46dd2a8_20 conda-forge
  libsodium 1.0.22 h280c20c_1 conda-forge
  libspatialite 5.1.0 gpl_hab3fe16_120 conda-forge
  libsqlite 3.53.1 h0c1763c_0 conda-forge
  libssh2 1.11.1 hcf80075_0 conda-forge
  libstdcxx 15.2.0 h934c35e_19 conda-forge
  libstdcxx-ng 15.2.0 hdf11a46_19 conda-forge
  libsystemd0 257.13 h084b8d7_1 conda-forge
  libthrift 0.22.0 h7d032f7_2 conda-forge
  libtiff 4.7.1 h9d88235_1 conda-forge
  libucxx 0.49.00 cuda12_260408_8d47a9ff rapidsai
  libudev1 257.13 h084b8d7_1 conda-forge
  libutf8proc 2.11.3 hfe17d71_0 conda-forge
  libuuid 2.42.1 h5347b49_0 conda-forge
  libuv 1.52.1 h280c20c_0 conda-forge
  libwebp-base 1.6.0 hd42ef1d_0 conda-forge
  libxcb 1.17.0 h8a09558_0 conda-forge
  libxcrypt 4.4.36 hd590300_1 conda-forge
  libxgboost 3.2.0 rapidsai_haf08eaf_1 rapidsai
  libxml2 2.15.3 h49c6c72_0 conda-forge
  libxml2-16 2.15.3 hca6bf5a_0 conda-forge
  libxml2-devel 2.15.3 h49c6c72_0 conda-forge
  libzlib 1.3.2 h25fd6f3_2 conda-forge
  libzopfli 1.0.3 h9c3ff4c_0 conda-forge
  linkify-it-py 2.1.0 pyhcf101f3_0 conda-forge
  llvmlite 0.46.0 py314h946fb2a_0 conda-forge
  locket 1.0.0 pyhd8ed1ab_0 conda-forge
  lz4 4.4.5 py314hd4c109c_1 conda-forge
  lz4-c 1.10.0 h5888daf_1 conda-forge
  lzo 2.10 h280c20c_1002 conda-forge
  mapclassify 2.10.0 pyhd8ed1ab_1 conda-forge
  markdown 3.10.2 pyhcf101f3_0 conda-forge
  markdown-it-py 4.2.0 pyhd8ed1ab_0 conda-forge
  markupsafe 3.0.3 py314h67df5f8_1 conda-forge
  matplotlib-base 3.10.9 py314h1194b4b_0 conda-forge
  matplotlib-inline 0.2.2 pyhd8ed1ab_0 conda-forge
  mdit-py-plugins 0.6.1 pyhd8ed1ab_0 conda-forge
  mdurl 0.1.2 pyhd8ed1ab_1 conda-forge
  minizip 4.2.1 hb71707f_0 conda-forge
  mistune 3.2.1 pyhcf101f3_0 conda-forge
  msgpack-python 1.1.2 py314h9891dd4_1 conda-forge
  multidict 6.7.1 pyh62beb40_0 conda-forge
  multipledispatch 0.6.0 pyhd8ed1ab_1 conda-forge
  munkres 1.1.4 pyhd8ed1ab_1 conda-forge
  muparser 2.3.5 h5888daf_0 conda-forge
  narwhals 2.21.2 pyhcf101f3_0 conda-forge
  natsort 8.4.0 pypi_0 pypi
  nbclient 0.10.4 pyhd8ed1ab_0 conda-forge
  nbconvert-core 7.17.1 pyhcf101f3_0 conda-forge
  nbformat 5.10.4 pyhd8ed1ab_1 conda-forge
  nccl 2.30.4 he02bfcb_0 nvidia
  ncurses 6.6 hdb14827_0 conda-forge
  nest-asyncio 1.6.0 pyhd8ed1ab_1 conda-forge
  networkx 3.6.1 pyhcf101f3_0 conda-forge
  nh3 0.3.5 py310hd8a072f_1 conda-forge
  nlohmann_json 3.12.0 h54a6638_1 conda-forge
  nodejs 26.2.0 he4ff34a_0 conda-forge
  notebook-shim 0.2.4 pyhd8ed1ab_1 conda-forge
  numba 0.64.0 py314h8169c2f_0 conda-forge
  numba-cuda 0.24.0 py314h118ebf0_1 conda-forge
  numcodecs 0.16.5 pypi_0 pypi
  numpy 2.4.6 py314h2b28147_0 conda-forge
  nvidia-ml-py 13.595.45 pyhd8ed1ab_1 conda-forge
  nvtx 0.2.15 py314h5bd0f2a_0 conda-forge
  nx-cugraph 26.04.00 py_260409_45918e26 rapidsai
  openjpeg 2.5.4 h55fea9a_0 conda-forge
  openjph 0.27.3 h8d634f6_0 conda-forge
  openssl 3.6.2 h35e630c_0 conda-forge
  orc 2.3.0 h21090e2_0 conda-forge
  overrides 7.7.0 pyhd8ed1ab_1 conda-forge
  packaging 26.2 pyhc364b38_0 conda-forge
  pandas 2.3.3 py314ha0b5721_2 conda-forge
  pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
  panel 1.9.2 pyhd8ed1ab_1 conda-forge
  panel-core 1.9.2 pyha770c72_1 conda-forge
  panel-material-ui 0.11.2 pyhd8ed1ab_0 conda-forge
  param 2.4.0 pyhc455866_0 conda-forge
  parso 0.8.7 pyhcf101f3_0 conda-forge
  partd 1.4.2 pyhd8ed1ab_0 conda-forge
  patsy 1.0.2 pypi_0 pypi
  pcre2 10.47 haa7fec5_0 conda-forge
  pexpect 4.9.0 pyhd8ed1ab_1 conda-forge
  pillow 12.2.0 py314h8ec4b1a_0 conda-forge
  pip 26.1.2 pyh145f28c_0 conda-forge
  platformdirs 4.10.0 pyhcf101f3_0 conda-forge
  polars 1.38.1 pyh6a1acc5_0 conda-forge
  polars-runtime-32 1.38.1 py310hffdcd12_0 conda-forge
  proj 9.8.1 he0df7b0_0 conda-forge
  prometheus-cpp 1.3.0 ha5d0236_0 conda-forge
  prometheus_client 0.25.0 pyhd8ed1ab_0 conda-forge
  prompt-toolkit 3.0.52 pyha770c72_0 conda-forge
  propcache 0.5.2 pyh7db6752_0 conda-forge
  psutil 7.2.2 py314h0f05182_0 conda-forge
  pthread-stubs 0.4 hb9d3cd8_1002 conda-forge
  ptyprocess 0.7.0 pyhd8ed1ab_1 conda-forge
  pure_eval 0.2.3 pyhd8ed1ab_1 conda-forge
  py-xgboost 3.2.0 rapidsai_pyh75519c6_1 rapidsai
  pyarrow 24.0.0 py314hdafbbf9_0 conda-forge
  pyarrow-core 24.0.0 py314h969be7f_0_cpu conda-forge
  pycparser 3.0 pyhcf101f3_0 conda-forge
  pyct 0.6.0 pyhd8ed1ab_0 conda-forge
  pygments 2.20.0 pyhd8ed1ab_0 conda-forge
  pylibcudf 26.04.00 cuda12_cp311_abi3_260408_f9c3cf19 rapidsai
  pylibcugraph 26.04.00 cuda12_cp311_abi3_260408_03ba9c2d rapidsai
  pylibraft 26.04.00 cuda12_cp311_abi3_260408_b01e3028 rapidsai
  pynndescent 0.6.0 pypi_0 pypi
  pyogrio 0.12.1 py314hab130c5_1 conda-forge
  pyparsing 3.3.2 pyhcf101f3_0 conda-forge
  pyproj 3.7.2 py314h7ce3bca_4 conda-forge
  pysocks 1.7.1 pyha55dd90_7 conda-forge
  python 3.14.5 habeac84_100_cp314 conda-forge
  python-confluent-kafka 2.14.0 py314h5bd0f2a_1 conda-forge
  python-dateutil 2.9.0.post0 pyhe01879c_2 conda-forge
  python-fastjsonschema 2.21.2 pyhe01879c_0 conda-forge
  python-gil 3.14.5 h4df99d1_100 conda-forge
  python-json-logger 3.2.1 pyh332efcf_0 conda-forge
  python-tzdata 2026.2 pyhd8ed1ab_0 conda-forge
  python_abi 3.14 8_cp314 conda-forge
  pytz 2026.2 pyhcf101f3_0 conda-forge
  pyviz_comms 3.0.6 pyhd8ed1ab_0 conda-forge
  pyyaml 6.0.3 py314h67df5f8_1 conda-forge
  pyzmq 27.1.0 py312hda471dd_3 conda-forge
  qhull 2020.2 h434a139_5 conda-forge
  raft-dask 26.04.00 cuda12_cp311_abi3_260408_b01e3028 rapidsai
  rapids 26.04.00 cuda12_260409_a77e25d8 rapidsai
  rapids-dask-dependency 26.04.00 260408_8071384e rapidsai
  rapids-logger 0.2.3 h98325ef_0 rapidsai
  rapids-singlecell-cu12 0.15.1 pypi_0 pypi
  rapids-xgboost 26.04.00 cuda12_260409_a77e25d8 rapidsai
  rav1e 0.8.1 h1fbca29_0 conda-forge
  rdma-core 63.0 h192683f_1 conda-forge
  re2 2025.11.05 h5301d42_1 conda-forge
  readline 8.3 h853b02a_0 conda-forge
  referencing 0.37.0 pyhcf101f3_0 conda-forge
  requests 2.34.2 pyhcf101f3_0 conda-forge
  rfc3339-validator 0.1.4 pyhd8ed1ab_1 conda-forge
  rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge
  rfc3987-syntax 1.1.0 pyhe01879c_1 conda-forge
  rich 15.0.0 pyhcf101f3_0 conda-forge
  rmm 26.04.00 cuda12_cp311_abi3_260408_48b36cc6 rapidsai
  rpds-py 2026.5.1 py314h1bee95f_0 conda-forge
  s2n 1.7.3 hc5a330e_0 conda-forge
  scanpy 1.12.1 pypi_0 pypi
  scikit-image 0.26.0 np2py314hda1ea4c_0 conda-forge
  scikit-learn 1.8.0 np2py314hf09ca88_1 conda-forge
  scikit-misc 0.5.2 pypi_0 pypi
  scipy 1.16.3 py314hf07bd8e_2 conda-forge
  scverse-misc 0.0.7 pypi_0 pypi
  seaborn 0.13.2 pypi_0 pypi
  send2trash 2.1.0 pyha191276_1 conda-forge
  session-info2 0.4.1 pypi_0 pypi
  setuptools 82.0.1 pyh332efcf_0 conda-forge
  shapely 2.1.2 py314hbe3edd8_2 conda-forge
  simpervisor 1.0.0 pyhd8ed1ab_1 conda-forge
  six 1.17.0 pyhe01879c_1 conda-forge
  snappy 1.2.2 h03e3b7b_1 conda-forge
  sniffio 1.3.1 pyhd8ed1ab_2 conda-forge
  sortedcontainers 2.4.0 pyhd8ed1ab_1 conda-forge
  soupsieve 2.8.4 pyhd8ed1ab_0 conda-forge
  sqlite 3.53.1 hbc0de68_0 conda-forge
  stack_data 0.6.3 pyhd8ed1ab_1 conda-forge
  statsmodels 0.14.6 pypi_0 pypi
  streamz 0.6.6 pyh332efcf_0 conda-forge
  svt-av1 4.0.1 hecca717_0 conda-forge
  tblib 3.2.2 pyhcf101f3_0 conda-forge
  terminado 0.18.1 pyhc90fa1f_1 conda-forge
  threadpoolctl 3.6.0 pyhecae5ae_0 conda-forge
  tifffile 2026.5.15 pyhd8ed1ab_0 conda-forge
  tinycss2 1.4.0 pyhd8ed1ab_0 conda-forge
  tk 8.6.13 noxft_h366c992_103 conda-forge
  tomli 2.4.1 pyhcf101f3_0 conda-forge
  toolz 1.1.0 pyhd8ed1ab_1 conda-forge
  tornado 6.5.6 py314h5bd0f2a_0 conda-forge
  tqdm 4.67.3 pyh8f84b5b_0 conda-forge
  traitlets 5.15.0 pyhcf101f3_0 conda-forge
  treelite 4.7.0 py314he2c2258_0 conda-forge
  typing-extensions 4.15.0 h396c80c_0 conda-forge
  typing_extensions 4.15.0 pyhcf101f3_0 conda-forge
  typing_utils 0.1.0 pyhd8ed1ab_1 conda-forge
  tzdata 2025c hc9c84f9_1 conda-forge
  uc-micro-py 2.0.0 pyhcf101f3_0 conda-forge
  ucx 1.19.1 h63b5c0b_0 conda-forge
  ucxx 0.49.00 cuda12_cp311_abi3_260408_8d47a9ff rapidsai
  umap-learn 0.5.12 pypi_0 pypi
  unicodedata2 17.0.1 py314h5bd0f2a_0 conda-forge
  uri-template 1.3.0 pyhd8ed1ab_1 conda-forge
  uriparser 0.9.8 hac33072_0 conda-forge
  urllib3 2.7.0 pyhd8ed1ab_0 conda-forge
  wcwidth 0.7.0 pyhd8ed1ab_0 conda-forge
  webcolors 25.10.0 pyhd8ed1ab_0 conda-forge
  webencodings 0.5.1 pyhd8ed1ab_3 conda-forge
  websocket-client 1.9.0 pyhd8ed1ab_0 conda-forge
  wget 3.2 pypi_0 pypi
  xarray 2026.4.0 pyhc364b38_0 conda-forge
  xerces-c 3.3.0 hd9031aa_1 conda-forge
  xgboost 3.2.0 rapidsai_pyh3d5ed11_1 rapidsai
  xorg-libxau 1.0.12 hb03c661_1 conda-forge
  xorg-libxdmcp 1.1.5 hb03c661_1 conda-forge
  xyzservices 2026.3.0 pyhd8ed1ab_0 conda-forge
  yaml 0.2.5 h280c20c_3 conda-forge
  yarl 1.24.2 pyh7db6752_0 conda-forge
  zarr 3.2.1 pypi_0 pypi
  zeromq 4.3.5 h09e67af_11 conda-forge
  zfp 1.0.1 h909a3a2_5 conda-forge
  zict 3.0.0 pyhd8ed1ab_1 conda-forge
  zipp 4.1.0 pyhcf101f3_0 conda-forge
  zlib 1.3.2 h25fd6f3_2 conda-forge
  zlib-ng 2.3.3 hceb46e0_1 conda-forge
  zstd 1.5.7 hb78ec9c_6 conda-forge

[Intron7]

Please upgrade to the latest cupy and rapids-singlecell version. Cupy <14.1 doesn't support 64 bit indptr. So does rsc<0.15.1. you might still oom. If that happens please look at the out of core options with dask

[pfren1998]

Thank you very much for your help. After upgrading, the issue was indeed resolved and the function now runs successfully.

Since our dataset is still growing, we expect that after merging all future data we may eventually run into out-of-memory (OOM) issues. You mentioned that we could use the out-of-core options with Dask, so I followed the tutorial here:

https://rapids-singlecell.readthedocs.io/en/latest/notebooks/06_out-of-core.html

However, I encountered some issues while trying to reproduce the workflow. The code and corresponding results are shown below.

import dask
import time
import gc
from dask_cuda import LocalCUDACluster
from dask.distributed import Client
import rmm
import cupy as cp
from rmm.allocators.cupy import rmm_cupy_allocator

preprocessing_gpus="0,1"
cluster = LocalCUDACluster(CUDA_VISIBLE_DEVICES=preprocessing_gpus,
                           threads_per_worker=10,
                           protocol="ucx",
                           rmm_pool_size= "10GB",
                           rmm_maximum_pool_size = "20GB",
                           rmm_allocator_external_lib_list= "cupy",
                          )

client = Client(cluster)

import rapids_singlecell as rsc
import scanpy as sc
import anndata as ad

from packaging.version import parse as parse_version
import zarr
if parse_version(ad.__version__) < parse_version("0.12.0rc1"):
    from anndata.experimental import read_elem_as_dask as read_dask
else:
    from anndata.experimental import read_elem_lazy as read_dask

SPARSE_CHUNK_SIZE = 20_000
f = zarr.open('sc_merged.zarr')
X = f["X"]
shape = X.attrs["shape"]
adata = ad.AnnData(
    X = read_dask(X, (SPARSE_CHUNK_SIZE, shape[1])),
    obs = ad.io.read_elem(f["obs"]),
    var = ad.io.read_elem(f["var"])
)
adata.obs["n_genes_by_counts"].min()
>>>250
adata.obs["n_genes_by_counts"].max()
>>>9817

rsc.get.anndata_to_GPU(adata)
rsc.pp.calculate_qc_metrics(adata)
adata.obs["n_genes_by_counts"].min()
>>>0
adata.obs["n_genes_by_counts"].max()
>>>0

As you can see, after transferring the data to the GPU, the calculated number of genes detected per cell becomes zero for all cells, which seems quite unexpected. Do you have any suggestions on what might be causing this issue or how I could further debug it?

Any advice would be greatly appreciated. Thank you again for your help.

[mxposed]

I see same error on different operation:

---------------------------------------------------------------------------
CUDADriverError                           Traceback (most recent call last)
Cell In[27], [line 1](vscode-notebook-cell:?execution_count=27&line=1)
----> [1](vscode-notebook-cell:?execution_count=27&line=1) rsc.pp.neighbors(gene_ds, n_neighbors=40, n_pcs=15, method='jaccard', random_state=948502)

File /usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/rapids_singlecell/preprocessing/_neighbors/__init__.py:215, in neighbors(adata, n_neighbors, n_pcs, use_rep, random_state, algorithm, metric, metric_kwds, algorithm_kwds, method, key_added, copy)
    213 n_obs = adata.shape[0]
    214 distances = _build_sparse_distances(knn_indices, knn_dist, n_obs=n_obs)
--> [215](https://vscode-remote+ssh-002dremote-002biris-002dgpu.vscode-resource.vscode-cdn.net/usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/rapids_singlecell/preprocessing/_neighbors/__init__.py:215) connectivities = _calc_connectivities(
    216     knn_indices,
    217     knn_dist,
    218     n_obs=n_obs,
    219     n_neighbors=n_neighbors,
    220     random_state=random_state,
    221     metric=metric,
    222     method=method,
    223 )
    224 if connectivities.nnz >= np.iinfo(np.int32).max:
    225     connectivities = connectivities.get().tocsr()

File /usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/rapids_singlecell/preprocessing/_neighbors/_neighbors.py:305, in _calc_connectivities(knn_indices, knn_dist, n_obs, n_neighbors, random_state, metric, method)
    299     return _get_connectivities_gauss(
    300         knn_indices,
    301         knn_dist,
    302         n_obs=n_obs,
    303     )
    304 if method == "jaccard":
--> [305](https://vscode-remote+ssh-002dremote-002biris-002dgpu.vscode-resource.vscode-cdn.net/usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/rapids_singlecell/preprocessing/_neighbors/_neighbors.py:305)     return _get_connectivities_jaccard(
    306         knn_indices,
    307         n_obs=n_obs,
    308         n_neighbors=n_neighbors,
    309     )
    310 return _get_connectivities_umap(
    311     knn_indices,
    312     knn_dist,
   (...)    316     metric=metric,
    317 )

File /usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/rapids_singlecell/preprocessing/_neighbors/_neighbors.py:246, in _get_connectivities_jaccard(knn_indices, n_obs, n_neighbors)
    244 i_idx = rows_adj
    245 j_idx = cols_adj
--> [246](https://vscode-remote+ssh-002dremote-002biris-002dgpu.vscode-resource.vscode-cdn.net/usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/rapids_singlecell/preprocessing/_neighbors/_neighbors.py:246) rows_i = adjacency[i_idx]
    247 rows_j = adjacency[j_idx]
    248 shared = cp.asarray(rows_i.multiply(rows_j).sum(axis=1)).ravel()

File /usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupyx/scipy/sparse/_index.py:381, in IndexMixin.__getitem__(self, key)
    379         return self._get_arrayXint(row, col)
    380     elif isinstance(col, slice):
--> [381](https://vscode-remote+ssh-002dremote-002biris-002dgpu.vscode-resource.vscode-cdn.net/usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupyx/scipy/sparse/_index.py:381)         return self._get_arrayXslice(row, col)
    382 else:  # row.ndim == 2
    383     if isinstance(col, _int_scalar_types):

File /usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupyx/scipy/sparse/_csr.py:678, in csr_matrix._get_arrayXslice(self, row, col)
    676     cols = cupy.arange(start, stop, step, self.indices.dtype)
    677     return self._get_arrayXarray(row, cols)
--> [678](https://vscode-remote+ssh-002dremote-002biris-002dgpu.vscode-resource.vscode-cdn.net/usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupyx/scipy/sparse/_csr.py:678) return self._major_index_fancy(row)._minor_slice(col)

File /usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupyx/scipy/sparse/_compressed.py:520, in _compressed_sparse_matrix._major_index_fancy(self, idx)
    516 if self.nnz == 0 or M == 0:
    517     return self._empty_like(new_shape)
    519 return self.__class__._from_parts(
--> [520](https://vscode-remote+ssh-002dremote-002biris-002dgpu.vscode-resource.vscode-cdn.net/usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupyx/scipy/sparse/_compressed.py:520)     *_index._csr_row_index(
    521         self.data, self.indices, self.indptr, idx),
    522     shape=new_shape)

File /usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupyx/scipy/sparse/_index.py:111, in _csr_row_index(Ax, Aj, Ap, rows)
    107 nnz = int(Bp[-1])  # synchronize!
    109 out_rows = _csr_indptr_to_coo_rows(nnz, Bp)
--> [111](https://vscode-remote+ssh-002dremote-002biris-002dgpu.vscode-resource.vscode-cdn.net/usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupyx/scipy/sparse/_index.py:111) Bj, Bx = _csr_row_index_ker(out_rows, rows, Ap, Aj, Ax, Bp)
    112 return Bx, Bj, Bp

File cupy/_core/_kernel.pyx:932, in cupy._core._kernel.ElementwiseKernel.__call__()

File cupy/_core/_kernel.pyx:957, in cupy._core._kernel.ElementwiseKernel._get_elementwise_kernel()

File cupy/_util.pyx:67, in cupy._util.memoize.decorator.ret()

File cupy/_core/_kernel.pyx:740, in cupy._core._kernel._get_elementwise_kernel()

File cupy/_core/_kernel.pyx:112, in cupy._core._kernel._get_simple_elementwise_kernel_from_code()

File cupy/_core/core.pyx:2592, in cupy._core.core.compile_with_cache()

File /usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupy/cuda/compiler.py:589, in _compile_module_with_cache(source, options, arch, extra_source, backend, enable_cooperative_groups, name_expressions, log_stream, jitify, to_ltoir)
    585     return _compile_with_cache_hip(
    586         source, options, arch, extra_source, backend,
    587         name_expressions, log_stream, cache_in_memory)
    588 else:
--> [589](https://vscode-remote+ssh-002dremote-002biris-002dgpu.vscode-resource.vscode-cdn.net/usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupy/cuda/compiler.py:589)     return _compile_with_cache_cuda(
    590         source, options, arch, extra_source, backend,
    591         enable_cooperative_groups, name_expressions, log_stream,
    592         cache_in_memory, jitify, to_ltoir)

File /usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupy/cuda/compiler.py:671, in _compile_with_cache_cuda(source, options, arch, extra_source, backend, enable_cooperative_groups, name_expressions, log_stream, cache_in_memory, jitify, to_ltoir)
    669     return cubin
    670 else:
--> [671](https://vscode-remote+ssh-002dremote-002biris-002dgpu.vscode-resource.vscode-cdn.net/usersoftware/peerd/markovn/.conda/envs/virchow-test/lib/python3.13/site-packages/cupy/cuda/compiler.py:671)     mod.load(cubin)
    672     if name_expressions:
    673         mod._enumerate_and_build_mapping(
    674             tuple(name_expressions))

File cupy/cuda/function.pyx:369, in cupy.cuda.function.Module.load()

File cupy/cuda/function.pyx:371, in cupy.cuda.function.Module.load()

File cupy_backends/cuda/api/driver.pyx:227, in cupy_backends.cuda.api.driver.moduleLoadData()

File cupy_backends/cuda/api/driver.pyx:64, in cupy_backends.cuda.api.driver.check_status()

CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

2.9M cells

fresh versions:
rsc 0.15.2
cupy 14.1.1

[Intron7]

@pfren1998 i saw something like this aswell. can you try tcp as the protocol.

[Intron7]

@mxposed i was able to reproduce the error. I'm working on a fix. it has to do with massive indices for the jaccard indexing

[pfren1998]

tcp works very well, really cool! Thank you!

[Intron7]

@pfren1998 please talk to your it department there is something wrong with pci routing on your node

[pfren1998]

@Intron7 Thanks again for your kind support!