Migrating to ncls-cpp for performant interval and search operations (#152)

Bulk of the work was done in the IRanges package. Also update relevant tests & documentation.

Author: jkanche

.github/workflows/publish-pypi.yml

@@ -1,6 +1,3 @@
-# This workflow will install Python dependencies, run tests and lint with a single version of Python
-# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
-
 name: Publish to PyPI
 
 on:
@@ -19,10 +16,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Set up Python 3.11
+      - name: Set up Python 3.12
         uses: actions/setup-python@v5
         with:
-          python-version: 3.11
+          python-version: 3.12
 
       - name: Install dependencies
         run: |
@@ -33,6 +30,14 @@ jobs:
         run: |
           tox
 
+      - name: Build Project and Publish
+        run: |
+          python -m tox -e clean,build
+
+      # This uses the trusted publisher workflow so no token is required.
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+
       - name: Build docs
         run: |
           tox -e docs
@@ -45,11 +50,3 @@ jobs:
           branch: gh-pages # The branch the action should deploy to.
           folder: ./docs/_build/html
           clean: true # Automatically remove deleted files from the deploy branch
-
-      - name: Build Project and Publish
-        run: |
-          python -m tox -e clean,build
-
-      # This uses the trusted publisher workflow so no token is required.
-      - name: Publish to PyPI
-        uses: pypa/gh-action-pypi-publish@release/v1

.github/workflows/run-tests.yml

@@ -1,32 +1,73 @@
-name: Run tests
+name: Test the library
 
 on:
   push:
-    branches: [master]
+    branches:
+      - master # for legacy repos
+      - main
   pull_request:
+    branches:
+      - master # for legacy repos
+      - main
+  workflow_dispatch: # Allow manually triggering the workflow
+  schedule:
+    # Run roughly every 15 days at 00:00 UTC
+    # (useful to check if updates on dependencies break the package)
+    - cron: "0 0 1,16 * *"
+
+permissions:
+  contents: read
+
+concurrency:
+  group: >-
+    ${{ github.workflow }}-${{ github.ref_type }}-
+    ${{ github.event.pull_request.number || github.sha }}
+  cancel-in-progress: true
 
 jobs:
-  build:
-    runs-on: ubuntu-latest
+  test:
     strategy:
       matrix:
-        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
-
-    name: Python ${{ matrix.python-version }}
+        python: ["3.9", "3.10", "3.11", "3.12", "3.13"]
+        platform:
+          - ubuntu-latest
+          - macos-latest
+          # - windows-latest
+    runs-on: ${{ matrix.platform }}
+    name: Python ${{ matrix.python }}, ${{ matrix.platform }}
     steps:
       - uses: actions/checkout@v4
 
-      - name: Setup Python
-        uses: actions/setup-python@v5
+      - uses: actions/setup-python@v5
+        id: setup-python
         with:
-          python-version: ${{ matrix.python-version }}
-          cache: "pip"
+          python-version: ${{ matrix.python }}
 
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install tox
+          pip install tox coverage
 
-      - name: Test with tox
-        run: |
+      - name: Run tests
+        run: >-
+          pipx run --python '${{ steps.setup-python.outputs.python-path }}'
           tox
+          -- -rFEx --durations 10 --color yes --cov --cov-branch --cov-report=xml  # pytest args
+
+      - name: Check for codecov token availability
+        id: codecov-check
+        shell: bash
+        run: |
+          if [ ${{ secrets.CODECOV_TOKEN }} != '' ]; then
+            echo "codecov=true" >> $GITHUB_OUTPUT;
+          else
+            echo "codecov=false" >> $GITHUB_OUTPUT;
+          fi
+
+      - name: Upload coverage reports to Codecov with GitHub Action
+        uses: codecov/codecov-action@v5
+        if: ${{ steps.codecov-check.outputs.codecov == 'true' }}
+        env:
+          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+          slug: ${{ github.repository }}
+          flags: ${{ matrix.platform }} - py${{ matrix.python }}

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## Version 0.7.0
+
+- Changes to switch to LTLA/nclist-cpp in the iranges package for overlap and search operations.
+
 ## Version 0.6.2 - 0.6.3
 
 - Implement biocutil's `extract_row_names` generic.
@@ -92,7 +96,7 @@ This is a complete rewrite of both these classes following the functional paradi
 
 The package does not rely on pandas anymore. While we try to provide backwards compatibility to construct a GenomicRanges object from a pandas dataframe using the `from_pandas` method, please note that the default constructor to genomic ranges does not accept a pandas data frame anymore!
 
-Most range based methods have been reimplemented and the heavy lifting is done in the [IRanges package](https://github.com/BiocPy/IRanges) for interval operations. The package indirectly depends on [NCLS](https://github.com/pyranges/ncls) interval tree data structure to perform search and overlap operations.
+Most range based methods have been reimplemented and the heavy lifting is done in the [IRanges package](https://github.com/BiocPy/IRanges) for interval operations. The package depends on [nclist-cpp](https://github.com/LTLA/nclist-cpp) data structure to perform search and overlap operations.
 
 Tests, documentation and readme has been updated to reflect these changes.
 

README.md

@@ -24,6 +24,9 @@ pip install genomicranges[optional]
 
 ### From Bioinformatic file formats
 
+> [!NOTE]
+> When reading genomic formats, `ends` are expected to be inclusive to be consistent with Bioconductor representations (& gff). If they are not, we recommend subtracting 1 from the `ends`.
+
 #### From `biobear`
 
 Although the parsing capabilities in this package are limited, the [biobear](https://github.com/wheretrue/biobear) library is designed for reading and searching various bioinformatics file formats, including FASTA, FASTQ, VCF, BAM, and GFF, or from an object store like S3. Users can esily convert these representations to `GenomicRanges` (or [read more here](https://www.wheretrue.dev/docs/exon/biobear/genomicranges-integration)):
@@ -44,8 +47,8 @@ print(len(gg), len(df))
 ```
 
     ## output
-    ## 77 77
-
+    ## 77 77> [!NOTE]
+> `ends` are expected to be inclusive to be consistent with Bioconductor representations. If they are not, we recommend subtracting 1 from the `ends`.
 
 #### UCSC or GTF file
 

docs/tutorial.md

@@ -8,7 +8,7 @@ kernelspec:
 
 `GenomicRanges` is a Python package designed to handle genomic locations and facilitate genomic analysis. It is similar to Bioconductor's [GenomicRanges](https://bioconductor.org/packages/release/bioc/html/GenomicRanges.html) and uses the [IRanges](https://github.com/BiocPy/IRanges) package under the hood to manage and provide interval-based arithmetic operations.
 
-An `IRanges` holds a **start** position and a **width**, and is typically used to represent coordinates along a genomic sequence. The interpretation of the **start** position depends on the application; for sequences, the **start** is usually a 1-based position, but other use cases may allow zero or even negative values, e.g., circular genomes. Ends are considered inclusive. `IRanges` uses [nested containment lists](https://github.com/pyranges/ncls) under the hood to perform fast overlap and search-based operations.
+An `IRanges` holds a **start** position and a **width**, and is typically used to represent coordinates along a genomic sequence. The interpretation of the **start** position depends on the application; for sequences, the **start** is usually a 1-based position, but other use cases may allow zero or even negative values, e.g., circular genomes. Ends are considered inclusive. `IRanges` uses [LTLa/nclist-cpp](https://github.com/LTLA/nclist-cpp) under the hood to perform fast overlap and search-based operations.
 
 The package provides a `GenomicRanges` class to specify multiple genomic elements, typically where genes start and end. Genes are themselves made of many subregions, such as exons, and a `GenomicRangesList` enables the representation of this nested structure.
 

pyproject.toml

@@ -11,17 +11,14 @@ version_scheme = "no-guess-dev"
 line-length = 120
 src = ["src"]
 # exclude = ["tests"]
-extend-ignore = ["F821"]
+lint.extend-ignore = ["F821"]
 
-[tool.ruff.pydocstyle]
+[tool.ruff.lint.pydocstyle]
 convention = "google"
 
 [tool.ruff.format]
 docstring-code-format = true
 docstring-code-line-length = 20
 
-[tool.ruff.per-file-ignores]
+[tool.ruff.lint.per-file-ignores]
 "__init__.py" = ["E402", "F401"]
-
-[tool.black]
-force-exclude = "__init__.py"

setup.cfg

@@ -50,7 +50,7 @@ python_requires = >=3.9
 install_requires =
     importlib-metadata; python_version<"3.8"
     biocframe>=0.6.2
-    iranges>=0.4.2
+    iranges>=0.5.1
     biocutils>=0.2.1
     numpy