Can only do stranded operations when both PyRanges contain strand info

[fairliereese]

Hi @endrebak and @marco-mariotti. This issue is a follow-up for the discussion in #374. As usual, apologies for the delays.

I have a minimal example including a small bed file that produces the error with version 0.1.2. As per my previous comment, the key difference here is converting between Pandas DataFrames and PyRanges objects:

import pyranges as pr
pr.__version__

0.1.2

# no conversion between data types; runs fine
df = pr.read_bed('test.bed')
df = df.merge(strand=True,
              count=True,
              count_col='n_genes',
              slack=0)

# converting from PyRanges to Pandas DataFrame and back; error
df = pr.read_bed('test.bed')
df = df.df
df = pr.PyRanges(df)
df = df.merge(strand=True,
              count=True,
              count_col='n_genes',
              slack=0) 

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[18], line 4
      2 df = df.df
      3 df = pr.PyRanges(df)
----> 4 df = df.merge(strand=True,
      5               count=True,
      6               count_col='n_genes',
      7               slack=0)

File [~/miniconda3/envs/cerberus_3/lib/python3.9/site-packages/pyranges/pyranges_main.py:3139](http://localhost:8890/lab/tree/data/pseudomasked_genomic_isoquant_guided/raw/~/miniconda3/envs/cerberus_3/lib/python3.9/site-packages/pyranges/pyranges_main.py#line=3138), in PyRanges.merge(self, strand, count, count_col, by, slack)
   3136     kwargs["sparse"] = {"self": True}
   3137     from pyranges.methods.merge import _merge
-> 3139     df = pyrange_apply_single(_merge, self, **kwargs)
   3140 else:
   3141     kwargs["sparse"] = {"self": False}

File [~/miniconda3/envs/cerberus_3/lib/python3.9/site-packages/pyranges/multithreaded.py:320](http://localhost:8890/lab/tree/data/pseudomasked_genomic_isoquant_guided/raw/~/miniconda3/envs/cerberus_3/lib/python3.9/site-packages/pyranges/multithreaded.py#line=319), in pyrange_apply_single(function, self, **kwargs)
    317 function, get, _merge_dfs = get_multithreaded_funcs(function, nb_cpu=nb_cpu)
    319 if strand:
--> 320     assert self.stranded, "Can only do stranded operation when PyRange contains strand info"
    322 results = []
    324 if strand:

AssertionError: Can only do stranded operation when PyRange contains strand info

As I mentioned in this comment, I was able to circumvent this behavior in my branch by commenting out a check for different categories in the strand column (see this commit in my branch). I would be happy to help put a more permanent solution into place for this library but wanted to check with you all first before committing to something.

[marco-mariotti]

thanks @fairliereese .
Would you be so kind as to check if this issue remains in the upcoming version 1 of pyranges?
https://github.com/pyranges/pyranges_1.x

There, pyranges is a direct subclass of dataframe. We're focusing our development on that.
You'll need to change 'merge' to 'merge_overlaps', as per
https://pyranges1.readthedocs.io/en/latest/migration_guide.html

[mlbendall]

Thanks @fairliereese for this, I am having a related issue when doing a PyRanges -> DataFrame -> PyRanges conversion:

>>> import pyranges as pr
>>> pr.__version__
'0.1.2'

Also using PyRanges version 0.1.2. Now create a very simple PyRanges with strand information:

>>> dummy_pr = pr.PyRanges(chromosomes=['chr1'], strands=['+'], starts=[5], ends=[10])
>>> dummy_pr
+--------------+-----------+-----------+--------------+
| Chromosome   |     Start |       End | Strand       |
| (category)   |   (int64) |   (int64) | (category)   |
|--------------+-----------+-----------+--------------|
| chr1         |         5 |        10 | +            |
+--------------+-----------+-----------+--------------+
Stranded PyRanges object has 1 rows and 4 columns from 1 chromosomes.
For printing, the PyRanges was sorted on Chromosome and Strand.
>>> dummy_pr.stranded
True

The resulting PyRanges is stranded. Now convert the PyRanges to a DataFrame and back to a PyRanges. (This is a toy example, my use case would be to perform some manipulations on the DataFrame in Pandas before converting back)

>>> converted_pr = pr.PyRanges(dummy_pr.df)
>>> converted_pr
+--------------+-----------+-----------+--------------+
| Chromosome   |     Start |       End | Strand       |
| (category)   |   (int64) |   (int64) | (category)   |
|--------------+-----------+-----------+--------------|
| chr1         |         5 |        10 | +            |
+--------------+-----------+-----------+--------------+
Unstranded PyRanges object has 1 rows and 4 columns from 1 chromosomes.
For printing, the PyRanges was sorted on Chromosome.
>>> converted_pr.stranded
False

The converted PyRanges has stranded == False and stranded operations will not work.

>>> merged_pr = converted_pr.merge(strand=True, count=True)
Traceback (most recent call last):
  File "/Users/bendall/Applications/PyCharm Professional.app/Contents/plugins/python/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "/Users/bendall/mambaforge/envs/test_pyranges/lib/python3.10/site-packages/pyranges/pyranges_main.py", line 3139, in merge
    df = pyrange_apply_single(_merge, self, **kwargs)
  File "/Users/bendall/mambaforge/envs/test_pyranges/lib/python3.10/site-packages/pyranges/multithreaded.py", line 320, in pyrange_apply_single
    assert self.stranded, "Can only do stranded operation when PyRange contains strand info"
AssertionError: Can only do stranded operation when PyRange contains strand info

I think the solution proposed by @fairliereese works to correctly call a PyRanges "stranded" when every row has a proper strand (using only the second condition). However the root of the issue is that the dtype for "Strand" is set to categorical with ".", "-", and "+" as categories (see this line in pyranges.methods.init), so the set difference (set(df.Strand.cat.categories) - set("+-")) will always contain '.' and will evaluate to True (at least for DataFrames coming directly from a PyRanges, see this line).

A short term solution is to simply remove the categorical from Strand, so your round trip goes from PyRanges -> DataFrame -> retyped DataFrame -> PyRanges. For example:

>>> dummy_pr = pr.PyRanges(chromosomes=['chr1'], strands=['+'], starts=[5], ends=[10])
>>> dummy_pr.stranded
True
>>> fixed_pr = pr.PyRanges(dummy_pr.df.astype({'Strand': str}))
>>> fixed_pr
+--------------+-----------+-----------+--------------+
| Chromosome   |     Start |       End | Strand       |
| (category)   |   (int64) |   (int64) | (category)   |
|--------------+-----------+-----------+--------------|
| chr1         |         5 |        10 | +            |
+--------------+-----------+-----------+--------------+
Stranded PyRanges object has 1 rows and 4 columns from 1 chromosomes.
For printing, the PyRanges was sorted on Chromosome and Strand.
>>> fixed_pr.stranded
True

Excited for pyranges1, thanks @marco-mariotti @endrebak