Numerical Types issue

[rpadmanabhan]

Hi,
Thanks for maintaining this great library. We are running into an issue on Windows with some of the methods. We first noticed this in the original pyranges package and see that there are already a couple of issues on this topic there. We were hoping this new re-write might have fixed that but we see a similar problem :

>>> foo
index    |    Chromosome    Start     End       Strand      gene_name
int64    |    category      uint32    uint32    category    object
-------  ---  ------------  --------  --------  ----------  ---------------
0        |    GL000008.2    83331     200303    +           ENSG00000296732
1        |    GL000008.2    152368    170429    -           ENSG00000296775
2        |    GL000008.2    177109    208612    -           ENSG00000296756
3        |    GL000009.2    5120      35307     +           ENSG00000297619
...      |    ...           ...       ...       ...         ...
54855    |    chrY          26121657  26135789  +           ENSG00000293280
54856    |    chrY          26409157  26508246  +           ENSG00000294742
54857    |    chrY          26493591  26497550  +           ENSG00000294758
54858    |    chrY          56881848  56882115  +           ENSG00000305394
PyRanges with 54859 rows, 5 columns, and 1 index columns.
Contains 69 chromosomes and 2 strands.
>>> foo.cluster_overlaps()
Traceback (most recent call last):
  File "C:\venvs\pyranges1_test\Lib\site-packages\ruranges\__init__.py", line 1209, in _resolve_rust_fn
    suffix, tgt_grp, tgt_pos = _SUFFIX_TABLE[(grp_dt, pos_dt)]
                               ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
KeyError: (dtype('uint32'), dtype('uint32'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\venvs\pyranges1_test\Lib\site-packages\pyranges\core\pyranges_main.py", line 745, in cluster_overlaps
    result = super().cluster_overlaps(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venvs\pyranges1_test\Lib\site-packages\pyranges\range_frame\range_frame.py", line 251, in cluster_overlaps
    cluster, idx = ruranges.cluster(  # type: ignore[attr-defined]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venvs\pyranges1_test\Lib\site-packages\ruranges\__init__.py", line 474, in cluster
    return _dispatch_unary(
           ^^^^^^^^^^^^^^^^
  File "C:\venvs\pyranges1_test\Lib\site-packages\ruranges\__init__.py", line 1236, in _dispatch_unary
    rust_fn, grp_t, pos_t = _resolve_rust_fn(prefix, groups.dtype if groups is not None else None, starts.dtype)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venvs\pyranges1_test\Lib\site-packages\ruranges\__init__.py", line 1211, in _resolve_rust_fn
    raise TypeError(f"Unsupported dtype pair: {grp_dt}, {pos_dt}") from exc
TypeError: Unsupported dtype pair: uint32, uint32

Do you think this is something you would be able to address to add support for Windows in the near future ? If you had any immediate ideas/suggestions on how to address the issue I can look into it as well.

Thank you
Raghav

[endrebak]

I have no way of testing on Windows, sorry.

[marco-mariotti]

Unfortunately we cannot support Windows at this moment. We may do later on, when our dev team gets larger.

Leaving the issue open a little longer to:

• clarify OS compatibility in Installation page

[endrebak]

Come to think of it, I do not see why this should break on Windows. The error messages just says that it does not find a compiled function to run cluster for groups with type uint32 and position types with uint32. I do not see how it is windows specific.

[endrebak]

These are the functions defined:

define_cluster_numpy!(cluster_numpy_u64_i64, u64, i64);
define_cluster_numpy!(cluster_numpy_u32_i64, u32, i64);
define_cluster_numpy!(cluster_numpy_u32_i32, u32, i32);
define_cluster_numpy!(cluster_numpy_u32_i16, u32, i16);
define_cluster_numpy!(cluster_numpy_u16_i64, u16, i64);
define_cluster_numpy!(cluster_numpy_u16_i32, u16, i32);
define_cluster_numpy!(cluster_numpy_u16_i16, u16, i16);
define_cluster_numpy!(cluster_numpy_u8_i64,  u8,  i64);
define_cluster_numpy!(cluster_numpy_u8_i32,  u8,  i32);
define_cluster_numpy!(cluster_numpy_u8_i16,  u8,  i16);

It is trying to look up uint32, uint32. We have a u32 and an i32 version of the function. Did you set the datatypes of the dataframe to u32 for the Starts and Ends? Can you try just using regular ints?

[endrebak]

There is really no reason it should not work on windows. Can you try to reproduce by setting the datatypes to uint32 for start and end Mario? I do not have access to a computer for a little while.

I think writing it does not work on Windows is premature. I think it should work on Windows and that this is due to something else. We can try adding CI for windows

[marco-mariotti]

I have no experience with Windows, and no easy access. Perhaps @rpadmanabhan or some other non-core dev that ended in this issue page wants to give it a shot?
Establishing CI for windows is a nice goal, would allow at least to detect (if not debug) Win-specific problems when they arise

[endrebak]

But I do not think this is a Windows problem. I think that if you created a data frame on Mac OS where the starts and ends were uint you'd run into the same problem. So do not write anything about this not working for Windows because I think it isn't true. It should work on Windows

[endrebak]

I have a Linux machine that would be able to run Windows but I am not going to pay for a Windows license 😂

[endrebak]

This is not a windows issue. pyranges should work perfectly well on windows.

I can reproduce by doing

In [7]: gr = gr.astype({"Start": np.uint32, "End": np.uint32})

In [9]: gr.cluster_overlaps()
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/code/ruranges/ruranges/__init__.py:1209, in _resolve_rust_fn(prefix, grp_dt, pos_dt)
   1208 try:
-> 1209     suffix, tgt_grp, tgt_pos = _SUFFIX_TABLE[(grp_dt, pos_dt)]
   1210 except KeyError as exc:

KeyError: (dtype('uint32'), dtype('uint32'))

[endrebak]

@marco-mariotti I think another selling point of pyranges1 is that it is Windows compatible. The error above is due to us not supporting all possible dtype combinations:

define_chromsweep_numpy!(chromsweep_numpy_u64_i64, u64, i64);
define_chromsweep_numpy!(chromsweep_numpy_u32_i64, u32, i64);
define_chromsweep_numpy!(chromsweep_numpy_u32_i32, u32, i32);
define_chromsweep_numpy!(chromsweep_numpy_u32_i16, u32, i16);
define_chromsweep_numpy!(chromsweep_numpy_u16_i64, u16, i64);
define_chromsweep_numpy!(chromsweep_numpy_u16_i32, u16, i32);
define_chromsweep_numpy!(chromsweep_numpy_u16_i16, u16, i16);
define_chromsweep_numpy!(chromsweep_numpy_u8_i64,  u8,  i64);
define_chromsweep_numpy!(chromsweep_numpy_u8_i32,  u8,  i32);
define_chromsweep_numpy!(chromsweep_numpy_u8_i16,  u8,  i16);

[marco-mariotti]

Is this still an issue?

[endrebak]

I opened a related issue about us not covering all combinations of dtypes.

But we might want to keep it open until that person in your lab proves it works on windows.

[marco-mariotti]

Renaming issue to reflect its nature.