Error when running read_gtf() on RefSeq MANE GTF

[isaacvock]

Hello,

I downloaded a GTF file from the MANE project (link; code to download specific GTF: wget https://ftp.ncbi.nlm.nih.gov/refseq/MANE/MANE_human/release_1.4/MANE.GRCh38.v1.4.refseq_genomic.gtf.gz) and then attempted to read it into a Python session with pyranges (version 0.1.4) with pr.read_gtf('path/to/gtf'). This threw the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/isaacvock/mambaforge/envs/biopython/lib/python3.12/site-packages/pyranges/readers.py", line 380, in read_gtf
    gr = read_gtf_full(
         ^^^^^^^^^^^^^^
  File "/home/isaacvock/mambaforge/envs/biopython/lib/python3.12/site-packages/pyranges/readers.py", line 423, in read_gtf_full
    extra = _to_rows(df.Attribute, ignore_bad=ignore_bad)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/isaacvock/mambaforge/envs/biopython/lib/python3.12/site-packages/pyranges/readers.py", line 460, in to_rows
    rowdicts.append({k: v for k, v in parse_kv_fields(line)})
                                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/isaacvock/mambaforge/envs/biopython/lib/python3.12/site-packages/pyranges/readers.py", line 442, in parse_kv_fields
    return [kv.replace('""', '"NA"').replace('"', "").split(None, 1) for kv in line.rstrip("; ").split("; ")]
                                                                               ^^^^^^^^^^^
AttributeError: 'float' object has no attribute 'rstrip'

Error reproduces in two separate conda environments with version 0.1.4 installed and two systems (one running WSL and the other Ubuntu). I confirmed that I can read the gtf file without error in R using rtracklayer::import(). Any help resolving this issue would be much appreciated.

[EDIT]: The error also appears in version 0.0.129. Also, the code runs with no error if you use the Ensembl MANE annotation rather than the RefSeq one.

Best,
Isaac

[endrebak]

Quick fix:

Store the parsed file in R as a tsv and import in PyRanges.

[endrebak]

I'll have a look.

Also, we have a new pyranges out: https://github.com/pyranges/pyranges_1.x

It is faster and better and it makes the PyRanges a subclass of pandas dataframes so they are not custom objects anymore

[endrebak]

@marco-mariotti I propose we have a debug version of the gtf/gff functions. Where you can pass in a debug flag. I do not want it in the regular code as it would make the function slower.

[endrebak]

Can reproduce:

In [1]: import pyranges as pr

In [2]: df = pr.read_gff("MANE.GRCh38.v1.4.refseq_genomic.gtf.gz")
ERROR:pyranges.readers:The following line is not parseable as gtf:
nan

To ignore bad lines use ignore_bad=True.
Traceback (most recent call last):
  File "/Users/endrebakkenstovner/ruranges_new_venv/lib/python3.13/site-packages/pyranges/readers.py", line 455, in to_rows
    rowdicts.append(dict(parse_kv_fields(line)))
                    ~~~~^^^^^^^^^^^^^^^^^^^^^^^
ValueError: dictionary update sequence element #0 has length 1; 2 is required
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 1
----> 1 df = pr.read_gff("MANE.GRCh38.v1.4.refseq_genomic.gtf.gz")

File ~/ruranges_new_venv/lib/python3.13/site-packages/pyranges/readers.py:353, in read_gtf(f, nrows, full, duplicate_attr, ignore_bad)
    350 _skiprows = find_first_data_line_index(path)
    352 if full:
--> 353     gr = read_gtf_full(
    354         path,
    355         nrows=nrows,
    356         skiprows=_skiprows,
    357         duplicate_attr=duplicate_attr,
    358         ignore_bad=ignore_bad,
    359     )
    360 else:
    361     gr = read_gtf_restricted(path, _skiprows, nrows=None)

File ~/ruranges_new_venv/lib/python3.13/site-packages/pyranges/readers.py:423, in read_gtf_full(f, nrows, skiprows, chunksize, duplicate_attr, ignore_bad)
    421 dfs = []
    422 for df in df_iter:
--> 423     extra = _to_rows(df.Attribute.astype(str), ignore_bad=ignore_bad)
    424     _df = df.drop("Attribute", axis=1)
    425     extra = extra.set_index(_df.index)

File ~/ruranges_new_venv/lib/python3.13/site-packages/pyranges/readers.py:455, in to_rows(anno, ignore_bad)
    453 try:
    454     for line in anno:
--> 455         rowdicts.append(dict(parse_kv_fields(line)))
    456 except ValueError:
    457     if not ignore_bad:

ValueError: dictionary update sequence element #0 has length 1; 2 is required

[endrebak]

Your file is not valid gtf:

gunzip -c MANE.GRCh38.v1.4.refseq_genomic.gtf.gz | tail -2
chr19_KI270930v1_alt	BestRefSeq	stop_codon	134959	134961	.	-	0	gene_id "KIR2DL5B_5"; transcript_id "NM_001018081.2"; db_xref "Ensembl:ENSP00000483362.1"; db_xref "GeneID:553128"; db_xref "GenBank:NP_001018091.2"; db_xref "HGNC:HGNC:16346"; db_xref "MIM:615727"; gbkey "CDS"; gene "KIR2DL5B"; product "killer cell immunoglobulin-like receptor 2DL5B precursor"; protein_id "NP_001018091.2"; tag "MANE Select"; exon_number "8";
###

The error message says:

ERROR:pyranges.readers:The following line is not parseable as gtf:
nan

To ignore bad lines use ignore_bad=True.

You can use the ignore_bad argument to read the file.

[endrebak]

Thanks for reporting this btw :)

[isaacvock]

Thank you for your quick response! Reading the gtf file in R, writing to a tsv, reading it in Python via pandas, and converting to a PyRanges object worked, so thank you for that suggestion.

On my end, running read_gtf() with ignore_bad=True did not work, throwing the same error as before:

>>> gtf = pr.read_gtf('../Data/MANE.GRCh38.v1.4.refseq_genomic.gtf.gz', ignore_bad=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/isaacvock/mambaforge/envs/biopython/lib/python3.12/site-packages/pyranges/readers.py", line 380, in read_gtf
    gr = read_gtf_full(
         ^^^^^^^^^^^^^^
  File "/home/isaacvock/mambaforge/envs/biopython/lib/python3.12/site-packages/pyranges/readers.py", line 423, in read_gtf_full
    extra = _to_rows(df.Attribute, ignore_bad=ignore_bad)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/isaacvock/mambaforge/envs/biopython/lib/python3.12/site-packages/pyranges/readers.py", line 460, in to_rows
    rowdicts.append({k: v for k, v in parse_kv_fields(line)})
                                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/isaacvock/mambaforge/envs/biopython/lib/python3.12/site-packages/pyranges/readers.py", line 442, in parse_kv_fields
    return [kv.replace('""', '"NA"').replace('"', "").split(None, 1) for kv in line.rstrip("; ").split("; ")]
                                                                               ^^^^^^^^^^^
AttributeError: 'float' object has no attribute 'rstrip'

Best,
Isaac

[endrebak]

Strange. The problem was that the file ended with triple hashes which is not valid GTF. Anyways, I'm happy you got it working