read_gtf() fails on RefSeq GTF due to semicolons inside quoted attribute values

[JackCurragh]

pyranges1.read_gtf() currently fails on valid RefSeq GTF files where an attribute value contains a semicolon inside quotes.

Example attribute field from a RefSeq GTF:

gene_id "SELENOO"; transcript_id "NM_001115017.5"; db_xref "GeneID:417745"; gbkey "CDS"; gene "SELENOO"; note "UGA stop codon recoded as selenocysteine; The RefSeq protein has 1 substitution compared to this genomic sequence"; product "protein adenylyltransferase SelO, mitochondrial"; protein_id "NP_001108489.5"; transl_except "(pos:20486313..20486315,aa:Sec)"; exon_number "1";

Running:

import pyranges1 as pr
pr.read_gtf("GCF_016699485.2_bGalGal1.mat.broiler.GRCg7b_genomic.gtf")

produces:

ValueError: dictionary update sequence element #5 has length 1; 2 is required

This appears to come from parse_kv_fields() splitting the attribute column on ; without accounting for quoted values.

A quote-aware split (only splitting on semicolons outside quotes) would resolve this. Would you like me to submit a PR?

[marco-mariotti]

Thanks Jack. Yes, a PR fix would be great

[endrebak]

A quote aware split would likely be much slower. But we should support this. So I propose I do it. If it is slower I'll put it behind a boolean guard (parse_quotes?)

[endrebak]

I just tested it with timeit in the REPL and a regular expression that handles quotes is only 15% slower than the current function. And the attribute parsing isn't such a large part of the function either so the runtime will only increase with a few percent probably.

[endrebak]

I think it is likely that this version will be just as fast because when you write it as a regular expression you don't need all the splits and strips that we currently have.

[endrebak]

I see the file referenced in the issue (GCF_016699485.2) ends with ###

Is that valid gtf?

[endrebak]

This new version is 14% faster!

diff --git a/pyranges1/readers.py b/pyranges1/readers.py
index 5ca9dc7..b37a8e5 100644
--- a/pyranges1/readers.py
+++ b/pyranges1/readers.py
@@ -435,7 +435,8 @@ def read_gtf_full(

 def parse_kv_fields(line: str) -> list[list[str]]:
     """Parse GTF attribute column."""
-    return [kv.replace('""', '"NA"').replace('"', "").split(None, 1) for kv in line.rstrip("; ").split("; ")]
+    parts = line.split('"')
+    return [(parts[i - 1].rsplit(";", 1)[-1].strip(), parts[i]) for i in range(1, len(parts), 2)]

The change is that instead of splitting on ; we split on " as a hack.

[endrebak]

Works on the previously failing file:

endrebakkenstovner@Mac ~> gtime -v python test_gtf.py  # Downloads/ncbi_dataset/data/GCF_016699485.2/genomic.gtf
index    |    Chromosome    Source    Feature     Start    End      Score    Strand      Frame    gene_id       ...
int64    |    str           str       category    int64    int64    str      category    str      str           ...
-------  ---  ------------  --------  ----------  -------  -------  -------  ----------  -------  ------------  -----
0        |    NC_052532.1   Gnomon    gene        13549    19518    .        -           .        LOC124418406  ...
1        |    NC_052532.1   Gnomon    transcript  13549    19518    .        -           .        LOC124418406  ...
2        |    NC_052532.1   Gnomon    exon        19483    19518    .        -           .        LOC124418406  ...
3        |    NC_052532.1   Gnomon    exon        18067    18219    .        -           .        LOC124418406  ...
...      |    ...           ...       ...         ...      ...      ...      ...         ...      ...           ...
2297008  |    NC_053523.1   RefSeq    stop_codon  14961    14964    .        -           0        J6367_mgp01   ...
2297009  |    NC_053523.1   RefSeq    gene        15485    15553    .        -           .        J6367_mgt22   ...
2297010  |    NC_053523.1   RefSeq    transcript  15485    15553    .        -           .        J6367_mgt22   ...
2297011  |    NC_053523.1   RefSeq    exon        15485    15553    .        -           .        J6367_mgt22   ...
PyRanges with 2297012 rows, 34 columns, and 1 index columns. (25 columns not shown: "transcript_id", "db_xref", "gbkey", ...).
Contains 151 chromosomes and 2 strands.
	Command being timed: "python test_gtf.py"
	User time (seconds): 15.86
	System time (seconds): 2.18
	Percent of CPU this job got: 119%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.05
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 1860016
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 79
	Minor (reclaiming a frame) page faults: 468099
	Voluntary context switches: 2
	Involuntary context switches: 1097
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 3
	Page size (bytes): 16384
	Exit status: 0

[marco-mariotti]

Hi, just a note, before pushing to main repo, make sure the new version works with some gtf files from ensembl and from NCBI at least Marco Mariotti

…

On Sat, Mar 14, 2026, 08:18 endrebak ***@***.***> wrote: *endrebak* left a comment (pyranges/pyranges1#138) <#138 (comment)> Works on the previously failing file: ***@***.*** ~> gtime -v python test_gtf.py # Downloads/ncbi_dataset/data/GCF_016699485.2/genomic.gtf index | Chromosome Source Feature Start End Score Strand Frame gene_id ... int64 | str str category int64 int64 str category str str ... ------- --- ------------ -------- ---------- ------- ------- ------- ---------- ------- ------------ ----- 0 | NC_052532.1 Gnomon gene 13549 19518 . - . LOC124418406 ... 1 | NC_052532.1 Gnomon transcript 13549 19518 . - . LOC124418406 ... 2 | NC_052532.1 Gnomon exon 19483 19518 . - . LOC124418406 ... 3 | NC_052532.1 Gnomon exon 18067 18219 . - . LOC124418406 ... ... | ... ... ... ... ... ... ... ... ... ... 2297008 | NC_053523.1 RefSeq stop_codon 14961 14964 . - 0 J6367_mgp01 ... 2297009 | NC_053523.1 RefSeq gene 15485 15553 . - . J6367_mgt22 ... 2297010 | NC_053523.1 RefSeq transcript 15485 15553 . - . J6367_mgt22 ... 2297011 | NC_053523.1 RefSeq exon 15485 15553 . - . J6367_mgt22 ... PyRanges with 2297012 rows, 34 columns, and 1 index columns. (25 columns not shown: "transcript_id", "db_xref", "gbkey", ...). Contains 151 chromosomes and 2 strands. Command being timed: "python test_gtf.py" User time (seconds): 15.86 System time (seconds): 2.18 Percent of CPU this job got: 119% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.05 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 1860016 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 79 Minor (reclaiming a frame) page faults: 468099 Voluntary context switches: 2 Involuntary context switches: 1097 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 3 Page size (bytes): 16384 Exit status: 0 — Reply to this email directly, view it on GitHub <#138 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACTXRIEXQ7KA6JCUAGWMIXT4QUBTZAVCNFSM6AAAAACWRG3MV6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANJZHA4TIMBYGY> . You are receiving this because you commented.Message ID: ***@***.***>

[endrebak]

I have done so.

By the way I get an 80 percent speedup with Cython. PyRanges was already the fastest at GTF parsing, even faster than rust libraries.

I think this is worth putting in a library (pyreaders?). We can have the compiled version behind a bool flag.

33 seconds vs 58 for the largest gencode gtf.

[endrebak]

Can you check if it is fixed now? 1.3.4 out on pypi

[JackCurragh]

This issue can likelt be closed as quoted attribute values are handled correctly.

However, this refseq gtf fails to load because the last line is ###. The commented lines seem to be handled correctly but now ### gets passed to parse_kv_fields as nan leading to AttributeError: 'float' object has no attribute 'split'. Even with ignore_bad=True...

import pyranges1 as pr

gtf = "GCF_016699485.2_bGalGal1.mat.broiler.GRCg7b_genomic.gtf"
gr = pr.read_gtf(gtf, ignore_bad=True)
print("Loaded successfully")
print(gr)

python -m pip freeze
natsort==8.4.0
numpy==2.4.3
pandas==3.0.1
pyranges1==1.3.4
python-dateutil==2.9.0.post0
ruranges==0.1.2
six==1.17.0
tabulate==0.10.0

[endrebak]

I know but that is invalid GTF :) I think GFF3 can separate features like that but not gtf

[endrebak]

Feel free to open an issue for that other bug