Hello,
I’ve been testing longcallD for somatic structural variant detection, for tumor only, and am seeing poor performance.
I first filtered the output VCF for the SOMATIC and SVTYPE tags. I then evaluated the unfiltered VCF, and finally used Severus as a comparator. The corresponding TrueVari summary reports are included below.
Do you have any suggestions for what might be going wrong?
One notable observation is that 35 of 149 truth variants appear to have been incorrectly labeled as germline.
Truth dataset
NIST_HG008-T_somatic-stvar_DraftBenchmark_V0.3-20250220/GRCh38_HG008-T-V0.3_somatic-stvar_PASS.draftbenchmark.vcf.gz 31a0db52bb62a4ba501f6b3b03e71d9f
NIST_HG008-T_somatic-stvar_DraftBenchmark_V0.3-20250220/GRCh38_HG008-T-V0.3_somatic-stvar.draftbenchmark.bed 64c029a88a747abb35e7fbee60538a23
longcallD command
longcallD call \
--threads 16 \
-s human_GRCh38_no_alt_analysis_set.fasta \
HG008.tumor.aligned.hiphase.bam \
> phased.vcf
Filtered (SOMATIC + SVTYPE)
{
"TP-base": 5,
"TP-comp": 5,
"FP": 250,
"FN": 142,
"precision": 0.0196078431372549,
"recall": 0.034013605442176874,
"f1": 0.024875621890547265,
"base cnt": 147,
"comp cnt": 255,
"TP-comp_TP-gt": 5,
"TP-comp_FP-gt": 0,
"TP-base_TP-gt": 5,
"TP-base_FP-gt": 0,
"gt_concordance": 1.0,
"gt_matrix": {
"(0, 1)": {
"(0, 1)": 2,
"(1, 0)": 1
},
"(0, 1, 0)": {
"(1, 0)": 2
}
}
}
Unfiltered VCF
{
"TP-base": 40,
"TP-comp": 40,
"FP": 33051,
"FN": 107,
"precision": 0.0012087878879453629,
"recall": 0.272108843537415,
"f1": 0.002406883687345809,
"base cnt": 147,
"comp cnt": 33091,
"TP-comp_TP-gt": 32,
"TP-comp_FP-gt": 8,
"TP-base_TP-gt": 32,
"TP-base_FP-gt": 8,
"gt_concordance": 0.8,
"gt_matrix": {
"(0, 1)": {
"(0, 1)": 17,
"(1, 0)": 13,
"(1, 1)": 8
},
"(0, 1, 0)": {
"(1, 0)": 2
}
}
}
Severus comparison
{
"TP-base": 111,
"TP-comp": 111,
"FP": 118,
"FN": 36,
"precision": 0.4847161572052402,
"recall": 0.7551020408163265,
"f1": 0.5904255319148936,
"base cnt": 147,
"comp cnt": 229,
"TP-comp_TP-gt": 111,
"TP-comp_FP-gt": 0,
"TP-base_TP-gt": 111,
"TP-base_FP-gt": 0,
"gt_concordance": 1.0,
"gt_matrix": {
"(0, 1)": {
"(0, 1)": 95
},
"(1, 0)": {
"(0, 1)": 7
},
"(0, 0, 0, 1)": {
"(0, 1)": 2
},
"(0, 0, 1, 0)": {
"(0, 1)": 1
},
"(0, 1, 0)": {
"(0, 1)": 3
},
"(0, 0, 1)": {
"(0, 1)": 3
}
}
}
Hello,
I’ve been testing longcallD for somatic structural variant detection, for tumor only, and am seeing poor performance.
I first filtered the output VCF for the SOMATIC and SVTYPE tags. I then evaluated the unfiltered VCF, and finally used Severus as a comparator. The corresponding TrueVari summary reports are included below.
Do you have any suggestions for what might be going wrong?
One notable observation is that 35 of 149 truth variants appear to have been incorrectly labeled as germline.
Truth dataset
longcallD command
longcallD call \ --threads 16 \ -s human_GRCh38_no_alt_analysis_set.fasta \ HG008.tumor.aligned.hiphase.bam \ > phased.vcfFiltered (SOMATIC + SVTYPE)
{ "TP-base": 5, "TP-comp": 5, "FP": 250, "FN": 142, "precision": 0.0196078431372549, "recall": 0.034013605442176874, "f1": 0.024875621890547265, "base cnt": 147, "comp cnt": 255, "TP-comp_TP-gt": 5, "TP-comp_FP-gt": 0, "TP-base_TP-gt": 5, "TP-base_FP-gt": 0, "gt_concordance": 1.0, "gt_matrix": { "(0, 1)": { "(0, 1)": 2, "(1, 0)": 1 }, "(0, 1, 0)": { "(1, 0)": 2 } } }Unfiltered VCF
{ "TP-base": 40, "TP-comp": 40, "FP": 33051, "FN": 107, "precision": 0.0012087878879453629, "recall": 0.272108843537415, "f1": 0.002406883687345809, "base cnt": 147, "comp cnt": 33091, "TP-comp_TP-gt": 32, "TP-comp_FP-gt": 8, "TP-base_TP-gt": 32, "TP-base_FP-gt": 8, "gt_concordance": 0.8, "gt_matrix": { "(0, 1)": { "(0, 1)": 17, "(1, 0)": 13, "(1, 1)": 8 }, "(0, 1, 0)": { "(1, 0)": 2 } } }Severus comparison
{ "TP-base": 111, "TP-comp": 111, "FP": 118, "FN": 36, "precision": 0.4847161572052402, "recall": 0.7551020408163265, "f1": 0.5904255319148936, "base cnt": 147, "comp cnt": 229, "TP-comp_TP-gt": 111, "TP-comp_FP-gt": 0, "TP-base_TP-gt": 111, "TP-base_FP-gt": 0, "gt_concordance": 1.0, "gt_matrix": { "(0, 1)": { "(0, 1)": 95 }, "(1, 0)": { "(0, 1)": 7 }, "(0, 0, 0, 1)": { "(0, 1)": 2 }, "(0, 0, 1, 0)": { "(0, 1)": 1 }, "(0, 1, 0)": { "(0, 1)": 3 }, "(0, 0, 1)": { "(0, 1)": 3 } } }