Comparison of observed sequences to expected, the second

[d4straub]

This is a follow-up PR to #1021.

When expected abundances per sample are provided for matching sequences the following metrics will be generated for each sample in one file, and aggregated over all samples (mean, median, min, max) in another file:

• observed: Number of observed sequences
• expected: Number of expected sequences
• TP: Number of true positive sequences (observed and expected)
• FN: Number of false negative sequences (not observed but expected)
• FP: Number of false positive sequences (observed but not expected)
• recall: Recall of expected sequences = TP/(TP+FN)
• precision: Precision = TP/(TP+FP)
• F1: F1 score, the harmonic mean of the precision and recall = 2TP/(2TP+FP+FN)
• Fbeta: Fbeta score, the weighted F1 score. By default beta=2, i.e. weighs recall twice higher than precision
• fdr: False discovery rate = FP/(TP+FP)
• jaccard: Jaccard index = TP/(TP+FP+FN)
• TPs_exp: TP IDs (max 100) corresponding to expected sequences
• FNs_exp: FN IDs (max 100) corresponding to expected sequences
• FPs_obs: FP IDs (max 100) corresponding to observed sequences

A quick boxplot of some metrics is also generated, here an example (made-up data, not of a mock dataset):

Further PRs might add in tentative priority

(1) evaluate how abundances compare to expected
(2) compare taxonomic classification to expected

PR checklist

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
• If you've added a new tool - have you followed the pipeline conventions in the contribution docs
• If necessary, also make a PR on the nf-core/ampliseq branch on the nf-core/test-datasets repository.
• Make sure your code lints (nf-core pipelines lint).
• Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
• Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
• Usage Documentation in docs/usage.md is updated.
• Output Documentation in docs/output.md is updated.
• CHANGELOG.md is updated.
• README.md is updated (including new tool citations and authors/contributors).

[erikrikarddaniel]

Looks great, although I was a bit surprised that you chose the pplace_sheet test case. Two very small formatting suggestions; completely discretionary.

[d4straub]

although I was a bit surprised that you chose the pplace_sheet test case.

Yes, I just chose a quick test that uses the most used test data (input = "Samplesheet_standardized.tsv" or "Samplesheet.tsv"). But I agree that this isnt optimal. I think I should change the general Illumina PE dataset that we use for CI tests to a (down-sampled) mock dataset. And reorganize the CI test. Thats a large change as well though. Will plan that for a follow-up.

[erikrikarddaniel]

although I was a bit surprised that you chose the pplace_sheet test case.

Yes, I just chose a quick test that uses the most used test data (input = "Samplesheet_standardized.tsv" or "Samplesheet.tsv"). But I agree that this isnt optimal. I think I should change the general Illumina PE dataset that we use for CI tests to a (down-sampled) mock dataset. And reorganize the CI test. Thats a large change as well though. Will plan that for a follow-up.

I wouldn't say it's suboptimal; as long as everything is tested, fine. I was just a bit surprised.

[d4straub]

Thanks!