Skip to content

feat: add comprehensive real-world test datasets with VCF and BAM files #60

Description

@berntpopp

Summary

Add realistic test datasets to replace toy data with production-scale VCF and BAM files for comprehensive testing.

Current Limitations

  • Small toy VCF files (~10-50 variants)
  • No BAM file integration for IGV reports
  • Missing complex inheritance patterns
  • Limited annotation coverage

Proposed Datasets

  1. Clinical Trio - 45K variants, rich annotations, BAM files
  2. Research Cohort - 50 samples, 2M variants, population genetics
  3. Cancer Analysis - tumor-normal pairs, somatic variants
  4. Performance Benchmark - 1M+ variants for scalability testing

Implementation

  • Cloud storage integration (S3/Azure)
  • On-demand download for CI/CD
  • Realistic file sizes (100MB-10GB)
  • Comprehensive annotations (gnomAD, CADD, ClinVar)

Tasks

  • Set up cloud storage for large datasets
  • Create clinical trio dataset with BAM files
  • Generate research cohort for population genetics
  • Add performance benchmark datasets
  • Implement dataset manager with lazy loading

Success Criteria

  • 5 realistic datasets covering major use cases
  • BAM integration for IGV report testing
  • Performance benchmarks with production-scale data

Labels: enhancement, testing, data, infrastructure

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions