Pipeline Output: Sync and Backup

TSV Files

A series of tsv files are created during the running of the pipeline, they are split in to data and results files.

Results

All relevant files can be found in: /local-scratch/projects/genotype-phenotype-map/results/latest

coloc_pairwise_results.tsv.gz
coloc_clustered_results.tsv.gz
rare_results.tsv.gz
study_extractions.tsv.gz
studies_processed.tsv.gz
traits_processed.tsv.gz

If you need to get additional info:

• the variant annotation data: /local-scratch/projects/genotype-phenotype-map/data/variant_annotation/vep_annotations_hg38.tsv.gz
• gene info: /local-scratch/projects/genotype-phenotype-map/data/variant_annotation/gene_info.tsv

studies_processed: metadata about the study that was ingested. Each study is either a GWAS or a QTL measurement (ie. eqtlgen-cis-ENSG00000228463)

• study_name: unique string referencing study.
  • study_name <-> study_extractions$study
  • study_name <-> coloc_pairs$study_a and study_b

traits_processed: This can be ignored unless you care about grouping rare and common studies, or trait categories.

• category: high level category for trait
• study_name: related to trait in studies_processed. There are many rows in studies_processed to trait study_name. Mostly around grouping rare and common studies.

study_extractions: All the finemapped extractions per ld block related to a specific study

• study: this matches with studies_processed.study_name. There are many extractions per study (1 per ld block)
• unique_study_id: the unique id that is used in coloc and rare results
  • unique_study_id <-> coloc_pairs$unique_study_a and unique_study_b
  • unique_study_id <-> coloc_clusters$unique_study_id
• known_gene: gene that was marked from the QTL study
• ignore: finemapped studies that have been deemed dodgy, should always filter by ignore == F
• snp: the first snp in the credible set that susie found. If susie failed, the lowest p-val.

coloc_pairwise_results

• coloc_group_id: the unique identifier which groups all unique_study_ids
• group_threshold: strong for groups where h4 > 0.8, moderate for groups where h4 > 0.6
• snp: the "candidate snp" that has the cumulative highest score for causal snp

rare_results.tsv.gz:

If you want to use this in it's current form, you'll need to something like this first:

rare_results <- rare_results |>
    dplyr::mutate(rare_result_group_id=1:dplyr::n()) |>
    dplyr::mutate(candidate_snp=trimws(candidate_snp)) |>
    tidyr::separate_rows(traits, genes, min_ps, files, sep=", ") |>
    dplyr::rename(unique_study_id=traits, gene=genes, min_p=min_ps, file=files) |>

• traits: a list of unique_study_ids. unique_study_id <-> study_extractions$unique_study_id
• candidate_snp: the snp in which all the studies have a significant rare variant.
• gene: gene, as tagged by the rare variant data source

DB Files

From the above files, a series of DuckDB database files are created,

• Database definitions: pipeline_steps/database_definitions.R
• Script that turns tsv files into db files: pipeline_steps/create_db_from_results.R

Syncing data and results to various places

There is a script that allows you to sync and move the results to various places that you need.

Rscript scripts/sync_to_servers.R

There are various arguments to this command if you wish to only update some places

Sync DB Files to web server

Steps to update the databases:

• Rscript scripts/sync_to_servers.R --db_files
• This will copy the necessary *.db files to the oracle web server(s), and rename them to *_new.db.
• Make necessary changes to genotype-phenotype-api to adjust for the db changes
  • Run create_test_db_from_results.R if needed (see below)
• On the web server, run the command ./update_data_update_code_restart.sh
  • This moves the databases into the right place and deploys all code / containers (via docker swarm)

Sync static web files to web server

Similarly to the DB Files, this syncs all files in static_web and svgs, which include all the SVGs for the GWAS, SEO files, and more. These then get served directly from nginx.

• Rscript scripts/sync_to_servers.R --static_web_files
• No need for anything more, nginx will automatically start serving the new files

Sync all necessary data to GWAS upload server

A lot of data is needed to run the GWAS Upload pipeline, if any of the following data has been updated, please sync. If any new studies have been added to the GPMap, this should be synced.

• Any of the finemapped_results.tsv.gz or coloc_pairwise_results.tsv.gz files, this means any new studies that have been added to the GPMap.
  • Note that if you don't want to include some results, use --block_list
• vep_annotations_hg38.tsv.gz
• Any data in ld_reference_panel_hg38

Sync summary statistics to Oracle Bucket

Similarly, if there are additional studies added to the map, you need to sync those results to the oracle bucket. This is used so the user can download summary statistics information.

--summary_stats

Backup data and results to RDSF

This is not so much for any explicit use, but as a way to back up all data. WARNING: takes a long time.

Creating test databases for API development

If data or, more importantly, the databases schemas have been updated, you will probably want to update the test databases for API development.

Rscript create_test_db_from_results.R --study_ids 5020 54929

Then wherever you are developing the API, you will want to

$ genotype-phenotype-api: scp ieup1:/local-scratch/projects/genotype-phenotype-map/results/latest/*_small.db tests/test_data

Then you can update the code accordingly.

Multiple Versions of Results

There will be multiple versions of the results that will exist at the same time. This must be treated carefully. For example

• 1.0.0: results version for GPMap publication
• latest: all results, including those ingested for after the publication
• specific_data: all results including the unreleased dataset from a collaboration

To attempt to ensure there is no contamination between datasets, some data directories:

• There are not separate, different versions of the data, his would be unmanageable. Instead we are filtering data that gets compiled in the results. results directory:
• Both compile_results.R and sync_to_servers.R have a block_list input file. The idea is that this is a reversal of the study_list.csv file, where all data in this list gets ignored by these scripts, and therefore does not make it into the results.
• As mentioned above, each specific results directory would have its own block_list file, which allows for multiple living versions of the results.