Matthew parkin git true values

Dockerfile

@@ -1,7 +1,7 @@
-FROM python:3.8-slim
+FROM python:3.8
 WORKDIR /home/biolib
+COPY model.pkl /home/biolib
 COPY requirements.txt .
 RUN pip install -r requirements.txt
 COPY predict.py .
-COPY data/test.zip data/
-ENTRYPOINT ["python3", "predict.py"]
+ENTRYPOINT ["python3", "predict.py"]

README.md

@@ -7,7 +7,7 @@ Large-scale protein production for biotechnology and biopharmaceutical applicati
 
 The challenge is mentored by [Christopher Ing](https://github.com/cing) and [Mark Fingerhuth](https://github.com/markf94).
 
-## Aim of the challenge
+## Challenge Aim
 
 It is the objective of this project to use our provided dataset of protein structure and solubility value pairs in order to produce a solubility predictor with comparable accuracy to sequence-based predictors reported in the literature. The provided dataset to be used in this project is created by following the dataset curation procedure described in [the SOLart paper](https://academic.oup.com/bioinformatics/article/36/5/1445/5585748), and this hackathon project has a similar aim to this manuscript.
 
@@ -17,11 +17,7 @@ The process of generating the dataset is described in the SOLArt manuscript. At
 
 If issues are identified with individual structures, please refer to the Uniprot ID and manually investigate the best template. In some cases, we needed to improve structure correctness by modelling missing atoms/residues inside the Chemical Computing Group software MOE on a case-by-case basis.
 
-The dataset can be found in the `data/` subdirectory - it is already divided into `training/` and `test/` data. The `training/` data comes with `solubility_values.csv` and `solublity_values.yaml` (same content just different format) which both contain the solubility target values for all the PDB files provided in that directory. Note that each PDB file is named after the Uniprot identifier of the respective protein and the `protein` column in the `solubility_values.csv` also contains the Uniprot identifiers.
-
-The `test/` dataset consists of three different subdirectories (protein structures derived from different organisms and with different approaches) and you should **NOT** use them for any training. Only the `yeast_crystal_structs/` directory contains `solubility_values.csv` and `solublity_values.yaml` (same content just different format) files which you can use for some local testing & validation. In order to find out your performance on the entire test dataset you need to use the automated benchmarking system (see below).
-
-### Example output
+### Example Output
 Your code should output a file called `predictions.csv` in the following format:
 
 ```
@@ -34,13 +30,13 @@ whereby the `protein` column contains the Uniprot ID (corresponds to the filenam
 
 Note, that there are three (!) test subsets but you are expected to submit all the predictions in one file (not three) for the benchmarking system to work.
 
-## Automated benchmarking system
+## Benchmarking System
 The continuous integration script in `.github/workflows/ci.yml` will automatically build the `Dockerfile` on every commit to the `main` branch. This docker image will be published as your hackathon submission to `https://biolib.com/<YourTeam>/<TeamName>`. For this to work, make sure you set the `BIOLIB_TOKEN` and `BIOLIB_PROJECT_URI` accordingly as repository secrets. 
 
 To read more about the benchmarking system [click here](https://www.notion.so/Benchmarking-System-46bfaeea0119490cb611688b493c589a).
 
 ## Say thanks
 
-Give this repo a star: ![GitHub Repo stars](https://img.shields.io/github/stars/ProteinQure/cbh21-protein-solubility-challenge?style=social)
+Give this repo a star ![GitHub Repo stars](https://img.shields.io/github/stars/ProteinQure/cbh21-protein-solubility-challenge?style=social)
 
 Star the [ProteinQure](https://github.com/proteinqure) org on Github: ![GitHub Org's stars](https://img.shields.io/github/stars/ProteinQure?style=social)

benchmark.py

@@ -3,80 +3,52 @@
 """
 
 from __future__ import annotations
-import argparse
-import pprint
 from typing import Any
 
 import pandas as pd
+import argparse
 from sklearn.metrics import mean_squared_error
-from scipy.stats import spearmanr, pearsonr
+from scipy.stats import spearmanr
 
 
-def compute_metrics(predictions_file: str, target_files: dict[str, str]) -> dict[str, Any]:
+def compute_metrics(predictions_file: str, target_files: list) -> dict[str, Any]:
     """
-    Computes the test performance metrics on the three test data sets in
+    Computes the test performance metrics on the test data sets in
     `target_files`.
     """
 
     predictions_df = pd.read_csv(predictions_file)
     predictions_df.columns = ["protein", "prediction"]
+    targets_dfs = [pd.read_csv(target_file) for target_file in target_files]
+    targets_df = pd.concat(targets_dfs)
+    targets_df.columns = ["protein", "true_value"]
 
-    metrics = {}
-    for test_set_name, target_file in target_files.items():
-
-        # this allows users to run it with the `yeast_crystal_structs` test set only
-        if target_file is None:
-            continue
-
-        targets_df = pd.read_csv(target_file)
-        targets_df.columns = ["protein", "true_value"]
+    # merge the two dataframes
+    df = pd.merge(predictions_df, targets_df, on="protein")
 
-        # merge the two dataframes (gets rid of the rows we don't need)
-        df = pd.merge(predictions_df, targets_df, on="protein")
+    true_values = df["true_value"].to_numpy()
+    predictions = df["prediction"].to_numpy()
+    mse = mean_squared_error(y_true=true_values, y_pred=predictions)
+    correlation, pvalue = spearmanr(a=true_values, b=predictions)
 
-        true_values = df["true_value"].to_numpy()
-        predictions = df["prediction"].to_numpy()
-        rmse = mean_squared_error(y_true=true_values, y_pred=predictions, squared=False)
-        spearman_correlation, spearman_pvalue = spearmanr(a=true_values, b=predictions)
-        pearson_correlation, pearson_pvalue = pearsonr(x=true_values, y=predictions)
-
-        metrics.update(
-            {
-                test_set_name: {
-                    "rmse": rmse,
-                    "spearman_rank": {
-                        "correlation": spearman_correlation,
-                        "pvalue": spearman_pvalue,
-                    },
-                    "pearson": {
-                        "correlation": pearson_correlation,
-                        "pvalue": pearson_pvalue,
-                    },
-                }
-            }
-        )
-
-    return metrics
+    return {
+        "mean_squared_error": mse,
+        "spearman_rank": {
+            "correlation": correlation,
+            "pvalue": pvalue,
+        },
+        "num_points_used_for_metrics": f"{df.shape[0]}/{targets_df.shape[0]}",
+    }
 
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
     parser.add_argument("--predictions", default="predictions.csv")
     parser.add_argument("--yeast-crystal", default="data/test/yeast_crystal_structs/solubility_values.csv")
-    parser.add_argument("--yeast-modelled", default=None)
-    parser.add_argument("--ecoli-modelled", default=None)
+    parser.add_argument("--yeast-modelled", default="data/test/yeast_modelled_structs/solubility_values.csv")
+    parser.add_argument("--ecoli-modelled", default="data/test/ecoli_modelled_structs/solubility_values.csv")
     args = parser.parse_args()
-
-    pp = pprint.PrettyPrinter(indent=4, width=120)
-    print("TEST SET PERFORMANCES")
-    print("=====================")
-    pp.pprint(
-        compute_metrics(
-            args.predictions,
-            {
-                "yeast_crystal_structs": args.yeast_crystal,
-                "yeast_modelled_structs": args.yeast_modelled,
-                "ecoli_modelled_structs": args.ecoli_modelled,
-            },
-        ),
+    print(
+        "Test set performance: ",
+        compute_metrics(args.predictions, [args.yeast_crystal, args.yeast_modelled, args.ecoli_modelled]),
     )