Questions about Uni-FEP-Benchmarks data processing: ChEMBL versioning, assay filtering, target–PDB mapping, reference PDB selection

[0ut0fcontrol]

Hi maintainers,

Thanks for releasing Uni-FEP-Benchmarks.
I have a few technical questions about the Uni-FEP-Benchmarks data processing pipeline (ChEMBL + PDB → benchmarks):

1. ChEMBL version / update plan

• Was Uni-FEP-Benchmarks built using ChEMBL v35?
• For example, the latest EGFR assay CHEMBL5260455 is dated 2023-04-18, which is before the ChEMBL 35 release date (December 2024).
• Do you plan to update the benchmark following new ChEMBL releases (ChEMBL is now at v37)?

2. Assay / affinity endpoint filtering

• Which affinity types are included (Ki/Kd only, or also IC50/EC50, etc.)?
• What filters are applied for assay/data quality (assay type, confidence score, curated flags, removing ambiguous units, replicates/uncertainty/outlier handling, etc.)?

3. Target → PDB mapping

• How is a ChEMBL target mapped to PDB structures (e.g., UniProt/SIFTS mapping, sequence alignment/similarity search)?
• How are mutations, engineered constructs, missing segments, or isoforms handled?

4. Reference PDB selection

• When multiple PDB structures are available for the same target, what is the selection strategy for the reference structure?
• For targets with distinct conformational states (e.g., GPCR agonist vs antagonist), how do you ensure the selected structure matches the relevant state?

5. Reproducible pipeline / scripts

• Are there scripts or a reproducible workflow available for the ChEMBL + PDB → benchmark generation? If not, any pointers to the key steps would be appreciated.

Thanks

[rongfengzou]

Hi,

Thank you for your interest in Uni-FEP-Benchmarks and for the very thoughtful questions. I am happy to clarify the benchmark construction procedure point by point.

1. ChEMBL version / update plan

Uni-FEP-Benchmarks was built using ChEMBL v34, not ChEMBL v35. Therefore, assay records added after ChEMBL v34 were not included in the current benchmark. We agree that updating the benchmark with newer ChEMBL releases would be valuable, especially as ChEMBL continues to add more recent assay data. At the moment, we do not have a fixed release schedule for updates, but this is something we are considering for future versions.

2. Assay / affinity endpoint filtering

We included standardized activity records with Ki, Kd, and IC50 endpoints. EC50 and other functional readouts were not included in the current version.

The main assay/data filters were:

• standardized relation must be "=";
• standardized units must be nM;
• assay format must be single protein format;
• records with ChEMBL data-validity comments were removed;
• only records with usable standardized pChEMBL values were retained;
• covalent/reactive compounds were filtered out;
• each assay was further filtered to retain the largest chemically related compound cluster;
• benchmark sets were kept only when they had sufficient size and activity range, specifically more than 8 compounds after filtering and a pChEMBL range greater than 2.0.

We did not apply a separate replicate-averaging or uncertainty-modeling procedure beyond the standardized ChEMBL curation and the filters above.

3. Target → PDB mapping

The target mapping starts from the ChEMBL target and associated UniProt information. PDB structures were then collected through UniProt/PDB cross-references, and the ligands in those PDB structures were extracted and compared with the ChEMBL ligands.

For PDB matching, we used ligand similarity as an important criterion. Co-crystal ligands were compared with the benchmark compounds using molecular fingerprints, and only sufficiently similar ligand-bound structures were considered. We also checked whether mutations in the PDB structure were compatible with the assay variant information. Pocket-region mutations were considered especially important, and PDB structures with inconsistent mutation information were excluded.

Engineered constructs, missing segments, and isoform-related issues were handled conservatively. Cases with obvious inconsistencies between the assay target/variant and the available PDB structure were not used.

4. Reference PDB selection

When multiple PDB structures were available, the reference structure was selected based on several practical criteria rather than simply choosing the newest or highest-resolution structure. The main considerations were:

• similarity between the co-crystal ligand and benchmark ligands;
• consistency between PDB mutations and assay variant information;
• absence of obvious covalent or coordination artifacts involving the ligand;

For targets with distinct conformational states, such as GPCR agonist versus antagonist states, we aimed to select a structure whose ligand and binding-site conformation were consistent with the intended assay/ligand series. Ambiguous cases were treated cautiously and were excluded if the structural state could not be assigned reliably.

5. Reproducible pipeline / scripts

At this stage, we are not releasing the full construction scripts, since the workflow was developed as an internal benchmark-generation pipeline and still contains project-specific dependencies. However, we are happy to clarify the overall procedure and key filtering criteria. Please feel free to let us know if you have any further questions, and we would be glad to help.