Improve Documentation for Non-CellProfiler Datasets in Pycytominer

README.md

@@ -66,6 +66,24 @@ Pycytominer is primarily built on top of [pandas](https://pandas.pydata.org/docs
 
 Pycytominer currently supports [parquet](https://parquet.apache.org/) and compressed text file (e.g. `.csv.gz`) i/o.
 
+## Data structure
+
+Pycytominer processes all data using [Pandas](https://pandas.pydata.org/) DataFrames.
+
+Currently, Pycytominer fully supports data generated by [CellProfiler]('https://cellprofiler.org/'), adhering to its specific data structure and naming conventions.
+
+CellProfiler-generated image-based profiles typically consist of two main components:
+
+- **Metadata features:** This section contains information about the experiment, such as plate ID, well position, incubation time, perturbation type, and other relevant experimental details.
+
+- **Morphology features:** These are the quantified morphological features captured from microscopy images. Naming of these features is structured like this
+
+The feature naming scheme in Pycytominer follows a specific structure.
+
+- **Metadata features:** These feature names are prefixed with `Metadata_`, indicating that the data in these columns contain metadata information.
+
+- **Morphological features:** These follow CellProfiler’s naming conventions, where default compartments are labeled as "cells," "cytoplasm," and "nuclei." If users have different compartments in their dataset, they will need to manually specify those compartments using the `compartments` [parameter](https://pycytominer.readthedocs.io/en/stable/pycytominer.cyto_utils.html#pycytominer.cyto_utils.cells.SingleCells.compartments).
+
 ## API
 
 Pycytominer has five major processing functions:
@@ -97,6 +115,8 @@ Each processing function has unique arguments, see our [documentation](https://p
 
 The default way to use pycytominer is within python scripts, and using pycytominer is simple and fun.
 
+The example below demonstrates how to perform normalization with a dataset generated by [CellProfiler](https://cellprofiler.org/).
+
 ```python
 # Real world example
 import pandas as pd
@@ -114,6 +134,42 @@ normalized_df = pycytominer.normalize(
 )
 ```
 
+### Handling Non-CellProfiler Morphological Features in Pycytominer
+
+In some cases, raw morphological features may not be extracted from [CellProfiler](https://cellprofiler.org/).
+While Pycytominer fully supports features extracted by [`CellProfiler`](https://cellprofiler.org/), errors may occur when using features from other tools.
+
+To resolve this, you can manually specify the morphological features using the `features` [parameter](https://pycytominer.readthedocs.io/en/latest/pycytominer.html#pycytominer.normalize.normalize).
+This parameter is also available in other key steps, such as [`aggregate`](https://pycytominer.readthedocs.io/en/latest/pycytominer.html#pycytominer.aggregate.aggregate), [`feature_select`](https://pycytominer.readthedocs.io/en/latest/pycytominer.html#pycytominer.feature_select.feature_select), and [`consensus`](https://pycytominer.readthedocs.io/en/latest/pycytominer.html#pycytominer.feature_select.feature_select).
+
+Below is an example of loading data that is not from [`CellProfiler`](https://cellprofiler.org/), demonstrating how to handle non-CellProfiler features.
+
+```python
+# Real world example using a different dataset
+import pandas as pd
+import pycytominer
+
+commit = "da8ae6a3bc103346095d61b4ee02f08fc85a5d98"
+url = f"https://media.githubusercontent.com/media/broadinstitute/lincs-cell-painting/{commit}/profiles/2016_04_01_a549_48hr_batch1/SQ00014812/SQ00014812_augmented.csv.gz"
+
+# assuming this is dataset was not generated by CellProfiler
+df = pd.read_csv(url)
+
+# split the metadata and morphology features columns
+metadata_features = df.columns[0:26].tolist()
+morphology_features = df.columns[26::].tolist()
+
+# use the 'features' parameter to declare what features pycytominer should focus
+normalized_df = pycytominer.normalize(
+    profiles=df,
+    features=morphology_features,
+    method="standardize",
+    samples="Metadata_broad_sample == 'DMSO'"
+)
+```
+
+**Note:** We are actively working on enhancing `pycytominer` to support morphological features extracted from a variety of software tools!
+
 ### Pipeline orchestration
 
 Pycytominer is a collection of different functions with no explicit link between steps.
@@ -138,8 +194,10 @@ Therefore, we have included some custom tools in `pycytominer/cyto_utils` that p
 - [Data processing for image-based profiling](#data-processing-for-image-based-profiling)
   - [Installation](#installation)
   - [Frameworks](#frameworks)
+  - [Data structure](#data-structure)
   - [API](#api)
   - [Usage](#usage)
+    - [Handling Non-CellProfiler Morphological Features in Pycytominer](#handling-non-cellprofiler-morphological-features-in-pycytominer)
     - [Pipeline orchestration](#pipeline-orchestration)
   - [Other functionality](#other-functionality)
     - [CellProfiler CSV collation](#cellprofiler-csv-collation)