The mle_bench directory contains several benchmark tasks from MLE Bench:
aptos2019-blindness-detection/: Diabetic retinopathy detection taskhistopathologic-cancer-detection/: Cancer detection from histopathology imagessiim-isic-melanoma-classification/: Melanoma classification taskdog-breed-identification/: Dog breed classification task- More incoming...
Each task directory contains:
- A question file used to ask Curie (e.g., aptos2019-blindness-detection-question.txt)
- Curie's auto-generated experiment report (e.g. aptos2019-blindness-detection_report.md)
- Curie's experimentation logs
- Setup kaggle credential
~/.kaggle/kaggle.json - Install correct sqlite version (to fix the bug in mle-bench)
conda create --name sqlite3-49-0 python=3.11
conda activate sqlite3-49-0
conda install sqlite=3.49- Install
mlebench
git clone https://github.com/openai/mle-bench.git
cd mle-bench
git lfs fetch --all
git lfs pull
pip install -e .Run mlebench prepare with the specific task ID:
conda activate sqlite3-49-0
mlebench prepare -c <task-id> # e.g., dog-breed-identificationThe data will be saved to $HOME/.cache/mle-bench/data.
Choose one of the available tasks from the MLE-Bench (E.g. siim-isic-melanoma-classification).
Use the following command format:
cd Curie/
python3 -m curie.main -f benchmark/mle_bench/<task-dir>/<task>-question.txt --dataset_dir <abd_path_to_dataset> --task_config curie/configs/mle_config.json MLE Bench provides description.md for each problem. We use the following prompt to convert the description into a research question:
Convert this Kaggle competition into a question to the ai agent (be concise): introduce the problem, goal, and all necessary details to guide the agent to find the best performing model/configuration:
<description.md>