Add support for multilingual training - #1
Conversation
Summary of ChangesHello @monatis, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the system's capability for multilingual training by introducing a flexible dataset management system. It allows for the integration and processing of data from various language sources, crucial for developing models with broader linguistic understanding. The changes include refactoring data download logic, organizing datasets into language-specific directories, and updating the tokenizer evaluation to include a new language, all demonstrated through a new comprehensive training script. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for multilingual training by introducing a new Turkish dataset. The changes in nanochat/dataset.py generalize data loading to handle multiple datasets, and scripts/tok_eval.py is updated to evaluate the tokenizer on Turkish text. A new script speedrun-multi.sh is added to orchestrate the multilingual training process. My review focuses on improving the new dataset handling logic, code style, and ensuring the multilingual training script is robust and balanced.
| python -m nanochat.dataset -n 240 & | ||
| DATASET_DOWNLOAD_PID=$! | ||
| # train the tokenizer with vocab size 2**16 = 65536 on ~2B characters of data | ||
| python -m scripts.tok_train --max_chars=2500000000 | ||
| # evaluate the tokenizer (report compression ratio etc.) | ||
| python -m scripts.tok_eval | ||
|
|
||
| # ----------------------------------------------------------------------------- | ||
| # Base model (pretraining) | ||
|
|
||
| # Download the eval_bundle from s3 to evaluate CORE metric during training (~162MB) | ||
| EVAL_BUNDLE_URL=https://karpathy-public.s3.us-west-2.amazonaws.com/eval_bundle.zip | ||
| if [ ! -d "$NANOCHAT_BASE_DIR/eval_bundle" ]; then | ||
| curl -L -o eval_bundle.zip $EVAL_BUNDLE_URL | ||
| unzip -q eval_bundle.zip | ||
| rm eval_bundle.zip | ||
| mv eval_bundle $NANOCHAT_BASE_DIR | ||
| fi | ||
|
|
||
| # The d20 model is 561M parameters. | ||
| # Chinchilla says #tokens = 20X #params, so we need 561e6 * 20 = 11.2B tokens. | ||
| # Assume our tokenizer is 4.8 chars/token, this is 11.2B * 4.8 ~= 54B chars. | ||
| # At 250M chars/shard, this is 54B / 250M ~= 216 shards needed for pretraining. | ||
| # Round up to 240 for safety. At ~100MB/shard, this downloads ~24GB of data to disk. | ||
| # (The total number of shards available in the entire dataset is 1822.) | ||
| echo "Waiting for dataset download to complete..." | ||
| wait $DATASET_DOWNLOAD_PID |
There was a problem hiding this comment.
The logic for downloading datasets for pre-training is not suitable for a multilingual run. It only downloads a large number of shards for the English dataset (karpathy) by relying on a default argument, and does not download the Turkish (altai) data. This leads to a very imbalanced dataset for pre-training.
To fix this, I recommend downloading both datasets in parallel and waiting for both to complete. This requires changing how the PIDs are stored and waited upon.
Here's an example of how you could modify the script:
-
Replace lines 69-70 to start downloads for both datasets and store their PIDs:
python -m nanochat.dataset --dataset karpathy -n 240 & KARPATHY_PID=$! python -m nanochat.dataset --dataset altai -n 90 & ALTAI_PID=$!
-
Then, replace line 95 to wait for both PIDs to finish:
wait $KARPATHY_PID wait $ALTAI_PID
| ds_infos = { | ||
| "karpathy": { | ||
| "BASE_URL": "https://huggingface.co/datasets/karpathy/fineweb-edu-100b-shuffle/resolve/main", | ||
| "MAX_SHARD": 1822 # the last datashard is shard_01822.parquet | ||
| }, | ||
| "altai": { | ||
| "BASE_URL": "https://huggingface.co/datasets/altaidevorg/fineweb2-hq-turkish/resolve/main", | ||
| "MAX_SHARD": 90 | ||
| }, | ||
| } |
There was a problem hiding this comment.
The indentation in the ds_infos dictionary is inconsistent, which makes it harder to read. Please format it according to standard Python style for better maintainability.
| ds_infos = { | |
| "karpathy": { | |
| "BASE_URL": "https://huggingface.co/datasets/karpathy/fineweb-edu-100b-shuffle/resolve/main", | |
| "MAX_SHARD": 1822 # the last datashard is shard_01822.parquet | |
| }, | |
| "altai": { | |
| "BASE_URL": "https://huggingface.co/datasets/altaidevorg/fineweb2-hq-turkish/resolve/main", | |
| "MAX_SHARD": 90 | |
| }, | |
| } | |
| ds_infos = { | |
| "karpathy": { | |
| "BASE_URL": "https://huggingface.co/datasets/karpathy/fineweb-edu-100b-shuffle/resolve/main", | |
| "MAX_SHARD": 1822 # the last datashard is shard_01822.parquet | |
| }, | |
| "altai": { | |
| "BASE_URL": "https://huggingface.co/datasets/altaidevorg/fineweb2-hq-turkish/resolve/main", | |
| "MAX_SHARD": 90 | |
| }, | |
| } |
| parquet_paths = [os.path.join(data_dir, f) for f in parquet_files] | ||
| data_dir = MAIN_DATA_DIR if data_dir is None else data_dir | ||
| parquet_paths = sorted(glob.glob(f"{data_dir}/**/*.parquet")) | ||
|
|
There was a problem hiding this comment.
There is an unnecessary indented blank line here. Please remove it to improve code cleanliness.
| def download_single_file(args): | ||
| dataset_name, index = args | ||
| """ Downloads a single file index, with some backoff """ |
There was a problem hiding this comment.
The function signature for download_single_file has changed, but the docstring is now in the wrong position and hasn't been updated to reflect the new args parameter. The docstring should be the first statement in the function and should explain its parameters for better code clarity and maintainability.
| def download_single_file(args): | |
| dataset_name, index = args | |
| """ Downloads a single file index, with some backoff """ | |
| def download_single_file(args): | |
| """ | |
| Downloads a single file index, with some backoff. | |
| `args` is a tuple of (dataset_name, index). | |
| """ | |
| dataset_name, index = args |
…BAŞLAYACAK OLAN ÖĞRENCİLER | ODTÜ ÖĞRENCİ İŞLERİ .pdf
2 participants
No description provided.