In NVIDIA-NeMo/Curator#2281, we are updating the FastTextLangId filter to return with the model's default lower case (en) instead of the previous upper case transformation (EN). We should make sure this does not break the Nemotron-CC recipe filtering.
Also, we had added NVIDIA-NeMo/Curator#2199 which includes language filtering within the FastTextLangId stage instead of needing to add a separate language filter. So we should try to use that instead.
cc @ayushdg
In NVIDIA-NeMo/Curator#2281, we are updating the
FastTextLangIdfilter to return with the model's default lower case (en) instead of the previous upper case transformation (EN). We should make sure this does not break the Nemotron-CC recipe filtering.Also, we had added NVIDIA-NeMo/Curator#2199 which includes language filtering within the
FastTextLangIdstage instead of needing to add a separate language filter. So we should try to use that instead.cc @ayushdg