Moe kernel for latest transformers v5

[Tcc0403]

🐛 Describe the bug

experts will no longer be just a ModuleList of MLP layers. It's time to write moe kernel for moe layers!

https://github.com/huggingface/transformers/pull/41580/files#diff-0855b77fc27ad9449158a1c74953f909b011c00de7125f7c8e68d0ff209c092aR218

Reproduce

No response

Versions

transformers v5

[Ramshankar07]

Can I take this one? I'm inspired by https://github.com/Dao-AILab/sonic-moe

[Ramshankar07]

Hey @Tcc0403 ,

1. should I add the ExpertsInterface patches in https://github.com/linkedin/Liger-Kernel/blob/main/src/liger_kernel/transformers/monkey_patch.py to handle the v5 and a new MoE kernels

2. Should I remove the v4 compactability or to handle it?

[Tcc0403]

1. should I add the ExpertsInterface patches in https://github.com/linkedin/Liger-Kernel/blob/main/src/liger_kernel/transformers/monkey_patch.py to handle the v5 and a new MoE kernels

Yes, similar to _bind_rms_norm_module, _patch_swiglu_module, etc, we only replace forward, extra_repr, get_name so that weight init and so on can be handled normally in transformers side. Feel free to open a PR and we can discuss the interface there.

2. Should I remove the v4 compactability or to handle it?

Choose https://github.com/linkedin/Liger-Kernel/tree/transformers-5.0.0rc1 as base when creating your PR. You can drop v4 compactability on that branch.