[WIP][EP] Make uccl.ep a self-contained Python subpackage (.so + Python code)

[zhenhuang12]

Description

Turns uccl.ep into a self-contained Python subpackage: the native extension (ep_cpp.abi3.so, renamed from the old top-level ep.abi3.so so it no longer collides with the uccl.ep package name) now lives alongside the Python helpers (Buffer, EventOverlap, initialize_uccl, destroy_uccl, …) under uccl/ep/. The Python source of truth moves to ep/python/uccl_ep/ and is exposed via package_dir={"uccl.ep": "ep/python/uccl_ep"}. ep/bench/{buffer,utils}.py and ep/deep_ep_wrapper/deep_ep/ are reduced to thin
re-export shims over uccl.ep, so existing bench scripts and deep_ep imports keep working. After this change pip install uccl is enough to import uccl.ep and use the full Python API.

Important

deep_ep_wrapper no longer needs to be reinstalled when the uccl.ep native library changes.
Previously deep_ep_wrapper shipped its own full copy of buffer.py / utils.py that called into the native extension directly, so any change to the uccl.ep dynamic-library API (symbol rename, signature change, new helper, …) required rebuilding and reinstalling deep_ep_wrapper to stay in sync. With this PR, deep_ep_wrapper contains no implementation of its own — it simply re-exports from uccl.ep at import time. Upgrading uccl alone is sufficient; the existing deep_ep_wrapper install picks up the new API automatically as long as the public names it re-exports remain stable.

Not a behavior change

Only packaging / module layout is changing. The native C++/CUDA code, the Python Buffer class,
initialize_uccl / destroy_uccl, benchmark scripts, and CI workflows all keep the same public semantics.
Fixes # (issue)

Type of Change

• Bug fix
• New feature
• Documentation update

How Has This Been Tested?

• Unit tests
• Integration tests
• Manual testing

Checklist

• I have run format.sh to follow the style guidelines.
• I have run build.sh to verify compilation.
• I have removed redundant variables and comments.
• I have updated the documentation.
• I have added tests.

[YangZhou1997]

There seems to be some conflict, cc @zhenhuang12

[zhenhuang12]

There seems to be some conflict, cc @zhenhuang12

@YangZhou1997 Thanks! I'm continuing to support this feature—the build changes are fairly large and I'd appreciate a review. I've moved the compilation of build_inner into the Makefile, while leaving the packaging scripts unchanged. Now we can install uccl directly via pip install . --no-build-isolation, and then use the ep module with from uccl import ep. The ep module is a complete Python package, and it has been decoupled from the deep_ep_wrapper dependency.

cc @MaoZiming

[YangZhou1997]

@zhenhuang12 is it possible to avoid using a Makefile, which is usually less maintainable than a shell script like build_inner.sh?

[zhenhuang12]

@zhenhuang12 is it possible to avoid using a Makefile, which is usually less maintainable than a shell script like build_inner.sh?

@YangZhou1997 I think we could replace the Makefile with a build.sh script, which is essentially equivalent to writing a ShellExtension to call build.sh in setup.py. Currently build_inner.sh covers both build and packaging steps, but we can separate out the build part from it.

Do you think using build.sh instead of Makefile is a better approach? I'd like to hear your opinion.

[zhenhuang12]

@YangZhou1997 If you think the current PR change is too large, I can revert it and only keep the "ep python self-containerd" part. Support for ShellExtension or MakeExtension can then be introduced in a later PR.

[YangZhou1997]

Hi @zhenhuang12, the PR size is okay for me. I think if we can use a ShellExtension in the setup.py, that would be better, so that we can keep all building functions in the existing build.sh and build_inner.sh (and get rid of Makefile in the top level)