New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix numpy and XGMI 1-hop detection #67

Merged

mawong-amd merged 2 commits into main from fix_numpy_and_xgmi1hop_detection

Jun 25, 2024

mawong-amd commented Jun 25, 2024

Temporarily fix numpy < 2.0.0 as Numpy 2.0 breaks ROCm PyTorch. Counterpart to vllm-project#5582

Fix XGMI 1-hop detection: previous version has the following problems

Ignores the device_ids passed in.
Each device only checks to see if it's 1-hop XGMI connected to all other devices. Instead, each device should check that all devices are 1-hop XGMI connected to all other devices. This prevents the odd case where some devices are 1-hop XGMI connected to all other devices, but others are not, which would result in not every device enabling custom_all_reduce and hence deadlock.

mawong-amd added 2 commits

June 25, 2024 18:36


          Fix 1-hop XGMI detection

2e20bba


          Fix numpy versioning

a2661ca

mawong-amd merged commit 3e7b0b6 into main

9 of 13 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet