-
Notifications
You must be signed in to change notification settings - Fork 6.9k
Enable MPS backend for bitsandbytes quantization #13361
Description
Is your feature request related to a problem? Please describe.
Bitsandbytes now has basic support for the Apple MPS backend, as I can tell by bitsandbytes-foundation/bitsandbytes#1818 and
bitsandbytes-foundation/bitsandbytes#1875.
The issue is that diffusers does not allow me to use the quantization on Apple hardware, because of the error No GPU found. A GPU is needed for quantization. from here
diffusers/src/diffusers/quantizers/bitsandbytes/bnb_quantizer.py
Lines 64 to 65 in f2be8bd
| if not (torch.cuda.is_available() or torch.xpu.is_available()): | |
| raise RuntimeError("No GPU found. A GPU is needed for quantization.") |
Describe the solution you'd like.
I modified the above lines to
if not (torch.cuda.is_available() or torch.xpu.is_available() or torch.mps.is_available()):
raise RuntimeError("No GPU found. A GPU is needed for quantization.")and tested the change with the quantized version of FLUX.2-dev as described in https://github.com/black-forest-labs/flux2/blob/main/docs/flux2_dev_hf.md#4-bit-transformer-and-4-bit-text-encoder-20g-of-vram, and all worked fine.
I am wondering if that solution could be adopted in the repository. I can open a PR if that is the case.
Describe alternatives you've considered.
None.
Additional context.
Add any other context or screenshots about the feature request here.