Does it work on CPU?
Once again thank you for creating such a great model. Does it work on CPU or does it require modern architecture (ampere or later), how about intel or AMD GPUs?
Also, same question about 1.58bit, Falcon3 10B 1.58bit model will that work on CPU?
Hi
@LLMToaster
Thanks for your question !
Yes, you can either run it with HF transformers on CPU (though being not optimized), or use bitnet.cpp (recommended)
For bitnet.cpp, you can go ahead and use: https://huggingface.co/tiiuae/Falcon-E-1B-Instruct-GGUF together with: https://github.com/microsoft/BitNet/pull/268 (right now the PR has not been merged)
For the 10B model, you can use: https://huggingface.co/tiiuae/Falcon3-10B-Instruct-1.58bit and no need to use https://github.com/microsoft/BitNet/pull/268 - bitnet main already supports Falcon3 models