Make it run on GTX1080, works!
#27
by
fawo
- opened
Hi All,
being disappointed by "needs CUDA capability 7 or better" I got a hint from Copilot:
output = model.generate(text, use_torch_compile=False, verbose=True)
Setting torch_compile = False works very well.
The GTX 1080 has to work hard, 3.7 - 4.5 tokens/s is a relaxing speed, but it works.
Have fun
Fabian