runtime error

Exit code: 1. Reason: mport ( File "/usr/local/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 15, in <module> import flash_attn_2_cuda as flash_attn_gpu ImportError: /usr/local/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEa The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/user/app/app.py", line 331, in <module> model = AutoModelForCausalLM.from_pretrained( File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 558, in from_pretrained model_class = get_class_from_dynamic_module( File "/usr/local/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 570, in get_class_from_dynamic_module return get_class_in_module(class_name, final_module, force_reload=force_download) File "/usr/local/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 267, in get_class_in_module module_spec.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/home/user/.cache/huggingface/modules/transformers_modules/dots-ocr-local/modeling_dots_ocr.py", line 5, in <module> from transformers.models.qwen2 import Qwen2ForCausalLM File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist File "/usr/local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1955, in __getattr__ module = self._get_module(self._class_to_module[name]) File "/usr/local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1969, in _get_module raise RuntimeError( RuntimeError: Failed to import transformers.models.qwen2.modeling_qwen2 because of the following error (look up to see its traceback): /usr/local/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEa

Container logs: