runtime error

Exit code: 1. Reason: as, ck, ck_splitk, cutlass, flash, flash3, triton_splitk File "/opt/conda/envs/appenv/lib/python3.10/site-packages/xformers/ops/fmha/flash.py", line 67, in <module> import flash_attn File "/opt/conda/envs/appenv/lib/python3.10/site-packages/flash_attn/__init__.py", line 3, in <module> from flash_attn.flash_attn_interface import ( File "/opt/conda/envs/appenv/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 15, in <module> import flash_attn_2_cuda as flash_attn_gpu ImportError: /opt/conda/envs/appenv/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationESs The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/workspace/finetune/gradio_app.py", line 13, in <module> from image_to_video import Image2Video, _resize_for_rectangle_crop, camera_pose_lerp File "/workspace/finetune/image_to_video.py", line 16, in <module> from diffusers import AutoencoderKLCogVideoX, CogVideoXDPMScheduler File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist File "/opt/conda/envs/appenv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 944, in __getattr__ value = getattr(module, name) File "/opt/conda/envs/appenv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 943, in __getattr__ module = self._get_module(self._class_to_module[name]) File "/opt/conda/envs/appenv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 955, in _get_module raise RuntimeError( RuntimeError: Failed to import diffusers.models.autoencoders.autoencoder_kl_cogvideox because of the following error (look up to see its traceback): /opt/conda/envs/appenv/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationESs ERROR conda.cli.main_run:execute(127): `conda run python finetune/gradio_app.py` failed. (See above for error)

Container logs: