Spaces:
Running
on
Zero
Running
on
Zero
Duplication of space fails
#5
by
jameshuntercarter
- opened
Error when cloning the space:
Exit code: 1. Reason: names, model_classes, model_resource, torch_dtype, device, infer)
File "/home/user/app/OmniAvatar/models/model_manager.py", line 97, in load_model_from_single_file
model = model.to_empty(device=device)
File "/home/user/.pyenv/versions/3.10.18/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1224, in to_empty
return self._apply(
File "/home/user/.pyenv/versions/3.10.18/lib/python3.10/site-packages/torch/nn/modules/module.py", line 928, in _apply
module._apply(fn)
File "/home/user/.pyenv/versions/3.10.18/lib/python3.10/site-packages/torch/nn/modules/module.py", line 928, in _apply
module._apply(fn)
File "/home/user/.pyenv/versions/3.10.18/lib/python3.10/site-packages/torch/nn/modules/module.py", line 928, in _apply
module._apply(fn)
[Previous line repeated 1 more time]
File "/home/user/.pyenv/versions/3.10.18/lib/python3.10/site-packages/torch/nn/modules/module.py", line 955, in _apply
param_applied = fn(param)
File "/home/user/.pyenv/versions/3.10.18/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1225, in <lambda>
lambda t: torch.empty_like(t, device=device), recurse=recurse
File "/home/user/.pyenv/versions/3.10.18/lib/python3.10/site-packages/torch/_prims_common/wrappers.py", line 309, in _fn
result = fn(*args, **kwargs)
File "/home/user/.pyenv/versions/3.10.18/lib/python3.10/site-packages/torch/_refs/__init__.py", line 5055, in empty_like
return torch.empty_permuted(
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 270.00 MiB. GPU 0 has a total capacity of 44.40 GiB of which 170.31 MiB is free. Including non-PyTorch memory, this process has 44.23 GiB memory in use. Of the allocated memory 43.80 GiB is allocated by PyTorch, and 13.07 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
What specs are needed?
this is a cuda memory issue, if you running it on anything lower that 80GB vram you probably want to try to turn on vram management back on again
change this line in app.py
# pipe.enable_vram_management(num_persistent_param_in_dit=args.num_persistent_param_in_dit)
to this
pipe.enable_vram_management(num_persistent_param_in_dit=args.num_persistent_param_in_dit)
also this
model_manager = ModelManager(device="cuda", infer=True)
model_manager.load_models(
[
self.args.dit_path.split(","),
self.args.vae_path,
self.args.text_encoder_path
],
torch_dtype=self.dtype,
device='cuda',
)
to
model_manager = ModelManager(device="cpu", infer=True)
model_manager.load_models(
[
self.args.dit_path.split(","),
self.args.vae_path,
self.args.text_encoder_path
],
torch_dtype=self.dtype,
device='cpu',
)
should hopefully fix your issue