It takes a long time, an hour and a third, and it doesn't end.

41min

Be careful with your tokens.

iam in colab t4

Hour and 7 minutes

Does it take all this time with everyone?

1 Hour and 24 minutes

1 Hour and 30 minutes

Building wheels for collected packages: mamba_ssm

how????

https://huggingface.co/spaces/tiiuae/falcon-mamba-playground

how ???

--2024-11-25 22:28:22-- https://huggingface.co/nvidia/Hymba-1.5B-Base/resolve/main/setup.sh
Resolving huggingface.co (huggingface.co)... 3.165.160.61, 3.165.160.12, 3.165.160.11, ...
Connecting to huggingface.co (huggingface.co)|3.165.160.61|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1243 (1.2K) [text/plain]
Saving to: ‘setup.sh.1’

setup.sh.1 100%[===================>] 1.21K --.-KB/s in 0s

2024-11-25 22:28:22 (956 MB/s) - ‘setup.sh.1’ saved [1243/1243]

Enter CUDA version (12.1 or 12.4): 12.1
setup.sh: line 13: conda: command not found
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.46.2)
Collecting transformers
Downloading transformers-4.46.3-py3-none-any.whl.metadata (44 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.1/44.1 kB 2.3 MB/s eta 0:00:00
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.16.1)
Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.26.2)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.26.4)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.2)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.2)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.9.11)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.32.3)
Requirement already satisfied: tokenizers<0.21,>=0.20 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.20.3)
Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.5)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.6)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (2024.10.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (4.12.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.8.30)
Downloading transformers-4.46.3-py3-none-any.whl (10.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/10.0 MB 81.5 MB/s eta 0:00:00
Installing collected packages: transformers
Attempting uninstall: transformers
Found existing installation: transformers 4.46.2
Uninstalling transformers-4.46.2:
Successfully uninstalled transformers-4.46.2
Successfully installed transformers-4.46.3
Collecting tiktoken
Downloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken) (2024.9.11)
Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.10/dist-packages (from tiktoken) (2.32.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (3.4.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (2024.8.30)
Downloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 20.5 MB/s eta 0:00:00
Installing collected packages: tiktoken
Successfully installed tiktoken-0.8.0
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (0.2.0)
Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (4.25.5)
Collecting ninja
Downloading ninja-1.11.1.2-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (5.3 kB)
Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (0.8.0)
Collecting triton
Downloading triton-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.3 kB)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (24.2)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from triton) (3.16.1)
Downloading ninja-1.11.1.2-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (422 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 422.9/422.9 kB 9.0 MB/s eta 0:00:00
Downloading triton-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (209.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.5/209.5 MB 5.5 MB/s eta 0:00:00
Installing collected packages: triton, ninja
Successfully installed ninja-1.11.1.2 triton-3.1.0
Cloning into 'mamba'...
remote: Enumerating objects: 648, done.
remote: Counting objects: 100% (300/300), done.
remote: Compressing objects: 100% (105/105), done.
remote: Total 648 (delta 228), reused 207 (delta 195), pack-reused 348 (from 1)
Receiving objects: 100% (648/648), 1.55 MiB | 4.47 MiB/s, done.
Resolving deltas: 100% (340/340), done.
Obtaining file:///content/mamba
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from mamba_ssm==2.2.2) (2.5.1+cu121)
Requirement already satisfied: ninja in /usr/local/lib/python3.10/dist-packages (from mamba_ssm==2.2.2) (1.11.1.2)
Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from mamba_ssm==2.2.2) (0.8.0)
Requirement already satisfied: triton in /usr/local/lib/python3.10/dist-packages (from mamba_ssm==2.2.2) (3.1.0)
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (from mamba_ssm==2.2.2) (4.46.3)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from mamba_ssm==2.2.2) (24.2)
Requirement already satisfied: setuptools>=61.0.0 in /usr/local/lib/python3.10/dist-packages (from mamba_ssm==2.2.2) (75.1.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->mamba_ssm==2.2.2) (3.16.1)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->mamba_ssm==2.2.2) (4.12.2)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->mamba_ssm==2.2.2) (3.4.2)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->mamba_ssm==2.2.2) (3.1.4)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->mamba_ssm==2.2.2) (2024.10.0)
Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch->mamba_ssm==2.2.2) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch->mamba_ssm==2.2.2) (1.3.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.10/dist-packages (from transformers->mamba_ssm==2.2.2) (0.26.2)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers->mamba_ssm==2.2.2) (1.26.4)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers->mamba_ssm==2.2.2) (6.0.2)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers->mamba_ssm==2.2.2) (2024.9.11)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers->mamba_ssm==2.2.2) (2.32.3)
Requirement already satisfied: tokenizers<0.21,>=0.20 in /usr/local/lib/python3.10/dist-packages (from transformers->mamba_ssm==2.2.2) (0.20.3)
Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers->mamba_ssm==2.2.2) (0.4.5)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers->mamba_ssm==2.2.2) (4.66.6)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->mamba_ssm==2.2.2) (3.0.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers->mamba_ssm==2.2.2) (3.4.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers->mamba_ssm==2.2.2) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers->mamba_ssm==2.2.2) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers->mamba_ssm==2.2.2) (2024.8.30)
Building wheels for collected packages: mamba_ssm
Building editable for mamba_ssm (pyproject.toml) ... done
Created wheel for mamba_ssm: filename=mamba_ssm-2.2.2-0.editable-cp310-cp310-linux_x86_64.whl size=15244 sha256=d4cc57ff6a414c19d0c77a99be8317f08eda37f33af155089cecf2348c0eb8ae
Stored in directory: /tmp/pip-ephem-wheel-cache-yyu274hm/wheels/fb/bd/db/47d39b5aa5f1c5ccf671710c66e7a084dd82fb317f85f52625
Successfully built mamba_ssm
Installing collected packages: mamba_ssm
Successfully installed mamba_ssm-2.2.2
Cloning into 'causal-conv1d'...
remote: Enumerating objects: 343, done.
remote: Counting objects: 100% (140/140), done.
remote: Compressing objects: 100% (45/45), done.
remote: Total 343 (delta 121), reused 95 (delta 95), pack-reused 203 (from 1)
Receiving objects: 100% (343/343), 81.12 KiB | 9.01 MiB/s, done.
Resolving deltas: 100% (176/176), done.

torch.version = 2.5.1+cu121

Traceback (most recent call last):
File "/content/causal-conv1d/setup.py", line 166, in
_, bare_metal_version = get_cuda_bare_metal_version(CUDA_HOME)
File "/content/causal-conv1d/setup.py", line 59, in get_cuda_bare_metal_version
raw_output = subprocess.check_output(
File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 503, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.10/subprocess.py", line 971, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/cuda-12.1/bin/nvcc'
Cloning into 'attention-gym'...
remote: Enumerating objects: 206, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 206 (delta 24), reused 26 (delta 17), pack-reused 162 (from 1)
Receiving objects: 100% (206/206), 84.78 KiB | 4.71 MiB/s, done.
Resolving deltas: 100% (84/84), done.
Processing /content/attention-gym
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from attn_gym==0.0.3.dev5+gbbf437e) (2.5.1+cu121)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->attn_gym==0.0.3.dev5+gbbf437e) (3.16.1)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->attn_gym==0.0.3.dev5+gbbf437e) (4.12.2)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->attn_gym==0.0.3.dev5+gbbf437e) (3.4.2)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->attn_gym==0.0.3.dev5+gbbf437e) (3.1.4)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->attn_gym==0.0.3.dev5+gbbf437e) (2024.10.0)
Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch->attn_gym==0.0.3.dev5+gbbf437e) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch->attn_gym==0.0.3.dev5+gbbf437e) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->attn_gym==0.0.3.dev5+gbbf437e) (3.0.2)
Building wheels for collected packages: attn_gym
Building wheel for attn_gym (pyproject.toml) ... done
Created wheel for attn_gym: filename=attn_gym-0.0.3.dev5+gbbf437e-py3-none-any.whl size=16103 sha256=84803e8b588ca35a6bf17ed9d5c18fabb9dd17d8b6871feeffcbb09aa8e0ec9d
Stored in directory: /root/.cache/pip/wheels/c0/fc/ed/76ed4737439db911521dfbcc4789939c202858ac32b917aa42
Successfully built attn_gym
Installing collected packages: attn_gym
Successfully installed attn_gym-0.0.3.dev5+gbbf437e
Collecting flash_attn
Downloading flash_attn-2.7.0.post2.tar.gz (2.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 35.0 MB/s eta 0:00:00
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Preparing metadata (setup.py) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Installation completed with CUDA 12.1.

it run
but

OSError: nvidia/Hymba-1.5B-Base does not appear to have a file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt or flax_model.msgpack.

it run
but

OSError: nvidia/Hymba-1.5B-Base does not appear to have a file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt or flax_model.msgpack.

We released the model weights and it should work now.

from transformers import LlamaTokenizer, AutoModelForCausalLM, AutoTokenizer, AutoModel, StopStringCriteria, StoppingCriteriaList
import torch

Load the tokenizer and model

repo_name = "nvidia/Hymba-1.5B-Base"

tokenizer = AutoTokenizer.from_pretrained(repo_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_name, trust_remote_code=True)
model = model.cuda().to(torch.bfloat16)

/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning:
The secret HF_TOKEN does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/selective_scan_interface.py:164: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead.
def forward(ctx, xz, conv1d_weight, conv1d_bias, x_proj_weight, delta_proj_weight,
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/selective_scan_interface.py:240: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda') instead.
def backward(ctx, dout):
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/triton/layer_norm.py:986: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead.
def forward(
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/triton/layer_norm.py:1045: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda') instead.
def backward(ctx, dout, *args):
/usr/local/lib/python3.10/dist-packages/mamba_ssm/distributed/tensor_parallel.py:26: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead.
def forward(ctx, x, weight, bias, process_group=None, sequence_parallel=True):
/usr/local/lib/python3.10/dist-packages/mamba_ssm/distributed/tensor_parallel.py:62: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda') instead.
def backward(ctx, grad_output):
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/triton/ssd_combined.py:758: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead.
def forward(ctx, zxbcdt, conv1d_weight, conv1d_bias, dt_bias, A, D, chunk_size, initial_states=None, seq_idx=None, dt_limit=(0.0, float("inf")), return_final_states=False, activation="silu",
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/triton/ssd_combined.py:836: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda') instead.
def backward(ctx, dout, *args):
model.safetensors: 100%
 3.05G/3.05G [01:13<00:00, 42.0MB/s]

ModuleNotFoundError Traceback (most recent call last)
in <cell line: 8>()
6
7 tokenizer = AutoTokenizer.from_pretrained(repo_name, trust_remote_code=True)
----> 8 model = AutoModelForCausalLM.from_pretrained(repo_name, trust_remote_code=True)
9 model = model.cuda().to(torch.bfloat16)

6 frames
~/.cache/huggingface/modules/transformers_modules/nvidia/Hymba-1.5B-Base/6abbf5eed08e3f17fa3a1013bda8c2e3107fcaaf/modeling_hymba.py in init(self, *args, **kwargs)
1191 # assert self.config.sliding_window is not None
1192
-> 1193 from torch.nn.attention.flex_attention import flex_attention, create_block_mask, and_masks, or_masks
1194 from functools import partial
1195

ModuleNotFoundError: No module named 'torch.nn.attention.flex_attention'

NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.

colab t4

t4 no amper
no flex_attention