24 1 9

s

Tom-Neverwinter

Tom-Neverwinter

AI & ML interests

Making improvements to help the world.

Recent Activity

reacted to csabakecskemeti's post with 🔥 19 days ago

I've built a small utility to split safetensors file by file. The issue/need came up when I've tried to convert the new Deepseek V3 model from FP8 to BF16. The only Ada architecture GPU I have is an RTX 4080 and the 16GB vram was just wasn't enough for the conversion. BTW: I'll upload the bf16 version here: https://huggingface.co/DevQuasar/deepseek-ai.DeepSeek-V3-Base-bf16 (it will take a while - days with my upload speed) If anyone has access the resources to test it I'd appreciate a feedback if it's working or not. The tool, is available from here: https://github.com/csabakecskemeti/ai_utils/blob/main/safetensor_splitter.py It's splitting every file to n pieces by the layers if possible, and create a new "model.safetensors.index.json" file. I've tested it with Llama 3.1 8B and multiple split sizes, and validated by using inference pipeline. use `--help` for usage Please note current version expects the model is already multiple file and have a "model.safetensors.index.json" layer-safetensor mapping file.

new activity about 1 month ago

Apollo-LMMs/README:model pulled

reacted to tomaarsen's post with ❤️ 3 months ago

📣 Sentence Transformers v3.2.0 is out, marking the biggest release for inference in 2 years! 2 new backends for embedding models: ONNX (+ optimization & quantization) and OpenVINO, allowing for speedups up to 2x-3x AND Static Embeddings for 500x speedups at 10-20% accuracy cost. 1️⃣ ONNX Backend: This backend uses the ONNX Runtime to accelerate model inference on both CPU and GPU, reaching up to 1.4x-3x speedup depending on the precision. We also introduce 2 helper methods for optimizing and quantizing models for (much) faster inference. 2️⃣ OpenVINO Backend: This backend uses Intel their OpenVINO instead, outperforming ONNX in some situations on CPU. Usage is as simple as `SentenceTransformer("all-MiniLM-L6-v2", backend="onnx")`. Does your model not have an ONNX or OpenVINO file yet? No worries - it'll be autoexported for you. Thank me later 😉 🔒 Another major new feature is Static Embeddings: think word embeddings like GLoVe and word2vec, but modernized. Static Embeddings are bags of token embeddings that are summed together to create text embeddings, allowing for lightning-fast embeddings that don't require any neural networks. They're initialized in one of 2 ways: 1️⃣ via Model2Vec, a new technique for distilling any Sentence Transformer models into static embeddings. Either via a pre-distilled model with `from_model2vec` or with `from_distillation` where you do the distillation yourself. It'll only take 5 seconds on GPU & 2 minutes on CPU, no dataset needed. 2️⃣ Random initialization. This requires finetuning, but finetuning is extremely quick (e.g. I trained with 3 million pairs in 7 minutes). My final model was 6.6% worse than bge-base-en-v1.5, but 500x faster on CPU. Full release notes: https://github.com/UKPLab/sentence-transformers/releases/tag/v3.2.0 Documentation on Speeding up Inference: https://sbert.net/docs/sentence_transformer/usage/efficiency.html

View all activity

Organizations

None yet

Tom-Neverwinter's activity

New activity in Apollo-LMMs/README about 1 month ago

model pulled

#1 opened about 1 month ago by

Tom-Neverwinter

New activity in multimodalart/flux-lora-the-explorer 5 months ago

how to make a lora

#2 opened 5 months ago by

guardiancc

New activity in meta-llama/Llama-3.1-8B-Instruct 6 months ago

Issues loading model with ooabooga textgenwebui

#20 opened 6 months ago by

Kenji776

New activity in Xenova/whisper-speaker-diarization 6 months ago

how do we run this?

#2 opened 6 months ago by

Tom-Neverwinter

New activity in lmstudio-community/DeepSeek-Coder-V2-Lite-Instruct-GGUF 6 months ago

GGUF for the 236B model

#4 opened 6 months ago by

amarmir

New activity in open-llm-leaderboard/open_llm_leaderboard 6 months ago

WizardLM-8x22B Evaluation failed

#823 opened 7 months ago by

llama-anon

New activity in DavidAU/L3-Stheno-Maid-Blackroot-Grand-HORROR-16B-GGUF 7 months ago

Evil.

#1 opened 7 months ago by

Reithan

New activity in TheDrummer/Llama-3SOME-8B-v2-GGUF 7 months ago

can we have some more details?

#1 opened 7 months ago by

Tom-Neverwinter

New activity in LoneStriker/DeepSeek-Coder-V2-Instruct-GGUF 7 months ago

How good is the gguf?

#3 opened 7 months ago by

Tom-Neverwinter

New activity in ddh0/Phi-3-mini-4k-instruct-bf16-GGUF 8 months ago

censored

#1 opened 8 months ago by

Tom-Neverwinter

New activity in LiteLLMs/Mixtral-8x22B-v0.1-GGUF 9 months ago

ram usage

#1 opened 9 months ago by

Tom-Neverwinter

New activity in bartowski/Mixtral-8x22B-v0.1-GGUF 9 months ago

resources to run

#3 opened 9 months ago by

Tom-Neverwinter

New activity in bartowski/Beyonder-4x7B-v3-exl2 10 months ago

3.0 bpw?

#1 opened 10 months ago by

CulturedMan

New activity in ai21labs/Jamba-v0.1 10 months ago

multiple gpu?

#3 opened 10 months ago by

bdambrosio

New activity in NeverSleep/Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss-GGUF 10 months ago

Missing config.json

#2 opened 12 months ago by

Cayleb

New activity in vikhyatk/moondream2 10 months ago

enter it into the leaderboard?

#6 opened 10 months ago by

Tom-Neverwinter

New activity in hpcai-tech/Open-Sora 10 months ago

safetensors?

#5 opened 10 months ago by

Tom-Neverwinter

New activity in google/gemma-7b 10 months ago

Dont download, google scuttled this model

#77 opened 10 months ago by

Tom-Neverwinter

New activity in NousResearch/Nous-Hermes-2-Yi-34B-GGUF 10 months ago

is this filtered(cencored) or not? can i run it on 3090?

#1 opened 10 months ago by

Avos0001

New activity in ShinojiResearch/Senku-70B-Full 10 months ago

Hardware

#5 opened 11 months ago by

Tom-Neverwinter