google/vit-base-patch16-224-in21k
Image Feature Extraction
•
Updated
•
4.34M
•
287
3000
gilangf3000
AI & ML interests
None yet
Recent Activity
liked
a model
13 days ago
google/vit-base-patch16-224-in21k
liked
a model
13 days ago
LukeJacob2023/nsfw-image-detector
updated
a Space
23 days ago
gilangf3000/text-classification
Organizations
gilangf3000's activity
liked
2
models
13 days ago
updated
2
Spaces
23 days ago
liked
a
Space
23 days ago
liked
a
dataset
24 days ago
reacted to
singhsidhukuldeep's
post with 🤗
about 1 month ago
Post
1524
Remember Gemini, GPT-4o, all being true multimodal models 🌟.
Now we have a paper 📄 describing an architecture that might achieve that!
Uni-MoE: a native multimodal, Unified Mixture of Experts (MoE) architecture 🏗️.
Uni-MoE integrates various modalities (text 📝, image 🖼️, audio 🎵, video 📹, speech 🗣️) using modality-specific encoders and connectors for a cohesive multimodal understanding.
Training Strategy:
1️⃣ Training cross-modality alignment with diverse connectors 🔄.
2️⃣ Training modality-specific experts using cross-modality instruction data 📊.
3️⃣Tuning the Uni-MoE framework with Low-Rank Adaptation (LoRA) on mixed multimodal data 🔧.
Technical Details:
Modality-Specific Encoders: CLIP for images 🖼️, Whisper for speech 🗣️, BEATs for audio 🎵.
MoE-Based Blocks: Shared self-attention layers, feed-forward networks (FFN) based experts, and sparse routers for token-level expertise allocation 🚀.
Efficient Training: Utilizes LoRA for fine-tuning pre-trained experts and self-attention layers 🛠️.
Uni-MoE outperforms traditional dense models on benchmarks like A-OKVQA, OK-VQA, VQAv2, MMBench, RACE-Audio, and English High School Listening Test 🏆.
The code is open-sourced as well: https://github.com/HITsz-TMG/UMOE-Scaling-Unified-Multimodal-LLMs/tree/master/Uni_MoE_v2
Paper: Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts (2405.11273)
Now we have a paper 📄 describing an architecture that might achieve that!
Uni-MoE: a native multimodal, Unified Mixture of Experts (MoE) architecture 🏗️.
Uni-MoE integrates various modalities (text 📝, image 🖼️, audio 🎵, video 📹, speech 🗣️) using modality-specific encoders and connectors for a cohesive multimodal understanding.
Training Strategy:
1️⃣ Training cross-modality alignment with diverse connectors 🔄.
2️⃣ Training modality-specific experts using cross-modality instruction data 📊.
3️⃣Tuning the Uni-MoE framework with Low-Rank Adaptation (LoRA) on mixed multimodal data 🔧.
Technical Details:
Modality-Specific Encoders: CLIP for images 🖼️, Whisper for speech 🗣️, BEATs for audio 🎵.
MoE-Based Blocks: Shared self-attention layers, feed-forward networks (FFN) based experts, and sparse routers for token-level expertise allocation 🚀.
Efficient Training: Utilizes LoRA for fine-tuning pre-trained experts and self-attention layers 🛠️.
Uni-MoE outperforms traditional dense models on benchmarks like A-OKVQA, OK-VQA, VQAv2, MMBench, RACE-Audio, and English High School Listening Test 🏆.
The code is open-sourced as well: https://github.com/HITsz-TMG/UMOE-Scaling-Unified-Multimodal-LLMs/tree/master/Uni_MoE_v2
Paper: Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts (2405.11273)
liked
a
model
about 1 month ago
liked
2
Spaces
about 1 month ago
liked
a
model
about 1 month ago
liked
9
Spaces
about 1 month ago
Running
2
📚
Vpstest
Running
1
🚀
Custom Youtube Algorithm
Running
3
📈
Algorithmic Lyrics Classifier
Running
7
🐠
Anime Aesthetic
Running
2
🚀
Bg Remove Anime Webcam
Running
14
💻
AnimeIns CPU
Running
697
🔍😊
Search Your Face Online
Track your online presence with reverse face search
Running
9
🛝
Datasets API Playground
Running
73
🤗