Tolga Cangöz's picture

Tolga Cangöz

tolgacangoz

·

standard_ai

AI & ML interests

AIGC

Recent Activity

reacted to Kseniase's post with ❤️ 1 day ago

8 types of RoPE As we always use Transformers, it's helpful to understand RoPE—Rotary Position Embedding. Since token order matters, RoPE encodes it by rotating token embeddings based on their position, so the model knows how to interpret which token comes first, second, and so on. Here are 8 types of RoPE that can be implemented in different cases: 1. Original RoPE -> https://huggingface.co/papers/2104.09864 Encodes token positions by rotating token embeddings in the complex plane via a position-based rotation matrix, thereby providing the self-attention mechanism with relative positional info. 2. LongRoPE -> https://huggingface.co/papers/2402.13753 Extends the context window of pre-trained LLMs to 2048k tokens, leveraging non-uniformities in positional interpolation with an efficient search. 3. LongRoPE2 -> https://huggingface.co/papers/2502.20082 Extends the effective context window of pre-trained LLMs to the target! length, rescaling RoPE guided by “needle-driven” perplexity. 4. Multimodal RoPE (MRoPE) -> https://huggingface.co/papers/2502.13923 Decomposes positional embedding into 3 components: temporal, height and width, so that positional features are aligned across modalities: text, images and videos. 5. Directional RoPE (DRoPE) -> https://huggingface.co/papers/2503.15029 Adds an identity scalar, improving how angles are handled without extra complexity. It helps balance accuracy, speed, and memory usage. 6. VideoRoPE -> https://huggingface.co/papers/2502.05173 Adapts RoPE for video, featuring 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable spacing. 7. VRoPE -> https://huggingface.co/papers/2502.11664 An another RoPE for video, which restructures positional indices and balances encoding for uniform spatial focus. 8. XPos (Extrapolatable Position Embedding) -> https://huggingface.co/papers/2212.10 Introduces an exponential decay factor into the rotation matrix, improving stability on long sequences.

reacted to Kseniase's post with 👀 1 day ago

8 types of RoPE As we always use Transformers, it's helpful to understand RoPE—Rotary Position Embedding. Since token order matters, RoPE encodes it by rotating token embeddings based on their position, so the model knows how to interpret which token comes first, second, and so on. Here are 8 types of RoPE that can be implemented in different cases: 1. Original RoPE -> https://huggingface.co/papers/2104.09864 Encodes token positions by rotating token embeddings in the complex plane via a position-based rotation matrix, thereby providing the self-attention mechanism with relative positional info. 2. LongRoPE -> https://huggingface.co/papers/2402.13753 Extends the context window of pre-trained LLMs to 2048k tokens, leveraging non-uniformities in positional interpolation with an efficient search. 3. LongRoPE2 -> https://huggingface.co/papers/2502.20082 Extends the effective context window of pre-trained LLMs to the target! length, rescaling RoPE guided by “needle-driven” perplexity. 4. Multimodal RoPE (MRoPE) -> https://huggingface.co/papers/2502.13923 Decomposes positional embedding into 3 components: temporal, height and width, so that positional features are aligned across modalities: text, images and videos. 5. Directional RoPE (DRoPE) -> https://huggingface.co/papers/2503.15029 Adds an identity scalar, improving how angles are handled without extra complexity. It helps balance accuracy, speed, and memory usage. 6. VideoRoPE -> https://huggingface.co/papers/2502.05173 Adapts RoPE for video, featuring 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable spacing. 7. VRoPE -> https://huggingface.co/papers/2502.11664 An another RoPE for video, which restructures positional indices and balances encoding for uniform spatial focus. 8. XPos (Extrapolatable Position Embedding) -> https://huggingface.co/papers/2212.10 Introduces an exponential decay factor into the rotation matrix, improving stability on long sequences.

liked a Space 11 days ago

Remade-AI/remade-effects

View all activity

Organizations

tolgacangoz's activity

upvoted a collection 11 days ago

Open-Sora 2.0

3 items • Updated 13 days ago • 10

upvoted a collection 19 days ago

Shot categorizer

Fine-tune of Florence-2 to generate shot categories, useful for data curation. Code: https://github.com/huggingface/movie-shot-categorizer. • 3 items • Updated 19 days ago • 2

upvoted an article 22 days ago

Article

You could have designed state of the art positional encoding

Nov 25, 2024

• 192

upvoted a collection 22 days ago

video-effects datasets

Smol datasets to emulate cool video effects like "squish", "dissolve", etc. Inspired by Pika effects. • 4 items • Updated Jan 28 • 3

upvoted a paper 23 days ago

Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget

Paper • 2407.15811 • Published Jul 22, 2024 • 2

upvoted a collection 23 days ago

AI Tools for Art - Feb '25

Tools & models from the 2nd issue of AI Tools for Art 🎉 Read more about February's releases: https://open.substack.com/pub/multimodalaiart • 18 items • Updated 24 days ago • 1

upvoted a paper 25 days ago

Towards an AI co-scientist

Paper • 2502.18864 • Published 27 days ago • 44

upvoted a collection 27 days ago

Remote VAE Inference Endpoints

Models and handler code used in https://huggingface.co/blog/remote_vae • 5 items • Updated 14 days ago • 4

upvoted a paper 28 days ago

AnyText2: Visual Text Generation and Editing With Customizable Attributes

Paper • 2411.15245 • Published Nov 22, 2024 • 1

upvoted an article 29 days ago

Article

Remote VAEs for decoding with HF endpoints 🤗

29 days ago

• 37

upvoted a paper 29 days ago

KernelBench: Can LLMs Write Efficient GPU Kernels?

Paper • 2502.10517 • Published Feb 14 • 2

upvoted a collection about 1 month ago

SigLIP 2

OpenCLIP and timm SigLIP 2 models • 45 items • Updated about 1 month ago • 14

upvoted 3 papers about 1 month ago

SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

Paper • 2502.14786 • Published Feb 20 • 133

Region-Adaptive Sampling for Diffusion Transformers

Paper • 2502.10389 • Published Feb 14 • 52

Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps

Paper • 2501.09732 • Published Jan 16 • 70

upvoted a collection about 1 month ago

DeepSeek-R1

8 items • Updated Jan 21 • 587

upvoted an article about 1 month ago

Article

Build awesome datasets for video generation

Feb 12

• 29

upvoted a paper about 1 month ago

Matryoshka Quantization

Paper • 2502.06786 • Published Feb 10 • 30

upvoted a collection about 2 months ago

video-effects

Fine-tunes of open video generation models like CogVideoX to emulate cool video effects like "squish", "dissolve", "cakeify", etc. Pika inspired. • 8 items • Updated 16 days ago • 6