Topic 23: What is LLM Inference, it's challenges and solutions for it
•
3
Ksenia Se
Kseniase
AI & ML interests
None yet
Recent Activity
posted
an
update
about 15 hours ago
Today, we spoke with Snowflake’s AI Research Team Leads, Yuxiong He and Samyam Rajbhandari (@samyam) (he is also one the researchers behind https://huggingface.co/papers/2401.08671 and other DeepSpeed papers)
Collaborating with their co-authors to reduce inference costs for enterprise-specific tasks, they observed that inputs are often significantly larger than outputs. This is because it’s in the nature of enterprises to analyze enormous amounts of information trying to extract valuable insights, which are much shorter. To address this, they developed SwiftKV https://huggingface.co/papers/2410.03960, an optimization that reduces LLM inference costs by up to 75% for Meta Llama LLMs, enhancing efficiency and performance in enterprise AI tasks.
Today they are open-sourcing SwiftKV (https://huggingface.co/Snowflake/Llama-3.1-SwiftKV-8B-Instruct) and ArcticTrainging Platform.
In our new episode "15 minutes with a Researcher" they explain how SwiftKV works, its applicability to other architectures, its limitations, and additional methods to further reduce computation costs in inference.
Watch the full 15 min interview here (https://youtu.be/9x1k7eXe-6Q?si=4_HQOyi1CPHgvlrx)
published
an
article
about 16 hours ago
Topic 23: What is LLM Inference, it's challenges and solutions for it
upvoted
an
article
4 days ago
🌁#83: GAN is back
Articles
Organizations
Kseniase's activity
upvoted
2
articles
4 days ago
Article
🌁#83: GAN is back
By
•
•
5Article
🅰️ℹ️ 1️⃣0️⃣1️⃣ **What is HtmlRAG, Multimodal RAG and Agentic RAG?**
By
•
•
6
upvoted
a
paper
4 days ago
upvoted
a
paper
5 days ago
upvoted
an
article
7 days ago
Article
🦸🏻#2: Your Go-To Vocabulary to Navigate the World of AI Agents and Agentic Workflows
By
•
•
9
upvoted
an
article
11 days ago
Article
🌁#82: AI and ML in Real Life
By
•
•
15
upvoted
a
paper
about 1 month ago
upvoted
a
collection
10 months ago