โ Hosting our own inference was not enough: now the Hub 4 new inference providers: fal, Replicate, SambaNova Systems, & Together AI.
Check model cards on the Hub: you can now, in 1 click, use inference from various providers (cf video demo)
Their inference can also be used through our Inference API client. There, you can use either your custom provider key, or your HF token, then billing will be handled directly on your HF account, as a way to centralize all expenses.
๐ธ Also, PRO users get 2$ inference credits per month!
If you haven't seen yet, we just released Inference Providers ๐
> 4 new serverless inference providers on the Hub ๐คฏ > Use your HF API key or personal key with all providers ๐ > Chat with Deepseek R1, V3, and more on HF Hub ๐ > We support Sambanova, TogetherAI, Replicate, and Fal.ai ๐ช
Best of all, we don't charge any markup on top of the provider ๐ซฐ Have you tried it out yet? HF Pro accounts get $2 of free usage for the provider inference.
Groundbreaking Research Alert: Can Large Language Models Really Understand Personal Preferences?
A fascinating new study from researchers at University of Notre Dame, Xi'an Jiaotong University, and Universitรฉ de Montrรฉal introduces PERRECBENCH - a novel benchmark for evaluating how well Large Language Models (LLMs) understand user preferences in recommendation systems.
Key Technical Insights: - The benchmark eliminates user rating bias and item quality factors by using relative ratings and grouped ranking approaches - Implements three distinct ranking methods: pointwise rating prediction, pairwise comparison, and listwise ranking - Evaluates 19 state-of-the-art LLMs including Claude-3.5, GPT-4, Llama-3, Mistral, and Qwen models - Uses Kendall's tau correlation to measure ranking accuracy - Incorporates BM25 retriever with configurable history items (k=4 by default)
Notable Findings: - Current LLMs struggle with true personalization, achieving only moderate correlation scores - Larger models don't always perform better - challenging conventional scaling laws - Pairwise and listwise ranking methods outperform pointwise approaches - Open-source models like Mistral-123B and Llama-3-405B compete well with proprietary models - Weight merging strategy shows promise for improving personalization capabilities
The research reveals that while LLMs excel at many tasks, they still face significant challenges in understanding individual user preferences. This work opens new avenues for improving personalized recommendation systems and highlights the importance of developing better evaluation methods.
A must-read for anyone interested in LLMs, recommender systems, or personalization technology. The team has made their benchmark and code publicly available for further research.
Ok, my 14B DeepSeek R1 merge with Qwen2.5 1M is really hot right nowโit's got 2.6k downloads! It's sitting pretty as the top trending model on the third page. ๐ฅ
I've made an uncensored version of DeepSeek-R1-Distill-Llama-8B with merge. It's passing the "say f***" censor test. Tested with lm-evaluation-harness on standard open llm leaderboard tests + hellaswag. Scores are improved in most. Details on the model card.