Mert Erbak's picture

Mert Erbak PRO

merterbak

·

AI & ML interests

Currently NLP and Image Processing

Recent Activity

reacted to mmhamdy's post with 🔥 about 9 hours ago

🎉 We're excited to introduce MemoryCode, a novel synthetic dataset designed to rigorously evaluate LLMs' ability to track and execute coding instructions across multiple sessions. MemoryCode simulates realistic workplace scenarios where a mentee (the LLM) receives coding instructions from a mentor amidst a stream of both relevant and irrelevant information. 💡 But what makes MemoryCode unique?! The combination of the following: ✅ Multi-Session Dialogue Histories: MemoryCode consists of chronological sequences of dialogues between a mentor and a mentee, mirroring real-world interactions between coworkers. ✅ Interspersed Irrelevant Information: Critical instructions are deliberately interspersed with unrelated content, replicating the information overload common in office environments. ✅ Instruction Updates: Coding rules and conventions can be updated multiple times throughout the dialogue history, requiring LLMs to track and apply the most recent information. ✅ Prospective Memory: Unlike previous datasets that cue information retrieval, MemoryCode requires LLMs to spontaneously recall and apply relevant instructions without explicit prompts. ✅ Practical Task Execution: LLMs are evaluated on their ability to use the retrieved information to perform practical coding tasks, bridging the gap between information recall and real-world application. 📌 Our Findings 1️⃣ While even small models can handle isolated coding instructions, the performance of top-tier models like GPT-4o dramatically deteriorates when instructions are spread across multiple sessions. 2️⃣ This performance drop isn't simply due to the length of the context. Our analysis indicates that LLMs struggle to reason compositionally over sequences of instructions and updates. They have difficulty keeping track of which instructions are current and how to apply them. 🔗 Paper: https://huggingface.co/papers/2502.13791 📦 Code: https://github.com/for-ai/MemoryCode

reacted to lysandre's post with ❤️ about 10 hours ago

SmolVLM-2 and SigLIP-2 are now part of `transformers` in dedicated releases! They're added on top of the v4.49.0 release, and can be installed from the following tags: `v4.49.0-SmolVLM-2` and `v4.49.0-SigLIP-2`. This marks a new beginning for the release process of transformers. For the past five years, we've been doing monthly releases featuring many models (v4.49.0, the latest release, features 9 new architectures). Starting with SmolVLM-2 & SigLIP2, we'll now additionally release tags supporting new models on a stable branch. These models are therefore directly available for use by installing from the tag itself. These tags will continue to be updated with fixes applied to these models. Going forward, continue expecting software releases following semantic versioning: v4.50.0 will have ~10 new architectures compared to v4.49.0, as well as a myriad of new features, improvements and bug fixes. Accompanying these software releases, we'll release tags offering brand new models as fast as possible, to make them accessible to all immediately.

liked a Space about 11 hours ago

gradio/theme-gallery

View all activity

Organizations

merterbak's activity

New activity in xai-org/grok-1 11 months ago

Fun Mode

#2 opened 11 months ago by

Fun Mode

#2 opened 11 months ago by