Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth

@SrRooT Thank you very much for taking the time to read our work and for sharing such thoughtful feedback. We really appreciate your close engagement with the Spanish samples.

All of the Drivelology examples in our dataset were collected directly from the Internet without modification, which explains why some contain typos (e.g., mosa → mosca) or non-standard punctuation. We recognise that this can sometimes distort the intended wordplay, and as you note, the models (and sometimes annotators) end up filling gaps from imperfect input.

On the cultural side, you are absolutely right that some samples reflect interactions between Mexican and European Spanish (not sure if it's Castilian), and that certain paradoxical or edgy references may be rooted in non-Spanish (e.g., French) framings of Spain's modern history. These are important observations: we did not curate such cases precisely because our goal was to capture how Drivelology circulates online "as-is," even if the humour lands awkwardly or uncomfortably in certain cultural contexts.

As we mention in the Limitations section of the paper, this work is ongoing, and the dataset will be expanded and refined. At present, we did not distinguish among Spanish varieties (Mexican vs. Castilian, etc.) mainly because we lacked annotators with both regional expertise and familiarity with Drivelological conventions. Even within other languages, many native speakers still struggle to interpret these texts (i.e. Drivelology in their own language), since the humour often depends on very specific rhetorical cues.

We very much welcome contributions from the community. If you or others are interested, we would be delighted to receive contributions or pull requests on our GitHub repository to help improve coverage and representation of Spanish varieties.

@Tomoomo When we conducted this study back in early 2025, we were definitely aiming to use the best models available. At the time, models like GPT-4.5 and Claude 3.7 were the top of the line. However, as we mentioned in our paper's Limitations section, running experiments at our scale with these models was unfortunately beyond our budget. The API costs are incredibly high, and we were often paying out-of-pocket to keep the project going. On the open-source side, we were limited by our hardware. We ran the experiments on a 4090 machine, and the Qwen3-14B model was the largest we could reliably fit and test. We would have loved to include even larger models if we had the resources.

Regarding your second point that smaller LLMs are "mostly useless" for complex language, our findings actually paint a more intricate and, we think, more compelling picture. The idea that performance simply scales with size isn't what we observed across the board. For instance, let's compare the performance of our open-source models against the larger proprietary ones on what was arguably our most difficult task: the "Hard" Narrative Selection (MCQA). As shown in Table 1, our qwen3-8b-instruct model achieved an accuracy of 26.78%. This result was a significant outlier, outperforming both claude-3.5-haiku (11.56%) and gpt-4o-mini (4.67%). This demonstrates that a smaller, 8B parameter model can develop a specialised capability for subtle reasoning that surpasses even the much larger, state-of-the-art models in a specific domain. Furthermore, when we analysed performance within the Qwen3 model family (from 4B to 14B parameters), the "bigger is always better" narrative was again challenged. As detailed in Table 2, the scaling effects were not linear. For the Drivelology Detection task, the 8B model, when prompted in Mandarin, achieved 78.81% accuracy, significantly outperforming the larger 14B model's 71.78%. We saw another non-linear pattern in the Tagging task, where performance dipped at the 8B size before recovering at 14B.

Taken together, these results suggest that the relationship between model size and the ability to understand complex Drivelological language is not a straight line. Rather than being "useless," smaller and mid-sized models can possess unique and sometimes superior capabilities for specific linguistics-related tasks.