Evaluating Large Language Models for Detecting Antisemitism

Accepted to EMNLP 2025 Main Conference

Below, we summarize our main findings and contributions across eight models we study:

• We present the first systematic evaluation of LLMs for antisemitism detection, demonstrating differences in utility (refusal rates, ambiguity, and repetitive generation) and performance traceable to model selection.
• Across nearly all models, our engineered Guided-CoT consistently outperforms Zero-Shot and Zero-Shot-CoT, regardless of decoding strategy, model size, or reasoning capability. Using Self-consistency, Guided-CoT improves positive-class F1-scores by at least 0.03 up to 0.13 compared to Zero-Shot-CoT and reduces refusal rates to nearly 0%, thus enhancing model utility.
• Providing additional context (in our case, the IHRA definition with contemporary examples as policy instead of a short definition) does not necessarily improve model performance under Zero-Shot or Zero-Shot-CoT prompts, with some models experiencing even a decrease in performance. In such cases where there is a need to provide a policy in the prompt, Guided-CoT can help.
• We introduce metrics to quantify model explanations and find that Zero-Shot prompts result in homogeneous responses across models, yet individual models distinguish between antisemitic and non-antisemitic cases significantly. In contrast, CoT-based prompts, especially Guided-CoT, highlight differences in explanations across all models, while these differences between positive and negative classes are not significant for most models.
• Qualitative analysis reveals that LLMs struggle to understand contextual cues in writing patterns. LLMs label posts as antisemitic solely because they contain stereotypical or offensive terms; additionally, LLMs mislabel quoted text and news-style reports, as well as neutral or critical opinions. Interestingly, LLMs flag typos (e.g., 'kikes' intended as 'likes') and proper nouns (e.g., 'Kiké') that resemble slurs as antisemitic.