AudSemThinker-QA

Corresponding paper: https://arxiv.org/abs/2505.14142

Model Description

AudSemThinker-QA is a specialized variant of AudSemThinker, meticulously fine-tuned for audio-based question answering tasks. It integrates a structured reasoning process based on auditory semantics, explicitly analyzing sound-generating agents, physical sound sources, generation mechanisms, and contextual cues.

This model is built upon the Qwen2.5-Omni-7B multimodal foundation model and is fine-tuned exclusively on the question answering subset of the novel AudSem dataset using Supervised Fine-Tuning (SFT). AudSemThinker-QA generates responses in a three-phase structure: a detailed <thinking> process, a listing of <semantic_elements>, and a concise <answer>.

How to Use

To use AudSemThinker-QA for audio question answering, you can load it using the transformers library. Ensure you have torch, torchaudio, and soundfile installed.

import soundfile as sf
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info
import torchaudio

# default: Load the model on the available device(s)
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    "gijs/audsemthinker-qa", 
    torch_dtype="auto", 
    device_map="auto",
    trust_remote_code=True,
    low_cpu_mem_usage=True
)

# We recommend enabling flash_attention_2 for better acceleration and memory saving.
# model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
#     "gijs/audsemthinker-qa",
#     torch_dtype="auto",
#     device_map="auto",
#     attn_implementation="flash_attention_2",
#     trust_remote_code=True,
#     low_cpu_mem_usage=True
# )

processor = Qwen2_5OmniProcessor.from_pretrained("gijs/audsemthinker-qa", trust_remote_code=True)

# Load and preprocess audio
audio_file = "path/to/your/audio.wav"
audio_input, sampling_rate = torchaudio.load(audio_file)
if sampling_rate != processor.feature_extractor.sampling_rate:
    audio_input = torchaudio.transforms.Resample(
        orig_freq=sampling_rate, 
        new_freq=processor.feature_extractor.sampling_rate
    )(audio_input)
audio_input = audio_input.squeeze().numpy()

# Example question
question = "What type of sound is present in the audio?"
user_prompt_text = f"You are given a question and an audio clip. Your task is to answer the question based on the audio clip. First, think about the question and the audio clip and put your thoughts in <think> and </think> tags. Then reason about the semantic elements involved in the audio clip and put your reasoning in <semantic_elements> and </semantic_elements> tags. Then answer the question based on the audio clip, put your answer in <answer> and </answer> tags.\nQuestion: {question}"

# Conversation format
conversation = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."}
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": audio_input},
            {"type": "text", "text": user_prompt_text}
        ],
    },
]

# Preparation for inference
text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
audios, images, videos = process_mm_info(conversation)
inputs = processor(
    text=text, 
    audio=audios, 
    images=images, 
    videos=videos, 
    return_tensors="pt", 
    padding=True
)
inputs = inputs.to(model.device).to(model.dtype)

# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=512)
response = processor.batch_decode(output_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
print(response[0])

# Expected output format for QA:
# <think>...detailed reasoning about the audio scene and question...</think>
# <semantic_elements>...list of identified semantic descriptors...</semantic_elements>
# <answer>...answer to the question...</answer>

Training Data

AudSemThinker-QA is fine-tuned on the Question Answering (QA) subset of the AudSem dataset. This subset primarily consists of audio-based multiple-choice and open-ended question answering examples, which were synthetically generated from YouTube closed captions through a robust multi-stage pipeline. The AudSem dataset is designed to minimize overlap with existing benchmarks.

Training Procedure

• Base Model: Qwen2.5-Omni-7B.
• Fine-tuning Paradigm: Supervised Fine-Tuning (SFT).
• Parameter-Efficient Fine-tuning: LoRA (Low-Rank Adaptation) applied to projection layers.
• Optimizer: AdamW.
• Learning Rate: 2e-04.
• Epochs: 1.
• Precision: bf16.
• Batch Size: 4.
• Hardware: Trained on a single H100 GPU.
• Training Time: Approximately 6 hours for the QA subset.
• Output Format: Trained to generate structured XML-like output with <think>, <semantic_elements>, and <answer> tags. The loss is computed only on the model completion part (assistant's response).

Evaluation Results

AudSemThinker-QA demonstrates strong performance in question-answering tasks across various audio domains, often outperforming general audio-language models on specific QA benchmarks.

Limitations and Bias

• Generalization: While strong in QA tasks, AudSemThinker-QA may exhibit trade-offs in performance on other task types (e.g., general audio captioning) compared to models trained on broader datasets.
• Speech Understanding: Similar to the base AudSemThinker, its speech understanding capabilities are not a primary focus and may be less developed.
• Data Contamination: While AudSem is designed to minimize overlap, the underlying Qwen2.5-Omni pretrained model might have encountered data present in test sets during its initial pretraining.

Ethical Considerations

• Data Sourcing: The AudSem dataset is primarily sourced from YouTube closed captions. While systematic checks for harmful content (e.g., child abuse, hate speech, sexual content, harassment) were performed and YouTube's community guidelines provide a safeguard, inherent biases or problematic content from the original video sources could potentially be present.
• Societal Impact: AudSemThinker-QA can contribute to positive societal impacts by enhancing audio-language understanding, particularly in question-answering scenarios. This could aid in various applications requiring precise information extraction from audio.

Citation

@misc{wijngaard2025audsemthinkerenhancingaudiolanguagemodels,
  title={AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound}, 
  author={Gijs Wijngaard and Elia Formisano and Michele Esposito and Michel Dumontier},
  year={2025},
  eprint={2505.14142},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2505.14142}, 
}