AIP-Speech

Abstract

Speech Large Language Models (Speech LLMs) are increasingly deployed in real-world acoustic environments, yet it remains unclear which measurable properties of background sounds best explain performance degradation. We present AIP-Speech, an inference-only benchmark that evaluates Speech LLM robustness under controlled mixtures of real-world background recordings. Rather than comparing robustness across background categories alone, we characterize each recording using interpretable acoustic descriptors and examine their associations with degradation. Across five open Speech LLMs, we find that simple spectral descriptors—including Zero Crossing Rate, Spectral Centroid, and Spectral Rolloff—consistently exhibit strong associations with degradation. We further quantify robustness disparities across accents and evaluate whether inference-time prompting improves resilience under adverse acoustic conditions. AIP-Speech provides a reproducible benchmark and descriptor-annotated background corpus for analyzing Speech LLM robustness.

Benchmark

How Speech LLMs behave under background interference remains poorly understood. Existing benchmarks report aggregate performance under a limited set of acoustic conditions, providing little insight into which measurable acoustic properties are associated with degradation—or whether interference affects demographic groups uniformly.

AIP‑Speech isolates the effect of background interference by holding spoken content fixed while systematically varying the background audio. A clean utterance is mixed with a controlled background recording at multiple signal‑to‑noise ratios (SNRs); paired clean–noisy predictions from five open Speech LLMs are then related to interpretable acoustic descriptors, demographic subgroup, and inference‑time prompting.

A reproducible benchmark and evaluation framework for auditing Speech LLMs under controlled, real‑world background interference.
Descriptor‑level analysis relating interpretable acoustic descriptors to degradation across ASR and keyword spotting.
An evaluation of robustness disparities across demographic (accent) groups, and of inference‑time prompting under acoustic interference.

FIG. 1 — clean speech + controlled background mixtures, evaluated across five Speech LLMs, paired for descriptor, fairness, and prompt‑steering analysis.

Descriptors

Each of the 20 background recordings is characterized by standard acoustic descriptors computed directly from its waveform (Zero Crossing Rate, spectral centroid/rolloff/bandwidth/flux/flatness/contrast, RMS energy variance, and Silero VAD speech fraction). Degradation—the drop in task performance between clean and noisy audio—is then related to each descriptor via Spearman rank correlation.

Correlations from Table 2. Zero Crossing Rate shows the strongest association with degradation for both tasks (r_s = 0.645 ASR, 0.723 KWS, pooled across models). Asterisked Qwen2‑Audio KWS values were measured under the unsteered baseline protocol.

Background corpus

AIP‑Speech uses a fixed set of 20 publicly available real‑world background recordings drawn from ESC‑50, MS‑SNSD, NOISEX‑92, and MUSAN, each annotated with the descriptors above. Click a column to sort; the table underlies the descriptor‑level correlations in §02.

Fairness

Background interference does not affect every accent subgroup equally. Mean ΔWER (noisy − clean, relative to each subgroup's own clean baseline) is reported across ten accent groups from Mozilla Common Voice, for all five evaluated models. Some models degrade fairly uniformly; others show pronounced sensitivity to specific accents.

Table 4. Positive values indicate greater degradation than the subgroup's clean baseline; negative values indicate the model performed better than its clean baseline on that subgroup's noisy audio.

Prompt steering

Each model is evaluated under a baseline instruction and under a steering prompt that asks it to attend only to the intended foreground speaker. Effectiveness is summarized as the Residual Effect Ratio, RER = Δ_prompt / Δ_baseline: values below the dashed line at 1.0 indicate the steering prompt reduced degradation.

Table 5. Values above 1 indicate that the steering prompt increased degradation relative to the baseline instruction. RER falls below 1 only for Qwen2.5‑Omni‑3B on ASR (0.812) and for Gemma 3n (0.767) and Phi‑4‑MM (0.549) on KWS, so no model improves on both tasks.

Limitations

Limited to inference‑time evaluation of five open Speech LLMs on ASR and keyword spotting; findings may not generalize to conversational or reasoning‑intensive speech tasks.
The 20 background recordings, while diverse, do not exhaust the full range of real‑world acoustic environments.
Fairness analysis is limited to English accent subgroups and relies on the demographic annotations of the underlying datasets.
Descriptor‑level analysis identifies statistical associations with degradation, not causal relationships between acoustic properties and model behavior.

Cite

@inproceedings{dsouza2026aipspeech,
  title     = {{AIP-Speech}: Auditing Speech {LLM} Robustness to Background Interference},
  author    = {Dsouza, Aiden and Mishra, Shreyansh and Suneja, Yashraj and Swain, Rohan and Borkar, Samyak and Jaishankar, Arjun and Urade, Tanmay},
  booktitle = {Proceedings of the 2026 Impact Speech Workshop: Identifying, Measuring, Preventing, and Assessing Consequences of Bias in Speech {LLM}s ({IMPACT-SPEECH})},
  year      = {2026},
  publisher = {Association for Computational Linguistics},
  note      = {To appear},
}