The prevailing view says syllabic rhythm is all about the jaw. We tested that — evidence from the MOCHA-TIMIT dataset (460 sentences, 9 British English speakers) by extracting syllabic and supra-syllabic modulations from acoustic, articulatory (EMA, EPG) and laryngographic (EGG) signals, and measuring pairwise phase consistency between articulatory and acoustic modulations. Results show that the larynx is the real driver of acoustic amplitude rhythmic modulations; the contribution of the jaw is completely mediated by the larynx.
@inproceedings{yuan_lancia_jep2026_rhythm,title={Articulatory Bases of Acoustic Amplitude Rhythmic Modulations},author={Lancia, Leonardo and Yuan, Zheng},month=jun,year={2026},booktitle={Journ{\'e}es d'{\'E}tudes sur la Parole (JEP) 2026},address={Montpellier, France},note={Poster \#73},}
Multi-Tier Forced Alignment for the MOCHA-TIMIT Corpus
MOCHA-TIMIT provides synchronized multimodal recordings (audio, EMA, EPG, laryngograph) for 9 speakers and 3,996 utterances, yet only 2 speakers had been manually cleaned for use. We present a five-stage validation pipeline—binary decoding, filename reconciliation, quality control, forced alignment, and multi-tier generation—that unlocks the full 9-speaker corpus. Comparing US and UK MFA models with G2P augmentation, the UK+G2P model achieves 80.5% phoneme accuracy, a 9.3% improvement over the US baseline driven by a 59% reduction in deletion rate. The final output is an enriched 8-tier TextGrid corpus with phonemic, syllabic, and prosodic annotations. Using this corpus, we demonstrate three downstream applications: laryngeal vs. jaw control of syllabic rhythm, cross-accent verification via multimodal acoustic–articulatory analysis, and an acoustic-to-articulatory proxy model.
@inproceedings{yuan_lancia_jep2026_mocha,title={Multi-Tier Forced Alignment for the {MOCHA-TIMIT} Corpus},author={Yuan, Zheng and Lancia, Leonardo},month=jun,year={2026},booktitle={Journ{\'e}es d''{\'E}tudes sur la Parole (JEP) 2026},address={Montpellier, France},note={Poster \#83},}
MythNER: Multi-Agent NER Extraction and Benchmarking in Chinese Myth Narratives
We introduce MythNER, a culturally grounded coarse-grained NER benchmark for Chinese mythology built from Ne Zha subtitle text with four labels (PER/LOC/ORG/OBJ) and conservative annotation criteria. We evaluate a zero-shot Chinese spaCy model, a supervised BERT token-classification baseline, and a multi-agent LLM extraction pipeline on the held-out Ne Zha Part 2 test set. Results show substantial domain shift for off-the-shelf NER, strong gains from supervised adaptation, and further improvements from agentic extraction under well-chosen constraints.
@misc{mythner2026,title={{MythNER}: Multi-Agent {NER} Extraction and Benchmarking in Chinese Myth Narratives},author={Yuan, Zheng and Feng, Ruitao and Shen, Qingye and Zhang, Fengyong and Zhang, Zhihua and Liang, Sheng},month=jan,year={2026},booktitle={Proceedings of the ACL ARR 2026 (Under Review)},howpublished={OpenReview submission \#8178},url={https://openreview.net/forum?id=Tpq4Acgbja},note={Under review},}
2025
Steer-MoE: Efficient Audio-Language Alignment with a Mixture-of-Experts Steering Module
Ruitao Feng, Bixi Zhang, Sheng Liang, and Zheng Yuan
Aligning pretrained audio encoders and Large Language Models offers a parameter-efficient path to building multimodal agents. We introduce Steer-MoE, which freezes both the audio encoder and the LLM decoder and trains only a lightweight steering module integrated within the encoder’s layers. This module uses a Mixture-of-Experts router to dynamically select and apply learned steering vectors, progressively transforming continuous audio representations into a space comprehensible to the LLM. Experiments on ASR, audio understanding, and a function-calling task show strong performance with high modularity and computational efficiency.
@misc{feng_etal_steermoe_arxiv,title={{Steer-MoE}: Efficient Audio-Language Alignment with a Mixture-of-Experts Steering Module},author={Feng, Ruitao and Zhang, Bixi and Liang, Sheng and Yuan, Zheng},month=oct,year={2025},booktitle={arXiv preprint arXiv:2510.13558},eprint={2510.13558},archiveprefix={arXiv},primaryclass={cs.SD},url={https://arxiv.org/abs/2510.13558},note={Submitted to ICASSP 2026},}
2024
ART: The Alternating Reading Task Corpus for Speech Entrainment and Imitation
We introduce the Alternating Reading Task (ART) Corpus, a collection of dyadic sentence reading for studying the entrainment and imitation behaviour in speech communication. The ART corpus features three experimental conditions - solo reading, alternating reading, and deliberate imitation - as well as three subcorpora encompassing French-, Italian-, and Slovak-accented English. This design allows systematic investigation of speech entrainment in a controlled and less spontaneous setting. Alongside detailed transcriptions, it includes English proficiency scores, demographics, and in-experiment questionnaires for probing linguistic, personal and interpersonal influences on entrainment. Our presentation covers its design, collection, annotation processes, initial analysis, and future research prospects.
@inproceedings{yuan-etal-2024-art,title={ART: The Alternating Reading Task Corpus for Speech Entrainment and Imitation},author={Yuan, Zheng and {de Jong}, Dorina and Beňuš, Štefan and Nguyen, No{\"e}l and Feng, Ruitao and Sabo, R{\'o}bert and Fadiga, Luciano and {D'Ausilio}, Alessandro},editor={Calzolari, Nicoletta and Kan, Min-Yen and Hoste, Veronique and Lenci, Alessandro and Sakti, Sakriani and Xue, Nianwen},booktitle={Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},month=may,year={2024},address={Torino, Italia},publisher={ELRA and ICCL},url={https://aclanthology.org/2024.lrec-main.137/},pages={1548--1562},}
Language Proficiency and F0 Entrainment: A Study of L2 English Imitation in Italian, French, and Slovak Speakers
We investigate how language proficiency affects fundamental frequency (F0) entrainment in L2 English speech imitation tasks. Our study examines speakers from three different L1 backgrounds (Italian, French, and Slovak) and reveals significant correlations between proficiency levels and prosodic convergence patterns.
@inproceedings{yuan24b_speechprosody,title={Language Proficiency and F0 Entrainment: A Study of L2 English Imitation in Italian, French, and Slovak Speakers},author={Yuan, Zheng and Beňuš, Štefan and {D'Ausilio}, Alessandro},month=jul,year={2024},booktitle={Speech Prosody 2024},pages={1265--1269},doi={10.21437/SpeechProsody.2024-255},issn={2333-2042},}
Breathing features and their impact on speech perception of COVID-19 patients
We investigate how COVID-19 affects breathing patterns during speech production and how these changes impact speech perception. Our study reveals significant alterations in respiratory control that influence both speech quality and listener comprehension.
@inproceedings{jiang24_speechprosody,title={Breathing features and their impact on speech perception of COVID-19 patients},author={Jiang, Xiaoming and Yu, Lixin and Dai, Leinuo and Chen, Jinyang and Yuan, Zheng},month=jul,year={2024},booktitle={Speech Prosody 2024},pages={66--70},doi={10.21437/SpeechProsody.2024-14},issn={2333-2042},}
2023
The ADAIO System at the BEA-2023 Shared Task: Shared Task Generating AI Teacher Responses in Educational Dialogues
Adaeze Adigwe and Zheng Yuan
In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Jul 2023
This paper presents the ADAIO team‘s system entry in the Building Educational Applications (BEA) 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues. The task aims to assess the performance of state-of-the-art generative models as AI teachers in producing suitable responses within a student-teacher dialogue. Our system comprises evaluating various baseline models using OpenAI GPT-3 and designing diverse prompts to prompt the OpenAI models for teacher response generation. After the challenge, our system achieved second place by employing a few-shot prompt-based approach with the OpenAI text-davinci-003 model. The results highlight the few-shot learning capabilities of large-language models, particularly OpenAI‘s GPT-3, in the role of AI teachers.
@inproceedings{adigwe-yuan-2023-adaio,title={The {ADAIO} System at the {BEA}-2023 Shared Task: Shared Task Generating {AI} Teacher Responses in Educational Dialogues},author={Adigwe, Adaeze and Yuan, Zheng},editor={Kochmar, Ekaterina and Burstein, Jill and Horbach, Andrea and Laarmann-Quante, Ronja and Madnani, Nitin and Tack, Ana{\"i}s and Yaneva, Victoria and Yuan, Zheng and Zesch, Torsten},booktitle={Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)},month=jul,year={2023},address={Toronto, Canada},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2023.bea-1.65/},doi={10.18653/v1/2023.bea-1.65},pages={796--804},}
The ART of Conversation: Measuring Phonetic Convergence and Deliberate Imitation in L2-Speech with a Siamese RNN
This paper presents a novel approach to measuring phonetic convergence and deliberate imitation in second language (L2) speech using a Siamese Recurrent Neural Network. We introduce the ART (Alternating Reading Task) paradigm and demonstrate how deep learning methods can effectively quantify speech convergence patterns in L2 learners.
@inproceedings{yuan23b_interspeech,title={The ART of Conversation: Measuring Phonetic Convergence and Deliberate Imitation in L2-Speech with a Siamese RNN},author={Yuan, Zheng and Pastore, Aldo and {de Jong}, Dorina and Xu, Hao and Fadiga, Luciano and {D'Ausilio}, Alessandro},month=aug,year={2023},booktitle={Interspeech 2023},pages={132--136},doi={10.21437/Interspeech.2023-2283},issn={2958-1796},}
