A Safety-Constrained Speaker-Adaptive Algorithm for Foreign Language Pronunciation Assessment in Correctional Education
Keywords:
automatic pronunciation assessment, speaker adaptation, residual adapter, forced alignment, prosody, uncertainty, fairnessAbstract
This study presents a safety-constrained, speaker-adaptive algorithm for automatically assessing foreign language pronunciation in correctional education. The method separates stable acoustic speaker characteristics from target pronunciation competence so that pitch range, timbre, habitual speaking rate, and recording channel are not incorrectly treated as pronunciation errors. Two to six unlabeled enrollment utterances are summarized with robust statistics and
passed to a small residual adapter placed above a frozenself-supervised speech encoder. The adapter is restricted to
a nuisance subspace by orthogonal projection, a preservation loss, and a trust region. A safety gate combines enrollment quality, forced alignment reliability,
predictive uncertainty, score agreement, and profile drift
to accept the adapted estimate, return the global estimate,
or request human review. A reproducible simulation used
one hundred sixty virtual speaker profiles, three hundred
twenty unlabeled enrollment records, and nine hundred
sixty evaluation samples under group-separated cross-
validation. The proposed adapter reduced mean absolute
error from 9.04 to 8.31 points and increased quadratic
weighted agreement from 0.586 to 0.668. Selective scoring
covered 84.7 percent of samples with a mean absolute error
of 7.98 points. The data are synthetic and do not represent
incarcerated individuals. Real deployment requires ethical
approval, independent expert annotation, external
validation, subgroup auditing, and strict separation from
legal or disciplinary decisions
References
P. Bell, J. Fainberg, O. Klejch, J. Li, S. Renals, and P. Swietojanski,
“Adaptation algorithms for neural network-based speech
recognition: An overview,” IEEE Open Journal of Signal
Processing, vol. 2, pp. 33–66, 2021, doi:
1109/OJSP.2020.3045349.
A. Bhatia, S. Sinha, S. Dingliwal, et al., “Do not stop self-
supervision: Accent adaptation of speech representations via
residual adapters,” in Proc. Interspeech, 2023, pp. 3362–3366, doi:
21437/Interspeech.2023-2192.
Z. Zhang, Y. Wang, and J. Yang, “BiCAPT: Bidirectional
computer-assisted pronunciation training with normalizing flows,”
in Proc. Interspeech, 2022, pp. 4511–4515, doi:
21437/Interspeech.2022-10937.
L. Kürzinger, D. Winkelbauer, L. Li, T. Watzel, and G. Rigoll,
“CTC segmentation of large corpora for German end-to-end speech
recognition,” in Speech and Computer, LNCS 12335, 2020, pp.
–278, doi: 10.1007/978-3-030-60276-5_27.
V. C. Mathad, T. J. Mahr, N. Scherer, et al., “The impact of forced
alignment errors on automatic pronunciation evaluation,” in Proc.
Interspeech, 2021, pp. 1922–1926, doi:
21437/Interspeech.2021-1403.
B. Lin, L. Wang, X. Feng, and J. Zhang, “Automatic scoring at
multi-granularity for second language pronunciation,” in Proc.
Interspeech, 2020, pp. 3022–3026, doi:
21437/Interspeech.2020-1282.
Y. Gong, Z. Chen, I. H. Chu, P. Chang, and J. Glass, “Transformer-
based multi-aspect, multi-granularity non-native English speaker
pronunciation assessment,” in Proc. ICASSP, 2022, pp. 7262–
, doi: 10.1109/ICASSP43922.2022.9746743.
F. A. Chao, T. H. Lo, T. I. Wu, Y. T. Sung, and B. Chen, “A
hierarchical context-aware modeling approach for multi-aspect and
multi-granular pronunciation assessment,” in Proc. Interspeech,
, pp. 974–978, doi: 10.21437/Interspeech.2023-550.
H. C. Pei, H. Fang, X. Luo, and X. S. Xu, “Gradformer: A
framework for multi-aspect, multi-granularity pronunciation
assessment,” IEEE/ACM Transactions on Audio, Speech, and
Language Processing, vol. 32, pp. 554–563, 2024, doi:
1109/TASLP.2023.3335807.
A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A
framework for self-supervised learning of speech representations,”
in Advances in Neural Information Processing Systems, vol. 33,
, pp. 12449–12460.
W. N. Hsu, B. Bolte, Y. H. H. Tsai, et al., “HuBERT: Self-
supervised speech representation learning by masked prediction of
hidden units,” IEEE ACM Transactions on Audio, Speech, and
Language Processing, vol. 29, pp. 3451–3460, 2021, doi:
1109/TASLP.2021.3122291.
S. Chen, C. Wang, Z. Chen, et al., “WavLM: Large-scale self-
supervised pre-training for full-stack speech processing,” IEEE
Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp.
–1518, 2022, doi: 10.1109/JSTSP.2022.3188113.
E. Kim, J. J. Jeon, H. Seo, and H. Kim, “Automatic pronunciation
assessment using self-supervised speech representation learning,”
in Proc. Interspeech, 2022, pp. 1411–1415, doi:
21437/Interspeech.2022-10245.
W. Dong, R. van Hout, C. Cucchiarini, and H. Strik, “Second
language prosody assessment by combining acoustic and neural
model features,” in Proc. Speech Prosody, 2024, pp. 165–169, doi:
21437/SpeechProsody.2024-34.
K. Fu, S. Gao, S. Shi, X. Tian, W. Li, and Z. Ma, “Phonetic and
prosody-aware self-supervised learning approach for non native
fluency scoring,” in Proc. Interspeech, 2023, pp. 949–953, doi:
21437/Interspeech.2023-587.
N. Markl, “Language variation and algorithmic bias in British
English automatic speech recognition,” in Proc. ACM FAccT,
, pp. 521–534, doi: 10.1145/3531146.3533117.
A. Koenecke, A. Nam, E. Lake, et al., “Racial disparities in
automated speech recognition,” Proceedings of the National
Academy of Sciences, vol. 117, no. 14, pp. 7684–7689, 2020, doi:
1073/pnas.1915768117.
J. Zhang, Z. Zhang, Y. Wang, et al., “speechocean762: An open
source non native English speech corpus for pronunciation
assessment,” in Proc. Interspeech, 2021, pp. 3710–3714, doi:
21437/Interspeech.2021-1259.
H. Ryu, S. Kim, and M. Chung, “A joint model for pronunciation
assessment and mispronunciation detection and diagnosis with
multi-task learning,” in Proc. Interspeech, 2023, pp. 959–963, doi:
21437/Interspeech.2023-337.
ISO IEC 25010:2023, Systems and Software Engineering—
Systems and Software Quality Requirements and Evaluation—
Product Quality Model. Geneva, Switzerland: International
Organization for Standardization, 2023
Downloads
Published
How to Cite
Issue
Section
Categories
License
Copyright (c) 2026 Oybek Kayumov, Nazokat Kayumova
This work is licensed under a Creative Commons Attribution 4.0 International License.
