In this paper, we propose novel structured language modeling methods for code mixing speech recognition by incorporating a well-known syntactic constraint for switching code, namely the Functional Head Constraint (FHC).Code mixing data is not abundantly available for training language models.Our proposed methods successfully alleviate this core problem for code mixing speech recognition by using bilingual data to train a structured language model with syntactic constraint.Linguists and bilingual speakers found that code switch do not happen between the functional head and its complements.We propose to learn the code mixing language model from bilingual data with this constraint in a weighted finite state transducer (WFST) framework.The constrained code switch language model is obtained by first expanding the search network with a translation model, and then using parsing to restrict paths to those permissible under the constraint.We implement and compare two approacheslattice parsing enables a sequential coupling whereas partial parsing enables a tight coupling between parsing and filtering.We tested our system on a lecture speech dataset with 16% embedded second language, and on a lunch conversation dataset with 20% embedded language.Our language models with lattice parsing and partial parsing reduce word error rates from a baseline mixed language model by 3.8% and 3.9% in terms of word error rate relatively on the average on the first and second tasks respectively.It outperforms the interpolated language model by 3.7% and 5.6% in terms of word error rate relatively, and outperforms the adapted language model by 2.6% and 4.6% relatively.Our proposed approach avoids making early decisions on codeswitch boundaries and is therefore more robust.We address the code switch data scarcity challenge by using bilingual data with syntactic structure.

In this paper, we propose for the first time to incorporate the linguistically well-known Functional Head Constraint into a code switch language model for speech recognition. Under this constraint, code switch cannot occur between the functional head and its complements. The constrained code switching language model is obtained by first expanding the search network with a translation model; and then using lattice-based parsing to restrict paths to those permissible under the Functional Head Constraint. We tested our system on two tasks of code switch speech recognition, namely lecture speech recognition and lunch conversation recognition. Our system reduces word error rates (WER) from a baseline mixed language model by 3.72% relative in the first task; and by 5.85% in the second task. It reduces WER from an interpolated language model by 2.51% in the first task; and by 4.57% in the second task. All results are statistically significant. In addition, our method reduces WER for both the matrix language and the embedded language.

We propose an integrated framework for large vocabulary continuous mixed language speech recognition that handles the accent effect in the bilingual acoustic model and the inversion constraint well known to linguists in the language model. Our asymmetric acoustic model with phone set extension improves upon previous work by striking a balance between data and phonetic knowledge. Our language model improves upon previous work by (1) using the inversion constraint to predict code switching points in the mixed language and (2) integrating a code-switch prediction model, a translation model and a reconstruction model together. This integration means that our language model avoids the pitfall of propagated error that could arise from decoupling these steps. Finally, a WFST-based decoder integrates the acoustic models, code-switch language model and a monolingual language model in the matrix language all together. Our system reduces word error rate by 1.88% on a lecture speech corpus and by 2.43% on a lunch conversation corpus, with statistical significance, over the conventional bilingual acoustic model and interpolated language model.

To train a code switching language model for mixed language speech recognition, we propose to assign weights to the sentence pairs in the parallel text data. The code switching language model which is composed of the code switching boundary prediction model, code switching translation model and reconstruction model is incorporated with a language for mixed language speech recognition. The code switching translation model which is trained using selected subsets of the sentence pairs in the parallel text data allows the decoder to make the decision whether a phrase is in the matrix language or in the embedded language. Moreover, we propose a weighting procedure while training the code switching translation model. We evaluate our methods on Mandarin-English code switching lecture speech and lunch conversations. Our proposed method reduces word error rate by a statistically significant 1.74% on the lecture speech, and by 1.29% on the lunch conversation over the conventional interpolated language model. Copyright © 2013 ISCA.

We propose to improve speech recognition performance on speaker-independent, mixed language speech by asymmetric acoustic modeling. Mixed language is either inter-sentential code switching from the source matrix language to a foreign language or intra-sentential code mixing between the matrix language and embedded foreign words or phrases. In either case, the foreign phrases are pronounced by the matrix language speaker with varying degrees of accent. Our pro posed system using selective decision tree merging between a bilingual model and an accented embedded speech model outperforms previous approaches of either using a bilingual model with model retraining by 21.51%, or using adaptation by 15.88%. It outperforms all models on both code mixing and code switching cases. We successfully improved recognition on embedded foreign speech without degrading the performance on the matrix language speech.