John Pavlopoulos, Leo Laugier, Alexandros Xenos, Jeffrey Sorensen, Ion Androutsopoulos. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

On the world wide web, toxic content detectors are a crucial line of defense against potentially hateful and offensive messages. As such, building highly effective classifiers that enable a safer internet is an important research area. Moreover, the web is a highly multilingual, cross-cultural community that develops its own lingo over time. As such, it is crucial to develop models that are effective across a diverse range of languages, usages, and styles. In this paper, we present the fundamentals behind the next version of the Perspective API from Google Jigsaw. At the heart of the approach is a single multilingual token-free Charformer model that is applicable across a range of languages, domains, and tasks. We demonstrate that by forgoing static vocabularies, we gain flexibility across a variety of settings. We additionally outline the techniques employed to make such a byte-level model efficient and feasible for productionization. Through extensive experiments on multilingual toxic comment classification benchmarks derived from real API traffic and evaluation on an array of code-switching, covert toxicity, emoji-based hate, human-readable obfuscation, distribution shift, and bias evaluation settings, we show that our proposed approach outperforms strong baselines. Finally, we present our findings from deploying this system in production.

Platforms that support online commentary, from social networks to news sites, are increasingly leveraging machine learning to assist their moderation efforts.But this process does not typically provide feedback to the author that would help them contribute according to the community guidelines.This is prohibitively time-consuming for human moderators to do, and computational approaches are still nascent.This work focuses on models that can help suggest rephrasings of toxic comments in a more civil manner.Inspired by recent progress in unpaired sequence-tosequence tasks, a self-supervised learning model is introduced, called CAE-T5 1 .CAE-T5 employs a pre-trained text-to-text transformer, which is fine tuned with a denoising and cyclic auto-encoder loss.Experimenting with the largest toxicity detection dataset to date (Civil Comments) our model generates sentences that are more fluent and better at preserving the initial content compared to earlier text style transfer systems which we compare with using several scoring systems and human evaluation.

Attack strategies used by online harassers have evolved over time to inflict increasing harm to their targets. In addition to scaling harassment through incitement and coordination, online communities that commonly engage in harassment are likely a source of "innovation" for harassment attack strategies. We use the incitements or calls to harassment posted by members of these communities as a lens through which to holistically measure and understand this ecosystem. We create a filtering pipeline to discover 14,679 incitements to harassment within four large-scale data sets of messages and posts that span multiple platforms.

Moderation is crucial to promoting healthy online discussions.Although several 'toxicity' detection datasets and models have been published, most of them ignore the context of the posts, implicitly assuming that comments may be judged independently.We investigate this assumption by focusing on two questions: (a) does context affect the human judgement, and (b) does conditioning on context improve performance of toxicity detection systems?We experiment with Wikipedia conversations, limiting the notion of context to the previous post in the thread and the discussion title.We find that context can both amplify or mitigate the perceived toxicity of posts.Moreover, a small but significant subset of manually labeled posts (5% in one of our experiments) end up having the opposite toxicity labels if the annotators are not provided with context.Surprisingly, we also find no evidence that context actually improves the performance of toxicity classifiers, having tried a range of classifiers and mechanisms to make them context aware.This points to the need for larger datasets of comments annotated in context.We make our code and data publicly available.

We introduce and illustrate a new approach to measuring and mitigating unintended bias in machine learning models. Our definition of unintended bias is parameterized by a test set and a subset of input features. We illustrate how this can be used to evaluate text classifiers using a synthetic test set and a public corpus of comments annotated for toxicity from Wikipedia Talk pages. We also demonstrate how imbalances in training data can lead to unintended bias in the resulting models, and therefore potentially unfair applications. We use a set of common demographic identity terms as the subset of input features on which we measure bias. This technique permits analysis in the common scenario where demographic information on authors and readers is unavailable, so that bias mitigation must focus on the content of the text itself. The mitigation method we introduce is an unsupervised approach based on balancing the training dataset. We demonstrate that this approach reduces the unintended bias without compromising overall model quality.

Yiqing Hua, Cristian Danescu-Niculescu-Mizil, Dario Taraborelli, Nithum Thain, Jeffery Sorensen, Lucas Dixon. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.

In this paper we describe the development of an accurate, small-footprint, large vocabulary speech recognizer for mobile de-vices. To achieve the best recognition accuracy, state-of-the-art deep neural networks (DNNs) are adopted as acoustic models. A variety of speedup techniques for DNN score computation are used to enable real-time operation on mobile devices. To reduce the memory and disk usage, on-the-fly language model (LM) rescoring is performed with a compressed n-gram LM. We were able to build an accurate and compact system that runs well below real-time on a Nexus 4 Android phone. Index Terms: Deep neural networks, embedded speech recog-nition, SIMD, LM compression.

Language models are important components of speech recognition and machine translation systems. Trained on billions of words, and consisting of billions of parameters, language models often are the single largest components of these systems. There have been many proposed techniques to reduce the storage requirements for language models. A technique based upon pointer-free compact storage of ordinal trees shows compression competitive with the best proposed systems, while retaining the full finite state structure, and without using computationally expensive block compression schemes or lossy quantization techniques. Index Terms: n-gram language models, unary data structures 1.

A novel distributed language model that has no constraints on the n-gram order and no practical constraints on vocabulary size is presented. This model is scalable and allows for an arbitrarily large corpus to be queried for statistical estimates. Our distributed model is capable of producing n-gram counts on demand. By using a novel heuristic estimate for the interpolation weights of a linearly interpolated model, it is possible to dynamically compute the language model probabilities. The distributed architecture follows the client-server paradigm and allows for each client to request an arbitrary weighted mixture of the corpus. This allows easy adaptation of the language model to particular test conditions. Experiments using the distributed LM for re-ranking N-best lists of a speech recognition system resulted in considerable improvements in word error rate (WER), while integration with a machine translation decoder resulted in significant improvements in translation quality as measured by the BLEU score.

Short vowels and other diacritics are not part of written Arabic scripts. Exceptions are made for important political and religious texts and in scripts for beginning students of Arabic. Script without diacritics have considerable ambiguity because many words with different diacritic patterns appear identical in a diacritic-less setting. We propose in this paper a maximum entropy approach for restoring diacritics in a document. The approach can easily integrate and make effective use of diverse types of information; the model we propose integrates a wide array of lexical, segment-based and part-of-speech tag features. The combination of these feature types leads to a state-of-the-art diacritization model. Using a publicly available corpus (LDC's Arabic Treebank Part 3), we achieve a diacritic error rate of 5.1%, a segment error rate 8.5%, and a word error rate of 17.3%. In case-ending-less setting, we obtain a diacritic error rate of 2.2%, a segment error rate 4.0%, and a word error rate of 7.2%.

We extend previous work on tree kernels to estimate the similarity between the dependency trees of sentences. Using this kernel within a Support Vector Machine, we detect and classify relations between entities in the Automatic Content Extraction (ACE) corpus of news articles. We examine the utility of different features such as Wordnet hypernyms, parts of speech, and entity types, and find that the dependency tree kernel achieves a 20% F1 improvement over a "bag-of-words" kernel.

The technique of embedding a digital signal into an audio recording or image using techniques that render the signal imperceptible has received significant attention. Embedding an imperceptible, cryptographically secure signal, or watermark, is seen as a potential mechanism that may be used to prove ownership or detect tampering. While there has been a considerable amount of attention devoted to the techniques of spread-spectrum signaling for use in image and audio watermarking applications, there has only been a limited study for embedding data signals in speech. Speech is an uncharacteristically narrow band signal given the perceptual capabilities of the human hearing system. However, using speech analysis techniques, one may design an effective data signal that can be used to hide an arbitrary message in a speech signal. Also included are experiments demonstrating the subliminal channel capacity of the speech data embedding technique developed here.

This paper describes a phrase splicing and variable substitution system which offers an intermediate form of automated speech production lying in-between the extremes of recorded utterance playback and full text-to-speech synthesis. The system incorporates a trainable speech synthesiser and an application specific set of pre-recorded phrases. The text to be synthesised is converted to a phone sequence using phone sequences present in the pre-recorded phrases wherever possible, and a pronunciation dictionary elsewhere. The synthesis inventory of the synthesiser is augmented with the synthesis information associated with the pre-recorded phrases used to construct the phone sequence. The synthesiser then performs a dynamic programming search over the augmented inventory to select a segment sequence to produce the output speech. The system enables the seamless splicing of pre-recorded phrases both with other phrases and with synthetic speech. It enables very high quality speech to be produced automatically within a limited domain.

CORRECTIVE TRAINING FOR SPEAKER ADAPTATIONXiuyang Yu and Wayne WardCenter for Spoken Language UnderstandingUniversity of Colorado, Boulder, ColoradoABSTRACTThis paper reports results on an experiment to use correctivetraining techniques for rapid acoustic speaker adaptation in asemi-continuous speech recognition system. Decoder outputis used to adjust HMM acoustic models to improvediscrimination between correct words and near misses.Twenty sentences are used as an adaptation set. A speechrecognizer is run on each utterance to generate a wordlattice. The lattice is pruned relative to the correct path. Theforward-backward algorithm is used to align each path in thelattice against the speech input and compute observationcounts. For each input frame, counts in correct models areadjusted upward, and counts in incorrect models are adjusteddownward. The adjusted counts are normalized to generatenew observation probabilities for the models. Theparameters being adjusted are the mixture weights for thesemi-continuous HMMs. The technique reduced word errorfor a test subject by 37% relative.INTRODUCTIONSpeech recognition systems based on Hidden MarkovModels typically experience significant performancedegradation when a speaker is not represented well in thedata that the system was trained on. Rapid speakeradaptation techniques can be very effective in improvingperformance for a novel speaker. These techniques use asmall number of sentences (20-40), whose transcript isknown, to quickly adapt HMM acoustic models to a newspeaker. Such techniques are important, since in order forlonger term adaptation to be able to work, the system mustfunction well enough for the user to be productive. Thesetechniques attempt to correct very poor models quickly toget the system to a usable point for the user. This paperreports results on an experiment to use corrective trainingtechniques to rapidly adapt HMM acoustic models to apoorly recognized speaker.CORRECTIVE TRAININGCorrective training was introduced for speaker-dependentisolated word recognition by [1] and extended to speaker-independent continuous speech recognition by [2]. HiddenMarkov Models are normally trained according to amaximum likelihood criterion; parameters are adjusted tomaximize the probability of the training set. This processdoes not directly minimize the word errors, it does thisindirectly by attempting to assign a high probability to thecorrect utterance. It makes no representation of whatprobability is assigned to near misses. Corrective trainingseeks to directly minimize the number of word errors byadjusting parameters so as to improve discriminationbetween correct words and near misses. The general processis:1. Generate a set of near misses, words which areconfusable with the correct words.2. Align the correct words against the input.3. Align a near miss against the input.4. Modify model such that correct words are more likelyand incorrect ones less likely.5. Repeat steps 2-3 for other near misses.Speech recognizers are used to generate the near misses. In[1] and [2], a sp eech recognizer was used to generate an n-best list for an isolated word or sentence. This list was usedas the set of near misses. Both the correct utterance and nearmiss are aligned against the input. Model parameters arethen adjusted to make the correct words more likely and theincorrect ones less likely.SPHINX-II OBSERVATION ESTIMATESIn order to describe how corrective training is used to adaptour model, it is first necessary to describe the basic model.Our experimental system uses the Carnegie MellonUniversity Sphinx-II system for speech recognition[3],[4],[5]. Sphinx-II uses semi-continuous Hidden MarkovModels [3] to model context dependent phones. Likecontinuous HMM systems, semi-continuous systems use aweighted sum of points on Gaussian Probability DensityFunctions to estimate observation probabilities. Thedifference is that, while continuous systems estimate a set ofdistributions for each HMM state in the system, semi-

We describe our development work on a telephonebased conversational system in the domain of mutual fund transactions. This system uses several components including robust large vocabulary continuous speech recognition, natural language understanding, dialog management, and text-to-speech synthesis technologies.

This paper presents a distance measure for evaluating the closeness of two sets of distributions. The job of finding the distance between two distributions has been addressed with many solutions present in the literature. To cluster speakers using the pre-computed models of their speech, a need arises for computing a distance between these models which are normally built of a collection of distributions such as Gaussians (e.g., comparison between two HMM models). The definition of this distance measure creates many possibilities for speaker verification, speaker adaptation, speaker segmentation and many other related applications. A distance measure is presented for evaluating the closeness of a collection of distributions with centralized atoms such as Gaussians (but not limited to Gaussians). Several applications including some in speaker recognition with some results are presented using this distance measure.

Described is a speaker separation system based on the maximum likelihood deconvolution (MLD). The maximum likelihood criteria is applied to the deconvolution of the combined signal to estimate the excitation signal of each speaker. Passing the estimated excitation signal through the corresponding vocal tract filter produces the restored speech of the particular speaker. The restored speech is the result of the convolution of the excitation signal with the vocal tract wavelet matrix. A simultaneous characterization of multiple speakers was accomplished with a new speech model.>

A new algorithm, the hierarchical speaker verification algorithm, is introduced. This algorithm employs a set of unique mapping functions determined from an enrolment utterance that characterize the target voice as a multidimensional martingale random walk process. For sufficiently long verification utterances, the central limit theorem insures that the accumulated scores for the target speaker will be distributed normally about the origin. Impostor speakers, which violate the martingale property, are distributed arbitrarily and widely scattered in the verification space. Excerpts of verification performance experiments are given and extensions to the algorithm for handling noisy channels and speaker template aging are discussed.>

Hidden Markov models (HMMs) have been used to model speech in many areas of speech processing. One characteristic of the HMM is that the probability of time spent in a particular state, or state occupancy, is geometrically distributed. This, however, becomes a serious limitation and results in inaccurate modeling when the HMMs are used for phoneme recognition. The authors use hidden semi-Markov models (HSMM) to overcome the above limitation. Semi-Markov models are a more general class of Markov chains in which the state occupancy can be explicitly modeled by an arbitrary probability mass distribution. The authors use non-parametric distributions to describe the state occupancies instead of parametric distributions such as gamma. Poisson or binomial, as analysis of actual data shows that the duration of some phonemes could not be approximated by any of the above. Preliminary tests conducted using only the linear prediction coding (LPC) cepstrum as features have shown that the use of HSMM increased the phoneme recognition accuracy to 53.7% from the 48.4% obtained using an HMM.>

Text-independent speaker verification systems typically depend upon averaging over a long utterance to obtain a feature set for classification. However, not all speech is equally suited to the task of speaker verification. An approach to text-independent speaker verification that uses a two-stage classifier is presented. The first stage consists of a speaker-independent phoneme detector trained to recognize a phoneme that is distinctive from speaker to speaker. The second stage is trained to recognize the frames of speech from the target speaker that are admitted by the phoneme detector. A common feature vector based on the linear predictive coding (LPC) cepstrum is projected in different directions for each of these pattern recognition tasks. Results of tests using the described speaker verification system are shown.>