Due to the limited attention given to geometric features of points in three-dimensional few-shot learning, the extracted and embedded features from point clouds lack sufficient distinction, resulting in significant intra-class variation and ambiguous inter-class differentiation of features. This hinders the development of few-shot semantic segmentation for point clouds. To address this issue, we propose a novel Feature-Rich Geometric Convolutional (FRGC) network. The FRGC executes edge and geometric convolution operations on point clouds through its feature extraction backbone network. This method models geometric relationships by capturing the distribution, magnitude, and direction of group deviations within feature vectors after extracting edge structure information. Furthermore, we introduce a cross-fusion module that employs a spatial cross-attention mechanism. This mechanism highlights regions with high spatial similarity between support and query features. By extracting discriminative features with the FRGC and facilitating targeted knowledge exchange and feature fusion between the support and query set via the cross-fusion module, we reduce intra-class variance while increasing inter-class differences, thereby significantly improving few-shot segmentation performance. Compared to state-of-the-art algorithms on the Stanford Large-Scale three-dimensional Indoor Spaces dataset and ScanNet dataset, our proposed method demonstrates performance improvements in mean Interaction over Union: 66.56% (＋3.79%) and 59.73% (＋3.22%) in 2-way 1-shot, 67.64% (−1.37%) and 60.78% (−1.32%) in 2-way 5-shot, 55.32% (＋2.54%) and 59.86% (＋4.42%) in 3-way 1-shot, 57.83% (−3.21%) and 63.61% (＋7.52%) in 3-way 5-shot.

Style is prone to experience but hard to formulate, even for design professionals. The current style classification (SC) methods can be categorized into mono-tasking (style) and multi-tasking (style, artists, genre, and other additional clues). However, the style strength spread-out phenomenon challenges them by the uncertain noises’ distribution and unstable distinguishable feature representation. The multiple sub-styles make the principal style recognition hard for current SC approaches with lower accuracy. This article proposes the community transferrable representation learning (CTRL) framework inspired by community detection and style transfer studies. The community feature is the transferrable feature embeddings in latent space through all samples within the same style concept. Therefore, instead of learning the unique style features, we introduce the learnable transferrable embeddings, whose distance will be smaller for the intraclass and larger for the interclass, to represent the principal style concept. There are two generative processes, p-AE and q-AE, as the cooperative operations to learn the community styles by the proposed diamond cells (D-cells) in a stacked way. The proposed community loss includes two terms: the invariance term is used to make the principal style embedding consistent, and the redundancy reduction term can decorate the different vector embeddings of sub-styles. The style classification and style feature learning model are optimized simultaneously in a semi-supervised way to learn the class-level style representation. We conduct experiments on six style datasets (three oil paintings, one architecture, one fashion, and one general image). Results show that the proposed framework brings a performance gain of 2%–7% in terms of accuracy compared with the SOTA approaches.

This study addresses the challenge of font retrieval in design by proposing a novel approach utilizing contrastive learning to establish a shared embedding space for texts and fonts. In contrast to previous methods limited to word-level queries, our method enables text-font retrieval at the sentence level. We collected text-font pair data from web pages and design templates on the Internet, finetuned the CLIP model on these pairs, and obtained text and font encoders for our application. The top-k fonts were then retrieved using the cosine distance between input text and font embeddings. Our approach offers three key advantages: (1) retrieving fonts with sentence-level text as input, which is intuitively consistent with design behaviors; (2) leveraging text-font pair data available on the Internet without manual annotation; and (3) scalability, the trained font encoder can encode new font candidates without retraining the model. We introduced an evaluation metric for font retrieval results. The results indicate that the retrieved fonts in the top 3 score better than those from baseline methods, and the top 1 retrieved font is competitive with the fonts selected by experienced graphic designers.

As an important part of intelligent manufacturing, robotic mixed-model assembly line needs to cost-effectively adjust its configuration to meet the dynamical manufacturing requirements. However, the existing reconfiguration method always considers the configurations of equipment in cyber space and ignores the interconnections between the physical and cyber spaces. Digital twin provides alternative ways to realize the interconnections between the physical and cyber spaces. In this paper, digital twin model of robotic mixed-model assembly line and the interconnections between physical and cyber spaces are realized by means of digital twin model. In addition, the mathematical model for minimizing the reconfiguration cost and load balancing is built, then the virtual reconfiguration problem is proposed. Afterwards, adaptive neighborhood search Bees algorithm, of which the adaptive neighborhood search strategy is included, is utilized to solve the proposed problem. Finally, the effectiveness of the proposed method is verified and the results show adaptive neighborhood search Bees algorithm generates better solutions compared with the other optimization algorithms.Note to Practitioners—The existing reconfiguration method is always realized in the cyber space and rarely integrates the physical industrial robot. Motivated by this, this paper combines the physical industrial robot with digital twin model of robotic mixed-model assembly line, and uses adaptive neighborhood search Bees algorithm to solve the reconfiguration problem. Based on the digital twin model of robotic mixed-model assembly line, the optimal solution of virtual reconfiguration method could be obtained considering the dynamic manufacturing requirements. Afterwards, the optimal solution can be sent to the physical robotic assembly line through the interconnection between the physical and cyber spaces. In addition, experiments under different cases show that the proposed method could generate the optimal solution when the manufacturing requirements change. In the future, the physical manufacturing process based on robotic mixed-model assembly line will be studied to make the proposed method more applicable.

In automated assembly tasks, ellipse detection is usually applied in the vision-based pose estimation of circular workpieces. However, the existing ellipse detection methods cannot effectively solve severe visual occlusion in cluttered environments. To address this problem, this paper proposes a Generative Adversarial Networks (GAN)-supported ellipse detection method against the occlusion condition. The trained GAN network can restore the occluded image of workpieces, so that the elliptical features can be detected robustly. In the experiments with different degrees of occlusion, the ellipse detection rate is above 90%, which shows better performance than other existing methods.

The fifth-generation mobile communication network (5G) is a significant infrastructure with the support of enhanced mobile broadband, ultra-reliable and low latency communication, and massive machine type communication. Due to the large-scale application of 5G in industrial control field, the security of 5G network has become an important issue. In order to efficiently perform security detection on 5G radio access network protocols, we propose an intelligent fuzzing algorithm for 5G NAS protocol based on predefined rules. Through the analysis of the 3GPP NAS protocol technical specification and captured packets, a message structure table is extracted based on the NAS message format and field properties. Different mutation strategies are then dynamically assigned to different key fields to realize the intelligence of the message mutation process. Furthermore, in order to evaluate the performance of the algorithm, we implement a fuzzing prototype system based on this intelligent mutation algorithm, and then conduct practical security detection on 5G NAS protocol in OAI, an open-source software radio simulation environment. Experimental results show that our proposed intelligent mutation algorithm has better performance in terms of protocol state coverage and the scale of test cases. In addition, five types of security vulnerabilities in OAI are also exposed in this paper, namely buffer overflow, use-after-free, infinite loop, memory access for uninitialized address, and memory access for NULL pointers. These vulnerabilities could result in denial of registration services to users.

This paper investigates the application of pre-trained BERT model in three classic text analysis tasks: Chinese grapheme-to-phoneme(G2P), text normalization(TN) and sentence punctuation annotation. Even though the full-sized BERT has prominent modeling power, there are two challenges for it in real applications: the requirement for annotated training data and the considerable computational cost. In this paper, we propose BERT-based low-latency solutions. To collect sufficient training corpus for G2P, we transfer knowledge from existing rule-based system to BERT through a large amount of unlabeled corpus. The new model could convert all characters directly from raw texts with higher accuracy. We also propose a hybrid two-stage text normalization pipeline which reduces the sentence error rate by 25% compared to the rule-based system. We offer both supervised and weakly supervised versions and find that the latter has only 1% accuracy drop from the former.

For efficient malware detection, there are more and more deep learning methods based on raw software binaries. Recent studies show that deep learning models can easily be fooled to make a wrong decision by introducing subtle perturbations to inputs, which attracts a large influx of work in adversarial attacks. However, most of the existing attack methods are based on manual features (e.g., API calls) or in the white-box setting, making the attacks impractical in current real-world scenarios. In this work, we propose a novel attack framework called GAPGAN, which generates adversarial payloads (padding bytes) with generative adversarial networks (GANs). To the best of our knowledge, it is the first work that performs end-to-end black-box attacks at the byte-level against deep learning based malware binaries detection. In our attack framework, we map input discrete malware binaries to continuous space, then feed it to the generator of GAPGAN to generate adversarial payloads. We append payloads to the original binaries to craft an adversarial sample while preserving its functionality. We propose to use a dynamic threshold for reducing the loss of the effectiveness of the payloads when mapping it from continuous format back to the original discrete format. For balancing the attention of the generator to the payloads and the adversarial samples, we use an automatic weight tuning strategy. We train GAPGAN with both malicious and benign software. Once the training is finished, the generator can generate an adversarial sample with only the input malware in less than twenty milliseconds. We apply GAPGAN to attack the state-of-the-art detector MalConv and achieve 100% attack success rate with only appending payloads of 2.5% of the total length of the data for detection. We also attack deep learning models with different structures under different defense methods. The experiments show that GAPGAN outperforms other state-of-the-art attack models in efficiency and effectiveness.

In this work, we propose minimum Bayes risk (MBR) training of RNN-Transducer (RNN-T) for end-to-end speech recognition.Specifically, initialized with a RNN-T trained model, MBR training is conducted via minimizing the expected edit distance between the reference label sequence and on-thefly generated N-best hypothesis.We also introduce a heuristic to incorporate an external neural network language model (NNLM) in RNN-T beam search decoding and explore MBR training with the external NNLM.Experimental results demonstrate an MBR trained model outperforms a RNN-T trained model substantially and further improvements can be achieved if trained with an external NNLM.Our best MBR trained system achieves absolute character error rate (CER) reductions of 1.2% and 0.5% on read and spontaneous Mandarin speech respectively over a strong convolution and transformer based RNN-T baseline trained on ∼21,000 hours of speech.

Remanufacturing helps to protect the environment and reduce the manufacturing cost by making full use of end-of-life products, and it has been paid much attention nowadays. Disassembly, which is the key step of remanufacturing, is always manually executed due to the uncertainties faced in disassembly process. Recently, robotic disassembly is the research hotspot of remanufacturing research field and it helps to promote the automatic and intelligent developments of disassembly process. Cloud manufacturing helps to improve manufacturing efficiency by encapsulating manufacturing resources into services. Combined with cloud manufacturing and robotic disassembly, the framework and enabling technologies of cloud robotic disassembly (CRD) is proposed in this paper. The enabling technologies of CRD are introduced from the perspective of perception methods, modeling and data storage, planning and controlling services, service scheduling and interaction methods. This paper could provide guidelines to build the CRD.

The traffic explosion and the rising of diverse requirements lead to many challenges for the mobile networks on flexibility, scalability, and deployability. To meet the challenges, end-to-end network slicing and mobile edge computing (MEC) are introduced into beyond fifth-generation (B5G) mobile communication networks. However, the edge servers are limited by computing and data processing capabilities, how to implement efficient load balancing and resource management in data center are still challenges for B5G network slices. In this paper, we first propose a model of load balancing for B5G network slices and formulate a total cost minimization problem, considering the energy cost, processing delay, backhaul bandwidth cost, and revenue loss associated with the backhaul delay. Then, a distributed algorithm based on Proximal Jacobian Alternating Direction Method of Multipliers (PJ-ADMM) is proposed to solve the formulated problem. Simulations are conducted to show that the proposed algorithm are effective in terms of reducing network operating cost.

In attention based end-to-end (A-E2E) speech recognition systems, the dependency between output tokens is typically formulated as an input-output mapping in decoder. Due to such dependency, decoding errors can easily propagate along output sequence. In this paper, we propose a token-wise training (TWT) method for A-E2E models. The new method is flexible and can be combined with a variety of loss functions. Applying TWT to multiple hypotheses, we propose a novel TWT in beam (TWTiB) training scheme. Trained on the benchmark Switchboard (SWBD) 300h corpus, TWTiB outperforms the previous best training scheme on the SWBD evaluation subset.

In this paper, we present a Sequence-to-Sequence Attentional Siamese Neural Network (Seq2Seq-ASNN) that leverages temporal alignment information for end-to-end speaker verification. In prior works of speaker discriminative neural networks, utterance-level evaluation/enrollment speaker representations are usually calculated. Our proposed model, utilizing a sequence-to-sequence (Seq2Seq) attention mechanism, maps the frame-level evaluation representation into enrollment feature domain and further generates an utterance-level evaluation-enrollment joint vector for final similarity measure. Feature learning, attention mechanism, and metric learning are jointly optimized using an end-to-end loss function. Experimental results show that our proposed model outperforms various baseline methods, including the traditional i-Vector/PLDA method, multi-enrollment end-to-end speaker verification models, d-vector approaches, and a self attention model, for text-dependent speaker verification on a Tencent internal voice wake-up dataset.

A method of attention-based end-to-end (E2E) automatic speech recognition (ASR) training, includes performing cross-entropy training of a model, based on one or more input features of a speech signal, performing beam searching of the model of which the cross-entropy training is performed, to generate an n-best hypotheses list of output hypotheses, and determining a one-best hypothesis among the generated n-best hypotheses list. The method further includes determining a character-based gradient and a word-based gradient, based on the model of which the cross-entropy training is performed and a loss function in which a distance between a reference sequence and the determined one-best hypothesis is maximized, and performing backpropagation of the determined character-based gradient and the determined word-based gradient to the model, to update the model.

Attackers often use domain generation algorithms (DGAs) to create various kinds of pseudorandom domains dynamically and select a part of them to connect with command and control servers, therefore it is important to automatically detect the algorithmically generated domains (AGDs). AGDs can be broken down into two categories: character-based domains and wordlist-based domains. Recently, methods based on machine learning and deep learning have been widely explored. However, much of the previous work perform well in detecting one kind of DGA families but poorly in classifying another kind. A general detection system which is applicable to both kinds of domains still remains a challenge. To address this problem, we propose a novel real-time detection method with high accuracy as well as high coverage. We first convey a domain name into a sequence of word-level or character-level components, then design a deep neural network based on temporal convolutional network to extract the implicit pattern and classify the domain into two or more categories. Our experimental results demonstrate that our model outperforms state-of-the-art approaches in both binary classification and multi-class classification, and shows a good performance in detecting different kinds of DGAs. Besides, the high training efficiency of our model makes it adjust to new malicious domains quickly.

Acoustic model and language model (LM) have been two major components in conventional speech recognition systems. They are normally trained independently, but recently there has been a trend to optimize both components simultaneously in a unified end-to-end (E2E) framework. However, the performance gap between the E2E systems and the traditional hybrid systems suggests that some knowledge has not yet been fully utilized in the new framework. An observation is that the current attention-based E2E systems could produce better recognition results when decoded with LMs which are independently trained with the same resource. In this paper, we focus on how to improve attention-based E2E systems without increasing model complexity or resorting to extra data. A novel training strategy is proposed for multi-task training with the connectionist temporal classification (CTC) loss. The sequence-based minimum Bayes risk (MBR) loss is also investigated. Our experiments on SWB 300hrs showed that both loss functions could significantly improve the baseline model performance. The additional gain from joint-LM decoding remains the same for CTC trained model but is only marginal for MBR trained model. This implies that while CTC loss function is able to capture more acoustic knowledge, MBR loss function exploits more word/character dependency.

Conventional acoustic models for automatic speech recognition (ASR) are usually constructed from sub-word unit (e.g., context-dependent phoneme, grapheme, wordpiece etc.). Recent studies demonstrate that connectionist temporal classification (CTC) based acoustic-to-word (A2W) models are also promising for ASR. Such structures have drawn increasing attention as they can directly target words as output units, which simplify ASR pipeline by avoiding additional pronunciation lexicon, or even language model. In this study, we systematically explore to use word as acoustic modeling unit for conversational speech recognition. By replacing senone alignment with word alignment in a convolutional bidirectional LSTM architecture and employing a lexicon-free weighted finite-state transducer (WFST) based decoding, we greatly simplify conventional hybrid speech recognition system. On Hub5-2000 Switchboard/CallHome test sets with 300-hour training data, we achieve a WER that is close to the senone based hybrid systems with a WFST based decoding.

This paper investigates the effectiveness of knowledge distillation in the context of multilingual models. We show that with knowledge distillation, Long Short-Term Memory(LSTM) models can be used to train standard feed-forward Deep Neural Network (DNN) models for a variety of low-resource languages. We then examine how the agreement between the teacher's best labels and the original labels affects the student model's performance. Next, we show that knowledge distillation can be easily applied to semi-supervised learning to improve model performance. We also propose a promising data selection method to filter un-transcribed data. Then we focus on knowledge transfer among DNN models with multilingual features derived from CNN+DNN, LSTM, VGG, CTC and attention models. We show that a student model equipped with better input features not only learns better from the teacher's labels, but also outperforms the teacher. Further experiments suggest that by learning from each other, the original ensemble of various models is able to evolve into a new ensemble with even better combined performance.

Multilingual (ML) representations play a key role in building speech recognition systems for low resource languages. The IARPA sponsored BABEL program focuses on building speech recognition (ASR) and keyword search (KWS) systems in over 24 languages with limited training data. The most common mechanism to derive ML representations in the BABEL program has been with the use of a two-stage network, the first stage being a convolutional network (CNN) from where multilingual features are extracted, expanded contextually and used as input to the second stage which can be a feed-forward DNN or a CNN. The final multilingual representations are derived from the second network. This paper presents two novel methods for deriving ML representations. The first is based on Long-Short Term Memory (LSTM) networks and the second is based on a very deep CNN (VGG-net). We demonstrate that ML features extracted from both models show significant improvement over the baseline CNN-DNN based ML representations, in terms of both speech recognition and keyword search performance and draw the comparison between the LSTM model itself and the ML representations derived from it on Georgian, the surprise language for the OpenKWS evaluation.

Abstract : Feature representations extracted from deep neural network-based multilingual frontends provide significant improvements to speech recognition systems in low resource settings. To effectively train these frontends, we introduce a data selection technique that discovers language groups from an available set of training languages. This data selection method reduces the required amount of training data and training time by approximately 40 , with minimal performance degradation. We present speech recognition results on 7 very limited language pack (VLLP) languages from the second option period of the IARPA Babel program using multilingual features trained on up to 10 languages. The proposed multilingual features provide up to 15 relative improvement over baseline acoustic features on the VLLP languages.

This paper examines the impact of multilingual (ML) acoustic representations on Automatic Speech Recognition (ASR) and keyword search (KWS) for low resource languages in the context of the OpenKWS15 evaluation of the IARPA Babel program. The task is to develop Swahili ASR and KWS systems within two weeks using as little as 3 hours of transcribed data. Multilingual acoustic representations proved to be crucial for building these systems under strict time constraints. The paper discusses several key insights on how these representations are derived and used. First, we present a data sampling strategy that can speed up the training of multilingual representations without appreciable loss in ASR performance. Second, we show that fusion of diverse multilingual representations developed at different LORELEI sites yields substantial ASR and KWS gains. Speaker adaptation and data augmentation of these representations improves both ASR and KWS performance (up to 8.7% relative). Third, incorporating un-transcribed data through semi-supervised learning, improves WER and KWS performance. Finally, we show that these multilingual representations significantly improve ASR and KWS performance (relative 9% for WER and 5% for MTWV) even when forty hours of transcribed audio in the target language is available. Multilingual representations significantly contributed to the LORELEI KWS systems winning the OpenKWS15 evaluation.

In this paper, we investigate the problem of automatically selecting textual keywords for keyword search development and tuning on audio data for any language. Briefly, the method samples candidate keywords in the training data while trying to match a set of target marginal distributions for keyword features such as keyword frequency in the training or development audio, keyword length, frequency of out-of-vocabulary words, and TF-IDF scores. The method is evaluated on four IARPA Babel program base period languages. We show the use of the automatically selected keywords for the keyword search system development and tuning. We show also that search performance is improved by tuning the decision threshold on the automatically selected keywords.

Similarity join plays an important role in many applications, such as data cleaning and integration, to address the poor data quality problem. Most of the existing studies focused on performing similarity join on static datasets but few studies realized running it on dynamic data streams. With the development of network technology, the data accessing paradigm has transferred from disk-oriented mode to online data streams, which makes performing similarity join in continuous query on data streams become a novel query processing paradigm. Different from static dataset, data stream is unbounded, continuous and unpredictable. The significant differences pose serious challenges, such as real-time query performance. To this end, we study the problem of continuous similarity join on data streams in this paper, which is based on edit distance metric and filter-and-verify framework with sliding-window semantics. Two subcases of this problem are studied, including self similarity join on a single data stream and similarity join on two streams. We introduced the basic window based sliding window model to facilitate the update of sliding window and its index. More details of our method, including signature extraction schemes, filtering and verification algorithms, re-evaluation strategies are discussed respectively. Finally, extensive experimental results show that our method works efficiently on real data streams.

Keyword search, in the context of low resource languages, has emerged as a key area of research. The dominant approach in keyword search is to use Automatic Speech Recognition (ASR) as a front end to produce a representation of audio that can be indexed. The biggest drawback of this approach lies in its the inability to deal with out-of-vocabulary words and query terms that are not in the ASR system output. In this paper we present an empirical study evaluating various approaches based on using confusion models as query expansion techniques to address this problem. We present results across four languages using a range of confusion models which lead to significant improvements in keyword search performance as measured by the Maximum Term Weighted Value (MTWV) metric.

Automatic speech recognition is a core component of many applications, including keyword search. In this paper we describe experiments on acoustic modeling, language modeling, and decoding for keyword search on a Cantonese conversational telephony corpus collected as part of the IARPA Babel program. We show that acoustic modeling techniques such as the bootstrapped-and-restructured model and deep neural network acoustic model significantly outperform a state-of-the-art baseline GMM/HMM model, in terms of both recognition performance and keyword search performance, with improvements of up to 11% relative character error rate reduction and 31% relative maximum term weighted value improvement. We show that while an interpolated Model M and neural network LM improve recognition performance, they do not improve keyword search results; however, the advanced LM does reduce the size of the keyword search index. Finally, we show that a simple form of automatically adapted keyword search performs 16% better than a preindexed search system, indicating that out-of-vocabulary search is still a challenge.

We present a system for keyword search on Cantonese conversational telephony audio, collected for the IARPA Babel program, that achieves good performance by combining postings lists produced by diverse speech recognition systems from three different research groups. We describe the keyword search task, the data on which the work was done, four different speech recognition systems, and our approach to system combination for keyword search. We show that the combination of four systems outperforms the best single system by 7%, achieving an actual term-weighted value of 0.517.

Spoken content in languages of emerging importance needs to be searchable to provide access to the underlying information. In this paper, we investigate the problem of extending data fusion methodologies from Information Retrieval for Spoken Term Detection on low-resource languages in the framework of the IARPA Babel program. We describe a number of alternative methods improving keyword search performance. We apply these methods to Cantonese, a language that presents some new issues in terms of reduced resources and shorter query lengths. First, we show score normalization methodology that improves in average by 20% keyword search performance. Second, we show that properly combining the outputs of diverse ASR systems performs 14% better than the best normalized ASR system.

Word class information has long been proven useful in language modeling (LM). However, the improved performance of class-based LMs over word n-gram models generally comes at the cost of increased decoding complexity and model size. In this paper, we propose a modified version of the Maximum Entropy token-based language model of [1] that matches the performance of the best existing class-based models, but which is as fast for decoding as a word n-gram model. In addition, while it is easy to statically combine word n-gram models built on different corpora into a single word n-gram model for fast decoding, it is unknown how to statically combine class-based LMs effectively. Another contribution of this paper is to propose a novel combination method that retains the gain of class-based LMs over word n-gram models. Experimental results on several spoken language translation tasks show that our model performs significantly better than word n-gram models with comparable decoding speed and only a modest increase in model size.

The preliminary case study presented in this paper forms part of a doctoral research in the built environment that aims to identify occupancy activities related to energy use. This is being undertaken using a monitoring framework focused upon the indoor environment and domestic electricity use. The data acquisition system is designed to adapt smart environment techniques for the on-site conditions of a real residential premise. A daily data sample from one monitored household is presented together with some qualitative and quantitative analysis. Future work, which will be required on the basis of the current progress, has been pointed out at the end of the paper.

Language model domain adaption usually uses a large quantity of auxiliary data in different genres and domains. It has mostly been relying on scoring functions for selection and it is typically independent of intended applications such as machine translation. In this paper, we present a novel domain adaptation approach that is directly motivated by the need of translation engine. We first identify interesting phrases by examining phrase translation tables, and then use those phrases as anchors to select useful and relevant sentences from general domain data, with the goal of improving domain coverage or providing additional contextual information. The experimental results on Farsi to English translation in military force protection domain and Chinese to English translation in travel domain show statistical significant gain using the reinforced language models over the baseline.

We present a novel language model capable of incorporating various types of linguistic information as encoded in the form of a token, a (word, label)-tuple. Using tokens as hidden states, our model is effectively a hidden Markov model (HMM) producing sequences of words with trivial output distributions. The transition probabilities, however, are computed using a maximum entropy model to take advantage of potentially overlapping features. We investigated different types of labels with a wide range of linguistic implications. These models outperform Kneser-Ney smoothed n-gram models both in terms of perplexity on standard datasets and in terms of word error rate for a large vocabulary speech recognition system.