In practical applications of semantic parsing, we often want to rapidly change the behavior of the parser, such as enabling it to handle queries in a new domain, or changing its predictions on certain targeted queries.While we can introduce new training examples exhibiting the target behavior, a mechanism for enacting such behavior changes without expensive model re-training would be preferable.To this end, we propose ControllAble Semantic Parser via Exemplar Retrieval (CASPER).Given an input query, the parser retrieves related exemplars from a retrieval index, augments them to the query, and then applies a generative seq2seq model to produce an output parse.The exemplars act as a control mechanism over the generic generative model: by manipulating the retrieval index or how the augmented query is constructed, we can manipulate the behavior of the parser.On the MTOP dataset, in addition to achieving stateof-the-art on the standard setup, we show that CASPER can parse queries in a new domain, adapt the prediction toward the specified patterns, or adapt to new semantic schemas without having to further re-train the model.

Dian Yu, Luheng He, Yuan Zhang, Xinya Du, Panupong Pasupat, Qi Li. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

We present a new problem: grounding natural language instructions to mobile user interface actions, and create three new datasets for it.For full task evaluation, we create PIX-ELHELP, a corpus that pairs English instructions with actions performed by people on a mobile UI emulator.To scale training, we decouple the language and action data by (a) annotating action phrase spans in HowTo instructions and (b) synthesizing grounded descriptions of actions for mobile user interfaces.We use a Transformer to extract action phrase tuples from long-range natural language instructions.A grounding Transformer then contextually represents UI objects using both their content and screen position and connects them to object descriptions.Given a starting screen and instruction, our model achieves 70.59% accuracy on predicting complete ground-truth action sequences in PIXELHELP.

Systems that can associate images with their spoken audio captions are an important step towards visually grounded language learning.We describe a scalable method to automatically generate diverse audio for image captioning datasets.This supports pretraining deep networks for encoding both audio and images, which we do via a dual encoder that learns to align latent representations from both modalities.We show that a masked margin softmax loss for such models is superior to the standard triplet loss.We fine-tune these models on the Flickr8k Audio Captions Corpus and obtain state-of-the-art results-improving recall in the top 10 from 29.6% to 49.5%.We also obtain human ratings on retrieval outputs to better assess the impact of incidentally matching image-caption pairs that were not associated in the data, finding that automatic evaluation substantially underestimates the quality of the retrieved results. * Work done as a member of the Google AI Residency Program.We address the problem of relating images to audio captions that describe them (Figure 1), building on previous research into learning from visually grounded, untranscribed speech (Harwath and Glass, 2015;Sun et al., 2016;Harwath et al., 2016;Chrupała et al., 2017; Kamper et al., 2017b;Chrupała, 2019;Harwath and Glass, 2019).Such problem settings provide opportunities both to improve our theoretical understanding of language

Yinfei Yang, Yuan Zhang, Chris Tar, Jason Baldridge. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

We address fine-grained multilingual language identification: providing a language code for every token in a sentence, including codemixed text containing multiple languages.Such text is prevalent online, in documents, social media, and message boards.We show that a feed-forward network with a simple globally constrained decoder can accurately and rapidly label both codemixed and monolingual text in 100 languages and 100 language pairs.This model outperforms previously published multilingual approaches in terms of both accuracy and speed, yielding an 800x speed-up and a 19.5% averaged absolute gain on three codemixed datasets.It furthermore outperforms several benchmark systems on monolingual language identification.

We introduce a neural method for transfer learning between two (source and target) classification tasks or aspects over the same domain. Rather than training on target labels, we use a few keywords pertaining to source and target aspects indicating sentence relevance instead of document class labels. Documents are encoded by learning to embed and softly select relevant sentences in an aspect-dependent manner. A shared classifier is trained on the source encoded documents and labels, and applied to target encoded documents. We ensure transfer through aspect-adversarial training so that encoded documents are, as sets, aspect-invariant. Experimental results demonstrate that our approach outperforms different baselines and model variants on two datasets, yielding an improvement of 27% on a pathology dataset and 5% on a review dataset.

Traditional syntax models typically leverage part-of-speech (POS) information by constructing features from hand-tuned templates.We demonstrate that a better approach is to utilize POS tags as a regularizer of learned representations.We propose a simple method for learning a stacked pipeline of models which we call "stack-propagation".We apply this to dependency parsing and tagging, where we use the hidden layer of the tagger network as a representation of the input tokens for the parser.At test time, our parser does not require predicted POS tags.On 19 languages from the Universal Dependencies, our method is 1.3% (absolute) more accurate than a state-of-the-art graph-based approach and 2.7% more accurate than the most comparable greedy model.* Research conducted at Google.

Yuan Zhang, David Gaddy, Regina Barzilay, Tommi Jaakkola. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016.

Accurate multilingual transfer parsing typically relies on careful feature engineering.In this paper, we propose a hierarchical tensor-based approach for this task.This approach induces a compact feature representation by combining atomic features.However, unlike traditional tensor models, it enables us to incorporate prior knowledge about desired feature interactions, eliminating invalid feature combinations.To this end, we use a hierarchical structure that uses intermediate embeddings to capture desired feature combinations.Algebraically, this hierarchical tensor is equivalent to the sum of traditional tensors with shared components, and thus can be effectively trained with standard online algorithms.In both unsupervised and semi-supervised transfer scenarios, our hierarchical tensor consistently improves UAS and LAS over state-of-theart multilingual transfer parsers and the traditional tensor model across 10 different languages.1

Tao Lei, Yuan Zhang, Lluís Màrquez, Alessandro Moschitti, Regina Barzilay. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2015.

Yuan Zhang, Chengtao Li, Regina Barzilay, Kareem Darwish. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2015.

Accurate scoring of syntactic structures such as head-modifier arcs in dependency parsing typically requires rich, highdimensional feature representations.A small subset of such features is often selected manually.This is problematic when features lack clear linguistic meaning as in embeddings or when the information is blended across features.In this paper, we use tensors to map high-dimensional feature vectors into low dimensional representations.We explicitly maintain the parameters as a low-rank tensor to obtain low dimensional representations of words in their syntactic roles, and to leverage modularity in the tensor for easy training with online algorithms.Our parser consistently outperforms the Turbo and MST parsers across 14 different languages.We also obtain the best published UAS results on 5 languages.1

Much of the recent work on depen-dency parsing has been focused on solv-ing inherent combinatorial problems as-sociated with rich scoring functions. In contrast, we demonstrate that highly ex-pressive scoring functions can be used with substantially simpler inference pro-cedures. Specifically, we introduce a sampling-based parser that can easily han-dle arbitrary global features. Inspired by SampleRank, we learn to take guided stochastic steps towards a high scoring parse. We introduce two samplers for traversing the space of trees, Gibbs and Metropolis-Hastings with Random Walk. The model outperforms state-of-the-art re-sults when evaluated on 14 languages of non-projective CoNLL datasets. Our sampling-based approach naturally ex-tends to joint prediction scenarios, such as joint parsing and POS correction. The resulting method outperforms the best re-ported results on the CATiB dataset, ap-proaching performance of parsing with gold tags.1 1

Dependency parsing with high-order features results in a provably hard decoding problem.A lot of work has gone into developing powerful optimization methods for solving these combinatorial problems.In contrast, we explore, analyze, and demonstrate that a substantially simpler randomized greedy inference algorithm already suffices for near optimal parsing: a) we analytically quantify the number of local optima that the greedy method has to overcome in the context of first-order parsing; b) we show that, as a decoding algorithm, the greedy method surpasses dual decomposition in second-order parsing; c) we empirically demonstrate that our approach with up to third-order and global features outperforms the state-of-the-art dual decomposition and MCMC sampling methods when evaluated on 14 languages of non-projective CoNLL datasets.1

Marketing strategies without emotion will not work. Emotion stimulates the mind 3,000 times quicker than rational thought. Such emotion invokes either a positive or a negative response and physical expressions. Understanding the underlying dynamics of users' emotions can efficiently help companies formulate marketing strategies and support after-sale services. While prior work has focused mainly on qualitative aspects, in this paper we present our research on quantitative analysis of how an individual's emotional state can be inferred from her historic emotion log and how this person's emotional state influences (or is influenced by) her friends in the social network. We statistically study the dynamics of individual's emotions and discover several interesting as well as important patterns. Based on this discovery, we propose an approach referred to as MoodCast to learn to infer individuals' emotional states. In both mobile-based social network and online virtual network, we verify the effectiveness of our proposed approach.

Human emotion is one important underlying force affecting and affected by the dynamics of social networks. An interesting question is “can we predict a person's mood based on his historic emotion log and his social network?”. In this paper, we propose a Mood Cast method based on a dynamic continuous factor graph model for modeling and predicting users' emotions in a social network. Mood Cast incorporates users' dynamic status information (e.g., locations, activities, and attributes) and social influence from users' friends into a unified model. Based on the historical information (e.g., network structure and users' status from time 0 to t-1), Mood Cast learns a discriminative model for predicting users' emotion status at time t. To the best of our knowledge, this work takes the first step in designing a principled model for emotion prediction in social networks. Our experimental results on both real social network and virtual web-based network show that we can accurately predict emotion status of more than 62% of users and 8+% improvement than the baseline methods.

Mobile is becoming a ubiquitous platform for context-aware intelligent computing. One fundamental but usually ignored issue is how to efficiently manage (e.g., index and query) the mobile context data. To this end, we present a unified framework and have developed a toolkit, referred to as MQuery. More specifically, the mobile context data is represented in the standard RDF (Resource Description Framework) format. We propose a compressed-index method which takes less than 50% of the memory cost (of the traditional method) to index the context data. Four query interfaces have been developed for efficiently querying the context data including: instance query, neighbor query, shortest path query, and connection subgraph query. Experimental results on two real datasets demonstrate the efficiency of MQuery.