Tasks are a fundamental unit of work in the daily lives of people, who are increasingly using digital means to keep track of, organize, triage, and act on them. These digital tools – such as task management applications – provide a unique opportunity to study and understand tasks and their connection to the real world, and through intelligent assistance, help people be more productive. By logging signals such as text, timestamp information, and social connectivity graphs, an increasingly rich and detailed picture of how tasks are created and organized, what makes them important, and who acts on them, can be progressively developed. Yet the context around actual task completion remains fuzzy, due to the basic disconnect between actions taken in the real world and telemetry recorded in the digital world. Thus, in this paper we compile and release a novel, real-life, large-scale dataset called MS-LaTTE that captures two core aspects of the context surrounding task completion: location and time. We describe our annotation framework and conduct a number of analyses on the data that were collected, demonstrating that it captures intuitive contextual properties for common tasks. Finally, we test the dataset on the two problems of predicting spatial and temporal task co-occurrence, concluding that predictors for co-location and co-time are both learnable, with a BERT fine-tuned model outperforming several other baselines. The MS-LaTTE dataset provides an opportunity to tackle many new modeling challenges in contextual task understanding and we hope that its release will spur future research in task intelligence more broadly.

Document authoring involves a lengthy revision process, marked by individual edits that are frequently linked to comments. Modeling the relationship between edits and comments leads to a better understanding of document evolution, potentially benefiting applications such as content summarization, and task triaging. Prior work on understanding revisions has primarily focused on classifying edit intents, but falling short of a deeper understanding of the nature of these edits. In this paper, we present explore the challenge of describing an edit at two levels: identifying the edit intent, and describing the edit using free-form text. We begin by defining a taxonomy of general edit intents and introduce a new dataset of full revision histories of Wikipedia pages, annotated with each revision’s edit intent. Using this dataset, we train a classifier that achieves a 90% accuracy in identifying edit intent. We use this classifier to train a distantly-supervised model that generates a high-level description of a revision in free-form text. Our experimental results show that incorporating edit intent information aids in generating better edit descriptions. We establish a set of baselines for the edit description task, achieving a best score of 28 ROUGE, thus demonstrating the effectiveness of our layered approach to edit understanding.

Naoki Otani, Michael Gamon, Sujay Kumar Jauhar, Mei Yang, Sri Raghu Malireddi, Oriana Riva. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

The unique limitations of mobile environments make content creation and editing difficult. Microtasking—breaking down complex tasks into subtasks—requires shorter attention spans and quick interactions, making it suitable for mobile usage scenarios. Writing is an ideal process for mobile microtasking because of its many subgoals, but little is known about how writers can use this decomposition through the evolution of a document. In this paper we present findings from a controlled, week long study to characterize how writers use mobile microtasks while authoring a document. We found that writers created microtasks for editing and inserting information that generally required minimal writing. These tasks were especially well suited for mobile devices with writers completing tasks on commutes or while waiting for meetings. Writers who microtasked found it easy to interact with their document and complete tasks, writing and editing their document more overall compared to writers who instead edited their document directly on their phone.

Xuchao Zhang, Dheeraj Rajagopal, Michael Gamon, Sujay Kumar Jauhar, ChangTien Lu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Emails in the workplace are often intentional calls to action for its recipients. We propose to annotate these emails for what action its recipient will take. We argue that our approach of action-based annotation is more scalable and theory-agnostic than traditional speech-act-based email intent annotation, while still carrying important semantic and pragmatic information. We show that our action-based annotation scheme achieves good inter-annotator agreement. We also show that we can leverage threaded messages from other domains, which exhibit comparable intents in their conversation, with domain adaptive RAINBOW (Recurrently AttentIve Neural Bag-Of-Words). On a collection of datasets consisting of IRC, Reddit, and email, our reparametrized RNNs outperform common multitask/multidomain approaches on several speech act related tasks. We also experiment with a minimally supervised scenario of email recipient action classification, and find the reparametrized RNNs learn a useful representation.

As the Web evolves towards a service-oriented architecture, application program interfaces (APIs) are becoming an increasingly important way to provide access to data, services, and devices. We study the problem of natural language interface to APIs (NL2APIs), with a focus on web APIs for web services. Such NL2APIs have many potential benefits, for example, facilitating the integration of web services into virtual assistants.

Ashequl Qadir, Michael Gamon, Patrick Pantel, Ahmed Hassan Awadallah. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016.

Models that learn to represent textual and knowledge base relations in the same continuous latent space are able to perform joint inferences among the two kinds of relations and obtain high accuracy on knowledge base completion (Riedel et al., 2013).In this paper we propose a model that captures the compositional structure of textual relations, and jointly optimizes entity, knowledge base, and textual relation representations.The proposed model significantly improves performance over a model that does not share parameters among textual relations with common sub-structure.

Research in automated grammatical error detection and correction has gained considerable momentum in the past few years. Although much progress has been made in this area, numerous challenges and opportunities exist for further research. For NLP researchers and students who are considering following or pursuing work in this area and for English language teaching practitioners and researchers who are interested in utilizing systems resulting from research in this area, this book by Leacock, Chodorow, Gamon, and Tetreault provides a timely, updated review of the state of the art in automatic learner error detection and correction.The introductory chapter first highlights the growth in prominence of the field of grammatical error detection and correction since the publication of the first edition of the volume. It then orients the reader to the book by detailing the changes from the first edition, providing a working definition of grammatical error, justifying the book's focus on English language learners, clarifying the use of the term “English language learner” to refer to learners of English as either a second or foreign language, putting forth a few claims about the relationship between NLP and Computer-Assisted Language Learning (CALL), specifying the intended audience of the book, and outlining the structure of the book.Chapter 2 provides the historical context for automated grammatical error detection and correction. The chapter first discusses how varying degrees of tolerance of grammatical errors are achieved in computational grammars used in grammar-checking and proofreading tools. It then offers a quick overview of data-driven and hybrid approaches to error detection.Chapter 3 summarizes the types of errors made by English language learners as reported in previous empirical and corpus-based learner error studies, touches upon the influence of L1 on learner English, and then delves into a detailed discussion of three specific problem areas for English language learners: prepositions, articles, and collocations.Chapter 4 focuses on the evaluation of error detection systems. The chapter first introduces traditional evaluation measures (e.g., precision, recall, F-score, accuracy, and kappa) and the evaluation measures adopted in recent shared tasks on grammatical error correction. It then discusses evaluation using a corpus of correct usage, questioning the value of this approach for indicating how well the system may perform on actual learner data. The advantages and challenges of evaluating system performance on learner writing are then examined. The authors advocate the use of multiple annotators and crowdsourcing as a means to improve the reliability of manual evaluation. The chapter concludes with a discussion of how statistical significance testing of differences in system performance may be performed and provides a checklist for consistent reporting of system results.Chapter 5 zooms in on data-driven approaches to detecting and correcting article and preposition errors. The chapter first looks at four types of information used by different systems, including lexical, syntactic, semantic, and source information. It then describes three prevailing types of data used to train statistical models of grammatical error correction, namely, well-formed text, artificial errors, and error-annotated learner corpora. Next, it discusses classification methods and n-gram statistics-related methods for error detection and correction. Finally, the chapter presents two end-to-end systems, Criterion and MSR ESL Assistant.Chapter 6 focuses on collocation errors. Following a brief discussion of the properties of collocations, the chapter introduces a range of metrics commonly used to measure the association strength between pairs of words and reviews a number of recent collocation error detection and correction systems.Chapter 7 moves beyond article, preposition, and collocation errors and examines rule-based and statistical methods for detecting and correcting verb-form, spelling, and punctuation errors and for identifying ungrammatical sentences. A small number of error detection systems for learners of languages other than English are also discussed.Chapter 8 covers issues with learner error annotation, describes examples of comprehensive and targeted annotation schemes, and proposes three methods for improving the efficiency of large-scale annotation: sampling, crowdsourcing, and mining online revision logs.Chapter 9 highlights several exciting emerging directions in the field of automated grammatical error correction. These include three recent shared tasks on grammatical error correction, the use of machine translation techniques for grammatical error correction, real-time crowdsourcing of grammatical error correction, and longitudinal studies on the efficacy of automated error correction systems for improving the writing of users of such systems.Chapter 10 concludes the book with several suggestions for future research in the field, including annotation for evaluation, error detection for underrepresented languages, research on understudied error types, L1-specific error detection modules, collaboration with second language learning and education groups, and applications of grammatical error correction. The book also includes an Appendix that contains a list of textual learner corpora with at least some publicly accessible URLs or references.I very strongly recommend this book to all NLP students and researchers who are interested in learning about, following, or pursuing research in automated grammatical error detection and correction. Not only does it offer a comprehensive systematic review of research in this field, but it also highlights real challenges and opportunities for fruitful future research. Practitioners and researchers in the language teaching and CALL community can also use this book to obtain a realistic understanding of the state of the art of the field. They will also most certainly welcome the volume's repeated emphasis on the importance for the NLP community to join efforts with the language teaching and CALL community to assess the effect of automated grammatical correction systems on improving the quality of language learners' writing.I have just two minor quibbles with this book. First, it seems that the title may better reflect the content of the book and the goals of the research field in question with the words “and correction” added after “detection.” Second, in both Chapter 3 and Chapter 6, native speakers' preference of powerful computer over strong computer is cited as an example of the arbitrariness of collocations. I do not intend to delve into the debate over whether collocations are arbitrary or linguistically motivated, but I note that the preference in this particular case is not arbitrary but semantically motivated (consider the semantic difference between powerful man and strong man). In fact, this preference merely reflects the fact that we are generally more concerned about the functional power of computers than their physical build, and it would be a false positive if a system marks strong computer as an error in a sentence that talks about a computer that is well-built and not easily breakable.

Natural touch interfaces, common now in devices such as tablets and smartphones, make it cumbersome for users to select text.There is a need for a new text selection paradigm that goes beyond the high acuity selection-by-mouse that we have relied on for decades.In this paper, we introduce such a paradigm, called Smart Selection, which aims to recover a user's intended text selection from her touch input.We model the problem using an ensemble learning approach, which leverages multiple linguistic analysis techniques combined with information from a knowledge base and a Web graph.We collect a dataset of true intended user selections and simulated user touches via a large-scale crowdsourcing task, which we release to the academic community.We show that our model effectively addresses the smart selection task and significantly outperforms various baselines and standalone linguistic analysis techniques.

This paper presents a deep semantic similarity model (DSSM), a special type of deep neural networks designed for text analysis, for recommending target documents to be of interest to a user based on a source document that she is reading.We observe, identify, and detect naturally occurring signals of interestingness in click transitions on the Web between source and target documents, which we collect from commercial Web browser logs.The DSSM is trained on millions of Web transitions, and maps source-target document pairs to feature vectors in a latent space in such a way that the distance between source documents and their corresponding interesting targets in that space is minimized.The effectiveness of the DSSM is demonstrated using two interestingness tasks: automatic highlighting and contextual entity search.The results on large-scale, real-world datasets show that the semantics of documents are important for modeling interestingness and that the DSSM leads to significant quality improvement on both tasks, outperforming not only the classic document models that do not use semantics but also state-of-the-art topic models.

We propose a system that determines the salience of entities within web documents. Many recent advances in commercial search engines leverage the identification of entities in web pages. However, for many pages, only a small subset of entities are central to the document, which can lead to degraded relevance for entity triggered experiences. We address this problem by devising a system that scores each entity on a web page according to its centrality to the page content. We propose salience classification functions that incorporate various cues from document content, web search logs, and a large web graph. To cost-effectively train the models, we introduce a soft labeling methodology that generates a set of annotations based on user behaviors observed in web search logs. We evaluate several variations of our model via a large-scale empirical study conducted over a test set, which we release publicly to the research community. We demonstrate that our methods significantly outperform competitive baselines and the previous state of the art, while keeping the human annotation cost to a minimum.

Social features are increasingly integrated within the search results page of the main commercial search engines. There is, however, little understanding of the utility of social features in traditional search. In this paper, we study utility in the context of social annotations, which are markings indicating that a person in the social network of the user has liked or shared a result document. We introduce a taxonomy of social relevance aspects that influence the utility of social annotations in search, spanning query classes, the social network, and content relevance. We present the results of a user study quantifying the utility of social annotations and the interplay between social relevance aspects. Through the user study we gain insights on conditions under which social annotations are most useful to a user. Finally, we present machine learned models for predicting the utility of a social annotation using the user study judgments as an optimization criterion. We model the learning task with features drawn from web usage logs, and show empirical evidence over real-world head and tail queries that the problem is learnable and that in many cases we can predict the utility of a social annotation.

We introduce an entity-centric search experience, called Active Objects, in which entity-bearing queries are paired with actions that can be performed on the entities. For example, given a query for a specific flashlight, we aim to present actions such as reading reviews, watching demo videos, and finding the best price online. In an annotation study conducted over a random sample of user query sessions, we found that a large proportion of queries in query logs involve actions on entities, calling for an automatic approach to identifying relevant actions for entity-bearing queries. In this paper, we pose the problem of finding actions that can be performed on entities as the problem of probabilistic inference in a graphical model that captures how an entity bearing query is generated. We design models of increasing complexity that capture latent factors such as entity type and intended actions that determine how a user writes a query in a search box, and the URL that they click on. Given a large collection of real-world queries and clicks from a commercial search engine, the models are learned efficiently through maximum likelihood estimation using an EM algorithm. Given a new query, probabilistic inference enables recommendation of a set of pertinent actions and hosts. We propose an evaluation methodology for measuring the relevance of our recommended actions, and show empirical evidence of the quality and the diversity of the discovered actions.

The psycholinguistic theory of communication accommodation accounts for the general observation that participants in conversations tend to converge to one another's communicative behavior: they coordinate in a variety of dimensions including choice of words, syntax, utterance length, pitch and gestures. In its almost forty years of existence, this theory has been empirically supported exclusively through small-scale or controlled laboratory studies. Here we address this phenomenon in the context of Twitter conversations. Undoubtedly, this setting is unlike any other in which accommodation was observed and, thus, challenging to the theory. Its novelty comes not only from its size, but also from the non real-time nature of conversations, from the 140 character length restriction, from the wide variety of social relation types, and from a design that was initially not geared towards conversation at all. Given such constraints, it is not clear a priori whether accommodation is robust enough to occur given the constraints of this new environment. To investigate this, we develop a probabilistic framework that can model accommodation and measure its effects. We apply it to a large Twitter conversational dataset specifically developed for this task. This is the first time the hypothesis of linguistic style accommodation has been examined (and verified) in a large scale, real world setting.

As users of social networking websites expand their network of friends, they are often flooded with newsfeed posts and status updates, most of which they consider to be "unimportant" and not newsworthy. In order to better understand how people judge the importance of their newsfeed, we conducted a study in which Facebook users were asked to rate the importance of their newsfeed posts as well as their friends. We learned classifiers of newsfeed and friend importance to identify predictive sets of features related to social media properties, the message text, and shared background information. For classifying friend importance, the best performing model achieved 85% accuracy and 25% error reduction. By leveraging this model for classifying newsfeed posts, the best newsfeed classifier achieved 64% accuracy and 27% error reduction.

A growing trend in commercial search engines is the display of specialized content such as news, products, etc. interleaved with web search results. Ideally, this content should be displayed only when it is highly relevant to the search query, as it competes for space with "regular" results and advertisements. One measure of the relevance to the search query is the click-through rate the specialized content achieves when displayed; hence, if we can predict this click-through rate accurately, we can use this as the basis for selecting when to show specialized content. In this paper, we consider the problem of estimating the click-through rate for dedicated news search results. For queries for which news results have been displayed repeatedly before, the click-through rate can be tracked online; however, the key challenge for which previously unseen queries to display news results remains. In this paper we propose a supervised model that offers accurate prediction of news click-through rates and satisfies the requirement of adapting quickly to emerging news events.

This paper presents a pilot study of the use of phrasal Statistical Machine Translation (SMT) techniques to identify and correct writing errors made by learners of English as a Second Language (ESL). Using examples of mass noun errors found in the Chinese Learner Error Corpus (CLEC) to guide creation of an engineered training set, we show that application of the SMT paradigm can capture errors not well addressed by widely-used proofing tools designed for native speakers. Our system was able to correct 61.81% of mistakes in a set of naturally-occurring examples of mass noun errors found on the World Wide Web, suggesting that efforts to collect alignable corpora of pre- and post-editing ESL writing samples offer can enable the development of SMT-based writing assistance tools capable of repairing many of the complex syntactic and lexical problems found in the writing of ESL learners.

We approach to correcting features in transferred linguistic representations in machine translation. The hybrid approach combines decision trees and transformation-based learning. Decision trees serve as a filter on the intractably large search space of possible interrelations among features. Transformation-based learning results in a simple set of ordered rules that can be compiled and executed after transfer and before sentence realization in the target language. We measure the reduction in noise in the linguistic representations and the results of human evaluations of end-to-end English-German machine translation.

We describe the implementation of two new language pairs (English-French and English-German) which use machine-learned sentence realization components instead of hand-written generation components. The resulting systems are evaluated by human evaluators, and in the technical domain, are equal to the quality of highly respected commercial systems. We comment on the difficulties that are encountered when using machine-learned sentence realization in the context of MT.

We show that it is possible to learn the contexts for linguistic operations which map a semantic representation to a surface syntactic tree in sentence realization with high accuracy. We cast the problem of learning the contexts for the linguistic operations as classification tasks, and apply straightforward machine learning techniques, such as decision tree learning. The training data consist of linguistic features extracted from syntactic and semantic representations produced by a linguistic analysis system. The target features are extracted from links to surface syntax trees. Our evidence consists of four examples from the German sentence realization system code-named Amalgam: case assignment, assignment of verb position features, extraposition, and syntactic aggregation.

We present a machine learning approach to evaluating the well-formedness of output of a machine translation system, using classifiers that learn to distinguish human reference translations from machine translations. This approach can be used to evaluate an MT system, tracking improvements over time; to aid in the kind of failure analysis that can help guide system development; and to select among alternative output strings. The method presented is fully automated and independent of source language, target language and domain.

We present an automated, system-internal evaluation technique for linguistic representations in a large-scale, multilingual MT system. We use machine-learned classifiers to recognize the differences between linguistic representations generated from transfer in an MT context from representations that are produced by “native” analysis of the target language. In the MT scenario, convergence of the two is the desired result. Holding the feature set and the learning algorithm constant, the accuracy of the classifiers provides a measure of the overall difference between the two sets of linguistic representations: classifiers with higher accuracy correspond to more pronounced differences between representations. More importantly, the classifiers yield the basis for error-analysis by providing a ranking of the importance of linguistic features. The more salient a linguistic criterion is in discriminating transferred representations from “native” representations, the more work will be needed in order to get closer to the goal of producing native-like MT. We present results from using this approach on the Microsoft MT system and discuss its advantages and possible extensions.