We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B, their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system. Though we train using 70% spoken-form data, our teacher models perform comparably to XLM-R and mT5 when evaluated on the written-form Cross-lingual Natural Language Inference (XNLI) corpus. We perform a second stage of pretraining on our teacher models using in-domain data from our system, improving error rates by 3.86% relative for intent classification and 7.01% relative for slot filling. We find that even a 170M-parameter model distilled from our Stage 2 teacher model has 2.88% better intent classification and 7.69% better slot filling error rates when compared to the 2.3B-parameter teacher trained only on public data (Stage 1), emphasizing the importance of in-domain data for pretraining. When evaluated offline using labeled NLU data, our 17M-parameter Stage 2 distilled model outperforms both XLM-R Base (85M params) and DistillBERT (42M params) by 4.23% to 6.14%, respectively. Finally, we present results from a full virtual assistant experimentation platform, where we find that models trained using our pretraining and distillation pipeline outperform models distilled from 85M-parameter teachers by 3.74%-4.91% on an automatic measurement of full-system user dissatisfaction.

We have recently witnessed tremendous success of Machine Learning (ML) in practical applications. Computer vision, speech recognition and language translation have all seen a near human level performance. We expect, in the near future, most business applications will have some form of ML. However, testing such applications is extremely challenging and would be very expensive if we follow today's methodologies. In this work, we present an articulation of the challenges in testing ML based applications. We then present our solution approach, based on the concept of Metamorphic Testing, which aims to identify implementation bugs in ML based image classifiers. We have developed metamorphic relations for an application based on Support Vector Machine and a Deep Learning based application. Empirical validation showed that our approach was able to catch 71% of the implementation bugs in the ML applications.

Test automation involves the automatic execution of test scripts instead of being manually run. This significantly reduces the amount of manual effort needed and thus is of great interest to the software testing industry. There are two key problems in the existing tools and methods for test automation - a) Creating an automation test script is essentially a code development task, which most testers are not trained on; and b) the automation test script is seldom readable, making the task of maintenance an effort intensive process. We present the Accelerating Test Automation Platform (ATAP) which is aimed at making test automation accessible to non-programmers. ATAP allows the creation of an automation test script through a domain specific language based on English. The English-like test scripts are automatically converted to machine executable code using Selenium WebDriver. ATAP's English-like test script makes it easy for non-programmers to author. The functional flow of an ATAP script is easy to understand as well thus making maintenance simpler (you can understand the flow of the test script when you revisit it many months later). ATAP has been built around the Eclipse ecosystem and has been used in a real-life testing project. We present the details of the implementation of ATAP and the results from its usage in practice.

Vagueness in software requirements documents can lead to several maintenance problems, especially when the customer and development team do not share the same language. Currently, companies rely on human translators to maintain communication and limit vagueness by translating the requirement documents by hand. In this paper, we describe two approaches that automatically identify vagueness in requirements documents in a multilingual environment. We perform two studies for calibration purposes under strict industrial limitations, and describe the tool that we ultimately deploy. In the first study, six participants, two native Portuguese speakers and four native Spanish speakers, evaluated both approaches. Then, we conducted a field study to test the performance of the best approach in real-world environments at two companies. We describe several lessons learned for research and industrial deployment.

The emergence of online labor markets has concentrated a lot of attention on the prospect of using crowdsourcing for software development, with a potential to reduce costs, improve time-to-market, and access high-quality skills on demand. However, crowdsourcing of software development is still not widely adopted. A key barrier to adoption is a lack of confidence that a task will be completed on time with the required quality standards. While good managers can develop good, intuitive estimates of task completion when assigning work to their team members, they might lack similar intuition for individuals drawn from an online crowd. The phrase, "Post and Hope" is thus sometimes used when talking about the crowdsourcing of software-development tasks. The objective of this paper is to show the value of replacing the traditional, intuitive assessment of a team's capability with a quantitative assessment of the crowd, derived through analysis of historical performance on similar tasks. This analysis will serve to transform "Post and Hope" to "Post and Expect." We demonstrate this by analyzing data about tasks performed on two popular crowdsourcing platforms: Topcoder and Upwork. Analysis of historical data from these platforms indicates that the platforms indeed demonstrate some level of predictability in task completion. We have identified certain factors that consistently contribute to task completion on both the platforms. Our findings suggest that a data-driven decision processes can play an important role in successful adoption of crowdsourcing practice for software development.

We present a method for the automatic extraction of glossary terms from unconstrained natural language requirements. The glossary terms are identified in two steps - a) compute units (which are candidates for glossary terms) b) disambiguate between the mutually exclusive units to identify terms. We introduce novel linguistic techniques to identify process nouns, abstract nouns and auxiliary verbs. The identification of units also handles co-ordinating conjunctions and adjectival modifiers. This requires solving co-ordination ambiguity and adjectival modifier ambiguity. The identification of terms among the units adapts an in-document statistical metric. We present an evaluation of our method over a real-life set of software requirements' documents and compare our results with that of a base algorithm. The intricate linguistic classification and the tackling of ambiguity result in superior performance of our approach over the base algorithm.