Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform. Our platform evaluates NLP models directly instead of relying on self-reported metrics or predictions on a single dataset. Under this paradigm, models are submitted to be evaluated in the cloud, circumventing the issues of reproducibility, accessibility, and backwards compatibility that often hinder benchmarking in NLP. This allows users to interact with uploaded models in real time to assess their quality, and permits the collection of additional metrics such as memory use, throughput, and robustness, which -- despite their importance to practitioners -- have traditionally been absent from leaderboards. On each task, models are ranked according to the Dynascore, a novel utility-based aggregation of these statistics, which users can customize to better reflect their preferences, placing more/less weight on a particular axis of evaluation or dataset. As state-of-the-art NLP models push the limits of traditional benchmarks, Dynaboard offers a standardized solution for a more diverse and comprehensive evaluation of model quality.

Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages. However, much of this work is English-Centric, training only on data which was translated from or to English.While this is supported by large sources of training data, it does not reflect translation needs worldwide. In this work, we create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages. We build and open-source a training data set that covers thousands of language directions with parallel data, created through large-scale mining. Then, we explore how to effectively increase model capacity through a combination of dense scaling and language-specific sparse parameters to create high quality models. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly translating between non-English directions while performing competitively to the best single systems from the Workshop on Machine Translation (WMT). We open-source our scripts so that others may reproduce the data, evaluation, and final M2M-100 model.

Speech synthesis technology has a wide range of applications such as voice assistants. In recent years waveform-level synthesis systems have achieved state-of-the-art performance, as they overcome the limitations of vocoder-based synthesis systems. A range of waveform-level synthesis systems have been proposed; this paper investigates the performance of hierarchical Recurrent Neural Networks (RNNs) for speech synthesis. First, the form of network conditioning is discussed, comparing linguistic features and vocoder features from a vocoder-based synthesis system. It is found that compared with linguistic features, conditioning on vocoder features requires less data and modeling power, and yields better performance when there is limited data. By conditioning the hierarchical RNN on vocoder features, this paper develops a neural vocoder, which is capable of high quality synthesis when there is sufficient data. Furthermore, this neural vocoder is flexible, as conceptually it can map any sequence of vocoder features to speech, enabling efficient synthesizer porting to a target speaker. Subjective listening tests demonstrate that the neural vocoder outperforms a high quality baseline, and that it can change its voice to a very different speaker, given less than 15 minutes of data for fine tuning.

In cloud computing, variability becomes a crucial ability of process-based cloud applications. Existing solutions either focused on modeling variability in the architectural model or tried to support dynamic variability management in implementation. An integrated approach that can inherit the virtues from both categories is expected. The paper aims to fill the gap by proposing a model-driven variability-based service composition approach. We propose VxUML to model the variability in the architecture model. Then, we define a set of model transformation rules to convert VxUML into VxBPEL (an extension to standard BPEL supporting variability at the implementation level). Finally, we implement a prototype tool, and present a case study to demonstrate the feasibility of our approach.

As more and more multi-tenant applications emerge in clouds, people increasingly recognize the importance of multi-tenant applications in realizing cloud benefit maximization. Service Level Agreement (SLA) is advocated widely to monitor these applications for multiple tenants to meet their service quality requirements. However, so far these applications provide limited multi-tenant monitoring supports, which prevents the applications from guaranteeing tenants' interests efficiently. In this paper, we propose an aspect-oriented approach to monitor multi-tenant applications based on tenant SLAs. Our approach includes monitoring code generation and runtime management. During code generation, the approach proposes an SLA feature model for tenants to specify variable requirements. Based on the requirements the approach selects code snippets, which are implemented as templates in advance, and splices them into an monitoring aspect. During runtime, the approach prioritizes aspects to determine execution order and updates monitoring status in term of tenant. An implemented prototype is used to evaluate the approach by case studies, which demonstrate the approach effectiveness in common situations.

As multi-tenant applications spring up in clouds, more and more people advocate using Service Level Agreement (SLA) in service delivery to fit tenants' non-functional needs e.g. Response time and budget limit. However, most of the present application optimizations based on SLA focuses on virtual machine-based (VM-based) computing service, while other services such as storage and cache are often neglected. In this paper, we propose an SLA-driven application optimization for cache service to help to meet tenants' needs better and improve cost-effectiveness, which can be taken as complementary to the existing work. The proposed approach, built on top of Platform-as-a-Service (PaaS), pays attention to evicted data. It considers both tenant SLA-evaluated status and data performance when weighting the evicted data with re-cache likelihoods, and then adjusts their re-cache priorities. At the beginning of every cycle it predicts tenant status and evicted data performance for the coming cycle by Holt-Winters double exponential smoothing. Our simulation experiments demonstrate the optimization effectiveness in improving cache cost-effectiveness and satisfying tenant SLAs.

Mode transformation is the key to model-based software engineering. When the model transformation is applied to industrial developments, its scalability becomes an important issue, since the model to be transformed may have a large size. To test the performance of model transformations, this paper proposes a randomized approach to generating large models as test inputs. First, the paper discusses the basic requirements and constraints for performance test input generation of the model transformation. Then, the paper presents our model generation algorithm. It can generate a model having a large size randomly and correctly within a reasonable time, according to the metamodel and user-defined constraints. Finally, an evaluation is also presented. And the result shows that our approach is more suitable for generating performance test inputs compared with existing model generation approaches.

With the popularity of pads, large touch screens, notebooks and smart mobiles, it is a reasonable requirement that modelers use such devices to models. However, there are few modeling tools that take full advantage of the devices. This paper discusses a Web based UML modeling tool with touch screens, based on our full analysis of human-machine interaction modes for software modeling. The tool provides more input means, i.e. combining gesture input and traditional keyboard and mouse input, and supports cross-platforms modeling by using HTML 5.

Cloud computing encourages application to migrate into it for economic of scale, where they rent shared resources to deliver services. Service Level Agreements(SLA) plays an important role in assisting various applications providing high-quality services to end users in cloud's complex and uncertain environments. Most of the existing work tries to support application claimed quality by help cloud make decisions of computing resources allocation during runtime. In this paper, we propose an approach for applications to maintain quality requirements by runtime cache adjustment in consideration of service level objectives (SLOs) and unpredictable workload in cloud, which can be taken as a complement to the existing work. Our approach includes application SLO modeling and mapping to monitor metrics during runtime, and an algorithm to adapting caches according to runtime status and SLOs. The approach has been applied to a real-world SNS application which proves effectiveness of our approach.

Estimating the cost is important for cloud application developers to services in clouds, and becomes even important when it needs remaining a certain service level at the same time. Though currently much work has been down to predict cost and performance of cloud applications, most of them perform either before application design or after the application construction, which leads to either imprecise estimation or irreparable design fault. In this paper, we propose an approach to estimate the cost of running typical applications in Amazon Web Service (AWS) cloud during design phase. We propose an UML Activity-extended model (AeModel) to describe execution of application service and introduce an extraction algorithm to extract information contained in the AeModel automatically. We propose a cost model on AWS, which can help developers to estimate operating cost during design phase and satisfy performance needs, with an algorithm to produce suitable purchase solutions automatically. We perform case studies using a web-based business application to show effectiveness of our approach, and find that our approach can help developers lessen cost by adjusting application models.

Bad smells are signs of potential problems in code. Detecting and resolving bad smells, however, remain time-consuming for software engineers despite proposals on bad smell detection and refactoring tools. Numerous bad smells have been recognized, yet the sequences in which the detection and resolution of different kinds of bad smells are performed are rarely discussed because software engineers do not know how to optimize sequences or determine the benefits of an optimal sequence. To this end, we propose a detection and resolution sequence for different kinds of bad smells to simplify their detection and resolution. We highlight the necessity of managing bad smell resolution sequences with a motivating example, and recommend a suitable sequence for commonly occurring bad smells. We evaluate this recommendation on two nontrivial open source applications, and the evaluation results suggest that a significant reduction in effort ranging from 17.64 to 20 percent can be achieved when bad smells are detected and resolved using the proposed sequence.

Recently model transformations quickly become large size and complex in structure. It is necessary to apply the modeling technique when developing transformations to manage the complexity. The paper proposes VisTML - a visual modeling language for model transformations. It provides six diagrams, which cover the whole lifecycle of transformation development and enable us to specify a transformation from various views and in different phases, i.e. the goal diagram, the transformation diagram, the rule diagram, the composition diagram, the test diagram, and the deployment diagram. We also provide a tool support and a real case of VisTML to illustrate the feasibility of VisTML. Besides, a comparison of VisTML, UML, and transML is presented at last.

Model Driven Development views application development as a continuous transformation of models of the target system. However, non-functional requirements, which are important for building user-satisfied software systems and have impacts on the software design, are not sufficiently considered in current MDD methods. This paper proposes an approach to modeling non-functional requirements, analyzing their impacts to the design and making a complement to the original design models for the MDD methods. First, existing UML models of the application's functional properties are taken as an input of the NFR modeling process, and with a NFR repository, the NFRs are refined and operationalized. Then based on the analysis result, modeling constructs are created to implement them, and the original UML models are reversely augmented by the result of NFR modeling. In particular, a metamodel is proposed to model the new constructs and facilitate the integration process. With this approach, we can finally get an integrated design model considering both functional requirements and non-functional requirements. The feasibility of the proposed approach is illustrated with an example of the design of a simplified credit card system.

Bad smell is a key concept in software refactoring. We have a bunch of bad smells, refactoring rules, and refactoring tools, but we do not know which kind of bad smells should be resolved first. The resolution of one kind of bad smells may have impact on the resolution of other bad smells. Con-sequently, different resolution orders of the same set of bad smells may require different effort, and/or lead to differ-ent quality improvement. In order to ease the work and maximize the effect of refactoring, we try to analyze the re-lationships among different kinds of bad smells, and their impact on resolution orders of these bad smells. With the analysis, we recommend a resolution order of common bad smells. The main contribution of this paper is to motivate the necessity to arrange resolution orders of bad smells, and recommend a resolution order of common bad smells.

A model is a blueprint of a system, which influences the quality of the system. A high quality model should specify not only the functional attributes of a system, i.e., what the system can do, but also the non-functional attributes, i.e., how well the system can do. Modeling for non-functional attributes, especially, the integration of nonfunctional attributes description with functional description and the checking for non-functional attributes, are rarely taken into account by the de facto modeling approaches and tools, while they support modeling and checking for the functional attributes well. In this paper, we extend UML class diagram by adding two model elements, i.e., the nonfunctional attributes notation and the constraint relationships table, for modeling non-functional attributes. An approach is given for checking consistency and satisfiability of the non-functional attributes in the extended UML class diagram. We use an example to demonstrate our proposal.

In order to define a graphical modeling language, it is necessary to define the graphical notation of the language in the process of metamodeling. So the defining of the no- tation has become one of the essential functions in meta- modeling tools. This paper proposed the Notation Defini- tion Metamodel (NDM) for metamodeling tools. NDM is used to define the graphical notation. It consists of three parts: basic figures and layouts, location relations and syn- tax bridges. NDM has been implemented in PKU Meta- Model Tool (PkuMMT). The paper made a case study to illustrate the feasibility of NDM. Besides, a comparison between PKU MMT and some metamodeling tools is pre- sented to show the capability and advantages of NDM.

Software refactoring is to restructure object-oriented software to improve its quality, especially extensibility, reusability and maintainability while preserving its external behaviors. For a special software system, there are usually quite a few refactorings available at the same time. But these refactorings may conflict with each other. In other words, carrying out a refactoring may disable other refactorings. Consequently, only a subset of the available refactorings can be applied together, and which refactorings will be applied depends on the schedule (application order) of the refactorings. Furthermore, carrying out different subsets of the refactorings usually leads to different improvement of software quality. As a result, in order to promote the improvement of software quality, refactorings should be scheduled rationally. However, how to schedule refactorings is rarely discussed. Usually, software engineers carry out refactorings immediately when they are found out. They do not wait until all applicable refactorings are found out and scheduled. In other words, the refactorings are not scheduled explicitly, and conflicts among them are not taken into consideration. Though more and more refactorings are formalized and automated by refactoring tools, refactoring tools apply refactorings usually in a nondeterministic fashion (in random). In this paper, we propose a scheduling approach to schedule conflicting refactorings to promote the improvement of software quality achieved by refactorings. Conflicts among refactorings are detected, and then a scheduling model is presented. And then a heuristic algorithm is proposed to solve the scheduling model. Results of experiments suggest that the proposed scheduling approach is effective in promoting the improvement of software quality

With the popularity of UML and MDA, models are replacing source code as core artifacts of software development and maintenance. But duplications in models reduce models' maintainability and reusability. To address the problem, we should detect duplications first. As an initial step to address the problem, we propose an approach to detect duplications in sequence diagrams. With special preprocessing, we convert 2-dimensional sequence diagrams into a 1-dimensional array. Then we construct a suffix tree of the array. We revise the traditional construction algorithm of suffix trees by proposing a special algorithm to detect common prefixes of suffixes. The algorithm ensures that every duplication detected with the suffix tree can be extracted into a separate reusable sequence diagram. With the suffix tree, duplications are found as refactoring candidates. With tool support, the proposed approach has been applied to real industrial projects, and the evaluation results suggest that the approach is effective.