SUMMARY As web applications become more complex and are used more pervasively, testing demands are increasing without corresponding automated support. One promising approach to automatic test generation is statistical model‐based testing, where logged user behaviour is used to build a usage‐based model of web application navigation, from which abstract test cases are generated. Executable test cases are then created by adding parameter values to the abstract test cases. Several researchers have proposed variations of this approach; however, no one has empirically examined the tradeoffs and implications of the different ways to represent user behaviour in a navigation model and the characteristics of the test cases automatically generated from different models. This paper reports on our exploratory study of automatically generated abstract test cases and the underlying usage‐based navigation models constructed from over 19,000 user sessions across five publicly deployed web applications. Our results suggest how web testers can easily configure statistical model‐based automatic test case generators for web applications toward generating tests closely related to user behaviour or toward new navigations without using large additional test resources. Copyright © 2013 John Wiley & Sons, Ltd.

Automatically creating test cases from statistical models of web application usage is an effective approach to generating test cases that represent actual usage. The models are typically generated from all collected user sessions. In this paper, we consider how grouping the user sessions -- specifically by the user's privilege -- creates different statistical models and the testing implications of those differences. We performed a study of user-privilege-specific navigation models and the resulting abstract test cases generated from over 19,000 user sessions to four deployed web applications. Our results suggest that grouping user sessions by the users' privileges results in smaller navigation models, which yield realistic test cases that represent users with that privilege well while also exploring navigations not seen in the input user sessions. In some cases, the user-privilege-specific models are significantly smaller, which allows the tester to either (a) generate relatively few test cases and still represent the user type well or (b) create test cases from a less abstract model -- without exorbitant model space costs or the need for additional models to generate executable test cases. However, the benefits are not universal for all applications, thus, we present guidance to testers on metrics to determine whether creating user-privilege-specific test cases will be advantageous.

Low undergraduate enrollments in computer science will not meet the future demand of employers. Some reasons for the low enrollments are computer science's nerdy image, lack of understanding of the field, and low motivation for learning programming. We propose to change the image of computer science by exposing students to applications of computing and its impact on their lives through reading and discussing recent news articles in 15 minutes of class. We call this component of our courses the Broader Issues in computer science. In this paper, we present our motivation for the Broader Issues component, how we implemented the component in CS0 and CS1 courses, and our evaluation of the component, including student reactions to and impact of the component.

While web applications expand in usage and complexity, testing demands are growing without corresponding automated support. One promising approach to automatic test generation is statistical model-based testing, where logged user behavior is used to build a usage-based model of web application navigation, from which abstract test cases are generated. Executable test cases are then created by adding parameter values to the abstract test cases. Several researchers have proposed variations of this approach, however, no one has empirically examined the tradeoffs and implications of the different ways to represent user behavior in a navigation model and the characteristics of the automatically generated test cases from different models. We report on our exploratory study of automatically generated abstract test cases and the underlying usage-based navigation models constructed from over 3500 user sessions across five publicly deployed web applications. Our results suggest how web testers can easily tune statistical model-based automatic test case generators for web applications toward generating tests closely related to user behavior or toward new navigations without using large additional test resources.

Software developers need automated techniques to maintain the correctness of complex, evolving Web applications. While there has been success in automating some of the testing process for this domain, there exists little automated support for verifying that the executed test cases produce expected results. We assist in this tedious task by presenting a suite of automated oracle comparators for testing Web applications. To effectively identify failures, each comparator is specialized to particular characteristics of the possibly nondeterministic Web applications' output in the form of HTML responses. We also describe combinations of comparators designed to achieve both high precision and recall in failure detection and a tool for helping testers to analyze the output of multiple oracles in detail. We present results from an evaluation of the effectiveness and costs of the oracle comparators. We also provide recommendations to testers on applying effective oracle comparators based on their application's characteristics.

The continuous use of the web for daily operations by businesses, consumers, and the government has created a great demand for reliable web applications. One promising approach to testing the functionality of web applications leverages user-session data collected by web servers. User-session-based testing automatically generates test cases based on real user profiles. The key contribution of this paper is the application of concept analysis for clustering user sessions and a set of heuristics for test case selection. Existing incremental concept analysis algorithms are exploited to avoid collecting and maintaining large user-session data sets and thus to provide scalability. We have completely automated the process from user session collection and test suite reduction through test case replay. Our incremental test suite update algorithm coupled with our experimental study indicate that concept analysis provides a promising means for incrementally updating reduced test suites in response to newly captured user sessions with little loss in fault detection capability and program coverage.

Test suite reduction uses test requirement coverage to determine if the reduced test suite maintains the original suite's requirement coverage. Based on observations from our previous experimental studies on test suite reduction, we believe there is a need for customized test requirements for Web applications. In this paper, we examine usage-based customized test requirements for the test suite reduction problem in Web application testing. We conduct an extensive experimental study to evaluate the tradeoffs between five classes of customized requirements with respect to reduced test suite size, program coverage and fault detection effectiveness. Our results show that the reduced suites' program coverage and fault detection effectiveness increases with the context or data associated with the reduction requirement. Based on our experimental results, we provide guidance to testers on the most useful test requirement for Web applications in general and provide intuition on factors testers need to consider when selecting test requirements

Automated cost-effective test strategies are needed to provide reliable, secure, and usable Web applications. As a software maintainer updates an application, test cases must accurately reflect usage to expose faults that users are most likely to encounter. User-session-based testing is an automated approach to enhancing an initial test suite with real user data, enabling additional testing during maintenance as well as adding test data that represents usage as operational profiles evolve. Test suite reduction techniques are critical to the cost effectiveness of user-session-based testing because a key issue is the cost of collecting, analyzing, and replaying the large number of test cases generated from user-session data. We performed an empirical study comparing the test suite size, program coverage, fault detection capability, and costs of three requirements-based reduction techniques and three variations of concept analysis reduction applied to two Web applications. The statistical analysis of our results indicates that concept analysis-based reduction is a cost-effective alternative to requirements-based approaches.

User-session-based testing of web applications gathers user sessions to create and continually update test suites based on real user input in the field. To support this approach during maintenance and beta testing phases, we have built an automated framework for testing web-based software that focuses on scalability and evolving the test suite automatically as the application's operational profile changes. This paper reports on the automation of the replay and oracle components for web applications, which pose issues beyond those in the equivalent testing steps for traditional, stand-alone applications. Concurrency, nondeterminism, dependence on persistent state and previous user sessions, a complex application infrastructure, and a large number of output formats necessitate developing different replay and oracle comparator operators, which have tradeoffs in fault detection effectiveness, precision of analysis, and efficiency. We have designed, implemented, and evaluated a set of automated replay techniques and oracle comparators for user-session-based testing of web applications. This paper describes the issues, algorithms, heuristics, and an experimental case study with user sessions for two web applications. From our results, we conclude that testers performing user-session-based testing should consider their expectations for program coverage and fault detection when choosing a replay and oracle technique.

This paper presents new mechanisms for dynamic resource management in a cluster manager called Cluster-on-Demand (COD). COD allocates servers from a common pool to multiple virtual clusters (vclusters), with independently configured software environments, name spaces, user access controls, and network storage volumes. We present experiments using the popular Sun GridEngine batch scheduler to demonstrate that dynamic virtual clusters are an enabling abstraction for advanced resource management in computing utilities and grids. In particular, they support dynamic, policy-based cluster sharing between local users and hosted Grid services, resource reservation and adaptive provisioning, scavenging of the idle resources, and dynamic instantiation of Grid services. These goals are achieved in a direct and general way through a new set of fundamental cluster management functions, with minimal impact on the Grid middleware itself.

Our work focuses on building tools to support collaborative software development. We are building a new programming environment with integrated software configuration management which provides a variety of features to help programming teams coordinate their work.