The RAND Database of Worldwide Terrorism Incidents (RDWTI) seeks to index information about all terrorist incidents that occur and are mentioned in worldwide news media, providing a useful resource for policy researchers and decision makers. We examined automated classification methods that could be used to identify news articles about terrorist incidents, thus enabling analysts to read a smaller number of news articles and maintain the database with less effort and cost. The support vector machine (SVM) and Lasso methods were only modestly successful, but a classifier based on the gradient boosting method (GBM) appeared to be very successful, correctly ranking 80% of the relevant articles at the “top of the pile” for examination by a human analyst.

OBJECTIVE: Prescription drugs can be associated with adverse effects (AEs) that are unrecognized despite evidence in the medical literature, as shown by rofecoxib's late recall in 2004. We assessed whether applying information mining to PubMed could reveal major drug-AE associations if articles testing whether drugs cause AEs are over-represented in the literature. DESIGN: MEDLINE citations published between 1949 and September 2009 were retrieved if they mentioned one of 38 drugs and one of 55 AEs. A statistical document classifier (using MeSH index terms) was constructed to remove irrelevant articles unlikely to test whether a drug caused an AE. The remaining relevant articles were analyzed using a disproportionality analysis that identified drug-AE associations (signals of disproportionate reporting) using step-up procedures developed to control the familywise type I error rate. MEASUREMENTS: Sensitivity and positive predictive value (PPV) for empirical drug-AE associations as judged against drug-AE associations subject to FDA warnings. RESULTS: In testing, the statistical document classifier identified relevant articles with 81% sensitivity and 87% PPV. Using data filtered by the statistical document classifier, base-case models showed 64.9% sensitivity and 42.4% PPV for detecting FDA warnings. Base-case models discovered 54% of all detected FDA warnings using literature published before warnings. For example, the rofecoxib-heart disease association was evident using literature published before 2002. Analyses incorporating literature mentioning AEs common to the drug class of interest yielded 71.4% sensitivity and 40.7% PPV. CONCLUSIONS: Results from large-scale literature retrieval and analysis (literature mining) compared favorably with and could complement current drug safety methods.

The paper summarizes the themes and goals of the Workshop on Advances in Model-Based Software Testing.

A new measure of performance, which uses both application integrity and traditional network response time, is proposed. Modern networked application services rely on a stack of network protocols and a host of other services many of which cross-organizational and corporate boundaries. We point out that traditional software quality assurance techniques don't scale up for post-deployment integrity checks for such applications and services. A new methodology to do non-stop post-production monitoring of networked application services for transactional integrity and time delay measurement is proposed. Specifically we describe the Telcordia/spl trade/ Application Assurance System, which we have created for measuring real-time performance of web-based applications used in commercial settings. The system measures both post-production application integrity and time delay. The measurements are carried out by sending synthetic end-user transactions and analyzing the responses. Statistical models for analyzing the data using single monitoring site as well as multiple monitoring sites are described. Creating synthetic end-user transactions is crucial for our method. The paper presents a method for generation of 'highly efficient' end-user transactions from a graphical model of the functionality of the system. Highly efficient transactions are generated using combinatorial designs. The graphical model is incrementally created using a recorder. We give several empirical examples of efficacy of this system and uses for finding performance problems.

Security is a critical consideration for an enterprise communications system. Different work contexts and corporate cultures may have different security requirements. In fact, such requirements may range from "no security" to protection from all possible attacks. In addition, as a communications session progresses, its context may dynamically change, and its security requirements may need to be changed accordingly. In this paper, we present our approach to providing security in enterprise communications in an efficient manner and report on the results of our initial performance study, which shows that encrypting and decrypting real-time media payloads end-to-end does not significantly degrade the quality of the end-user experience in real-time communications. Our approach is implemented in an enterprise communications system, called SEC, which is designed to enable enterprise employees to conduct secure, spontaneous, multimedia, and multi-party conferencing.

Model-based testing is a new and evolving technique for generating a suite of test cases from requirements.Testers using this approach concentrate on a data model and generation infrastructure instead of hand-crafting individual tests.Several relatively small studies have demonstrated how combinatorial test generation techniques allow testers to achieve broad coverage of the input domain with a small number of tests.We have conducted several relatively large projects in which we applied these techniques to systems with millions of lines of code.Given the complexity of testing, the modelbased testing approach was used in conjunction with test automation harnesses.Since no large empirical study has been conducted to measure efficacy of this new approach, we report on our experience with developing tools and methods in support of model-based testing.The four case studies presented here offer details and results of applying combinatorial test-generation techniques on a large scale to diverse applications.Based on the four projects, we offer our insights into what works in practice and our thoughts about obstacles to transferring this technology into testing organizations.

The paradigm of model based testing shifts the focus of testing from writing individual test cases to developing a model from which a test suite can be generated automatically. We report on our experience with model based testing of a highly programmable system that implements intelligent telephony services in the US telephone network. Our approach used automatic test case generation technology to develop sets of self checking test cases based on a machine readable specification of the messages in the protocol under test. The AETG/sup TM/ software system selected a minimal number of test data tuples that covered pairwise combinations of tuple elements. We found the combinatorial approach of covering pairwise interactions between input fields to be highly effective. Our tests revealed failures that would have been difficult to detect using traditional test designs. However, transferring this technology to the testing organization was difficult. Automatic generation of cases represents a significant departure from conventional testing practice due to the large number of tests and the amount of software development involved.

This paper describes a new approach to testing that uses combinatorial designs to generate tests that cover the pairwise, triple, or n-way combinations of a system's test parameters. These are the parameters that determine the system's test scenarios. Examples are system configuration parameters, user inputs and other external events. We implemented this new method in the AETG system. The AETG system uses new combinatorial algorithms to generate test sets that cover all valid n-way parameter combinations. The size of an AETG test set grows logarithmically in the number of test parameters. This allows testers to define test models with dozens of parameters. The AETG system is used in a variety of applications for unit, system, and interoperability testing. It has generated both high-level test plans and detailed test cases. In several applications, it greatly reduced the cost of test plan development.

Software testing is expensive, tedious and time consuming. Thus, the problem of making testing more efficient and mechanical, without losing its effectiveness, is very important. The Automatic Efficient Test Generator (AETG) is a new tool that mechanically generates efficient test sets from user defined test requirements. It is based on algorithms that use ideas from statistical experimental design theory to minimize the number of tests needed for a specific level of test coverage of the input test space. The savings due to AETG are substantial when compared to exhaustive testing or other methods of testing. AETG has been used in Bellcore for screen testing, interoperability testing and for protocol conformance testing. The paper describes the current system and it constructs and reports some preliminary results obtained during initial trials.>

For pt. I see ibid., vol.12, no.1, p.23-32 (1994). The software created by industrial, educational, and research organizations is increasingly large and complex. It also occupies a central role in the reliability and safety of many essential services. The authors examine the software development process and suggest opportunities for improving the process by using a combination of statistical and other process control techniques. Data, analysis of data, and tools for collecting data are crucial to the approach. Although the authors' views are based upon experiences with large telecommunications systems, they are likely to be useful to many other developers of large software systems.>

Developers of large software systems must decide how long software should be tested before releasing it. A common and usually unwarranted assumption is that the code remains frozen during testing. We present a stochastic and economic framework to deal with systems that change as they are tested. The changes can occur because of the delivery of software as it is developed, the way software is tested, the addition of fixes, and so on. Specifically, we report the details of a real time trial of a large software system that had a substantial amount of code added during testing. We describe the methodology, give all of the relevant details, and discuss the results obtained. We pay particular attention to graphical methods that are easy to understand, and that provide effective summaries of the testing process. Some of the plots found useful by the software testers include: the Net Benefit Plot, which gives a running chart of the benefit; the Stopping Plot, which estimates the amount of additional time needed for testing; and diagnostic plots. To encourage other researchers to try out different models, all of the relevant data are provided.>

It is noted that the developers of large software systems must decide how much software should be tested before releasing it. An explicit tradeoff between the costs of testing and releasing is considered. The former may include the opportunity cost of continued testing, and the latter may include the cost of customer dissatisfaction and of fixing faults found in the field. Exact stopping rules were obtained by Dalal and Mallows (J. Amer., Statist. Assoc., vol.83, p.872, 1988), under the assumption that the distribution of the fault finding rate is known. Here, two important variants where the fault finding distribution is not completely known are considered. They are (i) the distribution is exponential with unknown mean and (ii) the distribution is locally exponential with the rate changing smoothly over time. New procedures for both cases are presented. In case (i) it is shown how to incorporate information from related projects and subjective inputs. Several novel graphical procedures which are easy to implement are proposed, and these are illustrated for data from a large telecommunications software system.>