We propose a novel confidence scoring mechanism for deep neural networks based on a two-model paradigm involving a base model and a meta-model. The confidence score is learned by the meta-model observing the base model succeeding/failing at its task. As features to the meta-model, we investigate linear classifier probes inserted between the various layers of the base model. Our experiments demonstrate that this approach outperforms multiple baselines in a filtering task, i.e., task of rejecting samples with low confidence. Experimental results are presented using CIFAR-10 and CIFAR-100 dataset with and without added noise. We discuss the importance of confidence scoring to bridge the gap between experimental and real-world applications.

Predictive modeling is becoming increasingly common in healthcare. Existing healthcare cost prediction solutions are tailor-made to accomplish specific tasks for certain populations, hence requiring expensive modifications to adapt to a different task or population. In this paper, we present a modular and extensible solution for healthcare cost prediction, which can be easily configured for various prediction tasks and populations. Our solution incorporates efficient high-dimensional data handling, smart feature engineering, flexible predictive learning, individualized assessment of cost impacts of predictors, and a management system that allows for reuse of partial results. We configure two distinct applications using the proposed system and present results on prediction accuracy and cost impact assessment. The first application predicts healthcare costs for a commercial population, and the second predicts the cost of care for a Medicaid population using an entirely different set of data, predictors, and assumptions.

Detection of space-time clusters is an important function in various domains (e.g., epidemiology and public health). The pioneering work on the spatial scan statistic is often used as the basis to detect and evaluate such clusters. State-of-the-art systems based on this approach detect clusters with restrictive shapes that cannot model growth and shifts in location over time. We extend these methods significantly by using the flexible square pyramid shape to model such effects. A heuristic search method is developed to detect the most likely clusters using a randomized algorithm in combination with geometric shapes processing. The use of Monte Carlo methods in the original scan statistic formulation is continued in our work to address the multiple hypothesis testing issues. Our method is applied to a real data set on brain cancer occurrences over a 19 year period. The cluster detected by our method shows both growth and movement which could not have been modeled with the simpler cylindrical shapes used earlier. Our general framework can be extended quite easily to handle other flexible shapes for the space-time clusters.

Health activity monitoring (HAM) has received increasing attention due to the rapid advances of both hardware and software technologies and strong environmental and public health needs. In this paper, we describe the architecture and implementation of the Epi-SPIRE prototype, which is a novel health activity monitoring system that generates alerts from environmental, behavioral, and public health data sources. A model-based approach is used to develop disease and behavior models from multi-modal heterogeneous data sources. Furthermore, a model-based indexing technique has been developed to speed up the data access and retrieval. This system has been successfully applied to various genuine and simulated diseases outbreaks scenarios'.

Data on individuals and entities are being collected widely. These data can contain information that explicitly identifies the individual (e.g., social security number). Data can also contain other kinds of personal information (e.g., date of birth, zip code, gender) that are potentially identifying when linked with other available data sets. Data are often shared for business or legal reasons. This paper addresses the important issue of preserving the anonymity of the individuals or entities during the data dissemination process. We explore preserving the anonymity by the use of generalizations and suppressions on the potentially identifying portions of the data. We extend earlier works in this area along various dimensions. First, satisfying privacy constraints is considered in conjunction with the usage for the data being disseminated. This allows us to optimize the process of preserving privacy for the specified usage. In particular, we investigate the privacy transformation in the context of data mining applications like building classification and regression models. Second, our work improves on previous approaches by allowing more flexible generalizations for the data. Lastly, this is combined with a more thorough exploration of the solution space using the genetic algorithm framework. These extensions allow us to transform the data so that they are more useful for their intended purpose while satisfying the privacy constraints.

Forestry management requires careful and intensive planning efforts to ensure optimal yield, ecological stability, and regulatory compliance. We describe a method of identifying wetlands and producing maps of their extent from commonly available, remotely-sensed imagery. This method provides a large labor savings over both field inspections and manual photo inspections. The enhanced accuracy translates into better timber harvest planning and better conservation of the wetlands.

The task of evaluating and ranking items with multiple-attributes appears in many guises in commerce. Examples include evaluating responses to a request for quotes (RFQ) for some item and comparison shopping for an item within one or more catalogs. This task is straightforward if the value of the item can be explicitly specified by the evaluator as a function of the attribute values. However, a typical evaluator may not be able to provide the value function in explicit form. In contrast, it is intuitive for them to compare, say, two items and pick the preferable one based on all of the relevant attributes. In this paper we present a method, Q-Eval, that queries the evaluator with selected pairs of items and uses the responses to build a preference model for the evaluator. This model is then used to rank the items in order of the inferred preference. The evaluator can then pick the winning item or items by considering only the top few items in this ranked list. This should result in significant productivity improvement for the evaluator when the number of items to choose from is large. Our algorithm is novel in the way it attempts to derive a stable preference model with only a small number of user queries. This paper describes the algorithm and presents experimental results with real-life data to validate the approach.

Classi cation modeling (a.k.a.supervised learning) is an extremely useful analytical technique for developing predictive and forecasting applications.The explosive growth in data warehousing and internet usage has made large amounts of data potentially available for developing classi cation models.For example, natural language text is widely available in many forms (e.g., electronic mail, news articles, reports, and web page contents).Categorization of data is a common activity which can be automated to a large extent using supervised learning methods.Examples of this include routing of electronic mail, satellite image classi cation, and character recognition.However, these tasks require labeled data sets of suciently high quality with adequate instances for training the predictive models.Much of the on-line data, particularly the unstructured variety (e.g., text), is unlabeled.Labeling is usually a expensive manual process done by domain experts.Active learning is an approach to solving this problem and works by i d e n tifying a subset of the data that needs to be labeled and uses this subset to generate classi cation models.We present an active learning method that uses adaptive resampling in a natural way t o signi cantly reduce the size of the required labeled set and generates a classi cation model that achieves the high accuracies possible with current adaptive resampling methods.

This paper presents a new method (HOT) of generating oblique decision trees. Oblique trees have been shown to be useful tools for classification in some problem domains, producing accurate and intuitive solutions. The method can be incorporated into a variety of existing decision tree tools and the paper illustrates this with two very distinct tree generators. The key idea is a method of learning oblique vectors and using the corresponding families of hyperplanes orthogonal to these vectors to separate regions with distinct dominant classes. Experimental results indicate that the learnt oblique hyperplanes lead to compact and accurate oblique trees. HOT is simple and easy to incorporate into most decision tree packages, yet its results compare well with much more complex schemes for generating oblique trees.

Performance evaluation of processor designs using dynamic instruction traces is a critical part of the iterative design process. The widening gap between the billions of instructions in such traces for benchmark programs and the throughput of timers performing the analysis in the tens of thousands of instructions per second has led to the use of reduced traces during design. This opens up the issue of whether these traces are truly representative of the actual workload in these benchmark programs. The first key result in this paper is the introduction of a new metric, called the R-metric, to evaluate the representativeness of these reduced traces when applied to a wide class of processor designs. The second key result, is the development of a novel graph-based heuristic to generate reduced traces based on the notions incorporated in the metric. These ideas have been implemented in a prototype system (SMART) for generating representative and reduced traces. Extensive experimental results are presented on various benchmarks to demonstrate the quality of the synthetic traces and the uses of the R-metric.

This paper describes a system (AVPGEN) for generating tests (called architecture verification programs or AVP's) to check the conformance of processor designs to the specified architecture. To generate effective tests, AVPGEN uses novel concepts like symbolic execution and constraint solving, along with various biasing techniques. Unlike many earlier systems that make biased random choices, AVPGEN often chooses intermediate or final values and then solves for initial values that can lead to the desired values. A language called SIGL (symbolic instruction graph language) is provided in AVPGEN for the user to specify templates with symbolic constraints. The combination of user-specified constraints and the biasing functions is used to focus the tests on conditions that are interesting in that they are likely to activate various kinds of bugs. The system has been used successfully to debug many S/390 processors and is an integral part of the design process for these processors.>

High performance processor designs use techniques such as pipelining, multiple execution units, register renaming, bypass paths, and branch prediction to meet their goals. These techniques make them susceptible to design errors that are triggered only when executing complex sequences of instructions. We introduce a language called SIGL for specifying symbolic instruction graphs (SIGs) that can be used as templates for such test cases. SIGL allows specification of constraints in a high level template from which many test cases can be generated, all targeting some specific characteristic of a processor design. SIGL has been used successfully to check the conformance of various industrial processor designs to their architectural specifications.>

Various defects during fabrication have been shown in the literature to introduce delay faults in logic circuits. This paper analyzes the effects of these defects on the normal operation of logic circuits with the goal of developing an appropriate model for these faults. Single and multiple delay faults in this model are analyzed to determine if they are redundant with respect to the normal operation of the logic circuit. The relationships between delay redundancies and stuck-at redundancies are discussed. The redundancy identification techniques are applied to various benchmarks circuits and experimental data are presented.>

Gate delay faults can represent the effects of common point faults in logic circuits. A gate delay fault model based on the analysis of these effects in the normal operation of logic circuits is developed. The fault model allows for delay faults of both bounded and unbounded size. Techniques have been developed to identify single gate delay faults that do not have any effect on the normal operation of logic circuits. These techniques are applied to various benchmark circuits and indicate the existence of a surprisingly large number of such redundancies.>

The use of constraint solving and symbolic execution to generate highly probing test cases is discussed. A system called AVPGEN, which uses such techniques for debugging high-level machine designs and has already proved useful in finding errors, is described. AVPGEN generates large numbers of architecture verification programs. It consists of a generator and a symbolic simulator. The generator uses constraint solving, symbolic values and undo capabilities to set up conditions. These conditions may be architecture specific or design specific. The simulator in AVPGEN, although not bug-free, is considerably simpler than any of the hardware implementations it is used to verify.>

Delay testing, as opposed to static testing introduces the parameter of time as a new variable. Time impacts the way defects manifest themselves and are modeled as faults, as well as how defect sizes and system timing statistics interact with tester timing constraints in the detection of causes for dynamic system malfunctions. This paper briefly discusses some of the issues that had to be addressed in the development of a comprehensive system for delay testing in a Level Sensitive Scan Design environment.

We report an automatic test pattern generator that can handle designs with one million gates or more on medium size workstations. Run times and success rates, i.e. the fraction of faults that are resolved, are comparable to or better than those reported previously in the literature. No preprocessing is required and the amount of memory needed is less than 100 bytes per gate. The low memory requirements and high performance have been achieved by working with a larger but simpler search space, by simplifying decision making and backtracking and by using only implication techniques that are fast and that require no preprocessing.

The problems associated with optimization of the test application timing for a class of test equipment are identified. Two approaches to test application timing are introduced. The notion of slack is used to define the objective function for optimization. The optimization problem is shown to be NP-complete even for nonreconvergent-fanout circuits. Heuristics for the optimization problems are presented, and the results are compared with bounds on test circuits.>

The increasing emphasis on AC testing of integrated circuits is driven by the combination of tighter quality requirements and sensitivity of high performance circuits to delay defects. The areas of fault modeling, fault simulation and test generation as applied to AC testing have received most of the attention so far. The relatively unexplored side of AC test is the determination of the test application timing. Tight timings during test application are crucial to the success of the AC test. This paper formulates the problem of generating tight test application timings and presents some sample results using a heuristic algorithm.

Defects in integrated circuits can cause delay faults of various sizes. Testing for delay faults has the goal of detecting a large fraction of these faults for a wide range of fault sizes. Hence, an evaluation scheme for a delay fault test must not only compute whether or not a delay fault was detected, but also calculate the sizes of detected delay faults. Delay faults have the counterintuitive property that a test for a fault of one size need not be a test for a similar fault of a larger size. This makes it difficult to answer questions about the sizes of delay faults detected by a set of tests. A model for delay faults that answers such questions correctly, but with calculations simple enough to be done for large circuits, is presented.>

A method of synthesizing scan designs that are testable with pseudorandom patterns is presented. The logic is first simplified by various transformations in a logic synthesis system. A fault simulator is then used to guide the placement of control points and observation points. In order to reduce the overhead, control points are shared when possible and a condensation network is used with the observation points. Experimental results which indicate that pseudorandom testability can be achieved with small area overheads using simple techniques are presented.>

An approach to test for delay faults is presented. A variable size delay fault model is used to represent these failures. The nominal gate delays with the manufacturing tolerances are an integral part of the model and are used in the propagation of simplified waveforms through the logic network. The faulty waveforms are functions of the variable-size delay fault. For each fault and test pattern, a threshold is computed such that this fault is detected if its size exceeds epsilon . This threshold is used (along with the minimum slack at the fault site) to determine a metric called quality. The quality of detection for a fault measures how close the test came to exposing the ideally smallest-size fault at that point. This metric (together with the traditional fault coverage) gives a complete measure of the goodness of the test.>

For pt.I see ibid., p.857-66 (1988). A novel algebra is introduced for delay test generation. The algebra combines the nine natural logic values (00 , 01, 0X, 10, 11, 1X, X1, XX) with special attributes that record both heuristic choices and whatever information about waveforms is deducible algebraically (i.e. without numerical computations using actual gate delays). A test generator uses this algebra in an efficiently organized backtrack search. The test generator is linked to a delay fault simulator. Previous event-driven simulators have considered different types of events; one type of event is a change in faultless values from one test to another test, and the other type of event is a difference between faulty and faultless values. The presented simulator is driven by both types of events. Each generated test is simulated to determine the quality of detection.>

The usual block-oriented timing analysis for logic circuits does not take into account functional relations between signals. If functional relations are taken into consideration, it could be found that a long path is never activated. This results in more accurate delays. A comparison is made of three arrival time functions, A, B, and R. A is the arrival time as given by exhaustive simulation; B is the arrival time as calculated by a usual block-oriented algorithm; and R is the arrival time, that does functional analysis. It is shown that B contained in R contained in A. The first relation means that R is never more conservative than B and whenever the containment is proper, R is an improvement over B. The second relation means that R is correct in the sense that it will never assert a signal to be valid when it is not valid according to the ideal A. Experimental results showing how often R is an improvement over B are presented.>

SLS, a large-capacity, high-performance switch-level simulator developed to run on an IBM System/370 architecture is described. SLS uses a model which closely reflects the behavior of MOS circuits. The high performance is the result of mixing a compiled model with the more traditional approach of event-driven simulation control, together with very efficient algorithms for evaluating the steady-state response of the circuit. SLS is used for design verification/checking applications and for estimating fault coverage.>

We describe SLS, a large capacity, high performance switch level simulator, developed to run on an IBM System/370 architecture, that uses a model which closely reflects the behavior of MOS circuits. This performance is the result of mixing a compiled model with the more traditional approach of event-driven simulation control, together with very efficient algorithms for evaluating the steady state response of the circuit. SLS is used for design verification/checking applications and for estimating fault coverage.

This paper specifies procedures for defining a monitor circuit that can detect faults in microprogram sequencers. The monitor and the sequencer operate in parallel and errors are detected by comparing outputs from the monitor circuit with outputs from the sequencer. Faults that cause errors in the flow of control are detectable, as well as some faults that cause errors only in the microinstruction fields. The design procedure presented for monitors consists of four parts. First, a model of the program flow is constructed that only retains the information required to define a monitor. Second, faults in a specified fault set are modeled by the errors they cause in the program flow model. Third, the functional requirements of the monitor are specified in terms of partitions on the states of the program flow model. Fourth, the logic design of the monitor is completed.