The past 10 years have witnessed an explosion in deep learning neural network model development. The most common perceptual models with vision, speech, and text inputs are not general-purpose AI systems but tools. They automatically extract clues from inputs and compute probabilities of class labels. Successful applications require representative training data, an understanding of the limitations and capabilities of deep learning, and careful attention to a complex development process. The goal of this view is to foster an intuitive understanding of convolutional network deep learning models and how to use them with the goal of engaging a wider creative community. A focus is to make it possible for experts in areas such as health, education, poverty, and agriculture to understand the process of deep learning model development so they can help transition effective solutions to practice.

Abstract Previous methods of network anomaly detection have focused on defining a temporal model of what is "normal," and flagging the "abnormal" activity that does not fit into this pre-trained construct. When monitoring traffic to and from IP addresses on a large network, this problem can become computationally complex, and potentially intractable, as a state model must be maintained for each address. In this paper, we present a method of detecting anomalous network activity without providing any historical context. By exploiting the size of the network along with the minimal overhead of NetFlow data, we are able to model groups of hosts performing similar functions to discover anomalous behavior. As a collection, these anomalies can be further described with a few high-level characterizations and we provide a means for creating and labeling these categories. We demonstrate our method on a very large-scale network consisting of 30 million unique addresses, focusing specifically on traffic related to web servers.

A new tool named NAVIGATOR (Network Asset VIsualization: Graphs, ATtacks, Operational Recommendations) adds significant capabilities to earlier work in attack graph visualization. Using NAVIGATOR, users can visualize the effect of server-side, client-side, credential-based, and trustbased attacks. By varying the attacker model, NAVIGA-TOR can show the current state of the network as well as hypothetical future situations, allowing for advance planning. Furthermore, NAVIGATOR explicitly shows network topology, infrastructure devices, and host-level data while still conveying situational awareness of the network as a whole. This tool is implemented in Java and uses an existing C++ engine for reachability and attack graph calculations.

Geographical displays are commonly used for visualizing wide-spread malicious behavior of Internet hosts. Placing dots on a world map or coloring regions by the magnitude of activity often results in cluttered maps that invariably emphasize population-dense metropolitan areas in developed countries where Internet connectivity is highest. To uncover atypical regions, it is necessary to normalize activity by the local computer population. This paper presents EMBER (Extreme Malicious Behavior viewER), an analysis and display of malicious activity at the city level. EMBER uses a metric called Standardized Incidence Rate (SIR) that is the number of hosts exhibiting malicious behavior per 100,000 available hosts. This metric relies on available data that (1) Maps IP addresses to geographic locations, (2) Provides current city populations, and (3) Provides computer usage penetration rates. Analysis of several months of suspicious source IP addresses from DShield identifies cities with extremely high and low malicious activity rates on a day-by-day basis. In general, cities in a few Eastern European countries have the highest SIRs whereas cities in Japan and South Korea have the lowest. Many of these results are consistent with news reports describing local cyber security policies. A simulation that models how malware spreads preferentially within cities to local IP addresses replicates the long-tailed distribution of city SIRs that was found in the data. This simulation result agrees with past analyses in suggesting that malware often preferentially spreads to local regions with already high levels of malicious activity.

By accurately measuring risk for enterprise networks, attack graphs allow network defenders to understand the most critical threats and select the most effective countermeasures. This paper describes substantial enhancements to the NetSPA attack graph system required to model additional present-day threats (zero-day exploits and client-side attacks) and countermeasures (intrusion prevention systems, proxy firewalls, personal firewalls, and host-based vulnerability scans). Point-to-point reachability algorithms and structures were extensively redesigned to support "reverse" reachability computations and personal firewalls. Host-based vulnerability scans are imported and analyzed. Analysis of an operational network with 84 hosts demonstrates that client-side attacks pose a serious threat. Experiments on larger simulated networks demonstrated that NetSPA's previous excellent scaling is maintained. Less than two minutes are required to completely analyze a four-enclave simulated network with more than 40,000 hosts protected by personal firewalls.

Attack graphs are a valuable tool to network defenders, illustrating paths an attacker can use to gain access to a targeted network. Defenders can then focus their efforts on patching the vulnerabilities and configuration errors that allow the attackers the greatest amount of access. We have created a new type of attack graph, the multiple-prerequisite graph, that scales nearly linearly as the size of a typical network increases. We have built a prototype system using this graph type. The prototype uses readily available source data to automatically compute network reachability, classify vulnerabilities, build the graph, and recommend actions to improve network security. We have tested the prototype on an operational network with over 250 hosts, where it helped to discover a previously unknown configuration error. It has processed complex simulated networks with over 50,000 hosts in under four minutes

Passive network mapping has often been proposed as an approach to maintain up-to-date information on networks between active scans. This paper presents a comparison of active and passive mapping on an operational network. On this network, active and passive tools found largely disjoint sets of services and the passive system took weeks to discover the last 15% of active services. Active and passive mapping tools provided different, not complimentary information. Deploying passive mapping on an enterprise network does not reduce the need for timely active scans due to non-overlapping coverage and potentially long discovery times

The prevalent use of computers and internet has enhanced the quality of life for many people, but it has also attracted undesired attempts to undermine these systems. This special topic contains several research studies on how machine learning algorithms can help improve the security of computer systems.

Five modern static analysis tools (ARCHER, BOON, Poly-Space C Verifier, Splint, and UNO) were evaluated using source code examples containing 14 exploitable buffer overflow vulnerabilities found in various versions of Sendmail, BIND, and WU-FTPD. Each code example included a "BAD" case with and a "OK" case without buffer overflows. Buffer overflows varied and included stack, heap, bss and data buffers; access above and below buffer bounds; access using pointers, indices, and functions; and scope differences between buffer creation and use. Detection rates for the "BAD" examples were low except for Poly-Space and Splint which had average detection rates of 87% and 57%, respectively. However, average false alarm rates were high and roughly 50% for these two tools. On patched programs these two tools produce one warning for every 12 to 46 lines of source code and neither tool appears able to accurately distinguished between vulnerable and patched code.

An integral part of modeling the global view of network security is constructing attack graphs. Manual attack graph construction is tedious, error-prone, and impractical for attack graphs larger than a hundred nodes. In this paper we present an automated technique for generating and analyzing attack graphs. We base our technique on symbolic model checking algorithms, letting us construct attack graphs automatically and efficiently. We also describe two analyses to help decide which attacks would be most cost-effective to guard against. We implemented our technique in a tool suite and tested it on a small network example, which includes models of a firewall and an intrusion detection system.

Macroscope is a network-based intrusion detection system that uses bottleneck verification (BV) to detect user-to-superuser attacks. BV detects novel computer attacks by looking for users performing high privilege operations without passing through legal "bottleneck" checkpoints that grant those privileges. Macroscope's BV implementation models many common Unix commands, and has extensions to detect intrusions that exploit trust relationships, as well as previously installed Trojan programs. BV performs at a false alarm rate more than two orders of magnitude lower than a reference signature verification system, while simultaneously increasing the detection rate from roughly 20% to 80% of user-to-superuser attacks.

Lack of training data is a major problem that limits the performance of speech recognizers. Performance can often only be improved by expensive collection of data from many different talkers. This paper demonstrates that artificially transformed speech can increase the variability of training data and increase the performance of a wordspotter without additional expensive data collection. This approach was shown to be effective on a high-performance whole-word wordspotter on the Switchboard Credit Card database. The proposed approach used in combination with a discriminative training approach increased the figure of merit of the wordspotting system by 9.4% percentage points (62.5% to 71.9%). The increase in performance provided by artificially transforming speech was roughly equivalent to the increase that would have been provided by doubling the amount of training data. The performance of the wordspotter was also compared to that of human listeners who were able to achieve lower error rates because of improved consonant recognition.

This paper compares the word error rate of a speech recognizer using several signal processing front ends based on auditory properties. Front ends were compared with a control mel filter bank (MFB) based cepstral front end in clean speech and with speech degraded by noise and spectral variability, using the TI-105 isolated word database. MFB recognition error rates ranged from 0.5 to 26.9% in noise, depending on the SNR, and auditory models provided error rates as much as four percentage points lower. With speech degraded by linear filtering, MFB error rates ranged from 0.5 to 3.1%, and the reduction in error rates provided by auditory models was less than 0.5 percentage points. Some earlier studies that demonstrated considerably more improvement with auditory models used linear predictive coding (LPC) based control front ends. This paper shows that MFB cepstra significantly outperform LPC cepstra under noisy conditions. Techniques using an optimal linear combination of features for data reduction were also evaluated.>

A new approach to wordspotter training is presented which directly maximizes the figure of merit (FOM) defined as the average detection rate over a specified range of false alarm rates. This systematic approach to discriminant training for wordspotters eliminates the necessity of ad hoc thresholds and tuning. It improves the FOM of wordspotters tested using cross-validation on the credit-card speech corpus training conversations by 4 to 5 percentage points to roughly 70%. This improved performance requires little extra complexity during wordspotting and only two extra passes through the training data during training. The FOM gradient is computed analytically for each putative hit, back-propagated through HMM word models using the Viterbi alignment, and used to adjust RBF hidden node centers and state-weights associated with every node in HMM keyword models.>

Speech recognizers provide good performance for most users but the error rate often increases dramatically for a small percentage of talkers who are "different" from those talkers used for training. One expensive solution to this problem is to gather more training data in an attempt to sample these outlier users. A second solution, explored in this paper, is to artificially enlarge the number of training talkers by transforming the speech of existing training talkers. This approach is similar to enlarging the training set for OCR digit recognition by warping the training digit images, but is more difficult because continuous speech has a much larger number of dimensions (e.g. linguistic, phonetic, style, temporal, spectral) that differ across talkers. We explored the use of simple linear spectral warping to enlarge a 48-talker training data base used for word spotting. The average detection rate overall was increased by 2.9 percentage points (from 68.3% to 71.2%) for male speakers and 2.5 percentage points (from 64.8% to 67.3%) for female speakers. This increase is small but similar to that obtained by doubling the amount of training data.

Experiments demonstrated that sigmoid multilayer perceptron (MLP) networks provide slightly better risk prediction than conventional logistic regression when used to predict the risk of death, stroke, and renal failure on 1257 patients who underwent coronary artery bypass operations at the Lahey Clinic. MLP networks with no hidden layer and networks with one hidden layer were trained using stochastic gradient descent with early stopping. MLP networks and logistic regression used the same input features and were evaluated using bootstrap sampling with 50 replications. ROC areas for predicting mortality using preoperative input features were 70.5% for logistic regression and 76.0% for MLP networks. Regularization provided by early stopping was an important component of improved perfonnance. A simplified approach to generating confidence intervals for MLP risk predictions using an auxiliary "confidence MLP" was developed. The confidence MLP is trained to reproduce confidence intervals that were generated during training using the outputs of 50 MLP networks trained with different bootstrap samples.

The components of a speech message information retrieval system include an acoustic front end which provides an incomplete transcription of a spoken message, and a message classifier that interprets the incomplete transcription and classifies the message according to message category. The techniques and experiments described are concerned with the integration of these components and represent the first demonstration of a complete system that accepts speech messages as input and produces as estimated message class as output. The complete system has been implemented on special-purpose digital signal processing hardware and demonstrated using live speech input. The results obtained on a conversational speech task have demonstrated the feasibility of the technology and also illustrate the need for further work. Even with a perfect acoustic front end, a message classification accuracy of only 78% was obtained with a 126 keyword vocabulary.>

Many neural network classifiers provide outputs which estimate Bayesian a posteriori probabilities. When the estimation is accurate, network outputs can be treated as probabilities and sum to one. Simple proofs show that Bayesian probabilities are estimated when desired network outputs are 1 of M (one output unity, all others zero) and a squared-error or cross-entropy cost function is used. Results of Monte Carlo simulations performed using multilayer perceptron (MLP) networks trained with backpropagation, radial basis function (RBF) networks, and high-order polynomial networks graphically demonstrate that network outputs provide good estimates of Bayesian probabilities. Estimation accuracy depends on network complexity, the amount of training data, and the degree to which training data reflect true likelihood distributions and a priori class probabilities. Interpretation of network outputs as Bayesian probabilities allows outputs from multiple networks to be combined for higher level decision making, simplifies creation of rejection thresholds, makes it possible to compensate for differences between pattern class probabilities in training and test data, allows outputs to be used to minimize alternative risk functions, and suggests alternative measures of network performance.

Genetic algorithms were used for feature selection and creation in two pattern-classification problems. On a machine-version inspection task, it was found that genetic algorithms performed no better than conventional approaches to feature selection but required much more computation. On a difficult artificial machine-vision task, genetic algorithms were able to create new features (polynomial functions of the original features) which dramatically reduced classification error rates. Neural network and nearest-neighbor classifiers were unable to provide such low error rates using only the original features

A physiological front-end preprocessor for speech recognition was evaluated using a large isolated-word database in quiet and noise. The front-end was based on the ensemble interval histogram (EIH) model developed by O. Ghitza. This model provides phase or synchrony information similar to that available on the auditory nerve. A modified EIH front-end was implemented and was tested using the Lincoln robust hidden Markov model isolated-word recognizer with a multistyle database at various signal-to-noise ratios (SNRs). The modified EIH front-end performed as well as a conventional mel-filter-bank front-end for normal speech. It provided a slight improvement in error rate at very low SNRs but required substantially more computation than the mel-filter-bank front-end

The performance of current speech recognition systems is far below that of humans. Neural nets offer the potential of providing massive parallelism, adaptation, and new algorithmic approaches to problems in speech recognition. Initial studies have demonstrated that multilayer networks with time delays can provide excellent discrimination between small sets of pre-segmented difficult-to-discriminate words, consonants, and vowels. Performance for these small vocabularies has often exceeded that of more conventional approaches. Physiological front ends have provided improved recognition accuracy in noise and a cochlea filter-bank that could be used in these front ends has been implemented using micro-power analog VLSI techniques. Techniques have been developed to scale networks up in size to handle larger vocabularies, to reduce training time, and to train nets with recurrent connections. Multilayer perceptron classifiers are being integrated into conventional continuous-speech recognizers. Neural net architectures have been developed to perform the computations required by vector quantizers, static pattern classifiers, and the Viterbi decoding algorithm. Further work is necessary for large-vocabulary continuous-speech problems, to develop training algorithms that progressively build internal word models, and to develop compact VLSI neural net hardware.

Algorithms that are implementable by artificial neural networks show promise of augmenting the field of automatic speech recognition. A specific approach to the problem of isolated-word recognition was initiated by Tank and Hopfield. These ideas were applied to a concatenated systems consisting of a vector quantizer, a time concentrator with vector sequences as input and allophones as output and a final stage with allophone sequence as input and isolated words as output. Improvements in the system are discussed; included are the addition of data-dependent 'phenomenological' rules that yield improved results for a single-speaker 35-word vocabulary isolated-word recognition task.>

Artificial neural networks are of interest because algorithms used in many speech recognizers can be implemented using highly parallel neural net architectures and because new parallel algorithms are being development that are inspired by biological nervous systems. Some neural net approaches are resented for the problem of static pattern classification and time alignment. For static pattern classification, multi-layer perceptron classifiers trained with back propagation can form arbitrary decision regions, are robust, and train rapidly for convex decision regions. For time alignment, the Viterbi net is a neural net implementation of the Viterbi decoder used very effectively in recognition systems based on hidden Markov models (HMMs).>

There has been a resurgence of interest in neutral net models composed of many simple interconnected processing elements operating in parallel. The computational power of different neutral net models and the effectiveness of simple error correction training procedures have been demonstrated. Three important feed-forward models are described. Single- and multi-layer perceptrons which can be used for pattern classification are described, as well as Kohonen's feature map algorithm which can be used for clustering or as a vector quantizer. A major emphasis is placed on relating these models to existing classification and clustering algorithms.>

One limitation of hidden Markov model (HMM) recognizers is that subword models are not learned but must be prespecified before training. This can lead to excessive computation during recognition and/or poor discrimination between similar sounding words. A training procedure called discriminant clustering is presented that creates subword models automatically. Node sequences from whole-word models are merged using statistical clustering techniques. This procedure reduced the computation required during recognition for a 35-word vocabulary by roughly one-third while maintaining a low error rate. It was also found that five iterations of the forward-backward algorithm are sufficient and that adding nodes to HMM word models improves performance until the minimum word transition time becomes excessive.>

The authors describe an HMM-based isolated-word recognition system that dynamically adapts word model parameters to new speakers and to stress-induced speech variations. During recognition all input tokens presented to the system can be used to augment the current word model parameters. New tokens can be weighted so that adaptation simply increases the size of the training set, or tracks systematic changes by exponentially weighting all previously seen data. This system was tested on the 35-word 10710 token Lincoln stressed speech data base. Speaker adaptation experiments produced error rates equivalent to speaker-trained systems after the presentation of only a single new token per vocabulary word. Stress condition adaptation experiments produced results comparable to multistyle-trained systems after the presentation of several new tokens per vocabulary word.>

A new training procedure called multi-style training has been developed to improve performance when a recognizer is used under stress or in high noise but cannot be trained in these conditions. Instead of speaking normally during training, talkers use different, easily produced, talking styles. This technique was tested using a speech data base that included stress speech produced during a workload task and when intense noise was presented through earphones. A continuous-distribution talker-dependent Hidden Markov Model (HMM) recognizer was trained both normally (5 normally spoken tokens) and with multi-style training (one token each from normal, fast, clear, loud, and question-pitch talking styles). The average error rate under stress and normal conditions fell by more than a factor of two with multi-style training and the average error rate under conditions sampled during training fell by a factor of four.

This paper describes a two-stage isolated word speech recognition system that uses a Hidden Markov Model (HMM) recognizer in the first stage and a discriminant analysis system in the second stage. During recognition, when the first-stage recognizer is unable to clearly differentiate between acoustically similar words such as "go" and "no" the second-stage discriminator is used. The second-stage system focuses on those parts of the unknown token which are most effective at discriminating the confused words. The system was tested on a 35 word, 10,710 token stress speech isolated word data base created at Lincoln Laboratory. Adding the second-stage discriminating system produced the best results to date on this data base, reducing the overall error rate by more than a factor of two.