A new method of training deep neural networks including the convolutional network is proposed. The method deconvexifies the normalized risk-averting error (NRAE) gradually and switches to the risk-averting error (RAE) whenever RAE is computationally manageable. The method creates tunnels between the depressed regions around saddle points, tilts the plateaus, and eliminates nonglobal local minima. Numerical experiments show the effectiveness of gradual deconvexification as compared with unsupervised pretraining. After the minimization process, a statistical pruning method is used to enhance the generalization capability of the neural network under training. Numerical results show further reduction of the testing criterion.

Proper use of the normalized risk-averting error (NRAE) criterion has been shown to avoid nonglobal local minima effectively in the mean squared error (MSE) criterion. For training on large datasets, a pairwise algorithm for the NRAE criterion similar to the widely-used least mean square algorithm for the MSE criterion is proposed. The gradual deconvexification method employing this pairwise algorithm is tested on examples with built-in nonglobal local minima that are difficult to avoid and on recognition of handwritten numerals with the MNIST dataset. Numerical experiments show that the pairwise algorithm for the NRAE criterion is computationally more economical than the corresponding batch algorithm and delivers multilayer perceptrons with better performances than training methods based on the MSE criterion.

A method of training neural networks using the risk-averting error (RAE) criterion Jλ(w), which was presented in IJCNN 2001, has the capability to avoid nonglobal local minima, but suffers from a severe limitation on the magnitude of the risk-sensitivity index λ. To eliminating the limitation, an improved method using the normalized RAE (NRAE) Cλ(w) was proposed in ISNN 2012, but it requires a selection of a proper λ, whose range may be dependent on the application. A new training method called the gradual deconvexification (GDC) is proposed in this paper. It starts with a very large λ and gradually decreases it in the training process until a global minimum of Cλ(w) or a good generalization capability is achieved. GDC training method was tested on a large number of numerical examples and produced a very good result in each test.

This paper deals with an important probabilistic knowledge integration problem: revising a Bayesian network (BN) to satisfy a set of probability constraints representing new or more specific knowledge. We propose to solve this problem by adopting IPFP (iterative proportional fitting procedure) to BN. The resulting algorithm E-IPFP integrates the constraints by only changing the conditional probability tables (CPT) of the given BN while preserving the network structure; and the probability distribution of the revised BN is as close as possible to that of the original BN. Two variations of E-IPFP are also proposed: 1) E-IPFP-SMOOTH which deals with the situation where the probabilistic constraints are inconsistent with each other or with the network structure of the given BN; and 2) D-IPFP which reduces the computational cost by decomposing a global E-IPFP into a set of smaller local E-IPFP problems.

This paper presents a formal convergence proof for EIPFP, an algorithm that integrates low dimensional probabilistic constraints into a Bayesian network (BN) based on the mathematical procedure IPFP. It also extends E-IPFP to deal with constraints that are inconsistent with each other or with the BN structure.

This paper investigates the problem of belief update in Bayesian networks (BN) with uncertain evidence. Two types of uncertain evidences are identified: virtual evidence (reflecting the uncertainty one has about a reported observation) and soft evidence (reflecting the uncertainty of an event one observes). Each of the two types of evidence has its own characteristics and obeys a belief update rule that is different from hard evidence, and different from each other. The particular emphasis is on belief update with multiple uncertain evidences. Efficient algorithms for BN reasoning with consistent and inconsistent uncertain evidences are developed, and their convergences analyzed. These algorithms can be seen as combining the techniques of traditional BN reasoning, Pearl's virtual evidence method, Jeffrey's rule, and the iterative proportional fitting procedure.

This paper presents an efficient method, SMOOTH, for modifying a joint probability distribution to satisfy a set of inconsistent constraints. It extends the well-known "iterative proportional fitting procedure" (IPFP), which only works with consistent constraints. Comparing with existing methods, SMOOTH is computationally more efficient and insensitive to data. Moreover, SMOOTH can be easily integrated with Bayesian networks for Bayes reasoning with inconsistent constraints.

This paper reports our investigation on the problem of belief update in Bayesian networks (BN) using uncertain evidence. We focus on two types of uncertain evidences, virtual evidence (represented as likelihood ratios) and soft evidence (represented as probability distributions). We review three existing belief update methods with uncertain evidences: virtual evidence method, Jeffrey's rule, and IPFP (iterative proportional fitting procedure), and analyze the relations between these methods. This in-depth understanding leads us to propose two algorithms for belief update with multiple soft evidences. Both of these algorithms can be seen as integrating the techniques of virtual evidence method, IPFP and traditional BN evidential inference, and they have clear computational and practical advantages over the methods proposed by others in the past

Swoogle is a crawler-based indexing and retrieval system for the Semantic Web. It extracts metadata for each discovered document, and computes relations between documents. Discovered documents are also indexed by an information retrieval system which can use either character N-Gram or URIrefs as keywords to find relevant documents and to compute the similarity among a set of documents. One of the interesting properties we compute is ontology rank, a measure of the importance of a Semantic Web document.

The ultimate goal of service matching is to find the service provider(s) that would perform tasks of given description with the best overall degree of satisfaction. However, service description matching solves only part of the problem. Agents that match a given request may vary greatly in their actual capabilities to perform the tasks, and an agent may have strong and weak areas. In this work, we take a quantitative approach in which performance rating is considered an integral part of an agent's capability model and service distribution is taken into account in determining the degree of match. With the dynamic refinement of the agent capability model, the broker captures an agent's performance levels as well as its strong and weak areas. An experimental system has been designed and implemented within the OWL/OWL-S framework and the results statistics show significant advantage over other major levels of brokers.

Bayesian belief networks (BBN) are a widely studied graphical model for representing uncertainty and probabilistic interdependence among variables. One of the factors that restricts the model's wide acceptance in practical applications is that the general inference with BBN is NP-hard. This is also true for the maximum a posteriori probability (MAP) problem, which is to find the most probable joint value assignment to all uninstantiated variables, given instantiation of some variables in a BBN. To circumvent the difficulty caused by MAP's computational complexity, we suggest in this paper a neural network approximation approach. With this approach, a BBN is treated as a neural network without any change or transformation of the network structure, and the node activation functions are derived based on an energy function defined over a given BBN. Three methods are developed. They are the hill-climbing style discrete method, the simulated annealing method, and the continuous method based on the mean field theory. All three methods are for BBN of general structures, with the restriction that nodes of BBN are binary variables. In addition, rules for applying these methods to noisy-or networks are also developed, which may lead to more efficient computation in some cases. These methods' convergence is analyzed, and their validity tested through a series of computer experiments with two BBN of moderate size and complexity. Although additional theoretical and empirical work is needed, the analysis and experiments suggest that this approach may lead to effective and accurate approximation for MAP problems.

The maximum a posteriori probability (MAP) problem is to find the most probable instantiation of all uninstantiated variables, given an instantiation of a set of variables in a Bayesian belief network (BBN). MAP is known to be NP-hard. To circumvent the high computational complexity, we propose a neural network approach based on the mean field theory to approximate the MAP problem. In this approach, a given BBN is treated as a neural network with an energy function defined in such a way that the MAP solution corresponds to the global minimum energy state. The mean field equation is then derived. We also propose a method called resettling to further improve the solution accuracy. A series of computer experiment shows that this approach may lead to effective and accurate solutions to MAP problems.

We suggest a neural network approach to probabilistic inference in Bayesian belief networks (BBN). This is demonstrated by solving maximum a posteriori probability (MAP) problems, which are known to be NP-hard. In this approach, a belief network is treated as a neural network without any structural changes, and the node activation functions are derived based on the probabilistic calculus of the BBN. Three models are proposed and their convergence analyzed. Computer experiments with two non-trivial example BBN show that this approach may lead to effective approximation methods for MAP.

In this paper, we report our investigation on the behavior of some Bayesian belief network learning methods, when applied to realistic case data that do not accurately reflect the probability distribution of the underlying domain. The investigation focuses on two types of such data of practical interest, namely data excluding null or normal cases and data contaminated by noise, and their effects on the performance of two learning methods: the K2 algorithm, a representative method of Bayesian approach, and the extended Hebbian learning (EHL) method, representing those of neural network approach. Both analytical and experimental results show that when null cases are removed from the case database, the EHL method is able to accurately construct the underlying causal structure; whereas K2 algorithm may fail to properly handle root nodes. On the other hand, the EHL method is noise sensitive while K2 has certain inherent noise-resistance capability. Suggestions to extend both K2 and EHL to better cope with these problems are also made, and their effectiveness tested through a systematic computer simulation experiment. ©1999 John Wiley & Sons, Inc.

There have been various attempts to simulate the self-assembly process of lipid aggregates by computers. However, due to the computationally complex nature of the problem, previous simulations were often conducted with unrealistic simplifications of the molecules' morphology, intermolecular interactions, and the environment in which the lipid molecules interact. In this article, we present a new computational model in which each lipid is simulated by a more realistic amphiphilic particle consisting of a hydrophilic head and a long hydrophobic tail. The intermolecular interactions are approximated by a set of simple forces reflecting physical and chemical properties of lipids, for example, hydrophobicity and electrostatic forces, which are believed to be crucial for the formation of various aggregates. With a set of carefully selected parameters, this model is able to simulate successfully the formation of micelles in an aqueous environment and reversed micelle structures in an oil solvent from an initially randomly distributed set of lipidlike particles. This model can be used to study, at the microscopic level, the self-assembly of different protocell structures in the evolutionary process and the impact of environmental conditions on the formation of these structures. It may be further generalized to simulate the formation of other, more complex structures of amphiphilic molecules such as monolayer and bilayer aggregates.

This article presents a learning method for a special class of belief networks known as noisy-or networks. By extending the Hebbian rule of neural networks, two learning rules are developed to learn the probabilities of nodes and the internode causal strengths, respectively. The latter rule also learns structures of networks because a nonzero causal strength indicates the existence of a causal link. One distinct feature of this method is its ability to work in a sequential or incremental manner in which a network adjusts its parameters upon the arrival of every case description. As a result, this method is capable of not only constructing a causal knowledge base from a fixed set of case data but also dynamically adapting an existing knowledge base to a changing environment. To prove the convergence of learning, a Liapunov function is identified for the dynamic system defined by the learning rules. Computer experiment results show that this method is significantly faster than some existing learning methods for belief networks. © 1996 John Wiley & Sons, Inc.

This paper describes a connectionist approach to solving computationally difficult minimum vertex covering problems. This approach uses the graph representing the vertex covering problem as the connectionist network without any modifications (nodes of the connectionist network represent vertices and links represent edges of the given graph). The activation rule governing node behavior is derived by breaking down the global constraints on a solution into local constraints on individual nodes. The resulting model uses a competitive activation mechanism to carry out the computation where vertices compete not by explicit inhibitory links but through common resources (edges). Convergence and other properties of this model are formally established by introducing a monotonically non-increasing global energy function. Simulation results show that this model yields very high accuracy, significantly outperforming a well-known sequential approximation algorithm.

A competition-based connectionist model for solving diagnostic problems is described. The problems under consideration are computationally difficult in that multiple disorders may occur simultaneously and a global optimum in the space exponential to the total number of possible disorders is sought as a solution. To solve this problem, global optimization criteria are decomposed into local optimization criteria that are used to govern node activation updating in the connectionist model. Nodes representing disorders compete with each other to account for each 'individual' present manifestation, yet complement each other to account for all present manifestation, yet complement each other to account for all present manifestations through parallel node interactions. When equilibrium is reached, the network settles into a locally optimal state in which some disorder nodes (winners) are fully activated and compose the diagnosis for the given case, while all other disorder nodes are fully deactivated. A resettling process is proposed to improve accuracy. Three randomly generated examples of diagnostic problems, each of which has 1024 cases, were tested.>

Parsimonious covering theory is a formal model of abductive diagnostic problem-solving, Diagnostic knowledge is represented as a network of causal associations, and inferences are made using a sequential hypothesize-and-test procedure. In this article, we extend the methods available in parsimonious covering theory and related models to handle causal chaining (situations where multistep causal associations exist between disorders and manifestations). These extensions make parsimonious covering theory capable of representing a much broader range of real-world diagnostic problems.

The issue of how to effectively integrate and use symbolic causal knowledge with numeric estimates of probabilities in abductive diagnostic expert systems is examined. In particular, a formal probabilistic causal model that integrates Bayesian classification with a domain-independent artificial intelligence model of diagnostic problem solving (parsimonious covering theory) is developed. Through a careful analysis, it is shown that the causal relationships in a general diagnostic domain can be used to remove the barriers to applying Bayesian classification effectively (large number of probabilities required as part of the knowledge base, certain unrealistic independence assumptions, the explosion of diagnostic hypotheses that occurs when multiple disorders can occur simultaneously, etc.). Further, this analysis provides insight into which notions of "parsimony" may be relevant in a given application area. In a companion paper, Part Two, a computationally efficient diagnostic strategy based on the probabilistic causal model discussed in this paper is developed.

An important issue in diagnostic problem solving is how to generate and rank plausible hypotheses for a given set of manifestations. Since the space of possible hypotheses can be astronomically large if multiple disorders can be present simultaneously, some means is required to focus an expert system's attention on those hypotheses most likely to be valid. A domain-independent algorithm is presented that uses symbolic causal knowledge and numeric probabilistic knowledge to generate and evaluate plausible hypotheses during diagnostic problem solving. Given a set of manifestations known to be present, the algorithm uses a merit function for partially completed competing hypotheses to guide itself to the provably most probable hypothesis or hypotheses.