Bayesian Network Classifiers (BNCs) graphically model the probabilistic relationships among variables in the form of a directed acyclic graph (DAG). The log-likelihood function is commonly applied to evaluate the fit of DAG to data, and the significance of dependencies in DAG is measured by information metrics rather than probability metrics. However, the fit to data and the significance of dependencies may not hold from the perspective of probability theory. To address this issue, in this paper we propose to apply probability-likelihood function to provide a comprehensive and fundamental description of the mapping relationship between BNCs and data. Based on this, a heuristic search technique is introduced to construct a maximum probability-likelihood spanning tree by allowing for in-depth Bayesian inference of the probabilistic (in)dependence between attributes. Furthermore, instance-based learning is applied to enhance the interpretability and advance the classification capabilities of the learned BNCs in various domains and applications. Case analysis in the field of water quality monitoring shows that the proposed Average Weighted Bayesian Classifier (AWBC) can not only achieve accurate classification of water quality grades, but also fully capture and express the complex inherent dependence relationships among various water quality variables. Experimental evaluations conducted on 30 benchmark datasets from the University of California, Irvine (UCI) Machine Learning Repository demonstrate that, the final BNCs deliver superior or comparable classification performance compared to other BNC learners.

How to train learners over unbalanced data with asymmetric costs has been recognized as one of the most significant challenges in data mining. Bayesian network classifier (BNC) provides a powerful probabilistic tool to encode the probabilistic dependencies among random variables in directed acyclic graph (DAG), whereas unbalanced data will result in unbalanced network topology. This will lead to a biased estimate of the conditional or joint probability distribution, and finally a reduction in the classification accuracy. To address this issue, we propose to redefine the information-theoretic metrics to uniformly represent the balanced dependencies between attributes or that between attribute values. Then heuristic search strategy and thresholding operation are introduced to respectively learn refined DAGs from labeled and unlabeled data. The experimental results on 32 benchmark datasets reveal that the proposed highly scalable algorithm is competitive with or superior to a number of state-of-the-art single and ensemble learners.

Bayesian network classifiers (BNCs) provide a sound formalism for representing probabilistic knowledge and reasoning with uncertainty. Explicit independence assumptions can effectively and efficiently reduce the size of the search space for solving the NP-complete problem of structure learning. Strong conditional dependencies, when added to the network topology of BNC, can relax the independence assumptions, whereas the weak ones may result in biased estimates of conditional probability and degradation in generalization performance. In this paper, we propose an extension to the k-dependence Bayesian classifier (KDB) that achieves the bias/variance trade-off by verifying the rationality of implicit independence assumptions implicated. The informational and probabilistic dependency relationships represented in the learned robust topologies will be more appropriate for fitting labeled and unlabeled data, respectively. The comprehensive experimental results on 40 UCI datasets show that our proposed algorithm achieves competitive classification performance when compared to state-of-the-art BNC learners and their efficient variants in terms of zero-one loss, root mean square error (RMSE), bias and variance.

Attribute selection has been proved to be an effective trick to strengthen the classification capability of Bayesian network classifiers, such as Averaged n-Dependence Estimators (AnDE). However, conventional mutual information-based attribute ranking considers only the correlation between the attribute and the class, regardless of the redundancies among the attributes. In this paper, we propose a new ranking approach, called Maximin Conditional Mutual Information (MMCMI), which first minimises the conditional mutual information for any unsorted attribute with regard to the sorted attribute sequence, and then maximise the minimal conditional mutual information within all unsorted attributes. When ranking the very first attribute, the mutual information with the class is maximised within all attributes. Extensive empirical results demonstrate that the MMCMI ranking approach together with attribute selection framework achieves significantly superior classification performance and less classification time with respect to regular AnDE and the mutual information counterparts.

The independence assumptions help Bayesian network classifier (BNC), e.g., Naive Bayes (NB), reduce structure complexity and perform surprisingly well in many real-world applications. Semi-naive Bayesian techniques seek to improve the classification performance by relaxing the attribute independence assumption. However, the study of dependence rather than independence has received more attention during the past decade and the validity of independence assumptions needs to be further explored. In this paper, a novel learning technique, called Adaptive Independence Thresholding (AIT), is proposed to automatically identify the informational independence and probabilistic independence. AIT can respectively tune the network topologies of BNC learned from training data and testing instance under the framework of target learning. Zero-one loss, bias, variance and conditional log likelihood are introduced to compare the classification performance in the experimental study. The extensive experimental results on a collection of 36 benchmark datasets from the UCI machine learning repository show that AIT is more effective than other learning techniques (such as structure extension, attribute weighting) and helps make the final BNCs achieve remarkable classification improvements.

Bayesian network classifiers (BNCs) have proved their effectiveness and efficiency in the supervised learning framework. Numerous variations of conditional independence assumption have been proposed to address the issue of NP-hard structure learning of BNC. However, researchers focus on identifying conditional dependence rather than conditional independence, and information-theoretic criteria cannot identify the diversity in conditional (in)dependencies for different instances. In this paper, the maximum correlation criterion and minimum dependence criterion are introduced to sort attributes and identify conditional independencies, respectively. The heuristic search strategy is applied to find possible global solution for achieving the trade-off between significant dependency relationships and independence assumption. Our extensive experimental evaluation on widely used benchmark data sets reveals that the proposed algorithm achieves competitive classification performance compared to state-of-the-art single model learners (e.g., TAN, KDB, KNN and SVM) and ensemble learners (e.g., ATAN and AODE).

Of numerous proposals to refine naive Bayes by weakening its attribute independence assumption, averaged one-dependence estimators (AODE) has been shown to be able to achieve significantly higher classification accuracy at a moderate cost in classification efficiency. However, all one-dependence estimators (ODEs) in AODE have the same weights and are treated equally. To address this issue, model weighting, which assigns discriminate weights to ODEs and then linearly combine their probability estimates, has been proved to be an efficient and effective approach. Most information-theoretic weighting metrics, including mutual information, Kullback-Leibler measure and the information gain, place more emphasis on the correlation between root attribute (value) and class variable. We argue that the topology of each ODE can be divided into a set of local directed acyclic graphs (DAGs) based on the independence assumption, and multivariate mutual information is introduced to measure the extent to which the DAGs fit data. Based on this premise, in this study we propose a novel weighted AODE algorithm, called AWODE, that adaptively selects weights to alleviate the independence assumption and make the learned probability distribution fit the instance. The proposed approach is validated on 40 benchmark datasets from UCI machine learning repository. The experimental results reveal that, AWODE achieves bias-variance trade-off and is a competitive alternative to single-model Bayesian learners (such as TAN and KDB) and other weighted AODEs (such as WAODE).

Bagging has attracted much attention due to its simple implementation and the popularity of bootstrapping. By learning diverse classifiers from resampled datasets and averaging the outcomes, bagging investigates the possibility of achieving substantial classification performance of the base classifier. Diversity has been recognized as a very important characteristic in bagging. This paper presents an efficient and effective bagging approach, that learns a set of independent Bayesian network classifiers (BNCs) from disjoint data subspaces. The number of bits needed to describe the data is measured in terms of log likelihood, and redundant edges are identified to optimize the topologies of the learned BNCs. Our extensive experimental evaluation on 54 publicly available datasets from the UCI machine learning repository reveals that the proposed algorithm achieves a competitive classification performance compared with state-of-the-art BNCs that use or do not use bagging procedures, such as tree-augmented naive Bayes (TAN), k-dependence Bayesian classifier (KDB), bagging NB or bagging TAN.

Bayesian network classifiers (BNCs) are powerful tools to mine statistical knowledge from data and infer under conditions of uncertainty. However, most of the traditional BNCs focus on mining the dependency relationships existed in labeled data while neglecting the information hidden in unlabeled d ata, which may result in the biased decision boundaries. To address this issue, we introduce a new order-based greedy search heuristic based on mutual information for building efficient structures in tree-augmented naive Bayes (TAN), which is a highly accurate learner while maintaining simplicity and efficiency. Target learning is used to dynamically describe the dependency relationships in each unlabeled test instance. Extensive experimental results on UCI (University of California at Irvine) machine learning repository demonstrate that our proposed algorithm is a competitive alternative to state-of-the-art classifiers like weighted averaged TAN and k-dependence Bayesian classifier, as well as Random forest.

More and more applications have come with large data sets in the past decade. However, existing algorithms cannot guarantee to scale well on large data. Averaged n-Dependence Estimators (AnDE) allows for flexible learning from out-of-core data, by varying the value of n (number of super parents). Hence, AnDE is especially appropriate for large data learning. In this paper, we propose a sample-based attribute selection technique for AnDE. It needs one more pass through the training data, in which a multitude of approximate AnDE models are built and efficiently assessed by leave-one-out cross validation. The use of a sample reduces the training time. Experiments on 15 large data sets demonstrate that the proposed technique significantly reduces AnDE's error at the cost of a modest increase in training time. This efficient and scalable out-of-core approach delivers superior or comparable performance to typical in-core Bayesian network classifiers.