Discovering knowledge from social media becomes a trend in many domains such as tourism, where users' feedback and rating are the basis of recommendation systems. In this context, cluster analysis has been a major tool to disclose user groups by which the process of collaborative filtering can better determine a personalised suggestion. Matching this to the curse of big data is a challenge with previous studies either implementing conventional techniques on a distributed system or making use of data sampling. Specific to ensemble clustering, only a few aim to obtain both scalability and privacy preserving that are significant to handling social data. This paper presents a new bi-level framework of ensemble clustering in which an instance-segment based analysis is adopted to ensure data privacy and reduce the complexity of clustering the whole dataset. Unlike existing studies, instead of drawing a single clustering from each segment, multiple clusterings are selected to better represent instances therein. Based on published tourism datasets and different experimental settings, the new approach usually outperforms its baselines whilst being competitive to related methods found in the literature. Additional case studies on simulated big datasets and noisy variations are reported and discussed in addition to the analysis of algorithmic parameters.

Attack detection is one of the main features required in modern defence systems. Despite the ongoing research, it remains challenging for a typical mechanism like network-based intrusion detection system (NIDS) to catch up with evolving adversarial attacks. They specifically aim to confuse a machine-learning based predictor. Without the knowledge of adversarial patterns, the best approach is generalising signatures learned from a dataset of legitimate connections and known intrusions. This work focuses on analysing non-payload traffics so that the resulting techniques can be exploited to a range of network-based applications. It investigates a novel means to deal with the problem of imbalanced classes. An optimised undersampling method is introduced to select a subset of majority-class representatives initially created through an ensemble clustering procedure. A weighted combination of criteria representing distributions within and between classes is proposed as the objective function for a global optimisation using the artificial bee colony (ABC). This approach usually outperforms its baselines and other state-of-the-art undersampling models, with ABC being more effective using the global best strategy than a random selection of solutions or an iterative greedy search. The paper also details the parameter analysis offering a heuristic guide for potential taking up of the proposed techniques.

Malware has remained a consistent threat since its emergence, growing into a plethora of types and in large numbers. In recent years, numerous new malware variants have enabled the identification of new attack surfaces and vectors, and have become a major challenge to security experts, driving the enhancement and development of new malware analysis techniques to contain the contagion. One of the preliminary steps of malware analysis is to remove the abundance of counterfeit malware samples from the large collection of suspicious samples. This process assists in the management of man and machine resources effectively in the analysis of both unknown and likely malware samples. Hashing techniques are one of the fastest and efficient techniques for performing this preliminary analysis such as fuzzy hashing and import hashing. However, both hashing methods have their limitations and they may not be effective on their own, instead the combination of two distinctive methods may assist in improving the detection accuracy and overall performance of the analysis. This paper proposes a Fuzzy-Import hashing technique which is the combination of fuzzy hashing and import hashing to improve the detection accuracy and overall performance of malware analysis. This proposed Fuzzy-Import hashing offers several benefits which are demonstrated through the experimentation performed on the collected malware samples and compared against stand-alone techniques of fuzzy hashing and import hashing.

Cancer has been identified as the leading cause of death. It is predicted that around 20-26 million people will be diagnosed with cancer by 2020. With this alarming rate, there is an urgent need for a more effective methodology to understand, prevent and cure cancer. Micro array technology provides a useful basis of achieving this ultimate goal. In particular to cancer research, it has become almost routine to create gene expression profiles, which can discriminate patients into good and poor prognosis groups, and identify possible tumor subtypes. This classification or predictive model offers a useful tool for individualized treatment of disease. However, the accuracy of existing classifiers have been constrained by the curse of dimensionality typically observed in micro array data. In addition to gene selection, one may transform the original data to another variation, where only key gene components are included. Unlike conventional transformation-based techniques found in the literature, this paper presents a novel method that makes use of cluster ensembles, specifically the summarizing information matrix, as the transformed data for the following classification step. Among different state-of-the-art methods, the link-based cluster ensemble approach (LCE) provides a highly accurate clustering, and thus particularly employed here. The performance of this transformation model is evaluated on published micro array datasets and C4.5, in comparison with benchmark techniques. The findings suggest that the new model can improve the classification accuracy of original data and performs better than the other transformation methods investigated in the empirical study.

Cluster ensembles have been shown to be better than any standard clustering algorithm at improving accuracy. This meta-learning formalism helps users to overcome the dilemma of selecting an appropriate technique and the parameters for that technique, given a set of data. It has proven effective for many problem domains, especially microarray data analysis. Among different state-of-the-art methods, the link-based approach (LCE) recently introduced by [22], [23] provides a highly accurate clustering. This paper presents the improvement of LCE with a new link-based similarity measure being developed and engaged. Additional information that is already available in a network is included in the similarity assessment. As such, this refinement can increase the quality of the measures, hence the resulting cluster decision. The performance of this improved LCE is evaluated on synthetic and UCI benchmark datasets, in comparison with the original and several well-known cluster ensemble techniques. The findings suggest that the new model can improve the accuracy of LCE and performs better than the others investigated in the empirical study.

Cancer has been identified as the leading cause of death. It is predicted that around 20-26 million people will be diagnosed with cancer by 2020. As a result, there is an urgent need for a more effective methodology to prevent and cure cancer. Microarray technology provides a useful basis of achieving this ultimate goal. For cancer research, it has become almost routine to create gene expression profiles, which can discriminate patients into good and poor prognosis groups. This cluster analysis offers a useful basis for individualized treatment of disease. Cluster ensembles have been shown to be better than any standard clustering algorithm for such a task. This meta-learning formalism helps users to overcome the dilemma of selecting an appropriate technique and the parameters for that technique, given a set of data. Among different state-of-the-art methods, the link-based approach (LCE) provides a highly accurate clustering. This paper presents the improvement of LCE with a new link-based similarity measure being developed and engaged. Additional information that is already available in an information network is included in the similarity assessment. As such, this refinement can increase the quality of the measures, hence the resulting cluster decision. The performance of this improved LCE is evaluated on published microarray datasets, in comparison with the original LCE and several well-known cluster ensemble techniques. The findings suggest that the new model can improve the accuracy of LCE and performs better than the others investigated in the empirical study.

Although attempts have been made to solve the problem of clustering categorical data via cluster ensembles, with the results being competitive to conventional algorithms, it is observed that these techniques unfortunately generate a final data partition based on incomplete information. The underlying ensemble-information matrix presents only cluster-data point relations, with many entries being left unknown. The paper presents an analysis that suggests this problem degrades the quality of the clustering result, and it presents a new link-based approach, which improves the conventional matrix by discovering unknown entries through similarity between clusters in an ensemble. In particular, an efficient link-based algorithm is proposed for the underlying similarity assessment. Afterward, to obtain the final clustering result, a graph partitioning technique is applied to a weighted bipartite graph that is formulated from the refined matrix. Experimental results on multiple real data sets suggest that the proposed link-based method almost always outperforms both conventional clustering algorithms for categorical data and well-known cluster ensemble techniques.

Cluster ensembles have recently emerged as a powerful alternative to standard cluster analysis, aggregating several input data clusterings to generate a single output clustering, with improved robustness and stability. From the early work, these techniques held great promise; however, most of them generate the final solution based on incomplete information of a cluster ensemble. The underlying ensemble-information matrix reflects only cluster-data point relations, while those among clusters are generally overlooked. This paper presents a new link-based approach to improve the conventional matrix. It achieves this using the similarity between clusters that are estimated from a link network model of the ensemble. In particular, three new link-based algorithms are proposed for the underlying similarity assessment. The final clustering result is generated from the refined matrix using two different consensus functions of feature-based and graph-based partitioning. This approach is the first to address and explicitly employ the relationship between input partitions, which has not been emphasized by recent studies of matrix refinement. The effectiveness of the link-based approach is empirically demonstrated over 10 data sets (synthetic and real) and three benchmark evaluation measures. The results suggest the new approach is able to efficiently extract information embedded in the input clusterings, and regularly illustrate higher clustering quality in comparison to several state-of-the-art techniques.

The measure of data reliability has recently proven useful for a number of data analysis tasks. This paper extends the underlying metric to a new problem of soft subspace clustering. The concept of subspace clustering has been increasingly recognized as an effective alternative to conventional algorithms (which search for clusters without differentiating the significance of different data attributes). While a large number of crisp subspace approaches have been proposed, only a handful of soft counterparts are developed with the common goal of acquiring the optimal cluster-specific dimension weights. Most soft subspace clustering methods work based on the exploitation of k-means and greatly rely on the iteratively disclosed cluster centers for the determination of local weights. Unlike such wrapper techniques, this paper presents a filter approach which is efficient and generally applicable to different types of clustering. Systematical experimental evaluations have been carried out over a collection of published gene expression data sets. The results demonstrate that the reliability-based methods generally enhance their corresponding baseline models and outperform several well-known subspace clustering algorithms.

Clinical data has been employed as the major factor for traditional cancer prognosis. However, this classic approach may be ineffective for analyzing morphologically indistinguishable tumor subtypes. As such, the microarray technology emerges as the promising alternative. Despite a large number of microarray studies, the actual clinical application of gene expression data analysis remains limited due to the complexity of generated data and the noise level. Recently, the integrative cluster analysis of both clinical and gene expression data has shown to be an effective alternative to overcome the above-mentioned problems. This paper presents a novel method for using cluster ensembles that is accurate for analyzing heterogeneous biological data. It overcomes the problem of selecting an appropriate clustering algorithm or parameter setting of any potential candidate, especially with a new set of data. The evaluation on real biological and benchmark datasets suggests that the quality of the proposed model is higher than many state-of-the-art cluster ensemble techniques and standard clustering algorithms. Also, its performance is robust to the parameter perturbation, thus providing a reliable and useful means for data analysts and bioinformaticians. Online supplementary is available at http://users.aber.ac.uk/nii07/bibm2010.

MOTIVATION: It is far from trivial to select the most effective clustering method and its parameterization, for a particular set of gene expression data, because there are a very large number of possibilities. Although many researchers still prefer to use hierarchical clustering in one form or another, this is often sub-optimal. Cluster ensemble research solves this problem by automatically combining multiple data partitions from different clusterings to improve both the robustness and quality of the clustering result. However, many existing ensemble techniques use an association matrix to summarize sample-cluster co-occurrence statistics, and relations within an ensemble are encapsulated only at coarse level, while those existing among clusters are completely neglected. Discovering these missing associations may greatly extend the capability of the ensemble methodology for microarray data clustering. RESULTS: The link-based cluster ensemble (LCE) method, presented here, implements these ideas and demonstrates outstanding performance. Experiment results on real gene expression and synthetic datasets indicate that LCE: (i) usually outperforms the existing cluster ensemble algorithms in individual tests and, overall, is clearly class-leading; (ii) generates excellent, robust performance across different types of data, especially with the presence of noise and imbalanced data clusters; (iii) provides a high-level data matrix that is applicable to many numerical clustering techniques; and (iv) is computationally efficient for large datasets and gene clustering. AVAILABILITY: Online supplementary and implementation are available at: http://users.aber.ac.uk/nii07/bioinformatics2010. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.