ABSTRACT Attribute grouping serves as one of the effective steps in high‐dimensional outlier detection. However, existing outlier detection methods merely capture the local coupling between pairs of attribute values within each attribute group, and overlook the global coupling among combinations of multiple attribute values, thereby limiting the effectiveness of outlier detection. In this paper, a novel attribute grouping‐based outlier detection method for categorical data is proposed by using dynamic dimensional embedding representations to characterize the global coupling among all variables. First, within each attribute group, we dynamically construct the embedding dimensions of all attributes to avoid redundant representations caused by static dimensions. Second, Multi‐Head Self‐Attention is used to construct the dynamic dimensional embedding of any data object, effectively characterizing the global coupling. Then, the reconstruction error derived from the dynamic dimensional embedding is employed to quantify the outlier degree of data objects. Based on this, we propose an attribute grouping‐based outlier detection method, in which dynamic dimensional embedding representations are used for each attribute group. In the end, experimental results on the UCI and synthetic datasets validate that the algorithm has good outlier detection performance. Importantly, compared with the competitive methods, the algorithm bolsters the AUC index and the detection efficiency by an average of 7.76% and 48.49%, respectively.

ABSTRACT The core challenge of fine‐grained image recognition (FGIR) tasks is distinguishing highly similar subclasses within the same base category. Most CNN‐based deep learning methods typically focus on extracting information from local regions while overlook the inherent structure between subclasses and the complex relationships between features. This paper presents a two‐stage feature selection method based on partial order analysis (POA) and heterogeneity evaluation (HE) for FGIR tasks, guiding the model to focus on distinctive features while reducing uncertainty caused by interfering information. Specifically, in the POA stage, clustering first groups similar subcategories into a medium‐granularity category. Formal concept analysis then models their hierarchical partial order, identifying “shared features” among subcategories and “exclusive features” unique to each. This structured representation highlights key contrastive cues. In the HE stage, a novel heterogeneity index is introduced to measure the fluctuation of low‐level features within each fine‐grained category. This index guides the model to suppress pseudo‐discriminative features with high heterogeneity, mitigating the impact of noisy and unstable information on decision‐making. We perform comprehensive experiments on three commonly used benchmark datasets (CUB‐200‐2011, Stanford Cars, and FGVC‐Aircraft). Experimental results show that the proposed method outperforms classic FGIC methods, validating the effectiveness of our approach.

Subspace clustering is an effective way to analyze high-dimensional data. The main problems of the conventional subspace clustering techniques are as follows: first, conventional clustering methods can not describe categorical attribute space in more detail; second, most subspace-clustering techniques failure to process dynamic data effectively; finally, lack of effective noise recognition leads to the decline of the efficiency of incremental subspace-clustering analysis. We address the above problems by an incremental subspace-clustering algorithm — called ICE. With attribute subspace constructed by a rough set-based weight computing method, ICE obtains clustering results through initial and incremental clustering stage. Utilizing the original cluster results generated from initial clustering stage, we adopt merging and splitting operation to dynamic adjust cluster-structure in incremental clustering stage. Before achieving the final results, a polymerization-based noise recognition technique is employed to automatically identify noise from sparse clusters without human threshold intervention. We implement ICE on synthetic and real-world datasets. The experimental results reveal that incremental subspace-clustering method can achieves satisfactory performance on extensibility, accuracy and robustness.

The similarity metric has garnered widespread attention thanks to its potential applications in the fields of data mining, machine learning, and so on. Due to the interference of “distance concentration” caused by “Curse of dimensionality,” however, existing similarity metrics are inadequate in high-dimensional data analysis. In this study, we propose two innovative similarity metrics—Chebyshev Coulomb force and Chebyshev Coulomb resultant force—anchored on Chebyshev p-norms. In the initial phase, we eliminate dependency relationships among attributes by applying a metric matrix—and the theoretical analysis reveals that the Chebyshev p-norms is capable of mitigating the effect of “distance concentration” among high-dimensional data objects. Next, we devise two similarity metrics—Chebyshev Coulomb force and Chebyshev Coulomb resultant force—by adopting the metric matrix and Chebyshev p-norms. Chebyshev Coulomb force and Chebyshev Coulomb resultant force, being effective in characterizing the similarity among data objects, quantify the deviation of data objects from their respective dataset centers. Additionally, the two metrics alleviate the interference of “distance concentration.” Importantly, the discrepancy of data objects in attribute dimensions is captured by Chebyshev Coulomb force vector, rendering the similarity metric interpretable. By utilizing the UCI dataset, the experimental validation demonstrates the superiority of our similarity metrics, confirming their efficacy in mitigating the interference of “distance concentration.” Compared with the existing similarity metric approaches, the AUC index of outlier detection shows an average improvement of 8.18%—and the ARI, NMI, and F_score indices of clustering are revamped by averages 6.56%, 6.87%, and 6.01%, respectively.

Stacking pseudoinverse learner (SP) is an ensemble learning technology, and its generalization performance greatly affects the effect of image classification. Currently, most SPs randomly initialize the input weight matrix in a random subspace without limiting the random initial values, resulting in unstable training results and a decrease in generalization performance; in addition, training all samples at once may cause the classifier redundant and also affect the generalization performance of the model. To efficiently address the above issues, we propose a new framework called progressive stacking pseudoinverse learner (PSP), which aims to enhance the generalization performance of SP via active learning (AL) in random subspaces. Specifically, on the one hand, a random feature SP (RFSP) model is proposed, which constrains the random subspace by initializing the input weight matrix into different random specific distributions to improve the generalization performance of SP. On the other hand, an AL progressive (ALP) model based on RFSP is proposed. By iteratively selecting useful samples to optimize the classification results, the training sample information is effectively used to progressively enhance the generalization performance of the model. Experimental results on three public datasets show that our proposed PSP algorithm achieves better performance in accuracy, precision, recall, and$F1$score, and the results are competitive with state-of-the-art methods.

Traditional outlier detections are inadequate for high-dimensional data analysis due to the interference of distance tending to be concentrated (“curse of dimensionality”). Inspired by the Coulomb’s law, we propose a new high-dimensional data similarity measure vector, which consists of outlier Coulomb force and outlier Coulomb resultant force. Outlier Coulomb force not only effectively gauges similarity measures among data objects, but also fully reflects differences among dimensions of data objects by vector projection in each dimension. More importantly, Coulomb resultant force can effectively measure deviations of data objects from a data center, making detection results interpretable. We introduce a new neighborhood outlier factor, which drives the development of a high-dimensional outlier detection algorithm. In our approach, attribute values with a high deviation degree is treated as interpretable information of outlier data. Finally, we implement and evaluate our algorithm using the UCI and synthetic datasets. Our experimental results show that the algorithm effectively alleviates the interference of “Curse of Dimensionality”. The findings confirm that high-dimensional outlier data originated by the algorithm are interpretable.

This paper develops a parallel computing system - MiCS - for mutual information of big categorical data on the Spark computing platform. The MiCS algorithm is conductive to processing a large amount and strong repeatability of mutual-information calculation among feature pairs by applying a column-wise transformation scheme. And to improve the efficiency of the MiCS and the utilization rate of Spark cluster resources, we adopt a virtual partitioning scheme to achieve balanced load while mitigating the data skewness problem in the Spark Shuffle process.

In this paper, a novel trajectory clustering algorithm - TAD - is proposed to extract trajectory Stays based on spatial-temporal density analysis of data. Two new metrics - NMAST (Neighbourhood Move Ability and Stay Time) density function and NT (Noise Tolerance) factor - are defined in this algorithm. Firstly, NMAST integrates the characteristics of Neighbourhood Move Ability (NMA, extended from the concept of Move Ability MA), Stay Time (ST), and evaluation factor Eμ to measure the spatial-temporal density of data. Secondly, NT utilizes the features of noise to dynamically evaluate and reduce the influence of noise. The experimental results on Geolife dataset shows that the distributions hidden in data are extracted more realistically, especially for various complex or special trajectories with long-duration gaps. Furthermore, our analytical method of trajectory data is particularly applied in the spectra of LAMOST survey to analyse the variation characteristics of sky-background. The results show a regular distribution on observational date which is relatively concentrated in the month of 1, 10, 11, 12 in each year. The laws discovered in this work would provide a reasonable support for the designation of observational plans, and the new trajectory analysis method would also provide the services for the astronomical data analysis and then for the further studies of formation and evolution of the universe.

We propose a weighted outlier mining method called WATCH to identify outliers in high-dimensional categorical datasets. WATCH is composed of two distinctive modules: 1) feature grouping by the virtue of correlation measurement among features and 2) outlier mining by assigning scores to objects in each feature groups. At the heart of WATCH is the feature grouping module, which groups an array of features into multiple groups to discover various aspects of feature patterns in each group. The outlier mining module detects outliers from high-dimensional categorical datasets. Except for the number of outliers specified by users, WATCH is conducive to bypassing the optimization of any user-given parameter. We implement and evaluate WATCH using synthetic and real-world datasets. Our experimental results show that WATCH is a promising and practical algorithm to detect outliers in high-dimensional categorical datasets, because WATCH achieves high performance in terms of precision, efficiency, and interpretability.

In this paper, we propose a scalable mining algorithm to discover contextual outliers using relevant subspaces. We develop the mining algorithm using the MapReduce programming model running on a Hadoop cluster. Relevant subspaces, which effectively capture the local distribution of various datasets, are quantified using local sparseness of attribute dimensions. We design a novel way of calculating local outlier factors in a relevant subspace with the probability density of local datasets; this new approach can effectively reflect the outlier degree of a data object that does not satisfy the distribution of the local dataset in the relevant subspace. Attribute dimensions of a relevant subspace, and local outlier factors are expressed as vital contextual information, which improves the interpretability of outliers. Importantly, the selection of N data objects with the largest local outlier factor value is categorized as contextual outliers in our solution. To this end, our scalable mining algorithm, which incorporates the locality sensitive hashing distributed strategy, is implemented on a Hadoop cluster. The experimental results validate the effectiveness, interpretability, scalability, and extensibility of the algorithm using both synthetic data and stellar spectral data as experimental datasets.

Parallel clustering is an important research area of big data analysis. The conventional Hierarchical Agglomerative Clustering (HAC) techniques are inadequate to handle big-scale categorical datasets due to two drawbacks. First, HAC consumes excessive CPU time and memory resources; and second, it is non-trivial to decompose clustering tasks into independent sub-tasks executed in parallel. We solve these two problems by a MapReduce-based hierarchical subspace-clustering algorithm - called PAPU - using LSH-based data partitioning. PAPU is conducive to partitioning a large-scale dataset into multiple independent sub-datasets, into which similar data objects are mapped. Advocating parallel computing, PAPU obtains sub-clusters corresponding to respective attribute subspaces from independent chunks in the local clustering phase. To improve the accuracy of approximated clustering results, PAPU measures various scale clusters by applying the hierarchical clustering scheme to iteratively merge sub-clusters during the global clustering phase. We implement PAPU on a 24-node Hadoop computing platform. The experimental results reveal that hierarchical subspace-clustering coupled with the data-partitioning strategy achieves high clustering efficiency on both synthetic and real-world large-scale datasets. The experiments also demonstrate that PAPU delivers superior performance in terms of extensibility and scalability (e.g., a nearly linear speedup).

In this study, we discover that the data skewness problem imposes adverse impacts on MapReduce-based parallel kNN-join operations running clusters. We propose a data partitioning approach-called kNN-DP-to alleviate load imbalance incurred by data skewness. The overarching goal of kNN-DP is to equally divide data objects into a large number of partitions, which are processed by mappers and reducers in parallel. At the heart of kNN-DP is a data partitioning module, which dynamically and judiciously partitions data to optimize kNN-join performance by suppressing data skewness on Hadoop clusters. Data partitioning decisions largely depends on data properties (e.g., distributions), the analysis of which is highly expensive for a massive amount of data. To speed up the data-property analysis, we incorporate a sampling technique to profile the data distribution of a small sample dataset representing big datasets. After building a data-partitioning cost model for parallel kNN-joins, we derive the time-complexity upper and lower bounds of parallel kNN-join algorithms. The cost model offers us a guidance to systematically investigate kNN-DP's performance. kNN-DP obtains global nearest neighbors using local nearest neighbors. To improve the accuracy of such an approximation solution, we augment each node's local data by a small amount of redundant data. We develop two kNN-DP-based schemes called LSH+ and z-value+, which seamlessly integrate kNN-DP with the existing LSH and z-value algorithms for kNN-join computing. We implement and evaluate LSH+ and z-value+ on a 24-node Hadoop cluster driven by both synthetic and real-world high-dimensional datasets. The experimental results show that kNN-DP significantly improves the performance of LSH and z-value while offering high extensibility and scalability on Hadoop clusters.

Traditional parallel algorithms for mining frequent itemsets aim to balance load by equally partitioning data among a group of computing nodes. We start this study by discovering a serious performance problem of the existing parallel Frequent Itemset Mining algorithms. Given a large dataset, data partitioning strategies in the existing solutions suffer high communication and mining overhead induced by redundant transactions transmitted among computing nodes. We address this problem by developing a data partitioning approach called FiDoop-DP using the MapReduce programming model. The overarching goal of FiDoop-DP is to boost the performance of parallel Frequent Itemset Mining on Hadoop clusters. At the heart of FiDoop-DP is the Voronoi diagram-based data partitioning technique, which exploits correlations among transactions. Incorporating the similarity metric and the Locality-Sensitive Hashing technique, FiDoop-DP places highly similar transactions into a data partition to improve locality without creating an excessive number of redundant transactions. We implement FiDoop-DP on a 24-node Hadoop cluster, driven by a wide range of datasets created by IBM Quest Market-Basket Synthetic Data Generator. Experimental results reveal that FiDoop-DP is conducive to reducing network and computing loads by the virtue of eliminating redundant transactions on Hadoop nodes. FiDoop-DP significantly improves the performance of the existing parallel frequent-pattern scheme by up to 31 percent with an average of 18 percent.

Existing distance-based outlier mining methods do not consider the impact of each attribute's importance degree, thereby resulting in poor mining accuracies. To address this problem, we propose a new outlier mining algorithm – Miner* – that makes use of information entropy and Weighted Distance Sum to substantially improve mining accuracies. Miner* employs information entropy to determine weight values indicating the importance degrees of data attributes. An input dataset is reduced by Miner* through the neighbour-radius-based pruning technologies. Thus, Miner* obtains a candidate outlier set by removing any data objects that are unlikely to be outliers. Miner* calculates the weighted distance sum value Wkof each object in the candidate outlier set; Wkvalue ranks the top n to be regarded as outliers. Due to the sum of distance, which takes full advantage of the clustering characteristics of the dataset, edge distribution data objects and local outliers can be effectively mined out. To demonstrate the effectiveness of the Miner* algorithm, we implement Miner* in a prototype system to detect star spectrum data objects with abnormal characteristic lines. Our experimental results show that the algorithm in Miner* achieves high accuracy, high scalability, and low man-made influence by utilizing UCI and star spectrum dataset. Our results also confirm that Miner* is feasible and effective in mining spectrum data with abnormal characteristic lines from massive star spectrum dataset.

In this paper, we present a thermal-aware file assignment technique called TIGER for reducing the cooling cost of storage clusters in data centers. We show that peak inlet temperatures of storage nodes depend on not only CPU utilization but also I/O activities, which rely on file assignments in a cluster. The TIGER scheme aims to lower peak inlet temperatures of storage clusters by dynamic thermal management through file placements. TIGER makes use of cross-interference coefficients to estimate the re-circulation of hot air from the outlets to the inlets of data nodes. TIGER first calculates the thresholds of disks in each data node based on its contribution to heat re-circulation in a data center. TIGER undertakes two steps to achieve high I/O performance while reducing cooling cost. First, TIGER assigns groups of files with similar service times to shorten I/O response times. Second, TIGER ensures that load imbalance does not exceed a specified threshold. We evaluate performance of TIGER in terms of both cooling energy conservation and response time of a storage cluster. Our results confirm that TIGER reduces cooling-power requirements for clusters by offering about 10 to 15 percent cooling-energy savings without significantly degrading I/O performance.

Existing parallel mining algorithms for frequent itemsets lack a mechanism that enables automatic parallelization, load balancing, data distribution, and fault tolerance on large clusters. As a solution to this problem, we design a parallel frequent itemsets mining algorithm called FiDoop using the MapReduce programming model. To achieve compressed storage and avoid building conditional pattern bases, FiDoop incorporates the frequent items ultrametric tree, rather than conventional FP trees. In FiDoop, three MapReduce jobs are implemented to complete the mining task. In the crucial third MapReduce job, the mappers independently decompose itemsets, the reducers perform combination operations by constructing small ultrametric trees, and the actual mining of these trees separately. We implement FiDoop on our in-house Hadoop cluster. We show that FiDoop on the cluster is sensitive to data distribution and dimensions, because itemsets with different lengths have different decomposition and construction costs. To improve FiDoop's performance, we develop a workload balance metric to measure load balance across the cluster's computing nodes. We develop FiDoop-HD, an extension of FiDoop, to speed up the mining performance for high-dimensional data analysis. Extensive experiments using real-world celestial spectral data demonstrate that our proposed solution is efficient and scalable.

The diagnosability of a system is defined as the maximum number of faulty processors that the system can guarantee to identify, which plays an important role in measuring of the reliability of multiprocessor systems. In the work of Peng et al. in 2012, they proposed a new measure for fault diagnosis of systems, namely,$g$-good-neighbor conditional diagnosability. It is defined as the diagnosability of a multiprocessor system under the assumption that every fault-free node contains at least$g$fault-free neighbors, which can measure the reliability of interconnection networks in heterogeneous environments more accurately than traditional diagnosability. The$k$-ary$n$-cube is a family of popular networks. In this study, we first investigate and determine the$R_g$-connectivity of$k$-ary$n$-cube for$0\le g\le n.$Based on this, we determine the$g$-good-neighbor conditional diagnosability of$k$-ary$n$-cube under the PMC model and MM* model for$k\ge 4, n\ge 3$and$0\le g\le n.$Our study shows the$g$-good-neighbor conditional diagnosability of$k$-ary$n$-cube is several times larger than the classical diagnosability of$k$-ary$n$-cube.

In this paper, we present thermal-aware file assignment technique called TIGER for reducing cooling cost of storage clusters in data centers. TIGER first calculates the thresholds of disks in each node based on its contribution to heat recirculation in a data center. Next, TIGER assigns files to data nodes according to calculated thresholds. We evaluated performance of TIGER in terms of both cooling energy conservation and response time of a storage cluster. Our results confirm that TIGER reduces cooling-power requirements for clusters by offering about 10 to 15 percent cooling energy savings without significantly degrading I/O performance.