Traces and logs record the behaviors and interactions between microservices, making them indispensable for diagnosing system anomalies. However, the intricate and flexible architecture of microservices presents significant challenges for automated anomaly detection. Existing approaches often struggle to capture the internal dynamics and correlations among microservices, which limits their ability to comprehensively detect anomalies arising from synchronous or asynchronous calls. Therefore, we categorize three prevalent anomaly paradigms in microservice systems: intra-service, inter-service, and joint anomalies. To address these challenges, we propose a novel unsupervised learning framework for dual-view anomaly detection, named TraceHG. Specifically, it constructs the trace graph composed of logs, response times, and traces. Besides, TraceHG utilizes hypergraph transformation to establish two views, providing comprehensive information in microservices. Furthermore, we introduce a dual-view framework that leverages both a hypergraph-view and a graph-view encoders to explicitly learn associations among microservices. These two views, employing graph and HGNNs, capture representations of internal dynamics, invocations, and their associated contexts in microservices. By constructing two minimized hyperspheres and measuring the distances from their centers in the latent space, we identify anomalies based on the anomaly scores. Extensive experiments on public benchmark datasets demonstrate that the proposed framework outperforms several state-of-the-art baselines in microservice anomaly detection.

We consider the problem of real-time log anomaly detection for distributed system with deep neural networks by unsupervised learning. There are two challenges in this problem, including detection accuracy and analysis efficacy. To tackle these two challenges, we propose GLAD, a simple yet effective approach mining for anomalies in distributed systems. To ensure detection accuracy, we exploit the gradient features in a well-calibrated deep neural network and analyze anomalous pattern within log files. To improve the analysis efficacy, we further integrate one-class support vector machine (SVM) into anomalous analysis, which significantly reduces the cost of anomaly decision boundary delineation. This effective integration successfully solves both accuracy and efficacy in real-time log anomaly detection. Also, since anomalous analysis is based upon unsupervised learning, it significantly reduces the extra data labeling cost. We conduct a series of experiments to justify that GLAD has the best comprehensive performance balanced between accuracy and efficiency, which implies the advantage in tackling practical problems. The results also reveal that GLAD enables effective anomaly mining and consistently outperforms state-of-the-art methods on both recall and F1 scores.

Abstract. In this paper, we develop an efficient stochastic variance reduced gradient descent algorithm to solve the affine rank minimization problem consisting of finding a matrix of minimum rank from linear measurements. The proposed algorithm as a stochastic gradient descent strategy enjoys a more favorable complexity than that using full gradients. It also reduces the variance of the stochastic gradient at each iteration and accelerates the rate of convergence. We prove that the proposed algorithm converges linearly in expectation to the solution under a restricted isometry condition. Numerical experimental results demonstrate that the proposed algorithm has a clear advantageous balance of efficiency, adaptivity, and accuracy compared with other state-of-the-art algorithms.

Logs are valuable resources that record the health status of systems. Analyzing logs to uncover and investigate abnormal behaviors has become an essential approach of ensuring system security. However, some potential anomalies may be missed when performing log anomaly detection, and even a seemingly insignificant abnormal behavior can lead to a series of severe anomalies or continuous negative impact on the performance of systems. Therefore, the reliability and security of systems are facing significant challenges. To address this issue, in this study, we propose TransFlowLog, an Encoder-Decoder architecture-based approach for log anomaly detection. It utilizes the Transformer Encoder, a state-of-the-art sequence modeling technique, to comprehensively understand contextual relationships using self-attention mechanism. Moreover, we introduce the Interflow Decoder, which considers information exchange between channels in embedded log sequences. The Interflow Decoder enhances the features encoded by the Transformer Encoder in both sequence and channel dimensions, thereby capturing interdependence between different channels. Comparative experiments conducted on three real-world datasets demonstrate the effectiveness of the proposed method, as it achieves higher F1-score and reduces the number of false negatives.

Syslog-based anomaly detection is crucial for protecting the systems from malicious attacks or malfunctions. System logs are semi-structured text messages printed by logging statements to record the system’s run-time status, involving rich semantic information. However, the existing BERT-based log anomaly detection method is based on the log key sequence, does not consider the semantics of the log data, and discards the variable part, resulting in a high rate of missed detection. In this paper, we propose SemLog, a self-supervised framework for log anomaly detection based on BERT. By incorporating log semantics and variables and employing multi-feature fusion, we mitigate the independent assumption issue in the Masked Language Modeling model. The experimental results on three benchmarks show that SemLog achieves high performance compared with the state-of-the-art approaches for anomaly detection.

Deep neural networks (DNNs) have been widely employed to solve log anomaly detection and outperform a range of conventional methods. They have attained such striking success because they can usually explore and extract semantic information from a large volume of log data, which helps to infer complex log anomaly patterns more accurately. Despite its success in generalization accuracy, this data-driven approach can still suffer from a high vulnerability to adversarial attacks , which severely limits its practical use. To address this issue, several studies have proposed anomaly detectors to equip neural networks to improve their robustness. These anomaly detectors are built based on effective adversarial attack methods. Therefore, effective adversarial attack approaches are important for developing more efficient anomaly detectors, thereby improving neural network robustness. In this study, we propose two strong and effective black-box attackers, an attention-based and a gradient-based attacker, to defeat three target systems: MLP, AutoEncoder , and DeepLog. Our approach facilitates the generation of more effective adversarial examples with the help of the analysis of vulnerable logkeys. The proposed attention-based attacker leverages attention weights to achieve vulnerable logkeys and derive adversarial examples, which are implemented using our previously developed attention-based convolutional neural network model . The proposed gradient-based attacker calculates gradients based on potential vulnerable logkeys to seek an optimal adversarial sample. The experimental results showed that these two approaches significantly outperformed the state-of-the-art attacker model log anomaly mask (LAM). In particular, owing to its optimization, the proposed gradient-based attacker approach can significantly increase the misclassification rate on three target models, yields a 70% successful attack rate on DeepLog and greatly exceeds the baseline by 52%.

Abstract With versatility and complexity of computer systems, warning and errors are inevitable. To effectively monitor system’s status, system logs are critical. To detect anomalies in system logs, deep learning is a promising way to go. However, abnormal system logs in the real world are often difficult to collect, and effectively and accurately categorize the logs is an even time-consuming project. Thus, the data incompleteness is not conducive to the deep learning for this practical application. In this paper, we put forward a novel semi-supervised dual branch model that alleviate the need for large scale labeled logs for training a deep system log anomaly detector. Specifically, our model consists of two homogeneous networks that share the same parameters, one is called weak augmented teacher model and the other is termed as strong augmented student model. In the teacher model, the log features are augmented with small Gaussian noise, while in the student model, the strong augmentation is injected to force the model to learn a more robust feature representation with the guidance of teacher model provided soft labels. Furthermore, to further utilize unlabeled samples effectively, we propose a flexible label screening strategy that takes into account the confidence and stability of pseudo-labels. Experimental results show favorable effect of our model on prevalent HDFS and Hadoop Application datasets. Precisely, with only 30% training data labeled, our model can achieve the comparable results as the fully supervised version.

In the mathematical and engineering literature on signal processing and time-series analysis, there are two opposite points of view concerning the extraction of time-varying frequencies (commonly called instantaneous frequencies, IFs). One is to consider the given signal as a composite signal consisting of a finite number of subsignals that are oscillating, and the goal is to decompose the signal into the sum of the (unknown) subsignals, followed by extracting the IF from each subsignal; the other is first to extract from the given signal, the IFs of the (unknown) subsignals, from which the subsignals that constitute the given signal are recovered. Let us call the first the ``signal decomposition approach'' and the second the ``signal resolution approach.'' For the ``signal decomposition approach,'' rigorous mathematical theories on function decomposition have been well developed in the mathematical literature, with the most relevant one, called ``atomic decomposition'' initiated by R. Coifman, with various extensions by others, notably by D. Donoho, with the goal of extracting the signal building blocks, but without concern of which building blocks constitute any of the subsignals, and consequently, the subsignals along with their IFs cannot be recovered. On the other hand, the most popular of the decomposition approach is the ``empirical mode decomposition (EMD),'' proposed by N. Huang et al., with many variations by others. In contrast to atomic decomposition, all variations of EMD are ad hoc algorithms, without any rigorous mathematical theory. Unfortunately, all existing versions of EMD fail to resolve the inverse problem on the recovery of the subsignals that constitute the given composite signal, and consequently, extracting the IFs is not satisfactory. For example, EMD fails to extract even two IFs that are not far apart from each other. In contrast to the signal decomposition approach, the ``signal resolution approach'' has a very long history dated back to the Prony method, introduced by G. de Prony in 1795, for solving the inverse problem of time-invariant linear systems. On the other hand, for nonstationary signals, the synchrosqueezed wavelet transform (SST), proposed by I. Daubechies over a decade ago, with various extensions and variations by others, was introduced to resolving the inverse problem, by first extracting the IFs from some reference frequency, followed by recovering the subsignals. Unfortunately, the SST approximate IFs could not be separated when the target IFs are close to one another at certain time instants, and even if they could be separated, the approximation is usually not sufficiently accurate. For these reasons, some signal components could not be recovered, and those that could be recovered are usually inexact. More recently, we introduced and developed a more direct method, called signal separation operation (SSO), published in 2016, to accurately compute the IFs and to accurately recover all signal components even if some of the target IFs are close to each other. The main contributions of this article are twofold. First, the SSO method is extended from uniformly sampled data to arbitrarily sampled data. This method is localized as illustrated by a number of numerical examples, including components with different subsignal arrival and departure times. It also yields a short-term prediction of the digital components along with their IFs. Second, we present a novel theory-inspired implementation of our method as a deep neural network (DNN). We have proved that a major advantage of DNN over shallow networks is that DNN can take advantage of any inherent compositional structure in the target function, while shallow networks are necessarily blind to such structure. Therefore, DNN can avoid the so-called curse of dimensionality using what we have called the blessing of compositionality. However, the compositional structure of the target function is not uniquely defined, and the constituent functions are typically not known so that the networks still need to be trained end-to-end. In contrast, the DNN introduced in this article implements a mathematical procedure so that no training is required at all, and the compositional structure is evident from the procedure. We will disclose the extension of the SSO method in Sections II and III and explain the construction of the deep network in Section IV.

Sparse signal recoveries from multiple measurement vectors (MMV) with joint sparsity property have many applications in signal, image, and video processing. The problem becomes much more involved when snapshots of the signal matrix are temporally correlated. With signal's temporal correlation in mind, we provide a framework of iterative MMV algorithms based on thresholding, functional feedback and null space tuning. Convergence analysis for exact recovery is established. Unlike most of iterative greedy algorithms that select indices in a measurement/solution space, we determine indices based on an orthogonal subspace spanned by the iterative sequence. In addition, a functional feedback that controls the amount of energy relocation from the “tails” is implemented and analyzed. It is seen that the principle of functional feedback is capable to lower the number of iteration and speed up the convergence of the algorithm. Numerical experiments demonstrate that the proposed algorithm has a clearly advantageous balance of efficiency, adaptivity and accuracy compared with other state-of-the-art algorithms.

This paper addresses the problem of how routers efficiently select the optimal routes in large-scale networks. For the first time, it proposes Aggregation Tree Routing Model (ATRM) for dynamic topology changes. Aggregation Tree is first constructed in this model based on various parameter characteristics of links between routers, which greatly narrows the searching space of the routing procedure within much smaller domains. When network state changes, protocol calls the increment solution presented in this paper, updating the both two kinds of state changes in dynamic networks including link cost changes with time and topology structure changes with time. This solution is composed of three sub-algorithms-Link Deletion Algorithm, Link Addition Algorithm and Link Change Algorithm, and merely modifies affected information of links without re-routing in the whole network, as well as satisfying QoS constrains. The work achieves a logarithmic reduction in communication complexity and the simulation demonstrates that this model obtains high performance in routing accuracy as expected.