Automated log analysis is crucial to ensure the high availability and reliability of complex systems. The advent of Large Language Models (LLMs) in Natural Language Processing (NLP) has ushered in a new era of language model-driven automated log analysis, garnering significant interest. Within this field, two primary paradigms based on language models for log analysis have become prominent. Small Language Models (SLMs) (such as BERT) follow the pre-train and fine-tune paradigm, focusing on the specific log analysis task through fine-tuning on supervised datasets. On the other hand, LLMs (such as ChatGPT) following the in-context learning paradigm, analyze logs by providing a few examples in prompt contexts without updating parameters. Despite their respective strengths, both models exhibit inherent limitations. By comparing SLMs and LLMs, we notice that SLMs are more cost-effective but less powerful, whereas LLMs with large parameters are highly powerful but expensive and inefficient. To tradeoff between the performance and inference costs of both models in automated log analysis, this article introduces an adaptive log analysis framework known as AdaptiveLog, which effectively reduces the costs associated with LLM while ensuring superior results. This framework collaborates an LLM and an SLM, strategically allocating the LLM to tackle complex logs while delegating simpler logs to the SLM. Specifically, to efficiently query the LLM, we propose an adaptive selection strategy based on the uncertainty estimation of the SLM, where the LLM is invoked only when the SLM is uncertain. In addition, to enhance the reasoning ability of the LLM in log analysis tasks, we propose a novel prompt strategy by retrieving similar error-prone cases as the reference, enabling the model to leverage past error experiences and learn solutions from these cases. We evaluate AdaptiveLog on different log analysis tasks, Extensive experiments demonstrate that AdaptiveLog achieves state-of-the-art results across different tasks, elevating the overall accuracy of log analysis while maintaining cost efficiency. Our source code and detailed experimental data are available at https://github.com/LeaperOvO/AdaptiveLog-review .

Automated log analysis plays a vital role in software operations, with deep learning methods demonstrating effectiveness for analyzing logs from individual systems. However, existing methods face limitations in efficiency, adaptability, and knowledge preservation in system-incremental log analysis. Continual learning offers a solution by expanding the model’s ability to analyze logs from the increasing number of systems. For evaluating these methods in system-incremental log analysis, we introduce LogSI, a novel benchmark with four essential abilities for system-incremental log analysis. We perform a comprehensive evaluation of various baselines on LogSI, examining their robustness against different system permutations. Additionally, we conduct an in-depth study on the factors that influence their robustness. The datasets and source code of this paper can be found in https://github.com/nonauthor/LogSIbenchmark.

System-incremental log analysis, involves the ongoing training of a model using logs from diverse systems to enable effective resolution of log analysis tasks across an expanding array of systems. Existing continual learning methods, which are based on prompt tuning, have shown challenges in insufficient knowledge transfer and increasing catastrophic forgetting. To tackle these challenges, we present LogHPT, a novel continual learning method based on a hierarchical prompt tuning frame-work specifically tailored for system-incremental log analysis. LogHPT incorporates four types of prompt meticulously crafted to capture log knowledge across various granularities, thereby enhancing knowledge transfer. Subsequently, we employ a key-value mechanism to discern the most suitable prompts for the input logs. Additionally, we use general prompt learning based on knowledge distillation to mitigate catastrophic forgetting. To evaluate the performance of LogHPT, we conduct comprehensive experiments focusing on two fundamental subtasks: log parsing and log anomaly detection. The results show that LogHPT achieves state-of-the-art (SOTA) performance. The source code and datasets for this paper are accessible at the following link: https://github.com/nonauthor/LogHPT.

Logs play a critical role in providing essential information for system monitoring and troubleshooting. Recently, with the success of pre-trained language models (PLMs) and large language models (LLMs) in natural language processing (NLP), smaller PLMs (such as BERT) and LLMs (like GPT-4) have become the current mainstream approaches for log analysis. Despite the remarkable capabilities of LLMs, their higher cost and inefficient inference present significant challenges in leveraging the full potential of LLMs to analyze logs. In contrast, smaller PLMs can be fine-tuned for specific tasks even with limited computational resources, making them more practical. However, these smaller PLMs face challenges in understanding logs comprehensively due to their limited expert knowledge. To address the lack of expert knowledge and enhance log understanding for smaller PLMs, this paper introduces a novel and practical knowledge enhancement framework, called LUK, which acquires expert knowledge from LLMs automatically and then enhances the smaller PLM for log analysis with the expert knowledge. LUK can take full advantage of both types of models. Specifically, we design a multi-expert collaboration framework based on LLMs with different roles to acquire expert knowledge. In addition, we propose two novel pre-training tasks to enhance the log pre-training with expert knowledge. LUK achieves state-of-the-art results on different log analysis tasks, and extensive experiments demonstrate that expert knowledge from LLMs can be utilized more effectively to understand logs. Our source code and detailed experimental data are available athttps://github.com/LeaperOvO/LUK.

Logs as semi-structured text are rich in semantic information, making their comprehensive understanding crucial for automated log analysis. With the recent success of pre-trained language models in natural language processing, many studies have leveraged these models to understand logs. Despite their successes, existing pre-trained language models still suffer from three weaknesses. Firstly, these models fail to understand domain-specific terminology, especially abbreviations. Secondly, these models struggle to adequately capture the complete log context information. Thirdly, these models have difficulty in obtaining universal representations of different styles of the same logs. To address these challenges, we introduce KnowLog, a knowledge-enhanced pre-trained language model for log understanding. Specifically, to solve the previous two challenges, we exploit abbreviations and natural language descriptions of logs from public documentation as local and global knowledge, respectively, and leverage this knowledge by designing novel pre-training tasks for enhancing the model. To solve the last challenge, we design a contrastive learning-based pre-training task to obtain universal representations. We evaluate KnowLog by fine-tuning it on six different log understanding tasks. Extensive experiments demonstrate that KnowLog significantly enhances log understanding and achieves state-of-the-art results compared to existing pre-trained language models without knowledge enhancement. Moreover, we conduct additional experiments in transfer learning and low-resource scenarios, showcasing the substantial advantages of KnowLog. Our source code and detailed experimental data are available at https://github.com/LeaperOvO/KnowLog.

Large language models (LLMs) are becoming attractive as few-shot reasoners to solve Natural Language (NL)-related tasks. However, there is still much to learn about how well LLMs understand structured data, such as tables. Although tables can be used as input to LLMs with serialization, there is a lack of comprehensive studies that examine whether LLMs can truly comprehend such data. In this paper, we try to understand this by designing a benchmark to evaluate the structural understanding capabilities (SUC) of LLMs. The benchmark we create includes seven tasks, each with its own unique challenges, \eg, cell lookup, row retrieval, and size detection. We perform a series of evaluations on GPT-3.5 and GPT-4. We find that performance varied depending on several input choices, including table input format, content order, role prompting, and partition marks. Drawing from the insights gained through the benchmark evaluations, we proposeself-augmentation for effective structural prompting, such as critical value / range identification using internal knowledge of LLMs. When combined with carefully chosen input choices, these structural prompting methods lead to promising improvements in LLM performance on a variety of tabular tasks, \eg, TabFact(\uparrow2.31%), HybridQA(\uparrow2.13%), SQA(\uparrow2.72%), Feverous(\uparrow0.84%), and ToTTo(\uparrow5.68%). We believe that our open-source (please find code and data at https://github.com/microsoft/TableProvider) benchmark and proposed prompting methods can serve as a simple yet generic selection for future research.

The cold-start recommendation has been one of the most central problems in online platforms where new users or items arrive continuously. Although existing meta-learning based models with globally sharing knowledge show good performance in most cold-start scenarios, the ability to handle challenges on intention heterogeneity and prediction uncertainty is missing, and these two challenges are particularly evident in cold-start scenarios with fewer interaction data. To tackle the above challenges, in this paper, we present an uncertainty-aware Stochastic Meta Process with Doubly Intention learning (DISMP) for the cold-start recommendation, which has promising properties in uncertainty quantification. With the aid of the meta-learning stochastic process, DISMP can store general knowledge by capturing the relevance of different user-item pairs in terms of intentions and concepts, which is capable of rapid adaptation to new users and items. Furthermore, intentions with general and specific levels are extracted by doubly distinguishing the role of latent variables, which is able to capture the dependencies across different types of intentions and concepts. Empirical results show that our approach can achieve substantial improvement over the state-of-the-art baselines on cold-start recommendations with different perspectives.

User behavior data in recommendation are driven by the complex interactions of many intentions behind the user's decision making process. However, user behavior data tends to be sparse because of the limited user response and the vase combinations of users and items, which result in unclear user intentions and suffer from cold-start problem. The intentions are highly compound, and may range from high-level ones that govern user's intrinsic interests and realize the underlying reasons behind the user's decision making processes, to low-level one that characterize a user's extrinsic preference when executing intention to specific items. In this paper, we propose an intention neural process model (INP) for user cold-start recommendation (i.e., user with very few historical interactions), a novel extension of the neural stochastic process family using a general meta learning strategy with intrinsic and extrinsic intention learning for robust user preference learning. By regarding the recommendation process for each user as a stochastic process, INP defines distributions over functions, is capable of rapid adaptation to new users. Our approach learns intrinsic intentions by inferring the high-level concepts associated with user interests or purposes, while capturing the target preference of a user by performing self-supervised intention matching between historical items and target items in a disentangled latent space. Extrinsic intentions are learned by simultaneously generating the point-wise implicit feedback data and creates the pair-wise ranking list by sufficient exploiting both interacted and non-interacted items for each user. Empirical results show that our approach can achieve substantial improvement over the state-of-the-art baselines on cold-start recommendation.

News video is an important way of news communication, and people can easily get news from all over the world through the Internet. However, it lacks in the organization of news video content about temporality, presentation and relevance and fails to express the correlation between news events. In this paper, we present an event correlation model of news videos to organize news content. News content is clustered through topics, and the relationship between events is illustrated through relationship mappings. To achieve this purpose, an XLNet-based language model is presented to extract news keywords and their relationships. The clustering algorithm is designed to obtain news event topic clustering and named entity clustering. At the same time, we build the relationship mappings in news events to visualize the correlation between news events better. The user study is also designed to investigate the performance of our method in news reading and understanding. Compared with peer methods, the news information organized by our method achieves a higher user satisfaction level.

In hearing-loss community, sign language is a primary tool to communicate with people while there is communication gap between hearing-loss people with normal hearing people. Continuous sign language recognition, which can bridge the communication gap, is a challenging task because of the weakly supervised ordered annotations where no frame-level label is provided. To overcome this problem, connectionist temporal classification (CTC) is the most widely used method. However, CTC learning could perform bad if extracted features are not good. For better feature extraction, this work presents the novel self-attention-based fully-inception (SAFI) networks for vision-based end-to-end continuous sign language recognition. Considering the length of sign words differs from each other, we introduce fully inception network with different receptive field to extract dynamic clip-level features. To further boost the performance, the fully inception network with an auxiliary classifier is trained with aggregation cross entropy (ACE) loss. Then the self-attention networks as global sequential feature extractor is used to model the clip-level features with CTC. The proposed model is optimized by jointly training with ACE on clip-level feature learning and CTC on global sequential feature learning in an end-to-end fashion. The best method in the baselines achieves 35.6% WER on validation set and 34.5% WER on test set. It employs a better decoding algorithm for pseudo label to do the EM-like optimization to fine tune CNN module. In contrast, our approach focuses on the better feature extraction for end-to-end learning. To alleviate the overfitting on the limited dataset, we employ temporal elastic deformation to triple the real-world dataset RWTH-PHOENIX-Weather 2014. Experimental results on the real-world dataset RWTH-PHOENIX-Weather 2014 demonstrate the effectiveness of our approach which achieves 31.7% WER on validation set and 31.3% WER on test set.

Class imbalance is a challenging problem in many real-world applications such as fraudulent transactions detection in finance and diagnosis of rare diseases in medicine, which has attracted more and more attention in the community of machine learning and data mining. The main issue is how to capture the fundamental characteristics of the imbalanced data distribution. In particular, whether the hidden pattern can be truly mined from minority class is still a largely unanswered question after all it contains limited instances. The existing methods provide only a partial understanding of this issue and result in the biased and inaccurate classifiers. To overcome this issue, we propose a novel imbalanced classification framework with two stages. The first stage aims to accurately determine the class distributions by a supervised class distribution learning method under the Wasserstein auto-encoder framework. The second stage makes use of the generative adversarial networks to simultaneously generate instances according to the learnt class distributions and mine the discriminative structure among classes to train the final classifier. This proposed framework focuses on Supervised Class Distribution Learning for Generative Adversarial Networks-based imbalanced classification (SCDL-GAN). By comparing with the state-of-the-art methods, the experimental results demonstrate that SCDL-GAN consistently benefits the imbalanced classification task in terms of several widely-used evaluation metrics on five benchmark datasets.