The rapid advancement of AI has led to its integration into various areas, especially with Large Language Models (LLMs) significantly enhancing capabilities in Artificial Intelligence Generated Content (AIGC). However, the complexity of AI systems has also exposed their vulnerabilities, necessitating robust methods for Failure Analysis (FA) and Fault Injection (FI) to ensure resilience and reliability. Despite the importance of these techniques, there lacks a comprehensive review of FA and FI methodologies in AI systems. This study fills this gap by presenting a detailed survey of existing FA and FI approaches across six layers of AI systems. We systematically analyze 142 studies to answer three research questions including (1) what are the prevalent failures in AI systems, (2) what types of faults can current FI tools simulate, (3) what gaps exist between the simulated faults and real-world failures. Our findings reveal a taxonomy of AI system failures, assess the capabilities of existing FI tools, and highlight discrepancies between real-world and simulated failures. Moreover, this survey contributes to the field by providing a framework for fault diagnosis, evaluating the state-of-the-art in FI, and identifying areas for improvement in FI techniques to enhance the resilience of AI systems.

As large language models (LLMs) are widely applied across various domains, improving the quality of LLM inference services is essential. In this paper, we find that optimizing configuration parameters of LLM inference engines can significantly improve LLM inference performance in terms of latency and throughput. Therefore, we propose LLMConf, an automated performance tuning system that optimizes multiple LLM inference performance metrics by searching for the optimal configuration parameters of the LLM inference engine. We first introduce a knowledge-enhanced approach to identify the set of configuration parameters (LLMConfigs) that most significantly impact LLM performance from the adjustable parameters provided by the LLM inference engine. We then perform automated data collection to build functional relationships between LLMConfigs and each performance metric. Additionally, LLMConf employs a multi-objective optimization module to obtain optimal LLMConfigs for simultaneously optimizing multiple performance metrics. The experimental results show that LLMConf significantly outperforms existing methods. Compared to the default configuration parameters of the LLM inference engine, LLMConf achieves an average improvement of 20.1% across 7 key performance metrics. Moreover, experiments demonstrate that LLMConf has strong transferability across diverse datasets, varying concurrency levels and different LLM base models.

Consensus is critical for distributed databases as it ensures the consistency of states across nodes, reinforcing the robustness of the overall system. However, faults related to the consensus protocols such as Paxos can lead to serious issues in distributed databases. Such consensus issues impact the correctness and availability of these databases. Therefore, to automatically uncover consensus issues in distributed databases, we propose Conan, a framework designed with fuzzing-driven fault injection. Conan applies a state-guided fuzzing algorithm to effectively explore the fault search space. Moreover, Conan employs hybrid fault sequences that combines fine-grained message-level faults and coarse-grained system-level faults to enhance fault injection. We implement and evaluate Conan on 3 widely-used distributed databases, including etcd, rqlite and openGauss. Finally, Conan has successfully uncovered previously unknown consensus issues, some of which are not detected by existing approaches.