Wear debris analysis(WDA) is a critical predictive maintenance technique that provides key data for diagnosing wear faults in mechanical equipment. This technique helps ensure long-term stable operation and fault prediction for machinery. However, recent approaches in WDA rely heavily on data-driven methods and often neglect the most important expert knowledge, such as the morphology, size, and color of debris. This knowledge is essential for fault diagnosis, but traditional networks struggle to represent and integrate it effectively, in contrast, large language models (LLMs) and vision-language models (VLMs), with their exceptional generalization capabilities, are rapidly evolving into ideal tools for representing and integrating expert knowledge. Inspired by this, we propose a novel multimodal learning method that combines VLMs and LLMs to integrate expert knowledge into image information, extending general models to specialized industrial domains. Specifically, we use an LLMs to regenerate textual descriptions from expert knowledge and refine CLIP’s Layer Normalization parameters to enhance the model’s transferability to specialized datasets, such as ferrography images, for zero-shot learning. Additionally, we introduce an adaptive meta-learning strategy that allows the model to quickly adapt to ferrography images with limited samples, further improving performance. Experimental results on industrial datasets show that the proposed method effectively addresses the challenge of applying VLMs in data-limited specialized domains, significantly improving the accuracy and robustness of WDA.

The task of retrieving audio content relevant to lyric queries and vice versa plays a critical role in music-oriented applications. In this process, robust feature representations have to be learned for two modalities. Furthermore, interactions between different modalities should be properly captured at a fine-grained level. Existing approaches can effectively extract modal representations and perform retrieving between different modalities through alignment. However, these approaches model interactions between audio and lyrics in a coarse-grained manner. Especially the input features and interactions between enhanced representations produced by the alignment module are largely ignored, resulting in low-quality modality representations for final retrieval. This paper presents a novel method named CMRF that accomplishes cross-modal interactions via a reinforcement feedback procedure to learn high-quality multi-modal embeddings. Initially, we implicitly assimilate representations across distinct modalities via directional pairwise cross-modal attention. Subsequently, our approach recurrently identifies pivotal constituents within these elevated-level attributes to engage with the primary input features via reinforcement learning, thus augmenting the quality of multi-modal embeddings. In addition, we introduce a novel audio-lyrics datasetAL-song, which consists of paired audio with corresponding lyrics for the audio-lyrics retrieval task. The empirical findings derived from theAL-songdataset and the benchmark datasetSounddescssubstantiate the efficacy and efficiency of CMRF when juxtaposed with state-of-the-art methodologies.

Small moving objects at far distance always occupy only one or a few pixels in image and exhibit extremely limited visual features, which bring great challenges to motion detection. Highly evolved visual systems endow flying insects with remarkable ability to pursue tiny mates and prey, providing a good template to develop image processing method for small target motion detection. The insects' excellent sensitivity to small moving objects is believed to come from a class of specific neurons called small target motion detectors (STMDs). However, existing STMD-based methods often experience performance degradation when coping with complex natural scenes. In this paper, we propose a bio-inspired visual system with spatio-temporal feedback mechanism (called Spatio-Temporal Feedback STMD) to suppress false positive background movement while enhancing system responses to small targets. Specifically, the proposed visual system is composed of two complementary subnetworks and a feedback loop. The first subnetwork is designed to extract spatial and temporal movement patterns of cluttered background by neuronal ensemble coding. The second subnetwork is developed to capture small target motion information where its output and signal from the first subnetwork are integrated together via the feedback loop to filter out background false positives in a recurrent manner. Experimental results demonstrate that the proposed spatio-temporal feedback visual system is more competitive than existing methods in discriminating small moving targets from complex natural environments.

Collision detection is critical for autonomous vehicles or robots to serve human society safely. Detecting looming objects robustly and timely plays an important role in collision avoidance systems. The locust lobula giant movement detector (LGMD1) is specifically selective to looming objects which are on a direct collision course. However, the existing LGMD1 models cannot distinguish a looming object from a near and fast translatory moving object, because the latter can evoke a large amount of excitation that can lead to false LGMD1 spikes. This article presents a new visual neural system model (LGMD1) that applies a neural competition mechanism within a framework of separated ON and OFF pathways to shut off the translating response. The competition-based approach responds vigorously to monotonous ON/OFF responses resulting from a looming object. However, it does not respond to paired ON-OFF responses that result from a translating object, thereby enhancing collision selectivity. Moreover, a complementary denoising mechanism ensures reliable collision detection. To verify the effectiveness of the model, we have conducted systematic comparative experiments on synthetic and real datasets. The results show that our method exhibits more accurate discrimination between looming and translational events-the looming motion can be correctly detected. It also demonstrates that the proposed model is more robust than comparative models.

In low light conditions, image enhancement is critical for vision-based artificial systems since details of objects in dark regions are buried. Moreover, enhancing the low-light image without introducing too many irrelevant artifacts is important for visual tasks like motion detection. However, conventional methods always have the risk of “bad” enhancement. Nocturnal insects show remarkable visual abilities at night time, and their adaptations in light responses provide inspiration for low-light image enhancement. In this paper, we aim to adopt the neural mechanism of dark adaptation for adaptively raising intensities whilst preserving the naturalness. We propose a framework for enhancing low-light images by implementing the dark adaptation operation with proper adaptation parameters in R, G and B channels separately. Specifically, the dark adaptation in this paper consists of a series of canonical neural computations, including the power law adaptation, divisive normalization and adaptive rescaling operations. Experiments show that the proposed bio-inspired dark adaptation framework is more efficient and can better preserve the naturalness of the image compared to existing methods.

The LGMD1 neuron of locusts shows strong looming-sensitive property for both light and dark objects. Although a few LGMD1 models have been proposed, they are not reliable to inhibit the translating motion under certain conditions compare to the biological LGMD1 in the locust. To address this issue, we propose a bio-plausible model to enhance the collision-selectivity by inhibiting the translating motion. The proposed model contains three parts, the retina to lamina layer for receiving luminance change signals, the lamina to medulla layer for extracting motion cues via ON and OFF pathways separately, the medulla to lobula layer for eliminating translational excitation with neural competition. We tested the model by synthetic stimuli and real physical stimuli. The experimental results demonstrate that the proposed LGMD1 model has a strong preference for objects in direct collision course-it can detect looming objects in different conditions while completely ignoring translating objects.