Vision-Language Models (VLMs) have achieved impressive performance across various tasks, but often struggle to apply newly introduced visual concepts during inference. A common failure pattern is what we call Mixing Things Up: VLMs frequently confuse concept names, resulting in vague descriptions and failure to ground the concept correctly. Existing approaches mainly address person-related concepts through text prompts or tokenizer modifications. However, VLMs still miss or misinterpret untrained visual concepts, underscoring the need to learn new concepts directly from visual input, without relying on prior textual injection. To overcome these limitations, we propose BISCUIT (Basis-aligned Inference through Structured Concept Unification and Identification-aware Tuning), a two-step training method. Step I proposes a dual-stream structure-aware vision encoder that fuses RGB and edge-based embeddings within a shared basis space to enhance concept recognition. Step II enhances generation quality through identification-aware tuning, which encourages alignment between the generated text and the newly introduced visual concepts. Existing methods mainly focus on person concepts and lack comprehensive evaluation across diverse visual categories. We further propose a benchmark BiscuitVQA to evaluate VLMs performance on recognizing and applying novel image-introduced concepts across diverse concept types and task types, including real people, cartoons, animals, and symbolic content. We apply BISCUIT to LLaVA-1.5 and Qwen2.5-VL, achieving competitive results among open-source models and narrowing the gap to Gemini-2.5 and GPT-4o. Interestingly, our BISCUIT maintains strong generalization, showing minimal degradation on other downstream tasks.

In recent years, the significance of pre-trained transformer audio models has been increasingly recognized. However, existing pre-trained transformer audio models are based on single-channel audio. They cannot be directly applied to multi-channel audio for Sound Event Localization and Detection (SELD) tasks. To address this issue, in this paper, we propose SELD-SSAST, a novel model based on the single-channel Self-Supervised Audio Spectrogram Transformer (SSAST). Specifically, we first introduce a fusion feature that enables SSAST to learn the unique features in SELD problems effectively. Secondly, we input the multi-channel audio features into a single SSAST module to learn the temporal information across channels through channel-mixing. Finally, to enable SSAST to learn the relationships between multi-channel audio features, we propose a Convolutional Cross Attention (CCA) module to replace the Transformer’s Self-Attention and an intensity vector (IV) enhanced module to learn the differences between channel features. Our experiments show that using SELD-SSAST improved performance by 23.5% and 20.2% over the baseline on two datasets, respectively. Additionally, with the same data scale, SELD-SSAST outperforms the models in state-of-the-art (SOTA) methods on two datasets.

Vision-Language Models (VLMs) effectively align images and text, but they often struggle with fine-grained reasoning tasks. Fine-grained training also typically demand substantial GPU memory and a large number of trainable parameters. While existing Parameter-Efficient Fine-Tuning (PEFT) methods address these challenges, they also introduce new limitations, such as compromising generalization performance with additional parameters and causing overfitting with partial fine-tuning. To overcome these limitations, we propose History Tracker, a novel PEFT method designed to enhance fine-grained reasoning performance in VLMs. History Tracker leverages the stacked architecture of the image encoder, dividing its layers into shallow, middle, deep, and global groups. It then retrieves and compresses embeddings from these groups into compact tokens, reinforcing the distribution of image embeddings and allowing the text decoder to capture fine-grained details. Furthermore, we introduce a fine-grained dataset to fine-tune and evaluate models. This dataset focuses on tasks where highly similar text needs to be ranked based on its alignment with the given image. Experimental results demonstrate that History Tracker outperforms existing fine-tuning methods on both general benchmarks and fine-grained reasoning tasks. By fine-tuning only 7.03% of parameters, our method strikes a balance between fine-grained reasoning, generalization, and efficiency.

Vision-Language Models (VLMs) have achieved significant advances across various downstream tasks. However, as their performance improves, the increasing number of parameters results in slower prefilling speeds and longer inference times. To overcome these limitations, we observe that most VLMs do not require a large number of image tokens for inference, we propose BOLT (Basis-Oriented Lightweight Token-Trimming), a training-free and cross-attention-free token compression method. Unlike existing approaches, BOLT addresses the challenge of insufficient visual cues in textual prompts by leveraging token internal data distributions. We categorize tokens into three types: key tokens, proxy tokens, and remaining tokens. Then, by applying basis space similarity, we merge and filter the remaining tokens with the proxy tokens to retain the most informative ones. To account for the differences in VLM architectures and model sizes, we evaluate BOLT on LLaVA-Next-Llama3 and LLaVA-1.5 (7B and 13B). Our results show that BOLT achieves state-of-the-art performance, with a 90% token compression ratio leading to a 3.3× increase in pre-filling speed and a 1.5× improvement in inference speed, outperforming other methods.

Multi-source sound localization can find applications in many domains including auditory scene analysis, fault detection and diagnosis in manufacturing, augmented reality, etc. In far fields, 3D sound source localization is equivalent to finding the direction of arrival (DOA), namely, the azimuth and elevation angles of sound sources. Recent DOA estimation pipelines take multichannel audio inputs, extract spectral features from each channel and then feed them into a deep neural network. Unfortunately, the spectral features contain only the time-frequency information of the audio signals, while spatial information is only implicitly captured in the signals across different channels, which is highly dependent on the acoustic array geometry. To embed the spatial information of the sound source into the spectral feature representation, we propose a DSB-based spatial mapping method encode sound source location information. It can be combined with different feature extraction methods and machine learning models for DOA estimation. Furthermore, a redundancy removal procedure is proposed to accelerate DSB computation so that the pipeline can run in real-time on embedded GPUs, such as NVidia Jeston Nano. We conduct extensive experiments using two neural network models along with the DSB method on two datasets. The experiments demonstrate that the DOA errors can be effectively reduced using the DSB method. When combining DSB for feature extraction, the DOA errors are reduced by up to 19.24%. In addition, the feature extraction process is accelerated by up to 30.42% after the application of redundancy removal.

In this article, a new type-2 fuzzy-based modeling approach is proposed to assess human operators’ psychophysiological states for both safety and reliability of human–machine interface systems. Such a new modeling technique combines type-2 fuzzy sets with state tracking to update the rule base through a Bayesian process. These new configurations successfully lead to an adaptive, robust, and transparent computational framework that can be utilized to identify dynamic (i.e., real time) features without prior training. The proposed framework is validated on mental arithmetic cognitive real-time experiments with ten participants. It is found that the proposed framework outperforms other paradigms (i.e., an adaptive neuro-fuzzy inference system and an adaptive general type-2 fuzzy c-means modeling approach) in terms of disturbance rejection and learning capabilities. The proposed framework achieved the best performance compared to other models that have been presented in the related literature. Therefore, the new framework can be a promising development in human–machine interface systems. It can be further utilized to develop advanced control mechanisms, investigate the origins of human compromised task performance, and identify and remedy psychophysiological breakdown in the early stages.