Vision-Language Models (VLMs) have achieved impressive performance across various tasks, but often struggle to apply newly introduced visual concepts during inference. A common failure pattern is what we call Mixing Things Up: VLMs frequently confuse concept names, resulting in vague descriptions and failure to ground the concept correctly. Existing approaches mainly address person-related concepts through text prompts or tokenizer modifications. However, VLMs still miss or misinterpret untrained visual concepts, underscoring the need to learn new concepts directly from visual input, without relying on prior textual injection. To overcome these limitations, we propose BISCUIT (Basis-aligned Inference through Structured Concept Unification and Identification-aware Tuning), a two-step training method. Step I proposes a dual-stream structure-aware vision encoder that fuses RGB and edge-based embeddings within a shared basis space to enhance concept recognition. Step II enhances generation quality through identification-aware tuning, which encourages alignment between the generated text and the newly introduced visual concepts. Existing methods mainly focus on person concepts and lack comprehensive evaluation across diverse visual categories. We further propose a benchmark BiscuitVQA to evaluate VLMs performance on recognizing and applying novel image-introduced concepts across diverse concept types and task types, including real people, cartoons, animals, and symbolic content. We apply BISCUIT to LLaVA-1.5 and Qwen2.5-VL, achieving competitive results among open-source models and narrowing the gap to Gemini-2.5 and GPT-4o. Interestingly, our BISCUIT maintains strong generalization, showing minimal degradation on other downstream tasks.

Vision-Language Models (VLMs) effectively align images and text, but they often struggle with fine-grained reasoning tasks. Fine-grained training also typically demand substantial GPU memory and a large number of trainable parameters. While existing Parameter-Efficient Fine-Tuning (PEFT) methods address these challenges, they also introduce new limitations, such as compromising generalization performance with additional parameters and causing overfitting with partial fine-tuning. To overcome these limitations, we propose History Tracker, a novel PEFT method designed to enhance fine-grained reasoning performance in VLMs. History Tracker leverages the stacked architecture of the image encoder, dividing its layers into shallow, middle, deep, and global groups. It then retrieves and compresses embeddings from these groups into compact tokens, reinforcing the distribution of image embeddings and allowing the text decoder to capture fine-grained details. Furthermore, we introduce a fine-grained dataset to fine-tune and evaluate models. This dataset focuses on tasks where highly similar text needs to be ranked based on its alignment with the given image. Experimental results demonstrate that History Tracker outperforms existing fine-tuning methods on both general benchmarks and fine-grained reasoning tasks. By fine-tuning only 7.03% of parameters, our method strikes a balance between fine-grained reasoning, generalization, and efficiency.

Vision-Language Models (VLMs) have achieved significant advances across various downstream tasks. However, as their performance improves, the increasing number of parameters results in slower prefilling speeds and longer inference times. To overcome these limitations, we observe that most VLMs do not require a large number of image tokens for inference, we propose BOLT (Basis-Oriented Lightweight Token-Trimming), a training-free and cross-attention-free token compression method. Unlike existing approaches, BOLT addresses the challenge of insufficient visual cues in textual prompts by leveraging token internal data distributions. We categorize tokens into three types: key tokens, proxy tokens, and remaining tokens. Then, by applying basis space similarity, we merge and filter the remaining tokens with the proxy tokens to retain the most informative ones. To account for the differences in VLM architectures and model sizes, we evaluate BOLT on LLaVA-Next-Llama3 and LLaVA-1.5 (7B and 13B). Our results show that BOLT achieves state-of-the-art performance, with a 90% token compression ratio leading to a 3.3× increase in pre-filling speed and a 1.5× improvement in inference speed, outperforming other methods.

We propose an efficient lightweight fine-tuning method that simplifies model design and reduces parameters, focusing on optimizing Visual-Language Models (VLMs) for edge deployment. As VLMs evolve, the parameter size becomes increasingly challenging for edge devices. To overcome this limitation, we combine lightweighting and fine-tuning into a single step. We decompose large linear layers in the vision encoder and introduce smaller matrices in parallel, creating a new path.During fine tuning, performance is improved by reducing the matrix size and increasing the depth, gradually phasing out the original path. We deploy the lightened and fine-tuned model on a Jetson TX2 and shows comparable performance compared to VLMs with larger parameters.

The increasing integration of wearable devices in daily activities has elevated the need for robust authentication methods that safeguard user data. Traditional knowledge-based and biometric authentication techniques face challenges in wearable contexts, including privacy risks and hardware limitations. We propose TapID, a novel authentication approach that leverages the unique relaxation vibrations of wrist bone conduction following a tapping gesture. This method bypasses the need for intrusive data collection and expensive hardware. Our method employs an energy window extraction algorithm and cross-correlation to isolate biometric signals, followed by feature extraction and k-NN classification. Tested on a Raspberry Pi, TapID authenticated users with a 93% success rate in a preliminary trial involving ten individuals, demonstrating its potential for secure and user-friendly wearable authentication.