Visual Language Tracking (VLT) enables machines to perform tracking in real world through human-like language descriptions. However, existing VLT methods are limited to 2D spatial tracking or single-object 3D tracking and do not support multi-object 3D tracking within monocular video. This limitation arises because advancements in 3D multi-object tracking have predominantly relied on sensor-based data (e.g., point clouds, depth sensors) that lacks corresponding language descriptions. Moreover, natural language descriptions in existing VLT literature often suffer from redundancy, impeding the efficient and precise localization of multiple objects. We present the first technique to extend VLT to multi-object 3D tracking using monocular video. We introduce a comprehensive framework that includes (i) a Monocular Multi-object 3D Visual Language Tracking (MoMo-3DVLT) task, (ii) a large-scale dataset, MoMo-3DRoVLT, tailored for this task, and (iii) a custom neural model. Our dataset, generated with the aid of Large Language Models (LLMs) and manual verification, contains 8,216 video sequences annotated with both 2D and 3D bounding boxes, with each sequence accompanied by three freely generated, human-level textual descriptions. We propose MoMo-3DVLTracker, the first neural model specifically designed for MoMo-3DVLT. This model integrates a multimodal feature extractor, a visual language encoder-decoder, and modules for detection and tracking, setting a strong baseline for MoMo-3DVLT. Beyond existing paradigms, it introduces a task-specific structural coupling that integrates a differentiable linked-memory mechanism with depth-guided and language-conditioned reasoning for robust monocular 3D multi-object tracking. Experimental results demonstrate that our approach outperforms existing methods on the MoMo-3DRoVLT dataset. Our dataset and code are available at https://github.com/hongkai-wei/MoMo-3DVLT.

Visual-Language Tracking (VLT) is emerging as a promising paradigm to bridge the human-machine performance gap. For single objects, VLT broadens the problem scope to text-driven video comprehension. Yet, this direction is still confined to 2D spatial extents, currently lacking the ability to deal with 3D tracking in the confines of monocular video. Unfortunately, advances in 3D tracking mainly rely on expensive sensor inputs, e.g., point clouds, depth measurements, radar. Absence of language counterpart for the outputs of these mildly democratized sensors in the literature also hinders VLT expansion to 3D tracking. Addressing that, we make the first attempt towards extending VLT to 3D tracking based on monocular video. We present a comprehensive framework, introducing (i) the Monocular-Video-based 3D Visual Language Tracking (Mono3DVLT) task, (ii) a large-scale dataset for the task, called Mono3DVLT-V2X, and (iii) a customized neural model for the task. Our dataset is carefully curated, leveraging a Large Langauge Model (LLM) followed by human verification, composing natural language descriptions for 79,158 video sequences aiming at single object tracking, providing 2D and 3D bounding box annotations. Our neural model, termed Mono3DVLT-MT, is the first targeted approach for the Mono3DVLT task. Comprising the pipeline of multi-modal feature extractor, visual-language encoder, tracking decoder and a tracking head, our model sets a strong baseline for the task on Mono3DVLT-V2X. Experimental results show that our method significantly outperforms existing techniques on the Mono3DVLT-V2X dataset. Our dataset and code are available in https://github.com/hongkai-wei/Mono3DVLT.

The rapid advancement of artificial intelligence has propelled the healthcare industry into a new era of diagnostic precision. A pivotal component of this evolution is the accurate classification of 3D medical images, which necessitates extracting robust feature representations capable of effectively modeling long-range dependencies within the data. This paper introduces the Mamba Token Turing Machine (MTTM), a novel architecture that integrates the efficiency of Mamba with the memory mechanisms of the Token Turing Machine (TTM), effectively addressing limitations of Transformers in long-range dependency modeling. MTTM’s Memory-Augmented Processing Unit (MAPU) employs four blending methods, achieving state-of-the-art accuracy and efficiency on the MedMNIST v2 dataset, thereby advancing diagnostic precision in 3D medical image analysis. The code is available at https://github.com/hongkai-wei/MTTM.

Monocular 3D Visual Grounding (Mono3DVG) aims to predict the 3D localization of objects in monocular RGB images based on natural language descriptions. This task has broad applications in areas such as autonomous driving, human-computer interaction and robotic manipulation, making it both a significant and challenging task. To tackle this challenge, we propose a novel network architecture, the Monocular Selective Attention Learning Network (MSALNet). This network enhances the understanding and localization of objects by introducing an Adaptive Learning Module (ALM) and a Vision-Text Interaction Encoder. Specifically, the ALM further learns the features extracted from the 3D scene and textual descriptions, capturing the contextual relationships within the features. This enables the model to better understand the meaning of both the scene and the text. Meanwhile, the Vision-Text Interaction Encoder facilitates refined cross-modal interaction and fusion, promoting alignment between visual and textual information and providing more discriminative feature representations. Experimental results demonstrate that our method achieves competitive performance on the Mono3DRefer dataset.