Large Multimodal Models (LMMs) often face a modality representation gap during pretraining: while language embeddings remain stable, visual representations are highly sensitive to contextual noise (e.g., background clutter). To address this issue, we introduce a visual comprehension stage, which we call ViCToR (Visual Comprehension via Token Reconstruction), a novel pretraining framework for LMMs. ViCToR employs a learnable visual token pool and utilizes the Hungarian matching algorithm to select semantically relevant tokens from this pool for visual token replacement. Furthermore, by integrating a visual token reconstruction loss with dense semantic supervision, ViCToR can learn tokens which retain high visual detail, thereby enhancing the large language model's (LLM's) understanding of visual information. After pretraining on 3 million publicly accessible images and captions, ViCToR achieves state-of-the-art results, improving over LLaVA-NeXT-8B by 10.4%, 3.2%, and 7.2% on the MMStar, SEEDI, and RealWorldQA benchmarks, respectively.

Defocus deblurring is a challenging task due to the spatially varying blur. Recent works have shown impressive results in data-driven approaches using dual-pixel (DP) sensors. Quad-pixel (QP) sensors represent an advanced evolution of DP sensors, providing four distinct sub-aperture views in contrast to only two views offered by DP sensors. However, research on QP-based defocus deblurring is scarce. In this paper, we propose a novel end-to-end learning-based approach for defocus deblurring that leverages QP data. To achieve this, we design a QP defocus and all-in-focus image pair acquisition method and provide a QP Defocus Deblurring (QPDD) dataset containing 4,935 image pairs. We then introduce a Local-gate assisted Mamba Network (LMNet), which includes a two-branch encoder and a Simple Fusion Module (SFM) to fully utilize features of sub-aperture views. In particular, our LMNet incorporates a Local-gate assisted Mamba Block (LAMB) that mitigates local pixel forgetting and channel redundancy within Mamba, and effectively captures global and local dependencies. By extending the defocus deblurring task from a DP-based to a QP-based approach, we demonstrate significant improvements in restoring sharp images. Comprehensive experimental evaluations further indicate that our approach outperforms state-of-the-art methods.

Learning visual representations is foundational for a broad spectrum of downstream tasks. Although recent vision-language contrastive models, such as CLIP and SigLIP, have achieved impressive zero-shot performance via large-scale vision-language alignment, their reliance on global representations constrains their effectiveness for dense prediction tasks, such as grounding, OCR, and segmentation. To address this gap, we introduce Region-Aware Cluster Discrimination (RICE), a novel method that enhances region-level visual and OCR capabilities. We first construct a billion-scale candidate region dataset and propose a Region Transformer layer to extract rich regional semantics. We further design a unified region cluster discrimination loss that jointly supports object and OCR learning within a single classification framework, enabling efficient and scalable distributed training on large-scale data. Extensive experiments show that RICE consistently outperforms previous methods on tasks, including segmentation, dense detection, and visual perception for Multimodal Large Language Models (MLLMs). The pre-trained models have been released at https://github.com/deepglint/MVT.

After pre-training on extensive image-text pairs, Contrastive Language-Image Pre-training (CLIP) demonstrates promising performance on a wide variety of benchmarks. However, a substantial volume of multimodal interleaved documents remains underutilized for contrastive vision-language representation learning. To fully leverage these unpaired documents, we initially establish a Real-World Data Extraction pipeline to extract high-quality images and texts. Then we design a hierarchical retrieval method to efficiently associate each image with multiple semantically relevant realistic texts. To further enhance fine-grained visual information, we propose an image semantic augmented generation module for synthetic text production. Furthermore, we employ a semantic balance sampling strategy to improve dataset diversity, enabling better learning of long-tail concepts. Based on these innovations, we construct RealSyn, a dataset combining realistic and synthetic texts, available in three scales: 15M, 30M, and 100M. We compare our dataset with other widely used datasets of equivalent scale for CLIP training. Models pre-trained on RealSyn consistently achieve state-of-the-art performance across various downstream tasks, including linear probe, zero-shot transfer, zero-shot robustness, and zero-shot retrieval. Furthermore, extensive experiments confirm that RealSyn significantly enhances contrastive vision-language representation learning and demonstrates robust scalability. The code will be released in https://garygutc.github.io/RealSyn.

Despite the impressive progress of recent pretraining methods on multimodal tasks, existing methods are inherently biased towards either spatial modeling (e.g., CLIP) or temporal modeling (e.g., V-JEPA), limiting their joint capture of spatial details and temporal dynamics. To this end, we propose UniViT, a cluster-driven unified self-supervised learning framework that effectively captures the structured semantics of both image spatial content and video temporal dynamics through event-level and object-level clustering and discrimination. Specifically, we leverage offline clustering to generate semantic clusters across both modalities. For videos, multi-granularity event-level clustering progressively expands from single-event to structured multi-event segments, capturing coarse-to-fine temporal semantics; for images, object-level clustering captures fine-grained spatial semantics. However, while global clustering provides semantically consistent clusters, it lacks modeling of structured semantic relations (e.g., temporal event structures). To address this, we introduce a contrastive objective that leverages these semantic clusters as pseudo-label supervision to explicitly enforce structural constraints, including temporal event relations and spatial object co-occurrences, capturing structured semantics beyond categories. Meanwhile, UniViT jointly embeds structured object-level and event-level semantics into a unified representation space. Furthermore, UniViT introduces two key components: (i) Unified Rotary Position Embedding integrates relative positional embedding with frequency-aware dimension allocation to support position-invariant semantic learning and enhance the stability of structured semantics in the discrimination stage; and (ii) Variable Spatiotemporal Streams adapt to inputs of varying frame lengths, addressing the rigidity of conventional fixed-input approaches. Extensive experiments across varying model scales demonstrate that UniViT achieves state-of-the-art performance on linear probing, attentive probing, question answering, and spatial understanding tasks.

In the past decade, computer vision has developed rapidly, and its application scenarios are increasing. But in the process of its application, the limited embedded compute capability is still one of the most important reasons hindering its development. In contrast, with the continuous improvement of mobile computing capability in recent years, the reasoning of neural network models on mobile phones has become a closer and closer fact. The most of tasks of computer vision are continuous and fixed order of the calculation processes. According to the characteristic, we propose a method for collaborative embedded inference on mobile phones. This method divides computer vision tasks, moves part of the calculation to the mobile phone, and runs in a pipeline scheme to achieve the effect of accelerating inference. This method can realize the running acceleration of such tasks and reducing the computational burden of the embedded platform. Codes are available at https://github.com/yiyexy/pipeline.

In past work, deep learning researchers always designed hyperparameters such as model structure and learning rate first and then used the training set to train the weights in this model. While unrestricted model structure design leads to massive neuron redundancy in neural network models. By pruning these redundant neurons, not only can the storage be compressed effectively, but also the operation can be accelerated. In this paper, we propose a method to utilize the training set to prune the model structure during training: 1) train the initialized model and bring it to basic convergence; 2) feed the entire training set into the model and calculate the activations of neurons in each layer; 3) calculate the threshold for neuron pruning in each layer according to the pruning ratio, delete neurons whose activation value is lower than the threshold, and correspondingly delete the weights of the upper and lower layers; 4) further train the pruned model so that it eventually converges. This method of deleting redundant neurons not only greatly deletes the parameters in the model but also achieves model acceleration. We applied this method to some mainstream neural network models: VGGNet and ResNet, and achieved good results.

Dynamic channel pruning is a technique aimed at reducing the theoretical computational complexity and inference latency of convolutional neural networks. Dynamic channel pruning methods introduce complex additional modules for dynamically selecting channels for images. Due to the additional modules, dynamic channel pruning methods never achieve optimal acceleration effect in real world. To address this problem, we propose Consecutive Dynamic Channel Pruning (CDCP), a novel dynamic channel pruning framework unified for almost all dynamic pruning methods designed for continuous image processing. The core idea of CDCP stems from our observation that adjusting the network for all frames in semantically continuous scenes is unnecessary since adjacent frames often share similar network structures in dynamic channel pruning. CDCP introduces a simple binary classifier to determine whether the network structure needs to be adjusted for a new frame. Our method can also be used for semantically non-continuous image processing tasks with a slightly lower probability of model reuse. We validate the effectiveness of CDCP on three dynamic channel pruning methods and better acceleration effects are achieved when applied them with CDCP to the semantically continuous Waymo dataset, the nuScenes dataset, and the semantically discontinuous COCO dataset.

Software developers rely on support from a variety of resources---including other developers---but the coordination cost of finding another developer with relevant experience, explaining the context of the problem, composing a specific help request, and providing access to relevant code is prohibitively high for all but the largest of tasks. Existing technologies for synchronous communication (e.g. voice chat) have high scheduling costs, and asynchronous communication tools (e.g. forums) require developers to carefully describe their code context to yield useful responses. This paper introduces Codeon, a system that enables more effective task hand-off between end-user developers and remote helpers by allowing asynchronous responses to on-demand requests. With Codeon, developers can request help by speaking their requests aloud within the context of their IDE. Codeon automatically captures the relevant code context and allows remote helpers to respond with high-level descriptions, code annotations, code snippets, and natural language explanations. Developers can then immediately view and integrate these responses into their code. In this paper, we describe Codeon, the studies that guided its design, and our evaluation that its effectiveness as a support tool. In our evaluation, developers using Codeon completed nearly twice as many tasks as those who used state-of-the-art synchronous video and code sharing tools, by reducing the coordination costs of seeking assistance from other developers.

In the underlay spectrum sharing systems, secondary users (SUs) are allowed to transmit their data in the licensed spectrum band when primary users(PUs) are also transmitting, as long as the transmission of SUs do not interfere PUs' communications. In cognitive relay networks, the source and relay nodes both need to tune their transmit power to mitigate the interference to PU. In this paper, we investigate the outage performance of cognitive relay networks with PU's interference to signal ratio (ISR) constraint, where both average and peak ISR constraint are considered. Finally, We derive the exact outage probability in the scenario without cooperation and the upper bound of outage probability in the scenario with cooperation.

In spectrum sharing systems, a secondary user (SU) is permitted to share frequency bands with a primary user (PU) as long as its transmission does not interfere with the PU's communication. In this paper, the outage probability is investigated for the cognitive relay system over Nakagami-m fading channel. By applying the interference temperature constraints at the source nodes and relay nodes in secondary systems, we analyze the outage performance in two-hop underlay spectrum sharing with the best relay selection criterion. The probability density function (PDF) and cumulative distribution function (CDF) of the signal to noise ratio (SNR) at the SU's receiver are derived to obtain the closed-form upper bound of the outage probability of the secondary relay system. Simulations results demonstrate the validity and accuracy of the theoretical analysis.