Video generation using Large Language Models (LLMs) has shown promising potential, effectively leveraging the extensive LLM infrastructure to provide a unified framework for multimodal understanding and content generation. However, these methods face critical challenges, i.e., token redundancy and inefficiencies arising from long sequences, which constrain their performance and efficiency compared to diffusion-based approaches. In this study, we investigate the impact of token redundancy in LLM-based video generation by information-theoretic analysis and propose Vision Representation Compression (VRC), a novel framework designed to achieve more in both performance and efficiency with less video token representations. VRC introduces learnable representation compressor and decompressor to compress video token representations, enabling autoregressive next-sequence prediction in a compact latent space. Our approach reduces redundancy, shortens token sequences, and improves model's ability to capture underlying video structures. Our experiments demonstrate that VRC reduces token sequence lengths by a factor of 4, achieving more than 9~14x acceleration in inference while maintaining performance comparable to state-of-the-art video generation models. VRC not only accelerates the inference but also significantly reduces memory requirements during both model training and inference.

Contrastive Language-Image Pre-training (CLIP) has become a cornerstone in multimodal intelligence.However, recent studies discovered that CLIP can only encode one aspect of the feature space, leading to substantial information loss and indistinctive features.To mitigate this issue, this paper introduces a novel strategy that fine-tunes a series of complementary CLIP models and transforms them into a CLIP-MoE.Specifically, we propose a model-agnostic Diversified Multiplet Upcycling (DMU) framework for CLIP.Instead of training multiple CLIP models from scratch, DMU leverages a pre-trained CLIP and fine-tunes it into a diverse set with highly cost-effective multistage contrastive learning, thus capturing distinct feature subspaces efficiently.To fully exploit these fine-tuned models while minimizing computational overhead, we transform them into a CLIP-MoE, which dynamically activates a subset of CLIP experts, achieving an effective balance between model capacity and computational cost.Comprehensive experiments demonstrate the superior performance of CLIP-MoE across various zero-shot retrieval, zero-shot image classification tasks, and downstream Multimodal Large Language Model (MLLM) benchmarks when used as a vision encoder.Code is available at https: //github.com/OpenSparseLLMs/CLIP-MoE.

Large Vision-Language Models (LVLMs) have become pivotal at the intersection of computer vision and natural language processing.However, the full potential of LVLMs' Retrieval-Augmented Generation (RAG) capabilities remains underutilized.Existing works either focus solely on the text modality or are limited to specific tasks.Moreover, most LVLMs struggle to selectively utilize retrieved information and are sensitive to irrelevant or misleading references.To address these challenges, we propose a self-refinement framework designed to teach LVLMs to Selectively Utilize Retrieved Information (SURf).Specifically, when given questions that are incorrectly answered by the LVLM backbone, we obtain references that help correct the answers (positive references) and those that do not (negative references).We then fine-tune the LVLM backbone using a combination of these positive and negative references.Our experiments across three tasks and seven datasets demonstrate that our framework significantly enhances LVLMs' ability to effectively utilize retrieved multimodal references and improves their robustness against irrelevant or misleading information.The source code is available at https://github.com/GasolSun36/SURf. * Work done during internship at Shanghai AI Laboratory.† Both are corresponding authors.How many apples in the images? VQAVanilla: There are three apples.The image depicting five apples on a tree...The picture shows 7 apples .... leaves...Ours: There are four apples.Describe this image in details. Captioning Vanilla: A person walking in snow.The image depicting a...the skier is in a crouched position... The image captures a dynamic scene ..a skier dressed in a ...

Time-series and event sequences are widely collected data types in real-world applications. Modeling and forecasting of such temporal data play an important role in an informed decision-making process. A major limitation of previous methods is that they either focus on time-series or events, rather than the combination of the two worlds. In fact, the two types of data often provide complementary information, emphasizing the necessity of jointly modeling the both. In this paper, we propose the RNN-ODE collaborative model for joint modeling and forecasting of heterogeneous time-series and event sequence data, which combines several useful techniques from both Bayesian and deep learning for its interpretability. Specifically, we devise a tailored encoder to combine the advances in deep temporal point processes models and variational recurrent neural networks. To predict the probability of event occurrence over an arbitrary continuous-time horizon, we base our model on the mathematical foundation of Neural Ordinary Differential Equations (NODE). Extensive experimental results on simulations and real data sets show that compared with existing methods, our integrated approach can achieve more competitive forecasting performance of both time-series and event sequences.

In this paper, we present EatingTrak, an AI-powered sensing system using a wrist-mounted inertial measurement unit (IMU) to recognize eating moments in a near-free-living semi-wild setup. It significantly improves the SOTA in time resolution using similar hardware on identifying eating moments, from over five minutes to three seconds. Different from prior work which directly learns from raw IMU data, it proposes intelligent algorithms which can estimate the arm posture in 3D in the wild and then learns the detailed eating moments from the series of estimated arm postures. To evaluate the system, we collected eating activity data from 9 participants in semi-wild scenarios for over 113 hours. Results showed that it was able to recognize eating moments at three time-resolutions: 3 seconds and 15 minutes with F-1 scores of 73.7% and 83.8%, respectively. EatingTrak would introduce new opportunities in sensing detailed eating behavior information requiring high time resolution, such as eating frequency, snack-taking, on-site behavior intervention. We also discuss the opportunities and challenges in deploying EatingTrak on commodity devices at scale.

Time-series is ubiquitous across applications, such as transportation, finance and healthcare. Time-series is often influenced by external factors, especially in the form of asynchronous events, making forecasting difficult. However, existing models are mainly designated for either synchronous time-series or asynchronous event sequence, and can hardly provide a synthetic way to capture the relation between them. We propose Variational Synergetic Multi-Horizon Network (VSMHN), a novel deep conditional generative model. To learn complex correlations across heterogeneous sequences, a tailored encoder is devised to combine the advances in deep point processes models and variational recurrent neural networks. In addition, an aligned time coding and an auxiliary transition scheme are carefully devised for batched training on unaligned sequences. Our model can be trained effectively using stochastic variational inference and generates probabilistic predictions with Monte-Carlo simulation. Furthermore, our model produces accurate, sharp and more realistic probabilistic forecasts. We also show that modeling asynchronous event sequences is crucial for multi-horizon time-series forecasting.