Video-based large language models (Video-LLMs) have been recently introduced, targeting both fundamental improvements in perception and comprehension, and a diverse range of user inquiries. In pursuit of the ultimate goal of achieving artificial general intelligence, a truly intelligent Video-LLM model should not only see and understand the surroundings, but also possess human-level commonsense, and make well-informed decisions for users. To guide the development of such a model, the establishment of a robust and comprehensive evaluation system becomes crucial. To this end, this paper proposes Video-Bench, a new comprehensive benchmark along with a toolkit specifically designed for evaluating Video-LLMs. The benchmark comprises 10 meticulously crafted tasks, evaluating the capabilities of Video-LLMs across three distinct levels: video-exclusive understanding, prior knowledge-based question-answering, and comprehension and decision-making. In addition, we introduce an automatic toolkit tailored to process model outputs for various tasks, facilitating the calculation of metrics and conveniently generating final scores. We evaluate 9 representative Video-LLMs using Video-Bench. The findings reveal that current Video-LLMs still fall considerably short of achieving human-like comprehension and analysis of real-world video, and offer valuable insights for future research directions. The benchmark and toolkit are available at https://github.com/PKU-YuanGroup/Video-Bench.

This article studies the cell-free massive multi-input multi-output (MIMO) two-way relay systems with low-resolution analog-to-digital converters (ADCs). Primarily, we analyze the normalized mean square error (NMSE) and derive a closed-form expression for NMSE’s expectation${\mathrm {Exp}}_{\mathrm {nmse}}$. Asymptotic analysis showcases that, with an infinite number of access point (AP) antennas M or ideal ADCs,${\mathrm {Exp}}_{\mathrm {nmse}}$converges to a finite value. Particularly, as pilot power decreases with M in a power law, the channel estimation performance can be maintained at a desired level. Secondly, we derive two closed-form expressions of spectral efficiencies (SE) for multiple-access channel (MAC) phase and broadcasting (BC) phases, respectively. The corresponding analysis implies that, as AP number$L\rightarrow \infty $, the SE tends to infinity with MAC phase and approaches to constant with BC phase. Interestingly, all SE expressions can reduce to the conventional cases with high-resolution ADCs. Finally, based on geometric programming, an effective power successive approximation (PSA) scheme is provided to maximize the sum SE. Results prove that the proposed PSA scheme can significantly improve the SE compared to baseline benchmarks, particularly for median AP antenna regime. All the derived closed-form expressions are validated through Monte Carlo simulations.

Large Vision-Language Model (LVLM) has enhanced the performance of various downstream tasks in visual-language understanding.Most existing approaches encode images and videos into separate feature spaces, which are then fed as inputs to large language models.However, due to the lack of unified tokenization for images and videos, namely misalignment before projection, it becomes challenging for a Large Language Model (LLM) to learn multi-modal interactions from several poor projection layers.In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM.As a result, we establish a simple but robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images and videos, mutually enhancing each other.As a result, Video-LLaVA outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on MSRVTT, MSVD, TGIF, and ActivityNet, respectively.Additionally, our Video-LLaVA also achieves superior performances on a broad range of 9 image benchmarks.Notably, extensive experiments demonstrate that Video-LLaVA mutually benefits images and videos within a unified visual representation, outperforming models designed specifically for images or videos.We aim for this work to provide modest insights into the multi-modal inputs for the LLM.

The video-language (VL) pretraining has achieved remarkable improvement in multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, N ≥ 3) beyond vision and language. We thus propose LanguageBind, taking the language as the bind across different modalities because the language modality is well-explored and contains rich semantics. Specifically, we freeze the language encoder acquired by VL pretraining and then train encoders for other modalities with contrastive learning. As a result, all modalities are mapped to a shared feature space, implementing multi-modal semantic alignment. While LanguageBind ensures that we can extend VL modalities to N modalities, we also need a high-quality dataset with alignment data pairs centered on language. We thus propose VIDAL-10M with 10 Million data with Video, Infrared, Depth, Audio and their corresponding Language. In our VIDAL-10M, all videos are from short video platforms with complete semantics rather than truncated segments from long videos, and all the video, depth, infrared, and audio modalities are aligned to their textual descriptions. LanguageBind has achieved superior performance on a wide range of 15 benchmarks covering video, audio, depth, and infrared. Moreover, multiple experiments have provided evidence for the effectiveness of LanguageBind in achieving indirect alignment and complementarity among diverse modalities.

This paper studies the downlink spectral efficiency of a two-way cell-free massive multiple-input multiple-output system with few-bit analog-to-digital converts (ADCs). By utilizing minimum mean squared error channel estimation method and maximal-ratio transmission precoder, we derive a closed-form expression for the effective downlink signal-to-interference-plus-noise ratio (SINR), which applies to any number of access point (AP) antennas M and distortion factors of ADCs in both AP and user pair sides. Additionally, the obtained analytical expression specializes to the conventional one when the ideal ADCs are adopted. Moreover, the asymptotic performance and power scaling law of the effective SINR in high M regime are studied. The corresponding analysis indicates that increasing M can provide considerable gains to compensate the rate loss caused by non-ideal ADCs and also, appropriately cutting down the pilot power and transmission power will not affect the SINR performance in high M region. Finally, all the theoretical results are verified via simulations.