The integration of visual and textual information represents a promising direction in the advancement of language models.In this paper, we explore the dual modality of language-both visual and textual-within an autoregressive framework, pre-trained on both document images and texts.Our method employs a multimodal training strategy, utilizing visual data through next patch prediction with a regression head and/or textual data through next token prediction with a classification head.We focus on understanding the interaction between these two modalities and their combined impact on model performance.Our extensive evaluation across a wide range of benchmarks shows that incorporating both visual and textual data significantly improves the performance of pixel-based language models.Remarkably, we find that a unidirectional pixelbased model trained solely on visual data can achieve comparable results to state-of-the-art bidirectional models on several language understanding tasks.This work uncovers the untapped potential of integrating visual and textual modalities for more effective language modeling.We release our code, data, and model checkpoints at

Unmanned aerial vehicle (UAV) communications have a promising prospect in the next-generation network. Among various technologies, collaborative beamforming (CB) can enhance communication abilities and ranges. However, the CB method may be subject to interference issues in the physical en-vironment such as the wind and incorrect positioning, especially in outdoor conditions. In this paper, we aim to form a virtual antenna array (VAA) consists of UAV elements and simulate the interference of the physical environment on the positions of the UAV elements by considering positions errors. Then, we propose a model based on deep learning to predict the impact of errors on UAV-enabled VAA. Finally, aiming to repair the performance loss caused by the position errors of the UAV elements, we propose a performance recovery method by re-optimizing the excitation current weights of the UAV elements via the firefly algorithm (FA). Simulation results show that the position errors decrease the directivity and increase the maximum sidelobe level (SLL). Moreover, the proposed deep learning model can reflect the performance loss under different conditions. In addition, the proposed performance recovery method is effective.