Traditional and neural video codecs commonly encounter limitations in controllability and generality under ultra-low-bitrate coding scenarios. To overcome these challenges, we propose M3-CVC, a controllable video compression framework incorporating multimodal generative models. The framework utilizes a semantic-motion composite strategy for keyframe selection to retain critical information. For each keyframe and its corresponding video clip, a dialogue-based large multimodal model (LMM) approach extracts hierarchical spatiotemporal details, enabling both inter-frame and intra-frame representations for improved video fidelity while enhancing encoding interpretability. M3-CVC further employs a conditional diffusion-based, text-guided keyframe compression method, achieving high fidelity in frame reconstruction. During decoding, textual descriptions derived from LMMs guide the diffusion process to restore the original video’s content accurately. Experimental results demonstrate that M3-CVC significantly outperforms the state-of-the-art VVC standard in ultra-low bitrate scenarios, particularly in preserving semantic and perceptual fidelity.

Diffusion transformers have demonstrated remarkable performance in visual generation tasks, such as generating realistic images or videos based on textual instructions. However, larger model sizes and multi-frame processing for video generation lead to increased computational and memory costs, posing challenges for practical deployment on edge devices. Post-Training Quantization (PTQ) is an effective method for reducing memory costs and computational complexity. When quantizing diffusion transformers, we find that existing quantization methods face challenges when applied to text-to-image and video tasks. To address these challenges, we begin by systematically analyzing the source of quantization error and conclude with the unique challenges posed by DiT quantization. Accordingly, we design an improved quantization scheme: ViDiT-Q (**V**ideo \& **I**mage **Di**ffusion **T**ransformer **Q**uantization), tailored specifically for DiT models. We validate the effectiveness of ViDiT-Q across a variety of text-to-image and video models, achieving W8A8 and W4A8 with negligible degradation in visual quality and metrics. Additionally, we implement efficient GPU kernels to achieve practical 2-2.5x memory optimization and a 1.4-1.7x end-to-end latency speedup.

Bipedal robots, due to their anthropomorphic design, offer substantial potential across various applications, yet their control is hindered by the complexity of their structure. Currently, most research focuses on proprioception-based methods, which lack the capability to overcome complex terrain. While visual perception is vital for operation in human-centric environments, its integration complicates control further. Recent reinforcement learning (RL) approaches have shown promise in enhancing legged robot locomotion, particularly with proprioception-based methods. However, terrain adaptability, especially for bipedal robots, remains a significant challenge, with most research focusing on flat-terrain scenarios. In this paper, we introduce a novel mixture of experts teacher-student network RL strategy, which enhances the performance of teacher-student policies based on visual inputs through a simple yet effective approach. Our method combines terrain selection strategies with the teacher policy, resulting in superior performance compared to traditional models. Additionally, we introduce an alignment loss between the teacher and student networks, rather than enforcing strict similarity, to improve the student’s ability to navigate diverse terrains. We validate our approach experimentally on the Limx Dynamic P1 bipedal robot, demonstrating its feasibility and robustness across multiple terrain types.

Prevalent lossy image compression schemes can be divided into: 1) explicit image compression (EIC), including traditional standards and neural end-to-end algorithms; 2) implicit image compression (IIC) based on implicit neural representations (INR). The former is encountering impasses of leveling off bitrate reduction at a cost of tremendous complexity while the latter suffers from excessive smoothing quality as well as lengthy decoder models. In this paper, we propose an innovative paradigm, which we dub Unicorn (Unified Neural Image Compression with One Nnumber Reconstruction). By conceptualizing the images as index-image pairs and learning the inherent distribution of pairs in a subtle neural network model, Unicorn can reconstruct a visually pleasing image from a randomly generated noise with only one index number. The neural model serves as the unified decoder of images while the noises and indexes corresponds to explicit representations. As a proof of concept, we propose an effective and efficient prototype of Unicorn based on latent diffusion models with tailored model designs. Quantitive and qualitative experimental results demonstrate that our prototype achieves significant bitrates reduction compared with EIC and IIC algorithms. More impressively, benefitting from the unified decoder, our compression ratio escalates as the quantity of images increases. We envision that more advanced model designs will endow Unicorn with greater potential in image compression. The code will be made publicly available upon publication.

Recent research on pedestrian trajectory prediction based on deep learning has made significant progress. However, the previous methods do not deeply explore the relationship between scene information and trajectory. Moreover, the training model requires massive data and only targets specific scenes, so the prediction performance is poor when the new scene samples are limited. To solve the above problems, a meta-inverse reinforcement learning-based framework, dubbed Meta-IRLSOT++, is proposed in this work. IRLSOT++ improves the solid baseline IRLSOT, and the main contributions are as follows: (1) An inverse reinforcement learning framework is introduced to explore the trajectory–scene association to achieve task-level scene understanding, enhancing the correlation between trajectory and scene. The trajectory heat maps describing pedestrians dynamic characteristics are leveraged to align better the trajectory and scene semantic segmentation predicted by TopFormer. (2) A Transformer-based encoder–decoder network is proposed to fuse the trajectory and plan cues for better generating multi-modal trajectories with multi-head attention. The social graph attention and pseudo-oracle predictor are introduced to capture pedestrians’ social interactions and intent states, ameliorating trajectory prediction performance. (3) Meta-learning is leveraged to achieve collaborative training based on IRLSOT++ to improve the model generalization in new scenes, enabling fast adaptation of trajectory prediction. Experimental results on the Stanford Drone Dataset (SDD) indicate that IRLSOT++ can precisely forecast future trajectories by improving IRLSOT, decreasing Average Displacement Error/ Final Displacement Error (ADE/FDE) values from 9.66/13.05 to 8.36/12.28. Moreover, the meta-learning strategy quickly adapts IRLSOT++ to the new scene, achieving ADE/FDE values of 7.31/11.02 and 15.42/26.55 when T p r e d is set to 12 and 24, respectively. Both quantitative and qualitative experimental results show that Meta-IRLSOT++ has accuracy and fast adaptability, which is beneficial for real-world trajectory prediction tasks.

Two major challenges of 3D LiDAR Panoptic Segmentation (PS) are that point clouds of an object are surface-aggregated and thus hard to model the long-range dependency especially for large instances, and that objects are too close to separate each other. Recent literature addresses these problems by time-consuming grouping processes such as dual-clustering, mean-shift offsets and etc., or by bird-eye-view (BEV) dense centroid representation that downplays geometry. However, the long-range geometry relationship has not been sufficiently modeled by local feature learning from the above methods. To this end, we present SCAN, a novel sparse cross-scale attention network to first align multi-scale sparse features with global voxel-encoded attention to capture the long-range relationship of instance context, which is able to boost the regression accuracy of the over-segmented large objects. For the surface-aggregated points, SCAN adopts a novel sparse class-agnostic representation of instance centroids, which can not only maintain the sparsity of aligned features to solve the under-segmentation on small objects, but also reduce the computation amount of the network through sparse convolution. Our method outperforms previous methods by a large margin in the SemanticKITTI dataset for the challenging 3D PS task, achieving 1st place with a real-time inference speed.

Reliability has become a greater concern in recent years, because high-tech industrial processes with ever increasing levels of sophistication comprise most engineering systems today. To keep pace with this rapidly developing field, this paper provides a broad overview of recent research on reliability optimization problems and their solution methodologies. In particular, we address issues related to: 1) universal generating-function-based optimal multistate system design; 2) percentile life employed as a system performance measure; 3) multiobjective optimization of reliability systems, especially with uncertain component-reliability estimations; and 4) innovation and improvement in traditional reliability optimization problems, such as fault-tolerance mechanism and cold-standby redundancy-involved system design. New developments in optimization techniques are also emphasized in this paper, especially the methods of ant colony optimization and hybrid optimization. We believe that the interesting problems that are reviewed here are deserving of more attention in the literature. To that end, this paper concludes with a discussion of future challenges related to reliability optimization