Conceptual design is a critical stage in product development, which is a co-design process involving multidisciplinary collaboration based on prototypes. In this paper, we aim to propose a novel prototype paradigm that combines the distinct strengths of generative artificial intelligence (GAI) and spatial augmented reality (SAR), leveraging the expressive potential of SAR and the creative potential of GAI for co-design. To achieve this, we initially conducted a formative study with designers to explore how these technologies could be effectively combined to facilitate co-design. Based on our findings, we introduce ImmersiProtor, a prototype tool integrating multi-view SAR and component-layered GAI for co-design. On one hand, ImmersiProtor allows design team members to freely create and modify physical prototypes while automatically generating multi-view and high-fidelity renderings that are projected onto the surfaces of the physical prototype using SAR technology, enabling immersive communication and intuitive evaluation. On the other hand, ImmersiProtor introduces a component-layered generation and collaboration mode, offering both personal and shared team component resources. It ensures that individual team members can explore ideas independently without interference, while also supporting concept integration, evaluation, and iteration. We implemented ImmersiProtor, which involves a web-based application and an SAR design space. We conducted a user study to verify ImmersiProtor’s usability in supporting prototype and collaboration. Our results highlighted ImmersiProtor’s inherent strengths in enhancing intuition, promoting collaboration, and strengthening GAI controllability. We also explored the effect of mixed interaction on design and critically discuss its best practices for the HCI community.

Autoencoder-based neural compression methods leverage expressive models to fit large datasets but often incur considerable decoding complexity. Recently, overfitted codecs with reduced decoding complexity have gained attention as an alternative. However, they usually require access to entire videos or multiple frames simultaneously for encoding, resulting in substantial system delays. To address these limitations, we propose CNVC, a compact neural video codec that employs instance-level adaptation for efficient and flexible video compression. CNVC is fully overfitted (each frame is optimized independently using up to 45k iterations for maximum performance in this paper), building on the COOL-CHIC video model with substantial architectural and training enhancements. At a decoding complexity of just 1300 MACs per pixel, CNVC provides a more compact solution than previous autoencoder-based and overfitted codecs. Additionally, CNVC inherits the frame-wise overfitting mechanism of COOL-CHIC video, enabling flexible encoding configurations (e.g., low-delay1). In terms of compression efficiency, CNVC achieves significant bitrate reductions on HEVC and UVG datasets compared to COOL-CHIC video. To our knowledge, CNVC is the first compact neural codec to match HEVC (x265 slow setting) performance at such a decoding complexity level.

Screen content has emerged as a prominent medium in our increasingly connected world. However, compressed screen content images often suffer from unpleasant artifacts, significantly obstructing the comprehension of text and graphic regions. In this paper, we introduce a quality enhancement framework specifically designed for compressed screen content images. We first propose a dataset for enhancing the quality of screen content images affected by various levels of compression distortion, using state-of-the-art Versatile Video Coding with screen content coding techniques enabled. Given the unique characteristics of screen content images, our enhancement framework incorporates B-spline representation to mitigate the quality degradation caused by compression. Additionally, we focus on recovering distorted text by detecting text regions within the degraded image and generating a pristine textual map to guide the recovery process. Experimental results demonstrate that our proposed method effectively enhances the quality of reconstructed screen content images across different compression distortion levels, leading to the quantitative and qualitative improvement.

Recent exploration efforts in JVET (Joint Video Experts Team of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC29) has achieved progresses on neural network-based video coding (NNVC)11NNVC is also the name of the reference software for evaluating neural network-based video coding technologies in JVET. The project locates at https://vcgit.hhi.fraunhofer.de/jvet-ahg-nnvc/VVCSoftware_VTM. Latest version of NNVC features two normative deep tools, i.e., neural network-based intra prediction and neural network-based in-loop filtering. Specifically, the neural network-based filtering in NNVC supports three operating points, known as VLOP (very low-complexity operating point), LOP (low-complexity operating point), and HOP (high-complexity operating point). LOP filter receives more attention among these three due to its favorable performance-complexity trade-off. In this paper, we introduce CCLOP, a cross-component enhanced LOP filter. CCLOP builds upon LOP filter in NNVC but incorporates deep luma features for chroma filtering. We conduct extensive experiments to verify the effectiveness of CCLOP. Compared with NNVC-10, the latest reference software of NNVC, CCLOP achieves {-0.13%, −2.27%, −3.11%}, {-0.18%, −2.07%, −3.21%}, and {-0.03%, −1.81%, −2.51%} BD-rate changes on average for {Y, Cb, Cr} under random-access, low-delay, and all-intra configurations respectively, while maintaining the same complexity as existing LOP filter ([email protected] kMAC/pixel, [email protected] kMAC/pixel).

Neural compression methods often rely on highly expressive models to fit large datasets, resulting in significant decoding complexity. Overfitted codecs have been proposed as an alternative to reduce decoding complexity. However, these approaches typically lack flexibility in encoding configurations. To address this, we introduce CD, a neural video compression method that employs picture-wise overfitting. CD is built upon the Cool-chic video framework [1], but incorporates Decoupled representations for motion and residue. Additionally, we propose an effective training strategy for CD to further enhance its performance.

This study addresses the challenge of domain adaptation in learned image compression, focusing on shifting the model from natural images to user-generated content (UGC) domain. We propose a novel entropy adapter framework augmented with knowledge distillation techniques to improve performance. Unlike existing adapter-based methods that primarily enhance transformation modules, we identify the mismatch between the adapter-based transformation and the fixed entropy network. To resolve this, we introduce adapters within the hypernet and entropy model. Specifically, our decoupled entropy adapter features a deeper residual structure with two independent branches, enabling a separate refinement of mean and scale components. This design improves the accuracy of probability estimation and overall compression efficiency. To further enhance the effectiveness of the adapters, we incorporate a knowledge distillation (KD) strategy with a progressive loss function. It facilitates a smooth transition from KD loss to a rate-distortion (RD) loss in the training process, effectively transferring knowledge from a directly fine-tuned model to the student model. Consequently, this strengthens the adapter's learning capability and improves compression performance. Experimental results show that the proposed method achieves a significant 11.5% bitrate savings compared to the baseline model. Additionally, it demonstrates robust adaptability across diverse network architectures.

Resampling-based video coding is a well-established strategy for compressing high-resolution video under limited bandwidth. Recently, super resolution (SR) technologies has been explored for upsampling the low-resolution reconstructed video in the decoder side. Despite the promising rate-distortion improvements these SR model can offer, the significant complexity of neural network makes them impractical for real-world applications. In this paper, we propose a low complexity SR model for resampling-based video coding by applying CP decomposition to vanilla convolution to reduce complexity. To enhance the performance of the proposed low complexity SR, we utilize multiple decoded information generated during compression as additional auxiliary input of SR. This information provides valuable insights of the compressed picture, such as the texture and directional details. Additionally, to minimize the number of parameters in the neural network, a single model with split branches is designed to process luma and chroma components separately. Experimental results demonstrate that the proposed method achieves 3.72% and 3.65% BD-rate savings under random access and all-intra configurations compared to the latest video coding standard VVC/H.266, respectively. Compared to existing works, the proposed SR model exhibits a superior trade-off between complexity and coding performance.

This paper explores the application of enhancement filtering techniques in neural video compression. Specifically, we categorize these techniques into in-loop contextual filtering and out-of-loop reconstruction enhancement based on whether the enhanced representation affects the subsequent coding loop. In-loop contextual filtering refines the temporal context by mitigating error propagation during frame-by-frame encoding. However, its influence on both the current and subsequent frames poses challenges in adaptively applying filtering throughout the sequence. To address this, we introduce an adaptive coding decision strategy that dynamically determines filtering application during encoding. Additionally, out-of-loop reconstruction enhancement is employed to refine the quality of reconstructed frames, providing a simple yet effective improvement in coding efficiency. To the best of our knowledge, this work presents the first systematic study of enhancement filtering in the context of conditional-based neural video compression. Extensive experiments demonstrate a 7.71% reduction in bit rate compared to state-of-the-art neural video codecs, validating the effectiveness of the proposed approach.

Neural video codecs (NVCs), leveraging the power of end-to-end learning, have demonstrated remarkable coding efficiency improvements over traditional video codecs. Recent research has begun to pay attention to the quality structures in NVCs, optimizing them by introducing explicit hierarchical designs. However, less attention has been paid to the reference structure design, which fundamentally should be aligned with the hierarchical quality structure. In addition, there is still significant room for further optimization of the hierarchical quality structure. To address these challenges in NVCs, we propose EHVC, an efficient hierarchical neural video codec featuring three key innovations: (1) a hierarchical multi-reference scheme that draws on traditional video codec design to align reference and quality structures, thereby addressing the reference-quality mismatch; (2) a lookahead strategy to utilize an encoder-side context from future frames to enhance the quality structure; (3) a layer-wise quality scale with random quality training strategy to stabilize quality structures during inference. With these improvements, EHVC achieves significantly superior performance to the state-of-the-art NVCs. Code will be released in: https://github.com/bytedance/NEVC.

The past decade has witnessed the huge success of deep learning in well-known artificial intelligence applications such as face recognition, autonomous driving, and large language model like ChatGPT. Recently, the application of deep learning has been extended to a much wider range, with Neural Network-Based Video Coding (NNVC) being one of them. NNVC can be performed at two different levels: embedding neural network-based (NN-based) coding tools into a classical video compression framework or building the entire compression framework upon neural networks. This article elaborates our studies in response to the recent exploration efforts in JVET (Joint Video Experts Team of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC29) in the name of NNVC, falling in the former category. Specifically, in this article, we propose two advanced NN-based video coding technologies, i.e., NN-based intra prediction and NN-based in-loop filtering, which have been investigated for several meeting cycles in JVET and then adopted into the reference software, i.e., NNVC. In addition, we further propose a Small Ad-hoc Deep-Learning Library (SADL), which provides integer-based inference capabilities for neural networks to ensure interoperability across different systems. SADL has been adopted as the inference platform of all neural networks in NNVC. Extensive experiments on top of the NNVC have been conducted to evaluate the effectiveness of the proposed techniques. Compared with VTM-11.0_nnvc, the proposed two NN-based coding tools jointly achieve {11.94%, 21.86%, 22.59%}, {9.18%, 19.76%, 20.92%}, and {10.63%, 21.56%, 23.02%} BD-rate reductions on average for {Y, Cb, Cr} under random-access, low-delay, and all-intra configurations, respectively.

In this paper, a hybrid video compression framework is proposed that serves as a demonstrative showcase of deep learning-based approaches extending beyond the confines of traditional coding methodologies. The proposed hybrid framework is founded upon the Enhanced Compression Model (ECM), which is a further enhancement of the Versatile Video Coding (VVC) standard. We have augmented the latest ECM reference software with well-designed coding techniques, including block partitioning, deep learning-based loop filter, and the activation of block importance mapping (BIM) which was integrated but previously inactive within ECM, further enhancing coding performance. We evaluate the coding performance of the proposed framework with extensive experiments on the JVET dataset compared with ECM10.0 and VTM-11.0. Due to the testing environment and the coding complexity of the ECM, we did not conduct testing on Class A. The QPs are set as 22, 27, 32, 37, and 42. Compared with ECM-10.0, our method achieves 6.26%, 13.33%, and 12.33% BD-rate savings for the Y, U, and V components under random access (RA) configuration. The traditional hybrid coding framework combined with the three coding tools can further improve compression efficiency and has great potential for performance improvement.

Prototyping serves as a critical phase in the industrial conceptual design process, enabling exploration of problem space and identification of solutions. Recent advancements in large-scale generative models have enabled AI to become a co-creator in this process. However, designers often consider generative AI challenging due to the necessity to follow computer-centered interaction rules, diverging from their familiar design materials and languages. Physical prototype is a commonly used design method, offering unique benefits in prototype process, such as intuitive understanding and tangible testing. In this study, we propose ProtoDreamer, a mixed-prototype tool that synergizes generative AI with physical prototype to support conceptual design. ProtoDreamer allows designers to construct preliminary prototypes using physical materials, while AI recognizes these forms and vocal inputs to generate diverse design alternatives. This tool empowers designers to tangibly interact with prototypes, intuitively convey design intentions to AI, and continuously draw inspiration from the generated artifacts. An evaluation study confirms ProtoDreamer’s utility and strengths in time efficiency, creativity support, defects exposure, and detailed thinking facilitation.

Resampling-based video coding strategy is well-known for compressing high-resolution video with limited bandwidth. To keep the high frequency texture in chroma components, we propose a luma-only resampling-based video coding method, which includes three aspects. Firstly, the luma component is downsampled while the chroma components remain unchanged in the pre-processing module. Secondly, a convolutional neural network-based (CNN) module is adopted to remove coding artifacts and restore the reconstructed frame. Finally, a single channel based super resolution method, which utilizes CNN as basic structure, is proposed to restore the original color format. Extensive experimental results demonstrate the benefits of the proposed method. Compared with VTM-11.0, the proposed method achieves {8.63%, 7.82%, 8.34%} BD-rate reductions on 4K sequences for Y, U, and V components under all-intra configuration, respectively.

Down-sampling followed by an up-sampling is a well-known strategy to compress high-resolution pictures given a limited bandwidth in image as well as video coding. Recently, inspired by the latest advances of image super resolution (SR) technologies using convolutional neural network (CNN), CNN-based SR has been explored for resampling-based coding. However, the side information generated during the compression process is not utilized efficiently by the SR network in prior arts. In this paper, we propose a CNN-based SR method for video coding, where more side information is leveraged as a supplement to reconstruction samples. Specifically, we introduce prediction samples to be the auxiliary information, as it can provide the texture and directional information about the original picture. Considering the different characteristics, we design different networks for the luma and chroma components. When designing the chroma up-sampling CNN, the luma reconstruction is used as the auxiliary information of the chroma network, which exploits the cross-component correlation. Experimental results show that the proposed method achieves 11.07% BD-rate savings in all-intra configuration compared with VTM-11.0. Further experiments validate the effectiveness of using luma information to aid the chroma up-sampling process.

Recently, deep learning has been explored as a promising direction for image compression. Removing the spatial redundancy of the image is crucial for image compression and most learning based methods focus on removing the redundancy between adjacent pixels. Intuitively, to explore larger pixel range beyond adjacent pixel is beneficial for removing the redundancy. In this paper, we propose a fast yet effective method for end-to-end image compression by incorporating a novel spatial recurrent neural network. Block based LSTM is utilized to remove the redundant information between adjacent pixels and blocks. Besides, the proposed method is a potential efficient system that parallel computation on individual blocks is possible. Experimental results demonstrate that the proposed model outperforms state-of-the-art traditional image compression standards and learning based image compression models in terms of both PSNR and MS-SSIM metrics. It provides a 26.73% bits-reduction than High Efficiency Video Coding (HEVC), which is the current official state-of-the-art video codec.

Recently, the deep learning neural networks (DNNs) based filters have demonstrated their advantages to remove artifacts or improve the performance in the area of image/video coding. However, the existing DNNs-based filters can handle intra coding distortion or quality enhancement approaches, thus not suitable for inter frames without the Rate Distortion Optimization (RDO) strategy in encoder side, which is not desirable for the implementation on hardware since all the existing filters need to be executed to make the optimal choice for the current content in encoder side. Also the RDO algorithm can only make the local optimal choice, which does not represent the global optimum, because of the long-term dependency between the inter frames. In this paper, we propose an in-loop filter for inter frames to completely replace all the conventional filters in the codec, including de-blocking filter (DB), bilateral filter (BF), adaptive loop filter (ALF) and SAO (Sample Adaptive Offset) without the RDO strategy in encoder side. Moreover, the greedy heuristic is adopted to produce the optimal filter weights which approximates the global optimal solution in each stage during the off-line training. In the experiments, our method reduces the average BD-rate by 2.77%, 7.01%, 8.64% for luma and both chroma components with Random Access (RA) configuration by using only one set of parameters to handle multiple distortion and decrease the consumption of memory.

Vehicle type recognition (VTR) is a quite common requirement and one of the key challenges in real surveillance scenarios, such as intelligent traffic and unmanned driving. Usually coarse-grained and fine-grained VTRs are applied in different applications, and the challenge from multiple viewpoints is critical for both cases. In this paper, we propose a feedback-enhancement multi-branch CNN (FM-CNN) to solve the challenge in these two cases. The proposed FM-CNN takes three derivatives of an image as input and leverages the advantages of hierarchical details, feedback enhancement, model average, and stronger robustness to translation and mirroring. A single global cross-entropy loss is insufficient to train such a complex CNN and so we add extra branch losses to enhance feedbacks to each branch. Though reusing pre-trained parameters, we propose a novel parameter update method to adapt FM-CNN to task-specific local visual patterns and global information in new datasets. To test the effectiveness of FM-CNN, we create our own multi-view VTR (MVVTR) data set since there are no such data sets available. And, for fine-grained VTR, we use the CompCars data set. Compared with state-of-the-art classification solutions without special preprocessing, the proposed FM-CNN demonstrates better performance in both coarse-grained and fine-grained scenarios. For coarse-grained VTR, it achieves 94.9% Top-1 accuracy on the MVVTR data set. For fine-grained VTR, it achieves 91.0% Top-1 and 97.8% Top-5 accuracies on the CompCars data set.

As the demand and supply for 3D technologies grows, the quality assessment of stereoscopic content is both demanding and increasingly urgent. This paper proposes a novel 3D images quality assessment (IQA) algorithm which is based on the Binocular Rivalry model and the Summation and Difference model. Both the influence of distorted disparity information on perceptual quality and the binocular rivalry mechanism of HVS are considered in our algorithm. Since disparity map is not utilized in our algorithm, it saves the complex computation of disparity map and improves the robustness of our algorithm. Experiment results shows that our algorithm outperforms well-known 3D IQA metrics on LIVE database and delivers competitive performance on Waterloo database. The cross database validation indicates that our algorithm is stable on different database.