Existing retrieval-based methods have made significant strides in maintaining long-term conversations. However, these approaches face challenges in memory database management and accurate memory retrieval, hindering their efficacy in dynamic, real-world interactions. This study introduces a novel framework, COmpressive Memory-Enhanced Dialogue sYstems (COMEDY), which eschews traditional retrieval modules and memory databases. Instead, COMEDY adopts a “One-for-All” approach, utilizing a single language model to manage memory generation, compression, and response generation. Central to this framework is the concept of compressive memory, which integrates session-specific summaries, user-bot dynamics, and past events into a concise memory format. To support COMEDY, we collect the biggest Chinese long-term conversation dataset, Dolphin, derived from real user-chatbot interactions. Comparative evaluations demonstrate COMEDY’s superiority over traditional retrieval-based methods in producing more nuanced and human-like conversational experiences.

As generative technologies advance, visual content has evolved into a complex mix of natural and AI-generated images, driving the need for more efficient coding techniques that prioritize perceptual quality. Traditional codecs and learned methods struggle to maintain subjective quality at high compression ratios, while existing generative approaches face challenges in visual fidelity and generalization. To this end, we propose a novel generative coding framework leveraging diffusion priors to enhance compression performance at low bitrates. Our approach emplopys a pre-optimized encoder to generate generalized compressed-domain representations, integrated with pretrained model’s internal features via a lightweight adapter and an attentive fusion module. This framework effectively leverages existing pretrained diffusion models and enables efficient adaptation to different pretrained models for new requirements with minimal retraining costs. We also introduce a distribution renormalization method to further enhance reconstruction fidelity. Extensive experiments show that our method: (1) outperforms existing methods in visual fidelity across low bitrates, (2) improves compression performance by up to 79% over H.266/VVC, and (3) offers an efficient solution for AI-generated content while being adaptable to broader content types.

In this age of information, images are a critical medium for storing and transmitting information. With the rapid growth of image data amount, visual compression and visual data perception are two important research topics attracting a lot of attention. However, those two topics are rarely discussed together and follow separate research paths. Due to the compact compressed domain representation offered by learning-based image compression methods, there exists possibility to have one stream targeting both efficient data storage and compression, and machine perception tasks. In this paper, we propose a layered generative facial image compression model achieving high human vision-oriented image reconstructed quality, even at extreme compression ratios. To obtain analysis efficiency and flexibility, a task-agnostic learning-based compression model is proposed, which effectively supports various compressed domain-based analytical tasks while preserving outstanding reconstructed perceptual quality, compared with traditional and learning-based codecs. In addition, joint optimization schedule is adopted to acquire best balance point among compression ratio, reconstructed image quality, and downstream perception performance. Experimental results verify that our proposed compressed domain-based multi-task analysis method can achieve comparable analysis results against the RGB image-based methods with up to 99.6% bit rate saving (i.e., compared with taking original RGB image as the analysis model input). The practical ability of our model is further justified from model size and information fidelity aspects.

Utilizing large language models (LLMs) for data augmentation has yielded encouraging results in mathematical reasoning.However, these approaches face constraints in problem diversity, potentially restricting them to indomain/distribution data generation.To this end, we propose ControlMath, an iterative method involving an equation-generator module and two LLM-based agents.The module creates diverse equations, which the Problem-Crafter agent then transforms into math word problems.The Reverse-Agent filters and selects high-quality data, adhering to the "less is more" principle, achieving better results with fewer data points.This approach enables the generation of diverse math problems, not limited to specific domains or distributions.As a result, we collect ControlMathQA, which involves 190k math word problems.Extensive results prove that combining our dataset with indomain datasets like GSM8K can help improve the model's mathematical ability to generalize, leading to improved performances both within and beyond specific domains.

End-to-end image compression has achieved satisfactory results in recent studies. However, existing methods suffer from high complexity of complicated neural network computation and cannot be directly deployed on mobile devices due to the limitations of computing ability and storage. Therefore, considering the resource and computing ability constrains of the mobile devices, we make a trade-off in this paper between rate-distortion (R-D) performance, inference time, and model complexity. Then we design a novel lightweight perceptual image compression framework to alleviate the storage and complexity burden of mobile devices. Moreover, we design a hardware-friendly deployment scheme to apply the proposed compression framework on high-end mobile devices, which can achieve efficient image compression. Based on the above structures, we propose the first mobile system that achieves image compression on mobile devices. The supplementary material of our system demo is on https://sigport.org/documents/extreme-low-bitrate-Image-compression-system-mobile-deployment.

Nowadays, as critical conduits of communication, the information security of images and videos is particularly important. The existing encryption techniques usually transform images into high-frequency content that resembles noise, pre-senting significant challenges in achieving efficient compression. This paper presents an innovative collaborative approach that integrates image encryption and compression using a reversed diffusion model. This method, by reversing the typical process of diffusion models, adeptly changes encrypted high-frequency content into a domain that is more amenable to compression. Leveraging the reversible nature of the Denoising Diffusion Implicit Models (DDIM), our framework ensures the high-fidelity restoration of information. Our experimental findings demonstrate that this approach not only effectively encrypts images but also compresses the encrypted high-frequency noise content, outperforming Video Versatile Coding (VVC) in compression performance.

Nuo Chen, Linjun Shou, Jian Pei, Ming Gong, Bowen Cao, Jianhui Chang, Jia Li, Daxin Jiang. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Providing high video quality under the lowest possible bitrate constraint is one of the critical challenges in video coding technology. Inspired by the continuous development of motion imitation [1], the model-based video coding method is derived from extracting a series of features or parameters representing the person's motion and reconstructing each frame by motion imitation model at the decoder. Thus, we propose a Semantic Neural Rendering-based Video Coding framework (SNRVC) to transmit video at ultra-low bitrate while maintaining high subjective quality. At the encoder, we first extract the motion parameters with specific semantic meanings from each frame and then compress the first frame and the parameters of the subsequent frames by truncating to different decimals and differential pulse code modulation coding. Finally, the decoded image and parameters are fed into the motion imitator [2] to obtain each reconstructed frame consistent with the movements of the original frame. Our SNRVC can achieve better visual quality than traditional and model-based methods [3] at the ultra-low bitrate below 0.01 bpp.

Image compression approaches based on deep learning have achieved remarkable success. Existing studies mainly focus on human vision and machine analysis tasks taking reconstructed images as input. However, those methods need images to be decoded before performing downstream visual tasks, which motivates us to explore how to directly conduct visual analysis using the compressed data without decoding. The overview of our proposed model is shown as Fig. 1(a). Specifically, a task-agnostic learning-based compression model is proposed, which effectively supports various compressed domain-based analytical tasks meanwhile reserves outstanding re-constructed perceptual quality compared with traditional and learning-based codecs. To obtain the extremely compacted data representation with essential semantic infor-mation, we take the help of the generative model on decoder part. Then, we propose a multi-task learning model which can directly obtain semantic information from the compressed visual data. The pipeline of the proposed model is detailedly illus-trated in Fig. 1(b). In addition, joint optimization strategy is adopted to achieve the best balance point among compression efficiency, reconstructed image quality, and the downstream visual tasks' performance. Experimental results verify that our proposed compressed domain-based multi-task analysis model outperforms the reconstructed image-based method on transmission efficiency, saving more than ten times of bit-rate consumption while preserving comparable visual analysis precision (i.e., classification and segmentation tasks) when compared with RGB image input models, which is evaluated on the CelebA-HO dataset.

As an emerging compression scheme, conceptual coding usually encodes images into structural and textural representations and decodes them in a deep synthesis fashion. However, existing conceptual coding schemes ignore the structure of deep texture representation space, leading to a challenge of establishing efficient and faithful conceptual representations. In this paper, we firstly introduce contrastive learning into conceptual coding and propose Consistency-Contrast Learning (CCL) which optimizes the representation space by a consistency-contrast regularization. By modeling the original images and reconstructed images as "positive'' pairs and random images in a batch as "negative'' samples, CCL aims to align texture representation space with source images space relatively. Extensive experiments on diverse datasets demonstrate that: (1) the proposed CCL can achieve the best compression performance on the conceptual coding task; (2) CCL is superior to other popular regularization methods towards improving reconstruction quality; (3) CCL is general and can be applied to other tasks related to representation optimization and image reconstruction, such as GAN inversion.

Existing compression methods typically focus on the removal of signal-level redundancies, while the potential and versatility of decomposing visual data into compact conceptual components still lack further study. To this end, we propose a novel conceptual compression framework that encodes visual data into compact structure and texture representations, then decodes in a deep synthesis fashion, aiming to achieve better visual reconstruction quality, flexible content manipulation, and potential support for various vision tasks. In particular, we propose to compress images by a dual-layered model consisting of two complementary visual features: 1) structure layer represented by structural maps and 2) texture layer characterized by low-dimensional deep representations. At the encoder side, the structural maps and texture representations are individually extracted and compressed, generating the compact, interpretable, inter-operable bitstreams. During the decoding stage, a hierarchical fusion GAN (HF-GAN) is proposed to learn the synthesis paradigm where the textures are rendered into the decoded structural maps, leading to high-quality reconstruction with remarkable visual realism. Extensive experiments on diverse images have demonstrated the superiority of our framework with lower bitrates, higher reconstruction quality, and increased versatility towards visual analysis and content manipulation tasks.

Conceptual coding has been an emerging research topic recently, which encodes natural images into disentangled conceptual representations for compression. However, the compression performance of the existing methods is still suboptimal due to the lack of comprehensive consideration of rate constraint and reconstruction quality. To this end, we propose a novel end-to-end semantic prior modeling based conceptual coding scheme towards extremely low bitrate image compression, which leverages semantic-wise deep representations as a unified prior for entropy estimation and texture synthesis. Specifically, we employ semantic segmentation maps as structural guidance for extracting deep semantic prior, which provides fine-grained texture distribution modeling for better detail construction and higher flexibility in subsequent high-level vision tasks. Moreover, a cross-channel entropy model is proposed to further exploit the inter-channel correlation of the spatially independent semantic prior, leading to more accurate entropy estimation for rate-constrained training. The proposed scheme achieves an ultra-high 1000× compression ratio, while still enjoying high visual reconstruction quality and versatility towards visual processing and analysis tasks.

Motivated by the insight of Marr on generative image representations, we propose a layered conceptual image compression scheme by integrating the advantages of both variational auto-encoders (VAEs) and generative adversarial networks (GANs). In particular, the image is represented by two layers: the low-dimensional codes of the stochastic textures encoded by the VAE and the geometric structures characterized by edge maps. Subsequently, the edge maps and latent codes are compressed individually such that the final bit streams are formed in a combined manner. At the decoder side, the GAN synthesizes the decoded images on the basis of the latent codes and the reconstructed edge maps. Experimental results demonstrate that our proposed scheme achieves better visual reconstruction quality than the traditional image compression algorithms such as JPEG, JPEG2000 and HEVC (intra coding) in the low bit rate coding scenarios.