We present LARP, a novel video tokenizer designed to overcome limitations in current video tokenization methods for autoregressive (AR) generative models. Unlike traditional patchwise tokenizers that directly encode local visual patches into discrete tokens, LARP introduces a holistic tokenization scheme that gathers information from the visual content using a set of learned holistic queries. This design allows LARP to capture more global and semantic representations, rather than being limited to local patch-level information. Furthermore, it offers flexibility by supporting an arbitrary number of discrete tokens, enabling adaptive and efficient tokenization based on the specific requirements of the task. To align the discrete token space with downstream AR generation tasks, LARP integrates a lightweight AR transformer as a training-time prior model that predicts the next token on its discrete latent space. By incorporating the prior model during training, LARP learns a latent space that is not only optimized for video reconstruction but is also structured in a way that is more conducive to autoregressive generation. Moreover, this process defines a sequential order for the discrete tokens, progressively pushing them toward an optimal configuration during training, ensuring smoother and more accurate AR generation at inference time. Comprehensive experiments demonstrate LARPs strong performance, achieving state-of-the-art FVD on the UCF101 class-conditional video generation benchmark. LARP enhances the compatibility of AR models with videos and opens up the potential to build unified high-fidelity multimodal large language models (MLLMs). Project page: https://hywang66.github.io/larp/

Clustering is pivotal in deciphering cellular heterogeneity in single-cell RNA sequencing (scRNA-seq) data. However, it suffers from several challenges in handling the high dimensionality and complexity of scRNA-seq data. Especially when employing graph neural networks (GNNs) for cell clustering, the dependencies between cells expand exponentially with the number of layers. This results in high computational complexity, negatively impacting the model's training efficiency. To address these challenges, we propose a novel approach, called decoupled GNNs, based on multi-view contrastive learning (scDeGNN), for scRNA-seq data clustering. Firstly, this method constructs two adjacency matrices to generate distinct views, and trains them using decoupled GNNs to derive the initial cell feature representations. These representations are then refined through a multilayer perceptron and a contrastive learning layer, ensuring the consistency and discriminability of the learned features. Finally, the learned representations are fused and applied to the cell clustering task. Extensive experimental results on nine real scRNA-seq datasets from various organisms and tissues show that the proposed scDeGNN method significantly outperforms other state-of-the-art scRNA-seq data clustering algorithms across multiple evaluation metrics.

Diversified image color editing is typically modeled as a multimodal image-to-image translation (MMI2IT) problem with an impact on multiple applications such as photo enhancement and retouching. Although previous GAN-based algorithms successfully generate diverse edits with clear control, we observe two issues remaining: Firstly, they tend to apply the same color style to all kinds of input images when sampling with the same style latent, regardless of the input content and scenes. Secondly, they usually edit the color style globally in an image and fail to keep each semantic region and instance in harmonic colors individually. We attribute these issues to the strong independence between the style latent and the condition image in most current MMI2IT methods.To edit the raw image into a more harmonic direction with awareness of its global content and local semantics, we introduce auxiliary color restoration tasks by reducing the input color information and training jointly. We also increase the model’s capacity and enrich the noise’s locality with diffusion models. Furthermore, we propose a new set of metrics to measure the content-awareness of MMI2IT models, that is, how the generated style is adaptive to the input image’s content. Our model is also extensible to several downstream applications including exemplar-based color editing and language-guided color editing, without imposing extra demands on the already trained model.

Implicit neural representations (INR) have gained increasing attention in representing 3D scenes and images, and have been recently applied to encode videos (e.g., NeRV [1], E-NeRV [2]). While achieving promising results, existing INR-based methods are limited to encoding a handful of short videos (e.g., seven 5-second videos in the UVG dataset) with redundant visual content, leading to a model design that fits individual video frames independently and is not efficiently scalable to a large number of diverse videos. This paper focuses on developing neural representations for a more practical setup - encoding long and/or a large number of videos with diverse visual content. We first show that instead of dividing videos into small subsets and encoding them with separate models, encoding long and diverse videos jointly with a unified model achieves better compression results. Based on this observation, we propose D-NeRV, a novel neural representation framework designed to encode diverse videos by (i) decoupling clip-specific visual content from motion information, (ii) introducing temporal reasoning into the implicit neural network, and (iii) employing the task-oriented flow as intermediate output to reduce spatial redundancies. Our new model largely surpasses NeRV and traditional video compression techniques on UCF101 and UVG datasets on the video compression task. Moreover, when used as an efficient data-loader, D-NeRV achieves 3%-10% higher accuracy than NeRV on action recognition tasks on the UCF101 dataset under the same compression ratios.

The current privacy-preserving Graph Neural Networks (GNNs) cannot provide security and privacy guarantees against malicious adversaries without sacrificing accuracy and efficiency. For example, the Secure Multi-party Computation (MPC) can resist malicious adversaries while adding severe overhead. Trusted Execution Environment (TEE), such as Intel Software Guard Extension (SGX), can guarantee privacy and faithful execution without compromising efficiency. However, existing attacks can compromise the confidentiality of SGXs. Besides, the CPU-based structure of SGX restricts its extensibility that cannot perform collaborative computation with GPUs. To address the above issues, we propose a novel GNN training and inference framework to support data holders outsourcing their computation tasks to servers. First, we combine the advantage of MPC and the code integrity protection provided by SGXs to resist malicious adversaries without sacrificing efficiency. Second, we adopt a strategy that allows the servers to transfer the parallelizable computation task to the untrusted yet high-performance GPUs, further improving efficiency without hindering privacy. To the best of our knowledge, our proposal is the first privacy-preserving GNN framework against malicious adversaries without sacrificing accuracy and efficiency. Experiments on real-world citation datasets have demonstrated the performance of our framework regarding security, privacy, accuracy, and efficiency.

We propose a novel approach for multi-modal Image-to-image (I2I) translation. To tackle the one-to-many relationship between input and output domains, previous works use complex training objectives to learn a latent embedding, jointly with the generator, that models the variability of the output domain. In contrast, we directly model the style variability of images, independent of the image synthesis task. Specifically, we pre-train a generic style encoder using a novel proxy task to learn an embedding of images, from arbitrary domains, into a low-dimensional style latent space. The learned latent space introduces several advantages over previous traditional approaches to multi-modal I2I translation. First, it is not dependent on the target dataset, and generalizes well across multiple domains. Second, it learns a more powerful and expressive latent space, which improves the fidelity of style capture and transfer. The proposed style pre-training also simplifies the training objective and speeds up the training significantly. Furthermore, we provide a detailed study of the contribution of different loss terms to the task of multi-modal I2I translation, and propose a simple alternative to VAEs to enable sampling from unconstrained latent spaces. Finally, we achieve state-of-the-art results on six challenging benchmarks with a simple training objective that includes only a GAN loss and a reconstruction loss.

We propose a novel neural representation for videos (NeRV) which encodes videos in neural networks. Unlike conventional representations that treat videos as frame sequences, we represent videos as neural networks taking frame index as input. Given a frame index, NeRV outputs the corresponding RGB image. Video encoding in NeRV is simply fitting a neural network to video frames and decoding process is a simple feedforward operation. As an image-wise implicit representation, NeRV output the whole image and shows great efficiency compared to pixel-wise implicit representation, improving the encoding speed by $\textbf{25}\times$ to $\textbf{70}\times$, the decoding speed by $\textbf{38}\times$ to $\textbf{132}\times$, while achieving better video quality. With such a representation, we can treat videos as neural networks, simplifying several video-related tasks. For example, conventional video compression methods are restricted by a long and complex pipeline, specifically designed for the task. In contrast, with NeRV, we can use any neural network compression method as a proxy for video compression, and achieve comparable performance to traditional frame-based video compression approaches (H.264, HEVC \etc). Besides compression, we demonstrate the generalization of NeRV for video denoising. The source code and pre-trained model can be found at https://github.com/haochen-rye/NeRV.git.

The Multi-party Secure Computation (MPC)-based methods for privacy-preserving Graph Neural Networks (GNNs) are still challenged by high communication overhead. Moreover, the security guarantee of most MPC-based methods can only defend against the semi-honest adversary, while a few methods which can defend against the malicious adversary will cause a further increase in communication overhead. Moreover, Software Guard Extensions (SGX), which can provide the data confidentiality and code integrity, has been considered as a novel solution to privacy-preserving GNN. Unfortunately, previous work has shown that SGX is vulnerable to side-channel attacks that deprive its confidentiality and preserve only its integrity. To solve the above problems, we propose an n-party secure computation framework for GNNs using SGX. This framework can reduce the communication overhead and improve the security guarantee without relying on the confidentiality of SGX. Specifically, both data holders and the server hold SGX. Data holders enrich the data and train the model by MPC efficiently with the assistance of the server. SGX ensures integrity, where data holders and the server must execute according to protocols, so malicious adversaries cannot deviate from the protocol to breach privacy and security. Even if the confidentiality of SGX was breached, the adversary could only access the ciphertext in MPC instead of the plaintext. We conduct experiments on public datasets to demonstrate that our framework has achieved comparable performance with traditional GNNs and perform security analysis to validate that our framework satisfies security and privacy requirements.