Diffusion models achieved great success in image synthesis, but still face challenges in high-resolution generation. Through the lens of discrete cosine transformation, we find the main reason is that *the same noise level on a higher resolution results in a higher Signal-to-Noise Ratio in the frequency domain*. In this work, we present Relay Diffusion Model (RDM), which transfers a low-resolution image or noise into an equivalent high-resolution one for diffusion model via blurring diffusion and block noise. Therefore, the diffusion process can continue seamlessly in any new resolution or model without restarting from pure noise or low-resolution conditioning. RDM achieves state-of-the-art FID on CelebA-HQ and sFID on ImageNet 256$\times$256, surpassing previous works such as ADM, LDM and DiT by a large margin. All the codes and checkpoints are open-sourced at \url{https://github.com/THUDM/RelayDiffusion}.

Recovering three-dimensional (3D) structures such as object poses from limited two-dimensional (2D) information is an important research problem in computer vision, graphics, and robotics. The estimation of object pose from single images or multiple casual images could be ill-conditioned math problems. There is a popular family of algorithms of geometric sparse representation for 3D pose recovery (GSR-3D) that pretrains an overcomplete dictionary of 3D basis poses $B$, and then matches the detected 2D object pose $Y$ by jointly estimating the transformation $R$, projection $\Pi,$ and combination coefficients $c$, assuming $Y \approx \Pi \sum_i c_i R B_i$. In this paper, we make the first step of analyzing to which extent could we solve this ill-conditioned problem, and of understanding how the recovery error is affected by fundamental factors, e.g., dictionary size, observation noise, and running time. As these factors are implicit in objective functions, we analyze with the help of various sparse regularizers and a multistage optimizer, and prove that the recovery error $\mathcal L(l)$ decays w.r.t. the number of stages $l$ with a high probability, $Prob\left(\mathcal L(l) < \rho^{l-1} \mathcal L(0) + \delta \right) \geq 1- \epsilon$, where the constants $0< \rho <1, 0<\delta, 0<\epsilon \ll 1$ are related to the aforementioned factors. To the best of our knowledge, this is the first theoretical analysis in this line of research. Experiments are conducted to support our improvement upon previous regularization within the same framework. This will further characterize the trade-off between speed and accuracy towards real-time geometric inference in applications.

Generating diverse yet specific data is the goal of the generative adversarial network (GAN), but it suffers from the problem of mode collapse. We introduce the concept of normalized diversity which force the model to preserve the normalized pairwise distance between the sparse samples from a latent parametric distribution and their corresponding high-dimensional outputs. The normalized diversification aims to unfold the manifold of unknown topology and non-uniform distribution, which leads to safe interpolation between valid latent variables. By alternating the maximization over the pairwise distance and updating the total distance (normalizer), we encourage the model to actively explore in the high-dimensional output space. We demonstrate that by combining the normalized diversity loss and the adversarial loss, we generate diverse data without suffering from mode collapsing. Experimental results show that our method achieves consistent improvement on unsupervised image generation, conditional image generation and hand pose estimation over strong baselines.

Modern large-scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information such as stochastic gradients among different workers. In this paper, to reduce the communication cost, we propose a convex optimization formulation to minimize the coding length of stochastic gradients. The key idea is to randomly drop out coordinates of the stochastic gradient vectors and amplify the remaining coordinates appropriately to ensure the sparsified gradient to be unbiased. To solve the optimal sparsification efficiently, several simple and fast algorithms are proposed for an approximate solution, with a theoretical guarantee for sparseness. Experiments on $\ell_2$ regularized logistic regression, support vector machines, and convolutional neural networks validate our sparsification approaches.

Feature extraction is an important task in machine learning. In this paper, we present a simple and efficient method, named max-margin data shifting (MMDS), to process the data before feature extraction. By relying on a large-margin classifier, MMDS is helpful to enhance the discriminative ability of subsequent feature extractors. The kernel trick can be applied to extract nonlinear features from input data. We further analyze in detail the example of principal component analysis (PCA). The empirical results on multiple linear and nonlinear models demonstrate that MMDS can efficiently improve the performance of unsupervised extractors.