Diffusion models have emerged as mainstream framework in visual generation. Building upon this success, the integration of Mixture of Experts (MoE) methods has shown promise in enhancing model scalability and performance. In this paper, we introduce Race-DiT, a novel MoE model for diffusion transformers with a flexible routing strategy, Expert Race. By allowing tokens and experts to compete together and select the top candidates, the model learns to dynamically assign experts to critical tokens. Additionally, we propose per-layer regularization to address challenges in shallow layer learning, and router similarity loss to prevent mode collapse, ensuring better expert utilization. Extensive experiments on ImageNet validate the effectiveness of our approach, showcasing significant performance gains while promising scaling properties.

In this paper, we propose a simple yet effective transformer framework for self-supervised learning called DenseDINO to learn dense visual representations. To exploit the spatial information that the dense prediction tasks require but neglected by the existing self-supervised transformers, we introduce point-level supervision across views in a novel token-based way. Specifically, DenseDINO introduces some extra input tokens called reference tokens to match the point-level features with the position prior. With the reference token, the model could maintain spatial consistency and deal with multi-object complex scene images, thus generalizing better on dense prediction tasks. Compared with the vanilla DINO, our approach obtains competitive performance when evaluated on classification in ImageNet and achieves a large margin (+7.2% mIoU) improvement in semantic segmentation on PascalVOC under the linear probing protocol for segmentation.

Fake faces produced by deepfake techniques have attracted public concerns in recent years. Deepfake detection is a binary classification task that distinguishes fake faces from real ones. As the training data for deepfake detection is usually generated from real faces via various face forgery methods, it is difficult for a single binary decision boundary to distinguish fake faces. Besides that, the learned features often involve irrelevant information for identifying fake faces that are generated from diverse forgery methods. To deal with such challenges, unlike existing approaches that regard fake detection as a binary classification, we re-model the task as a multiclass forgery-domain classification task, where each forgery method is treated as a distinct class. This simplifies the complex decision boundary brought by the diversity of forgery patterns and provides more forgery-relevant information for the learning process. In addition, we introduce a non-negative constrained learning framework composed of non-negative features and a non-negative constrained classifier (NCC) to block irrelevant features with zero weights and enhance forgery-relevant features with positive weights, leading to a sparse structure of the classifier. Furthermore, to capture subtle and discriminative forgery-relevant features, we propose an integration module over augmented faces based on cross-attention. We demonstrate that our approach achieves competitive performance and generalization ability on widely-used benchmarks through extensive experiments.