The diagnosis of cancer primarily relies on pathological slides stained with hematoxylin and eosin (HE). These slides are typically prepared from tissue samples that have been fixed in formalin and embedded in paraffin (FFPE). However, the traditional process of staining FFPE samples with HE is time-consuming and resource-intensive. Recent advances in virtual staining technologies, driven by digital pathology and generative models, offer a promising alternative. However, the blurred structures in FFPE images pose unique challenges to achieving high-quality FFPE-to-HE virtual staining. In this context, we developed a novel Multiple Cell Semantics-guided supervised generative adversarial model, MCS-Stain. Specifically, the guidance consists of three components: 1) pretrained cell semantic guidance, aligning the powerful intermediate features of real and virtual images, embedded in the pretrained cell segmentation model (PCSM); 2) cell mask guidance, introducing comprehensible cell information which serves as part of the input to the discriminator through channel concatenation; 3) dynamic cell semantic guidance, aligning the dynamic intermediate features embedded in the generator during training. The comparative results on FFPE-to-HE datasets demonstrated that MCS-Stain outperforms existing state-of-the-art (SOTA) methods with substantial qualitative and quantitative improvements. Results across various PCSMs and data sources further confirmed its effectiveness and robustness. Notably, the dynamic cell semantic exhibits strong potential beyond FFPE-to-HE virtual staining, further demonstrated by virtual staining from HE images to immunohistochemical (IHC) images. In general, MCS-Stain presents a promising avenue to advance virtual staining techniques. Code is available at https://github.com/huyihuang/MCS-Stain.

Diffusion models have demonstrated superior performance in portrait animation. However, current approaches relied on either visual or audio modality to control character movements, failing to exploit the potential of mixed-modal control. This challenge arises from the difficulty in balancing the weak control strength of audio modality and the strong control strength of visual modality. To address this issue, we introduce MegActor-Sigma: a mixed-modal conditional diffusion transformer (DiT), which can flexibly inject audio and visual modality control signals into portrait animation. Specifically, we make substantial advancements over its predecessor, MegActor, by leveraging the promising model structure of DiT and integrating audio and visual conditions through advanced modules within the DiT framework. To further achieve flexible combinations of mixed-modal control signals, we propose a ``Modality Decoupling Control" training strategy to balance the control strength between visual and audio modalities, along with the ``Amplitude Adjustment" inference strategy to freely regulate the motion amplitude of each modality. Finally, to facilitate extensive studies in this field, we design several dataset evaluation metrics to filter out public datasets and solely use this filtered dataset for training. Extensive experiments demonstrate the superiority of our approach in generating vivid portrait animations.

Diffusion-based virtual staining methods of histopathology images have demonstrated outstanding potential for stain normalization and cross-dye staining (e.g., hematoxylin-eosin to immunohistochemistry). However, achieving pathology-correct cross-dye virtual staining with versatile tone controls poses significant challenges due to the difficulty of decoupling the given pathology and tone conditions. This issue would cause non-pathologic regions to be mistakenly stained like pathologic ones, and vice versa, which we term “pathology leakage.” To address this issue, we propose diffusion virtual staining Transformer (D-VST), a new framework with versatile tone control for cross-dye virtual staining. Specifically, we introduce a pathology encoder in conjunction with a tone encoder, combined with a two-stage curriculum learning scheme that decouples pathology and tone conditions, to enable tone control while eliminating pathology leakage. Further, to extend our method for billion-pixel whole slide image (WSI) staining, we introduce a novel frequency-aware adaptive patch sampling strategy for high-quality yet efficient inference of ultra-high resolution images in a zero-shot manner. Integrating these two innovative components facilitates a pathology-correct, tone-controllable, cross-dye WSI virtual staining process. Extensive experiments on three virtual staining tasks that involve translating between four different dyes demonstrate the superiority of our approach in generating high-quality and pathologically accurate images compared to existing methods based on generative adversarial networks and diffusion models. Our code and trained models will be released.