Aligning a pair of cross-modality images (e.g., MR-CT, CBCT-CT) is important, yet conventional approaches, including registration or image-to-image (I2I) translation methods often have limitations. To overcome these challenges, we introduce a "Register by Generation (RbG)" framework, a novel 2D deep learning approach designed to generate images that are structurally well-aligned with the fixed image while preserving the detailed intensity and contrast of the moving image, which we refer to as the reference image. Our approach operates in two sequential key stages: first, we employ a novel semi-global reference-augmented image synthesis network incorporating Patch Adaptive Instance Normalization (PAdaIN). This method leverages a down-sampled reference image to guide local adaptive synthesis, generating a more accurately aligned image with a reduced risk of hallucinations. In the second stage, we introduce a detailed refining reference-augmented network featuring a Deformation-Aware Cross-Attention (DACA) block, which aims to recover finer details and textures that may be missing from the initial stage. This unique component (DACA block) enables the transfer of corresponding relevant features from the reference image, effectively performing a "copy-and-paste" operation within the latent feature space. Additionally, we propose a novel combination of loss functions that enables self-supervised training on misaligned datasets, eliminating the need for pre-aligned data. We rigorously evaluate our method on multiple misaligned datasets using metrics focused on structural alignment and distributional consistency, demonstrating comprehensively superior performance. Furthermore, we test its robustness by simulating intentional misalignments in a well-aligned dataset. Additionally, experiments from a case study and downstream segmentation tasks highlight the broad applicability of our approach.

RegistFormer, our novel reference-augmented image synthesis framework, generates aligned pseudo-CT images (with respect to MR) from misaligned MR and CT pairs. RegistFormer addresses the limitations of intensity-based registration methods, which often fail due to dissimilar image features and complex deformation fields. Unlike conventional image-to-image (I2I) translation methods, our method uses a misaligned CT scan as an auxiliary input to guide the synthesis task through the Deformation-Aware Cross-Attention (DACA) mechanism. DACA integrates the deformation field from a registration method to aggregate spatially matched features from the misaligned CT into MR spatial coordinates. Additionally, we propose a novel combination of loss functions for training with datasets of misaligned MR-CT pairs in a self-supervised manner, eliminating the need for pre-aligned training data. Experiments were conducted with the synthRAD202311https://synthrad2023.grand-challenge.org/ MR-CT pelvis pair dataset. RegistFormer outperforms past state-of-the-art methods, including I2I, registration, and hybrid (registration + I2I), across metrics evaluating both structure alignment and distribution similarity. Moreover, RegistFormer demonstrates superior performance in zero-shot segmentation downstream tasks, highlighting its clinical value. Source code: https://github.com/danny4159/RegistFormer