Multi-camera systems have become extensively integrated into smartphones. One prominent implementation employs a color-monochrome dual-camera system, which synergistically combines data from both sensors to produce high-quality color images. The dual-camera system can capture asymmetric image pairs, consisting of relatively high-resolution monochrome images and low-resolution color images. Existing methods do not effectively address the issues of the inconsistency in spectral and spatial resolutions between color-monochrome image pairs in asymmetric color transfer. In this paper, we propose a novel framework based on YUV color space to address these issues. First, we conduct some analyses and experiments on several color spaces, demonstrating the effectiveness of asymmetric color transfer in YUV color space. Second, we design a YUV-color-space-based Asymmetric Color Transfer Network (YACT-Net) for reference-based colorization. In this network, we use a restoration module and a chrominance reconstruction module that utilize YUV information from image pairs to generate high-quality color images. Third, we create a Stereo Color Transfer (SCT) dataset, which includes both synthetic and real-world datasets, to evaluate the performance of YACT-Net. Experimental results demonstrate that YACT-Net achieves state-of-the-art performance in both quantitative and qualitative results. Our code and dataset are available athttps://github.com/fengshenghong/YACT-Net.