Since digital images are able to be encoded through the luminance-bandwidth-chrominance (YUV) mode, and the contribution of luminance information is greater than that of chrominance information for human visual perception, it can be inferred that the appropriate reduction of chrominance information in convolutional neural network does not disturb image object recognition. In this paper, we propose a new multi-scale non-uniform convolution called YuvConv, wherein the output feature map of the convolutional layer is regarded as an image. First, the output channels in the new convolution are divided into three kinds of components: Y, U, and V tensors. Then, the tensor Y is used to process luminance information, which is high-resolution and occupies more output channels. Next, the tensors U and V are low-resolution and use fewer channels to process chrominance information. Finally, the adjacent tensors (Y-U, Y-U-V, and U-V) are fused as the output of YuvConv. Experimental results indicate that the use of the YuvConv instead of the standard convolution can improve the performance of deep learning tasks, and it can also reduce memory consumption and computation cost.