By now, many works have been done on shadow removal for image manipulation. As a result, detecting shadow removal has become a critical part to reveal the traces of image manipulation. However, there are only a few works conducted on shadow removal detection, and these works cannot accurately localize the image regions where the shadows have been removed. In this paper, we present a novel model called Multi-level Feature Fusion Network (MFF-Net) for shadow removal detection. MFF-Net consists of two parts: a dual-branch feature extraction encoder and a dense prediction decoder. The encoder anchors the approximate position of the manipulated regions, while the decoder progressively fills in the details of the estimated shadow masks by integrating multi-level information. In the encoder part, a global modeling branch is constructed to capture long-range dependencies, while a local feature extraction branch is designed to extract local structural information. The features extracted by these two branches are integrated using a feature fusion module. In the decoder part, a multi-scale feature upsampling module is proposed to upsample the input features and integrate them with the low-level features obtained from the encoder part. Meanwhile, the cross attention mechanism is introduced to guide the multi-level feature fusion process. Finally, the features of different resolutions are employed to estimate the shadow masks in a coarse-to-fine manner. Extensive experiments on shadow removal detection demonstrate the superiority of MFF-Net over the state-of-the-art methods. The source code of MFF-Net is publicly available athttps://github.com/HITFuxiwen/MFF-Net.