Control of an aerial manipulator is challenging due to the decentralized dynamics of the aerial vehicle and the robotic arm. It is generally complex to adjust the controller of the aerial manipulator when replacing a different robotic arm. This paper presents a flexible control scheme for a quadrotor-based aerial manipulator equipped with a replaceable robotic arm. To analyze the dynamic characteristics during grasping, the model of the aerial manipulator is decentralized including the models of a quadrotor and the centroid of an n-DOF robotic arm. The interaction effect of a moving robotic arm on the quadrotor is considered by analyzing the varying centroid of the robotic arm. Based on the modeling of the aerial manipulator, a control scheme integrating a linear model predictive control (LMPC) and a feedforward controller is presented to accurately control the motion of the aerial platform. The LMPC controls the aerial vehicle to follow the desired trajectory, and a feedforward controller keeps the aerial platform hovering stably during grasping. Practical experiments with two different robotic arms are performed. Experimental results show that the proposed modeling and control scheme provides a flexible and effective approach for an aerial manipulator with a replaceable robotic arm.