The operation of traditional multi-object trackers on a moving autonomous aerial vehicle (AAV) faces many difficulties due to the irregular motion of AAV, the occlusion problem, and in particular arbitrarily oriented targets that are densely distributed with complex backgrounds. To solve these difficulties, this paper proposes a novel multi-object tracking framework, namely ArbiTrack, for a moving AAV to effectively detect and track arbitrarily oriented targets on the grounds. The proposed framework consists of an oriented object detection module to capture ground objects, a multi-scale context aggregation (MCA) module to improve the detection accuracy of small objects, and an adaptive motion switching (AMS) module to deal with the nonlinear complexity among AAV and ground objects. Historical information from multiple moments is used in this framework to learn the spatio-temporal characteristics so that the occlusion problem can be solved effectively. Experiments are conducted by using our OriDrone dataset and the public dataset UAVDT dataset. Results demonstrate that the proposed method achieves state-of-the-art tracking performance.