Drone applications require perception all around the vehicle to avoid obstacles during navigation. Due to the weight and computation limitations on UAVs, using a large number of sensors, such as numerous cameras, could be prohibitive. In such scenarios, usage of fisheye cameras with a wider field of view is very beneficial. Despite the usefulness of fisheye camera for UAV applications, not much work has been carried out to develop perception algorithms for fisheye camera. One of the main problems being the lack of publicly available omnidirectional datasets in relation to drone flight. With this paper, we address this gap by presenting AirFisheye dataset, which is applicable for tasks such as segmentation, depth estimation and depth completion, among other tasks required for autonomous drone navigation. Also, a generic framework for creating synthetic fisheye images is provided. Furthermore, we propose a novel occlusion correction algorithm that removes incorrectly projected LiDAR point clouds into the camera image due to the viewpoint variation of both sensors. We release about 26K images and LiDAR scans along with annotations. Baseline code and supporting scripts are available at https://collaborating.tuhh.de/ilt/airfisheye-dataset