Modern Internet of Things (IoT) systems are highly complex due to its mobile, ad-hoc and geographically distributed nature. Very often, an edge-cloud infrastructure is established to offer intelligent services in modern IoT systems. However, IoT edge devices are typically resource-constrained and can not perform sophisticated machine learning algorithm on board. Data sharing with a central server is a common approach of crowdsourcing, but also brings privacy and security concerns. The emerging federated learning offers a promising pathway to achieve an accurate model through distributed machine learning while ensuring data privacy. The existing federated learning process is not tailored to the mobile and adhoc nature of IoT systems where devices are of varying data and system qualities and may not be able to participate the entire training process. Therefore, in this paper, a new federated learning framework is proposed to support asynchronous model fusion with clustering-based participant selection. The proposed framework aims to accommodate the ad-hoc nature of IoT devices, and at the same time avoiding low quality or even malicious data from its participants to ensure model convergence and performance.