Recent hardware advances have led to the development and consumerization of mobile devices, which mainly include smartphones and various wearable devices. To protect the privacy of users, various user authentication mechanisms have been proposed. In particular, biometrics has been widely used for multi-factor authentication. However, biometrics-based authentication mechanisms usually require costly sensors deployed on devices, and rely on explicit user input and Internet connection for performing user authentication. In this article, we propose a system, called RISKCOG, which can authenticate the ownership of mobile devices unobtrusively and in a real-time manner by adopting a learning-based approach. Unlike previous studies on user authentication, for cross-platform deployment, maximum user privacy protection, and unobtrusive authentication, RISKCOG only relies on those widely available and privacy-insensitive motion sensors to capture the data related to the users' daily device usage. It requires no users' explicit input and has no requirement on the users' motion state or the device placement. RISKCOG is also usable in the environment without Internet access by performing offline user identity verification. We conduct comprehensive experiments on smartphones and smartwatches, which show that RISKCOG can authenticate device users rapidly and with high accuracy.