Due to the widespread use of mobile devices, it is essential to authenticate users on mobile devices to prevent sensitive information leakage. Biometrics-based authentication is prevalent on smart devices to verify the legitimacy of users, but is vulnerable to replay attacks. In this paper, we propose to leverage the distinctive finger tap gesture during unlocking smartphone to establish a secure multi-factor authentication system, named FingerVib. Compared with other biometric-based authentication systems, FingerVib does not require users to remember any complicated information (e.g., hand gestures, doodles) and the working type is unobtrusive. When users unlock their phones by tapping, FingerVib utilizes the microphone to record the sound produced by fingers tapping on the phone and adopts IMU (Inertial Measurement Unit) to extract the vibration of users’ smartphones. One key contribution is that we model the inherent correlation between sounds and vibration signals. Specifically, FingerVib captures two novel reactions to describe how the individual’s contact palm modulates signals in two different domains. Based on these two responses, we develop a real-time noise-resistant unlocking activity detection algorithm, which allows accurate unlocking signal segmentation even if the two modalities are interfered. Further, we develop a modal fusion model where the model extracts cross-modal features and acquires inter-modal correlation features to ensure consistent performance of inference even when modalities are disturbed. In a user study with 41 participants, FingerVib achieves an authentication accuracy of 98.53% and an average performance of 1.36% FAR, 2.76% FRR and 2.72% EER against replay attacks and impersonation attacks. FingerVib’s fusion approach improves identification performance by roughly 9.7% and 11.6% over Wavocie and AUDIOIMU, respectively, within existing multi-modal fusion systems. Extensive experimental results demonstrate the effectiveness and robustness of FingerVib under various conditions.