We propose a privacy-preserving face authentication system for smartphones that guarantees security against malicious clients. Using the proposed system, a face feature vector is stored on a remote server in encrypted form. To guarantee security against an honest-but-curious server who may try to learn the private feature vector, we perform a Euclidean distance-based matching score computation on encrypted feature vectors using homomorphic encryption. To provide security against malicious clients, we adopt a blinding technique. We implement the proposed system on a mobile client and a desktop server. Through an experiment with real-world participants, we demonstrate that secure face verification can be completed in real time (within 1.3 s) even when a smartphone is involved, with an Equal Error Rate (EER) of 3.04%. In further experiments with two public face datasets, CFP and ORL, face verification is completed in approximately 1 s with EER of 1.17% and 0.37%, respectively. Our system is two orders of magnitude faster than previous privacy-preserving face verification method with the same security assumptions and functionalities. To achieve this secure real-time computation, we improve the Catalano-Fiore transformation which converts a linear homomorphic encryption scheme into a quadratic scheme, and parallelize the decryption procedure of our system.