Convolutional neural networks (CNNs) have excellent and extensive applications in image recognition. With the continuous exploitation of data value and the proliferation of machine learning-as-a-service, convolutional neural network prediction schemes on privacy preservation have been introduced one after another, which makes much more attention focused on the privacy leakage and services offered to be efficient and light. Therefore, how to improve the convolutional neural prediction scheme on the premise of privacy preservation turns out to be an imperative research issue. In this paper, we propose a privacy-preserving convolutional neural network prediction scheme (PCP-LL) that supports low latency and lightweight users. The scheme starts from the perspective of lossless accuracy from underlying networks. First, we construct a secure activation function computing protocol (SActF) utilizing a commodity-based secure comparison protocol, which reduces the complexity and latency during the activation function computing under ciphertexts compared with common schemes. Second, to further support lightweight users, we introduce a secure output layer protocol (SOut) that enables users to obtain the prediction results without extra decryption after simple operations. Then, the scheme adopts the distributed two trapdoors public-key cryptosystem (DT-PKC) to achieve both data and model security, which well avoids security issues especially such as wiretapping by semi-honest participants commonly in secret sharing schemes. Finally, through relevant evaluations, the scheme not only achieves privacy preservation and low latency, but also supports lightweight users.