Lying is a common act in daily life and may have various degrees of falsehood. Deception detection has always been a fascinating area of research in which many studies have been conducted using subjects’ facial, verbal or bodily cues to spot potential deceit. However, none of the studies have investigated the physiological responses of observers in response to misleading statements with various degrees of falsehood. In this paper, we investigated this problem by first conducting designed experiments to collect participants’ physiological signals while they were watching stimulus videos with various falsehood levels. Then, the data was analysed using machine learning or deep learning models. Various challenges including relatively small amounts of training data and imbalanced classes have been addressed by implementing data augmentation. The results show that deep learning models, such as ResNet and VAE-LSTM, can predict the degree of falsehood with an F1-measure up to 0.83 from observers’ reactions when compared to the stimuli ground truth. This was attained when the model was trained with the most useful physiological signal in this study, Electrodermal Activity (EDA). This result indicates that observers’ physiological signals can be used as an indicator to determine the degree of falsehood for misleading statements. In the future, this system may be applied to provide an objective evaluation for deception detection.