Studies on schizophrenia assessments using deep learning typically treat it as a classification task to detect the presence or absence of the disorder, oversimplifying the condition and reducing its clinical applicability. This traditional approach overlooks the complexity of schizophrenia, limiting its practical value in healthcare settings. This study shifts the focus to individual symptom severity estimation using a multimodal approach that integrates speech, video, and text inputs. We develop unimodal models for each modality and a multimodal framework to improve accuracy and robustness. By capturing a more detailed symptom profile, this approach can help in enhancing diagnostic precision and support personalized treatment, offering a scalable and objective tool for mental health assessment.

This study investigates the speech articulatory coordination in schizophrenia subjects exhibiting strong positive symptoms (e.g. hallucinations and delusions), using two distinct channel-delay correlation methods. We show that the schizophrenic subjects with strong positive symptoms and who are markedly ill pose complex articulatory coordination pattern in facial and speech gestures than what is observed in healthy subjects. This distinction in speech coordination pattern is used to train a multimodal convolutional neural network (CNN) which uses video and audio data during speech to distinguish schizophrenic patients with strong positive symptoms from healthy subjects. We also show that the vocal tract variables (TVs) which correspond to place of articulation and glottal source outperform the Mel-frequency Cepstral Coefficients (MFCCs) when fused with Facial Action Units (FAUs) in the proposed multimodal network. For the clinical dataset we collected, our best performing multimodal network improves the mean F1 score for detecting schizophrenia by around 18% with respect to the full vocal tract coordination (FVTC) baseline method implemented with fusing FAUs and MFCCs.

Biology is well-known for its ability to communicate through (i) molecularly-specific signaling modalities and (ii) a globally-acting electrical modality associated with ion flow across biological membranes. Emerging research suggests that biology uses a third type of communication modality associated with a flow of electrons through reduction/oxidation (redox) reactions. This redox signaling modality appears to act globally and has features of both molecular and electrical modalities: since free electrons do not exist in aqueous solution, the electrons must flow through molecular intermediates that can be switched between two states - with electrons (reduced) or without electrons (oxidized). Importantly, this global redox modality is easily accessible through its electrical features using convenient electrochemical instrumentation. In this review, we explain this redox modality, describe our electrochemical measurements, and provide four examples demonstrating that redox enables communication between biology and electronics. The first two examples illustrate how redox probing can acquire biologically relevant information. The last two examples illustrate how redox inputs can transduce biologically-relevant transitions for patterning and the induction of a synbio transceiver for two-hop molecular communication. In summary, we believe redox provides a unique ability to bridge bio-device communication because simple electrochemical methods enable global access to biologically meaningful information. Further, we envision that redox may facilitate the application of information theory to the biological sciences.