Figure 1: General procedure of the Kids-SIT research tool.A child participant follows a fully automated conversation scenario with pre-recorded videos of an actress.The actress initiates a dialogue on meal preferences, talking about her and asking the participant on their favorite and disliked foods one after the other.The actress empathically waits for responses and automatically continues the conversation in the following.For the whole time, the participant behavior is video recorded through the front camera.

Autism Spectrum Condition (ASC) is characterized by social interaction difficulties that can be challenging to assess objectively in the diagnostic process. In this paper, we evaluate the capability of using videos of a standardized social interaction to differentiate non-verbal behaviors of individuals with and without ASC. We collected a large video dataset consisting of 164 participants with ASC (n = 83) and neurotypical individuals (n = 81) who completed the computer-based Simulated Interaction Task (SIT) in different studies including lab and home settings. To classify individuals with and without ASC, we trained uni-and multimodal machine learning models based on different modalities such as facial expressions, gaze behavior, head pose and voice features. Our results indicate that a multimodal late fusion approach achieved the highest accuracy (74%). In the unimodal setting, classification based on facial expressions (accuracy 73%) and voice features (accuracy 70%) were most effective. An explainability analysis of the most relevant features for the facial expression model indicated that features from all emotional parts as well as from both the speaking and listening part of the interaction were informative. Based on our results, we developed a scalable online version of the SIT to collect diverse data on a large scale for the development of machine learning models that can differentiate between different clinical conditions. Our study highlights the potential of machine learning on videos of standardized social interactions in supporting clinical diagnosis and the objective and effective measurement of differences in social interaction behavior.

Machine-learning-based stress detection systems differ with respect to the ground truth used for training the algorithms. It is unclear how models trained on different facets of the stress reaction (e.g., biological, psychological, social) can be compared, interpreted and applied. In this study, we investigate the influence of the stress label on the performance of machine learning models trained on either vocal characteristics or facial expressions extracted from videos. We collected videos from 40 male participants while being exposed to the Trier Social Stress Test (TSST) and assessed self-reported, live observed, video-annotated and neuro-endocrinological stress levels. We train three standard machine learning models to separately predict different stress labels using either voice or facial cues. Analyzing the relationships of different stress facets we found that observers' annotations were significantly positively associated (live vs. video annotated,$\boldsymbol{\rho}_{\mathbf{s}}\ =\ . 53$). Similarly, the neuro-endocrinological stress indices correlated with each other (cortisol vs. sAA,$\boldsymbol{\rho}_{\mathbf{s}}$=.39). Machine learning experiments resulted in predictions that were positively associated with panel-annotated stress levels showing significantly stronger correlations in voice-based models$(\boldsymbol{\rho}_{\mathbf{s}=}.54\ \mathbf{v}\mathbf{s}. \boldsymbol{\rho}_{\mathbf{S}}=.30)$. Predictions of self-reported stress were positively related to ground truth values for face-based$(\boldsymbol{\rho}_{\mathbf{s}}$=.24) but not for voice-based models. There was no evidence for successful predictions of video-annotations or endocrinological stress levels in both settings. We provide evidence that machine learning models trained on different stress assessments perform differently and should be interpreted and applied accordingly. Implications and recommendations for future work on video-based stress detection are discussed.

MOTIVATION: Medical images can provide rich information about diseases and their biology. However, investigating their association with genetic variation requires non-standard methods. We propose transferGWAS, a novel approach to perform genome-wide association studies directly on full medical images. First, we learn semantically meaningful representations of the images based on a transfer learning task, during which a deep neural network is trained on independent but similar data. Then, we perform genetic association tests with these representations. RESULTS: We validate the type I error rates and power of transferGWAS in simulation studies of synthetic images. Then we apply transferGWAS in a genome-wide association study of retinal fundus images from the UK Biobank. This first-of-a-kind GWAS of full imaging data yielded 60 genomic regions associated with retinal fundus images, of which 7 are novel candidate loci for eye-related traits and diseases. AVAILABILITY AND IMPLEMENTATION: Our method is implemented in Python and available at https://github.com/mkirchler/transferGWAS/. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.