The emergence of remote vital sign measurement techniques has provided an alternative approach for monitoring vital signs without direct physical contact. However, the performance of contactless methods such as remote photoplethysmography are dependent on the accuracy of face detection algorithms. The misalignment of the face pixels from frame to frame can introduce jitters in the generated rPPG signals, and in turn, interfere with vital sign estimations. Nonetheless, the investigation into the performance of face detectors mostly focused on quantifying the accuracy of face landmarks based on an individual image. How to assess facial alignments across video frames has largely remained unknown and understudied. To address this issue, this paper introduced three novel metrics for assessing face alignment in remote vital sign detection: (1) Circular Radius, (2) Mean Offset, and (3) Percentage of Impacted Pixels. We evaluated two face detectors using proposed metrics in static and motion scenarios, where static represents no facial movement and motion scenarios involve facial movements induced by breathing. Our experiments demonstrated that employing the superior face detector recommended by our metrics resulted in a noteworthy 12.5% reduction in second-level mean absolute error and a corresponding 3.0% improvement in 5%-accuracy for remote heart rate estimation.

Abstract Software design patterns have a considerable impact on the software development life cycle. Design pattern (DP) is a reliable and reusable solution provided by software experts to obtain quality software design. However, due to the large number of design patterns, selecting the appropriate one is quite difficult. To overcome this difficulty, several approaches with different methods have been presented to suggest the appropriate DP. Despite conducting a number of studies that have explored some aspects of this field, such as design pattern selection tools and techniques, there is a need for a deeper understanding, analysis, classification, and thorough review of the design pattern selection process. So far, no systematic review of design pattern selection approaches is available. This paper aims to classify existing approaches, provide several criteria for comparing approaches, analyze each one, and identify and analyze the most important elements in this field, including open issues, data sets, and so on. The present investigation paper will help future research to employ the existing approaches taking into account the specification of each one and it also raises awareness about the approaches used in previous research and their potential limitations.

The concept of movement synchrony is derived from the scientific study of interacting dyads in the autism field. Automated movement synchrony estimation has been achieved by utilizing deep learning models applied to other tasks, such as human activity recognition. To better adapt to the movement synchrony estimation task, we proposed a skeleton-based uncertainty-aware graph transformer incorporating joint confidence scores. We uniquely designed a joint position embedding shared between the same joints of interacting individuals and introduced a temporal similarity matrix in temporal attention computation considering the periodic intrinsic of body movements. To further improve the performance, we constructed a dataset for movement synchrony estimation using Human3.6M and pretrained our model on it via contrastive learning. We further applied knowledge distillation to alleviate information loss introduced by pose detector failure in a privacy-preserving way. Our method achieved an overall accuracy of on PT13, a dataset collected from autism therapy interventions, and surpassed its counterpart approaches by a good margin. This work also has implications for synchronous movement activity recognition in group settings, with broad applications in education and sports.

Movement synchrony refers to the dynamic temporal connection between the motions of interacting people. The applications of movement synchrony are wide and broad. For example, as a measure of coordination between teammates, synchrony scores are often reported in sports. The autism community also identifies movement synchrony as a key indicator of children's social and developmental achievements. In general, raw video recordings are often used for movement synchrony estimation, with the drawback that they may reveal people's identities. Furthermore, such privacy concern also hinders data sharing, one major roadblock to a fair comparison between different approaches in autism research. To address the issue, this paper proposes an ensemble method for movement synchrony estimation, one of the first deep-learning-based methods for automatic movement synchrony assessment under privacy-preserving conditions. Our method relies entirely on publicly shareable, identity-agnostic secondary data, such as skeleton data and optical flow. We validate our method on two datasets: (1) PT13 dataset collected from autism therapy interventions and (2) TASD-2 dataset collected from synchronized diving competitions. In this context, our method outperforms its counterpart approaches, both deep neural networks and alternatives.

Mutual gaze is one of the most significant, reliable, and observable social cues that we can use for establishing and maintaining successful social interactions. This cue has been actively used to assess the level of social behavior in the context of autism therapy. However, collecting gaze data manually and evaluating them is so challenging, which requires a lot of time and effort from therapy experts. To address these issues, in this paper, we introduce an automated mutual gaze detection framework, grounded based on previous works on automated gaze detection, as an effective predictive model for social visual behavior analysis and assessment in autism therapy. To evaluate the proposed gaze prediction framework, we prepare an in-house video dataset that captures social interactions between children with autism and their therapy trainers (N = 10, 30 video recordings). We estimate the mutual gaze ratio of children using our prediction model, then compared it with the social visual behavior scores that therapy experts manually annotated. The results showed that our framework provided mutual gaze ratio scores that reliably represent (or even replace) the therapy experts’ hand-coded social visual behavior scores through different analysis approaches: descriptive comparisons, correlation analysis, and regression prediction. We report our findings and discuss the implications of the proposed work in the context of visual behavior analysis for children with autism.

Movement synchrony refers to the dynamic temporal connection between the motions of interacting people. The automatic measurement of movement synchrony is worth studying for social behavior analysis applications, for instance, in play therapy of children in the autism spectrum. Existing approaches based on motion energy analysis are strongly reliant on the region of interest, and thus limit the interaction between individuals, especially for highly engaging activities like play therapy. Inspired by action quality assessment, a task to assess how well an action has been performed, in this paper, we propose an end-to-end deep learning method to integrate the following major tasks: (1) the automatic assessment of children’s performance in play therapy, and (2) the automatic estimation of movement synchrony between children and therapists, facilitated by an auxiliary task of intervention activity recognition. This multi-task paradigm generally improves the performance of our model across all tasks. Furthermore, when annotations are subjective, the typical exclusive annotation strategy may reduce tagging quality. As a result, we explored applying distribution learning to mitigate human bias in movement synchrony estimation. We allowed the second and third labels for each instance, namely the uncertainty-preserved annotation approach. We tested our method on Play Therapy 13 (PT13), a dataset collected from video recordings of play therapy interventions. The findings of the experiments indicated that our framework can accurately quantify movement synchronization and assess the quality of children’s actions in play therapy. Moreover, the uncertainty-preserved annotation approach produced a comparable outcome to standard methods at a far reduced cost, demonstrating its efficacy in mitigating biases.

With the rise of attention to global warming which brings in more extreme weather and climate conditions, the earth science education would be one of the crucial topics for the next generation. Mixed-Reality has been shown to offer more engaging and effective learning solutions on essential science topics, such as math, physics, and chemistry. However, there are few augmented reality and mixed reality applications on earth science subject. Also, collaborative learning has been shown to be beneficial for student learning by aspiring student curiosity, and the ability of cooperation. In this paper, we propose a Mixed Reality Tornado Simulator which offers an earth science education intervention in a collaborative mixed reality setting. Students and their instructor can wear see-through head-mounted displays to cooperate on learning the knowledge of the formation and its damage cause on human-built structures, farming, and vegetation by using our proposed mixed reality application. Also, for evaluating the learning performance in this mixed reality setting, we will study the students cognitive load using standard survey instruments. We will conduct a controlled study with two conditions to compare the proposed intervention in the head-mounted-display setting, versus a desktop setting to test usability and knowledge gain of the students in those settings.

Interactive training environments typically include feedback mechanisms designed to help trainees improve their performance through either guided or self-reflection. In this context, trainees are candidate teachers who need to hone their social skills as well as other pedagogical skills for their future classroom. We chose an avatar-mediated interactive virtual training system–TeachLivE–as the basic research environment to investigate the motions and embodiment of the trainees. Using tracking sensors, and customized improvements for existing gesture recognition utilities, we created a gesture database and employed it for the implementation of our real-time gesture recognition and feedback application. We also investigated multiple methods of feedback provision, including visual and haptics. The results from the conducted user studies and user evaluation surveys indicate the positive impact of the proposed feedback applications and informed body language. In this paper, we describe the context in which the utilities have been developed, the importance of recognizing nonverbal communication in the teaching context, the means of providing automated feedback associated with nonverbal messaging, and the preliminary studies developed to inform the research.

Left-right confusion occurs across the entire population and refers to an impeded ability to distinguish between left and right. In medicine this phenomenon is particularly relevant as left and right are always defined with respect to the patient's point of view, i.e. the doctor's right is the patient's left. Traditional anatomy learning resources such as illustrations in textbooks naturally consider this by consistently depicting the anatomy of a patient as seen by an observer standing in front. Augmented Reality Magic Mirrors (MM) are one example of novel anatomy teaching resources and show a user's digital mirror image augmented with virtual anatomy on a large display. As left and right appear to be reversed in such MM setups, similar to real-world physical mirrors, intriguing perceptual questions arise: is a non-reversing MM (NRMM) the more natural choice for the task of anatomy learning and do users even learn anatomy the wrong way with a traditional, reversing MM (RMM)? In this paper, we explore the perceptual differences between an NRMM and RMM design and present the first empirical study comparing these two concepts for the purpose of anatomy learning. Experimental results demonstrate that medical students perform significantly better at identifying anatomically correct placement of virtual organs in an NRMM. However, interaction was significantly more difficult compared to an RMM. We explore the underlying psychological effects and discuss the implications of using an NRMM on user perception, knowledge transfer, and interaction. This study is relevant for the design of future MM systems in the medical domain and lessons-learned can be transferred to other application domains.

Screen-based Augmented Reality (AR) systems can be built as a window into the real world as often done in mobile AR applications or using the Magic Mirror metaphor, where users can see themselves with augmented graphics on a large display. The term Magic Mirror implies that the display shows the users enantiomorph, i.e. the mirror image, such that the system mimics a real-world physical mirror. However, the question arises whether one should design a traditional mirror, or instead display the true mirror image by means of a non-reversing mirror? We discuss the perceptual differences between these two mirror visualization concepts and present a first comparative study in the context of Magic Mirror anatomy teaching.

Nonverbal behaviors such as facial expressions, eye contact, gestures, and body movements in general have strong impacts on the process of communicative interactions. Gestures play an important role in interpersonal communication in the classroom between student and teacher. To assist teachers with exhibiting open and positive nonverbal signals in their actual classroom, we have designed a multimodal teaching application with provisions for real-time feedback in coordination with our TeachLivE test-bed environment and its reflective application; ReflectLivE. Individuals walk into this virtual environment and interact with five virtual students shown on a large screen display. The recent research study is designed to have two settings (7-minute long each). In each of the settings, the participants are provided lesson plans from which they teach. All the participants are asked to take part in both settings, with half receiving automated real-time feedback about their body poses in the first session (group 1) and the other half receiving such feedback in the second session (group 2). Feedback is in the form of a visual indication each time the participant exhibits a closed stance. To create this automated feedback application, a closed posture corpus was collected and trained based on the existing TeachLivE teaching records. After each session, the participants take a post-questionnaire about their experience. We hypothesize that visual feedback improves positive body gestures for both groups during the feedback session, and that, for group 2, this persists into their second unaided session but, for group 1, improvements occur only during the second session.

Research in learning analytics and educational data mining has recently become prominent in the fields of computer science and education. Most scholars in the field emphasize student learning and student data analytics; however, it is also important to focus on teaching analytics and teacher preparation because of their key roles in student learning, especially in K-12 learning environments. Nonverbal communication strategies play an important role in successful interpersonal communication of teachers with their students. In order to assist novice or practicing teachers with exhibiting open and affirmative nonverbal cues in their classrooms, we have designed a multimodal teaching platform with provisions for online feedback. We used an interactive teaching rehearsal software, TeachLivE, as our basic research environment. TeachLivE employs a digital puppetry paradigm as its core technology. Individuals walk into this virtual environment and interact with virtual students displayed on a large screen. They can practice classroom management, pedagogy and content delivery skills with a teaching plan in the TeachLivE environment. We have designed an experiment to evaluate the impact of an online nonverbal feedback application. In this experiment, different types of multimodal data have been collected during two experimental settings. These data include talk-time and nonverbal behaviors of the virtual students, captured in log files; talk time and full body tracking data of the participant; and video recording of the virtual classroom with the participant. 34 student teachers participated in this 30-minute experiment. In each of the settings, the participants were provided with teaching plans from which they taught. All the participants took part in both of the experimental settings. In order to have a balanced experiment design, half of the participants received nonverbal online feedback in their first session and the other half received this feedback in the second session. A visual indication was used for feedback each time the participant exhibited a closed, defensive posture. Based on recorded full-body tracking data, we observed that only those who received feedback in their first session demonstrated a significant number of open postures in the session containing no feedback. However, the post-questionnaire information indicated that all participants were more mindful of their body postures while teaching after they had participated in the study.

As part of normal interpersonal communication, people send and receive messages with their body, especially with their hands. Gestures play an important role in teacher-student classroom interactions. In the domain of education, many research projects have focused on the study of such gestures either in real classrooms or in tutorial settings with experienced teachers. Novice teachers especially need to understand the messages they are sending through nonverbal communication as this can have a major effect on their ability to manage behaviors and deliver content. Such learning should optimally occur before experiencing the real classroom. To assist in this process, we have developed a virtual classroom environment- TeachLivE- and used it for teacher practice, reflection and assessment. This paper investigates the way teachers use gestures in the virtual classroom settings of TeachLivE. Biology and algebra teachers were evaluated in our study. Analysis of video recordings from real and virtual environment seems to indicate that algebra teachers gesture significantly more often than biology teachers. These results have implications for providing useful feedback to participant teachers.