In this article we present a multi-modal machine learning-based system, which we call ACORN, to analyze videos of school classrooms for the Positive Climate (PC) and Negative Climate (NC) dimensions of the CLASS [1] observation protocol that is widely used in educational research. ACORN uses convolutional neural networks to analyze spectral audio features, the faces of teachers and students, and the pixels of each image frame, and then integrates this information over time using Temporal Convolutional Networks. The audiovisual ACORN's PC and NC predictions have Pearson correlations of 0.55 and 0.63 with ground-truth scores provided by expert CLASS coders on the UVA Toddler dataset (cross-validation on$n=300$15-min video segments), and a purely auditory ACORN predicts PC and NC with correlations of 0.36 and 0.41 on the MET dataset (test set of$n=2000$videos segments). These numbers are similar to inter-coder reliability of human coders. Finally, using Graph Convolutional Networks we make early strides (AUC=0.70) toward predicting the specific moments (45-90sec clips) when the PC is particularly weak/strong. Our findings inform the design of automatic classroom observation and also more general video activity recognition and summary recognition systems.

To adapt materials for an individual learner, intelligent tutoring systems must estimate their knowledge or abilities. Depending on the content taught by the tutor, there have historically been different approaches to student modeling. Unlike common skill-based models used by math and science tutors, second language acquisition (SLA) tutors use memory-based models since there are many tasks involving memorization and retrieval, such as learning the meaning of a word in a second language. Based on estimated memory strengths provided by these memory-based models, SLA tutors are able to identify the optimal timing and content of retrieval practices for each learner to improve retention. In this work, we seek to determine whether skill-based models can be combined with memory-based models to improve student modeling and especially retrieval practice performance for SLA. In order to define skills in the context of SLA, we develop methods that can automatically extract multiple types of linguistic features from words. Using these features as skills, we apply skill-based models to a real-world SLA dataset. Our main findings are as follows. First, incorporating lexical features to represent individual words as skills in skill-based models outperforms existing memory-based models in terms of recall probability prediction. Second, incorporating additional morphological and syntactic features of each word via multiple-skill tagging of each word further improves the skill-based models. Third, incorporating semantic features, like word embeddings, to model similarities between words in a learner’s practice history and their effects on memory also improves the models and appears to be a promising direction for future research.

With the goal of giving teachers automated feedback about their classrooms, we investigate how to train automatic speech detectors of key phrases such as good job, thank you, please, and you’re welcome. This kind of language conveys support and respect from teacher to student and is one of the behavioral markers used in the established CLASS [1] classroom observation protocol. School classrooms are noisy and contain overlapping speech, presenting a highly challenging environment for automatic speech recognition (ASR), even for state-of-the-art approaches. We train deep neural networks using hierarchical multitask learning (MTL) on a modest-sized but highly-tailored dataset of classroom speech. Compared to 2 state-of-the-art ASR systems for general-purpose speech recognition (Google [2] and Deep-Speech [3]), our system delivers a substantially improved recall rate (50.4% versus 20.5%) while matching their precision (30%). Moreover, our system’s predictions correlate with several dimensions of the CLASS.

The human hand serves as an inspiration for robotic grippers. However, the dimensions of the human hand evolved under a different set of constraints and requirements than that of robots today. This paper discusses a method of kinematically optimizing the design of an anthropomorphic robotic hand. We focus on maximizing the workspace intersection of the thumb and the other fingers as well as maximizing the size of the largest graspable object. We perform this optimization and use the resulting dimensions to construct a flexible, underactuated 3D printed prototype. We verify the results of the optimization through experimentation, demonstrating that the optimized hand is capable of grasping objects ranging from less than 1 mm to 12.8 cm in diameter with a high degree of reliability. The hand is lightweight and inexpensive, weighing 333 g and costing less than 175 USD, and strong enough to lift over 1.1 lb (500 g). We demonstrate that the optimized hand outperforms an open-source 3D printed anthropomorphic hand on multiple tasks. Finally, we demonstrate the performance of our hand by employing a classification-based user intent decision system which predicts the grasp type using real-time electromyographic (EMG) activity patterns.

Drug-drug interaction related adverse events (DIAE) signals are a major public health issue. Drug safety analysts must sift through thousands of adverse event reports submitted daily to U.S. Food and Drug Administration (FDA) to discover unexpected DIAE signals, which if addressed can lead to life-saving actions. To facilitate the DIAE discovery from these massive data sets, we design several technological innovations that together are integrated into an interactive visual analytics system called DEVES 1. First, our state- of-the-art DIAE mining algorithm efficiently infers potential DIAE signals from these reports, and then ranks them based on their significance score. For interpretability of these inferred DIAE signals, domain knowledge of adverse events and already known drug interactions is extracted from external authoritative data sources and then seamlessly integrated with the inferred signal set. Guided by this augmented signal model, DEVES supports advanced signal analytics - empowering the analyst to interact with linked visualizations offering complementary perspectives into the signal set and its supporting evidence in the form of reports. Our demonstration showcases the technological innovations of DEVES using real-world FDA datasets, demonstrating that DEVES effectively supports the core regulatory tasks from signal screening, signal prioritization to signal validation.

Adverse drug reactions (ADRs) caused by drug-drug interactions (DDI) are a major cause of morbidity and mortality worldwide. There is a growing need for computing-supported methods that facilitate the automated signaling of DDI related ADRs (DIARs) that otherwise would remain undiscovered in millions of ADR reports. In this demonstration, we showcase our MeDIAR technology - an end-to-end DIAR signal generation, exploration and validation solution for pharmaceutical regulatory agencies to detect true DIAR signals from a drug surveillance database. MeDIAR's innovations include an efficient rule-driven learning algorithm for deriving DIAR signals from ADR reports, an innovative scoring methodology based on the proposed contextual association cluster model to rank the generated signals by their importance. Further, these ranked signals are augmented with meta information such as their significance level and their severity, along with links to their supporting ADR reports. Lastly, MeDIAR features an interactive visual analytics interface to support drug safety evaluators in reviewing and discovering unknown severe DIARs.

Mobile devices now rival desktop computers as a method of web browsing. Even so, many web applications are still designed with the desktop setting in mind. As screen sizes of mobile devices continue to get larger, operating smartphones single-handedly becomes increasingly difficult. This paper explores the possibility of automatic operating hand detection by capturing users' touch traces during normal browsing activities. Automatic operating hand detection would enable web applications to adapt their interfaces and better suit the user's laterality. Supervised classifiers were constructed for the primary goal of operating hand detection (left, right), but also to detect hand posture (thumb, index). Both classifiers use features extracted from the touch traces and button clicks on a dataset of 174 users. The resulting classifiers featured true positive rates of 96.0% and 70.1% respectively when tested using 10-fold cross-validation. While previous studies achieved similarly accurate results for operating hand (94.1%) and posture (82.6%) detection, the approach proposed by this paper is not platform-specific and does not rely on access to gyroscopes or accelerometers.