Facial expressions have been considered a metric reflecting a person’s engagement with a task. While the evolution of expression detection methods is consequential, the foundation remains mostly on image processing techniques that suffer from occlusion, ambient light, and privacy concerns. In this paper, we propose ExpresSense, a lightweight application for standalone smartphones that relies on near-ultrasound acoustic signals for detecting users’ facial expressions. ExpresSense has been tested on different users in lab-scaled and large-scale studies for both posed as well as natural expressions. By achieving a classification accuracy of over various basic expressions, we discuss the potential of a standalone smartphone to sense expressions through acoustic sensing.

The profuse popularity of video conferencing has led to a simultaneous rise in the opportunity for the participants to multitask. Productive multitasking, such as taking notes, browsing for relevant information, etc., can help promote the cognitive attentiveness of participants. However, existing approaches of tagging inattentive participants solely based on their visual concentration on the meeting app fail to work in such instances. This paper proposes EmotiConf -- a novel real-time framework to monitor participants' attentiveness and a non-real-time framework for visual multitask detection without explicitly relying on their visual concentration. EmotiConf utilizes an unconventional observation where the emotional states of attentive participants, captured through their facial expressions, correlate and also correspond to the vocal expression of the speaker and the intent of the speech. Accordingly, EmotiConf develops a software wrapper to tag the inattentive participants while also characterizing visual multitasking instances performed by them. A thorough evaluation of EmotiConf confirms its usability with a high score of >80.

Smartphones are ubiquitous nowadays, be it for setting a reminder, quick messaging, or an email reply, which requires typing through a soft-keyboard. However, people with medical issues like dactylitis, sarcopenia, joint pains might feel difficulty in typing, using the conventional approach. Existing gaze or voice-based approaches do not work well without commercial trackers or in noisy environments. In this paper, we develop a novel technique called Nosype, a contact-free text entry system for such users. Nosype’s core functionality lies in nose-tip tracking and projection and allows the users to draw alphanumeric characters in the air for constructing a text. With 11 users with different clinical conditions, on a lab-scale, we observe that Nosype can help in typing with an average typing error rate of 6.9% and a typing-speed of 6.31 words/minute. A large-scale usability study with 60 participants, including 10 participants having clinical disabilities, shows an average usability score of 77.708.

The rapid proliferation of Massive Open Online Courses (MOOC) has resulted in many-fold increase in sharing the global classrooms through customized online platforms, where a student can participate in the classes through her personal devices, such as personal computers, smartphones, tablets, etc. However, in the absence of direct interactions with the students during the delivery of the lectures, it becomes difficult to judge their involvements in the classroom. In academics, the degree of student's attention can indicate whether a course is efficacious in terms of clarity and information. An automated feedback can hence be generated to enhance the utility of the course. The precision of discernment in the context of human attention is a subject of surveillance. However, visual patterns indicating the magnitude of concentration can be deciphered by analyzing the visual emphasis and the way an individual visually gesticulates, while contemplating the object of interest. In this paper, we develop a methodology called Gestsatten which captures the learner's attentiveness from his visual gesture patterns. In this approach, the learner's visual gestures are tracked along with the region of focus. We consider two aspects in this approach -- first, we do not transfer learner's video outside her device, so we apply in-device computing to protect her privacy; second, considering the fact that a majority of the learners use handheld devices like smartphones to observe the MOOC videos, we develop a lightweight approach for in-device computation. A three level estimation of learner's attention is performed based on these information. We have implemented and tested Gestatten over 48 participants from different age groups, and we observe that the proposed technique can capture the attention level of a learner with high accuracy (average absolute error rate is 8.68%), which meets her ability to learn a topic as measured through a set of cognitive tests.