Automated grading in rubric-based, short-answer open-ended questions often mishandles partial credit, calibration, and actionable transparency, ultimately requiring teachers to reevaluate. This challenge is amplified in resource-constrained settings (e.g., with limited teachers and a large student population), resulting in weaker learning outcomes. To address this challenge, we present the Teacher-Cognition Multi-Agent Grading framework (TC-MAG), which mirrors teachers’ micro-steps via anchored LLM agents for rubric creation, guideline checks, blind double marking, arbitration, and cross-checking to calibrate confidence. Each step produces a concise explanation for targeted review. We first conducted a motivational study to inform the design of TC-MAG. Next, we validated the effectiveness of the TC-MAG framework on a dataset of 2,000 Singapore primary school students’ responses across 1–4-mark mathematics questions with teacher-adjudicated gold labels. TC-MAG attained deployment-level reliability (κ=0.968 on 1-mark; quadratic-weighted κ=0.936 on 2–4 marks) by outperforming human teachers (Δκ=+0.063, p<.001) and state-of-the-art LLM baselines (min Δκ=+0.012, p<.001). In a mixed-methods teacher study (N=14; 12.1 years’ experience), explanation format and TC-MAG ’s confidence score influenced whether teachers delegated grading to TC-MAG. Staged explanations yielded greater diagnosticity (LR+ 11.5 vs. 4.60 for summarized explanations), informing a progressive disclosure strategy of explanations based on confidence. Overall, TC-MAG offers replicable multi-agent framework and triage methods for classroom deployment while preserving teacher oversight.

Many video applications (e.g., gaming, meeting, tutoring) aim to improve the user's interaction experience based on continuously inferred user emotion. To infer user emotion, these apps typically deploy machine learning models, trained with continuously collected emotion ground truth labels. However, as continuous annotations are generally collected as emotion self-reports during video consumption (using an auxiliary device), they incur significant annotation effort. To address this problem, we propose PResUP, a framework that creates users' profile using physiological responses (e.g., GSR or galvanic skin re-sponse) and deploys an LSTM network to identify the opportune probing moments for emotion ground truth (emotion self-report) collection instead of continuous annotation. We evaluate the proposed approach on a large-scale publicly available dataset (CASE) containing the physiological signals of subjects during video consumption. The evaluation of PRes UP reveals that it reduces the probing rate by 30.3 % (on average), detects the opportune probing moments with a TPR (True Positive Rate) of 86.1 %, and yet maintains the quality of the emotion annotations as observed in the continuous self-reports. Furthermore, we evaluated the generalizability of PResUP on another public dataset (K-emocon); which reveals an average probing rate reduction of 25.71 %. These results underscore the efficiency of PResUP in reducing the continuous emotion annotation overhead.

The Experience Sampling Method (ESM) is widely used to collect emotion self-reports to train machine learning models for emotion inference. However, as ESM studies are time-consuming and burdensome, participants often withdraw in between. This unplanned withdrawal compels the researchers to discard the dropout participants’ data, significantly impacting the quality and quantity of the self-reports. To address this problem, we leverage only the self-reporting similarity across participants (unlike prior works that apply different machine learning approaches on additional modalities) for missing self-report estimation. In specific, we propose a Multi-task Learning (MTL) framework, MUSE, that constructs the missing self-reports of the dropout participants. We evaluate MUSE in two in-the-wild studies (N1=24, N2=30) of 6-week and 8-week duration, during which the participants reported four emotions (happy, sad, stressed, relaxed) using a smartphone application. The evaluation reveals that MUSE estimates the missing emotion self-reports with an average AUCROC of 84% (Study I) and 82% (Study II). A follow-up evaluation of MUSE for an emotion inference (downstream) task reveals no significant difference in emotion inference performance when estimated self-reports are used. These findings underscore the utility of MUSE in estimating missing self-reports in ESM studies and the applicability of MUSE for downstream tasks (e.g., emotion inference).

Capturing drivers’ affective responses given driving context and driver-pedestrian interactions remains a challenge for designing in-vehicle, empathic interfaces. To address this, we conducted two lab-based studies using camera and physiological sensors. Our first study collected participants’ (N = 21) emotion self-reports and physiological signals (including facial temperatures) toward non-verbal, pedestrian crossing videos from the Joint Attention for Autonomous Driving dataset. Our second study increased realism by employing a hybrid driving simulator setup to capture participants’ affective responses (N = 24) toward enacted, non-verbal pedestrian crossing actions. Key findings showed: (a) non-positive actions in videos elicited higher arousal ratings, whereas different in-video pedestrian crossing actions significantly influenced participants’ physiological signals. (b) Non-verbal pedestrian interactions in the hybrid simulator setup significantly influenced participants’ facial expressions, but not their physiological signals. We contribute to the development of in-vehicle empathic interfaces that draw on behavioral and physiological sensing to in-situ infer driver affective responses during non-verbal pedestrian interactions.

Smartphone keyboard interaction based emotion detection systems are used widely to provide value-added services such as mental health monitoring, keyboard layout optimization, guided response generation. At the core of these services lie a machine learning model, which automatically infers emotion based on keyboard interaction pattern. To train these models, the emotion ground truth labels are typically collected as emotion self-report by conducting an Experience Sampling Method (ESM) based study. However, as responding to repetitive self-report probes is time-consuming and fatigue-inducing, efficient self-report collection approaches are essential that avoid probing at inopportune moments and reduce survey fatigue. To address this problem, we propose an active learning based framework, ALOE (Active Learning based Opportunistic Experience Sampling for Emotion Self-report Collection) that automatically decides to avoid probing at the unfavorable moments based on the typing signatures captured from smartphone keyboard interaction sessions. We bootstrap the framework with a few labeled instances (typing session) and allow the learner to probe (or query) the user only when it is least confident about an instance (typing session) and retrain accordingly. This way, we reduce the number of probes required (and therefore user engagement) and yet probe at the opportune moments. We evaluate ALOE in a 3-week in-the-wild study involving 18 participants, who record their smartphone keyboard interaction patterns and emotion self-reports during this period. The experimental results demonstrate that ALOE requires 56% less inopportune self-reports to train the probing moment detection learning model and yet detects the probing moments accurately with an average F-score of 93%.

The ubiquity of smartphones and the widespread usage of text entry by soft keyboard in different instant messaging applications (e.g., WhatsApp, FB messenger) have opened the possibilities of inferring emotions from longitudinal typing data. To build this emotion inference engine, we apply machine learning models on features extracted from user’s typing patterns (not content). However, one major challenge encountered while developing the emotion inference model is the requirement of individual training data as typing patterns are often person-specific. In this paper, we investigate the possibility of combining typing pattern with emotion self-reporting to identify a group of similar users so that the training data among these users can be shared to fulfill the requirement of personalized dataset. We develop a framework AffectPro, which quantifies the typing interaction behavior (e.g., typing speed, error rate) and self-reporting pattern (e.g., emotion state transition probability) to construct the affective profiles of users. We evaluated AffectPro in a 6-week in-the-wild study involving 28 users, who used an Android application encompassing a custom keyboard to perform all their typing activities, and to report their instantaneous emotions. We extracted different typing signatures and self-report behavior details from the collected dataset (≈ 5000 typing sessions, ≈ 108 hours of typing data) to construct the affective profile of users. Our results demonstrate similarity across users in terms of typing signature, emotion self-reporting pattern, and a combination of both; which can be leveraged to share training data among similar users to overcome the challenges of personalized data collection.

Keyboard interaction patterns on a smartphone is the input for many intelligent emotion-aware applications, such as adaptive interface, optimized keyboard layout, automatic emoji recommendation in IM applications. The simplest approach, called the Experience Sampling Method (ESM), is to systematically gather self-reported emotion labels from users, which act as the ground truth labels, and build a supervised prediction model for emotion inference. However, as manual self-reporting is fatigue-inducing and attention-demanding, the self-report requests are to be scheduled at favorable moments to ensure high fidelity response. We, in this paper, perform fine-grain keyboard interaction analysis to determine suitable probing moments. Keyboard interaction patterns, both cadence, and latency between strokes, nicely translate to frequency and time domain analysis of the patterns. In this paper, we perform a 3-week in-the-wild study (N = 22) to log keyboard interaction patterns and self-report details indicating (in)opportune probing moments. Analysis of the dataset reveals that time-domain features (e.g., session length, session duration) and frequency-domain features (e.g., number of peak amplitudes, value of peak amplitude) vary significantly between opportune and inopportune probing moments. Driven by these analyses, we develop a generalized (all-user) Random Forest based model, which can identify the opportune probing moments with an average F-score of 93%. We also carry out the explainability analysis of the model using SHAP (SHapley Additive exPlanations), which reveals that the session length and peak amplitude have strongest influence to determine the probing moments.

Smartphones provide the capability to perform in-situ sampling of human behavior using Experience Sampling Method (ESM). Designing an ESM schedule involves probing the user repeatedly at suitable moments to collect self-reports. Timely probe generation to collect high fidelity user responses while keeping probing rate low is challenging. In mobile-based ESM, timeliness of the probe is also impacted by user's availability to respond to self-report request. Thus, a good ESM design must consider -probing frequency,timely self-report collection, andnotifying at opportune momentto ensure highresponse quality. We propose a two-phase ESM design, where the first phase (a) balances between probing frequency and self-report timeliness, and (b) in parallel, constructs a predictive model to identify opportune probing moments. The second phase uses this model to further improve response quality by eliminating inopportune probes. We use typing-based emotion detection in smartphone as a case study to validate proposed ESM design. Our results demonstrate that it reduces probing rate by 64 percent, samples self-reports timely by reducing elapsed time between self-report collection, and event trigger by 9 percent while detecting inopportune moments with an average accuracy of 89 percent. These design choices improve the response quality, as manifested by 96 percent valid response collection and a maximum improvement of 24 percent in emotion classification accuracy.

Most of the latest context-based applications capture the mobility of a user using Inertial Measurement Unit (IMU) sensors like accelerometer and gyroscope which do not need explicit user-permission for application access. Although these sensors provide highly accurate mobility context information, existing studies have shown that they can lead to undesirable leakage of location information. To evade this breach of location privacy, many of the state-of-the-art studies suggest to impose stringent restrictions over the usage of IMU sensors. However, in this paper, we show that typing and smartphone engagement patterns can act as an alternative modality to sniff the mobility context of a user, even if the IMU sensors are not sampled at all. We develop an adversarial framework, named ConType, which exploits the signatures exposed by typing and smartphone engagement patterns to track the mobility of a user. Rigorous experiments with in-the-wild dataset show that ConType can track the mobility contexts with an average micro-F1of 0.87 (±0.09), without using IMU data. Through additional experiments, we also show that ConType can track mobility stealthily with very low power and resource footprints, thus further aggravating the risk.

Characteristics of typing on smartphone keyboards among different individuals can elicit emotion, similar to speech prosody or facial expressions. Existing works on typing based emotion recognition rely on feature engineering to build machine learning models, while recent speech and facial expression based techniques have shown the efficacy of learning the features automatically. Therefore, in this work, we explore the effectiveness of such learning models in keyboard interaction based emotion detection. In this paper, we propose an end-to-end framework, which first uses a sequence-based encoding method to automatically learn the representation from raw keyboard interaction pattern and subsequently uses this representation to train a multi-task learning based neural network (MTL-NN)to identify different emotions. We carry out a 3-week in-the-wild study involving 24 participants using a custom keyboard capable of tracing users' interaction pattern during text entry. We collect interaction details like touch speed, error rate, pressure and self-reported emotions (happy, sad, stressed, relaxed) during the study. Our analysis on the collected dataset reveals that the representation learnt from the interaction pattern has an average correlation of 0.901 within the same emotion and 0.811 between different emotions. As a result, the representation is effective in distinguishing different emotions with an average accuracy (AUCROC)of 84%.

Typing based interfaces are common across many mobile applications, especially messaging apps. To reduce the difficulty of typing using keyboard applications on smartphones, smartwatches with restricted space, several techniques, such as auto-complete, auto-suggest, are implemented. Although helpful, these techniques do add more cognitive load on the user. Hence beyond the importance to improve the word recommendations, it is useful to understand the pattern of use of auto-suggestions during typing. Among several factors that may influence use of auto-suggest, the role of emotion has been mostly overlooked, often due to the difficulty of unobtrusively inferring emotion. With advances in affective computing, and ability to infer user's emotional states accurately, it is imperative to investigate how auto-suggest can be guided by emotion aware decisions. In this work, we investigate correlations between user emotion and usage of auto-suggest i.e. whether users prefer to use auto-suggest in specific emotion states. We developed an Android keyboard application, which records auto-suggest usage and collects emotion self-reports from users in a 3-week in-the-wild study. Analysis of the dataset reveals relationship between user reported emotion state and use of auto-suggest. We used the data to train personalized models for predicting use of auto-suggest in specific emotion state. The model can predict use of auto-suggest with an average accuracy (AUCROC) of 82% showing the feasibility of emotion-aware auto-suggestion.

In the last decade, social media has evolved as one of the leading platform to create, share, or exchange information; it is commonly used as a way for individuals to maintain social connections. In this online digital world, people use to post texts or pictures to express their views socially and create user-user engagement through discussions and conversations. Thus, social media has established itself to bear signals relating to human behavior. One can easily design user characteristic network by scraping through someone’s social media profiles. In this paper, we investigate the potential of social media in characterizing and understanding predominant drunk texters from the perspective of their social, psychological and linguistic behavior as evident from the content generated by them. Our research aims to analyze the behavior of drunk texters on social media and to contrast this with non-drunk texters. We use Twitter social media to obtain the set of drunk texters and non-drunk texters and show that we can classify users into these two respective sets using various psycholinguistic features with an overall average accuracy of 96.78% with very high precision and recall. Note that such an automatic classification can have far-reaching impact – (i) on health research related to addiction prevention and control, and (ii) in eliminating abusive and vulgar contents from Twitter, borne by the tweets of drunk texters.

Typing characteristics on smartphones can provide clues for emotion detection. Collecting large volumes of typing data is also easy on smartphones. This motivates the use of Deep Neural Network (DNN) to determine emotion states from smartphone typing. In this work, we developed a DNN model based on typing features to predict four emotion states (happy, sad, stressed, relaxed) and investigate its performance on a smartphone. The evaluation of the model in a 3-week study with 15 participants reveals that it can reliably detect emotions with an average accuracy of 80% with peak CPU utilization less than 15%.

Passenger comfort is a major factor influencing a commuter's decision to avail public transport. Existing studies suggest that factors like overcrowding, jerkiness, traffic congestion etc. correlate well to passenger's (dis)comfort. An online survey conducted with more than 300 participants from 12 different countries reveals that different personalized and context dependent factors influence passenger comfort during a travel by public transport. Leveraging on these findings, we identify correlations between comfort level and these dynamic parameters, and implement a smartphone based application, ComfRide, which recommends the most comfortable route based on user's preference honoring her travel time constraint. We use a 'Dynamic Input/Output Automata' based composition model to capture both the wide varieties of comfort choices from the commuters and the impact of environment on the comfort parameters. Evaluation of ComfRide, involving 50 participants over 28 routes in a state capital of India, reveals that recommended routes have on average 30% better comfort level than Google map recommended routes, when a commuter gives priority to specific comfort parameters of her choice.

In Affective Computing, different modalities, such as speech, facial expressions, physiological properties, smart-phone usage patterns, and their combinations, are applied to detect the affective states of a user. Keystroke analysis i.e. study of the typing behavior in desktop computer is found to be an effective modality for emotion detection because of its reliability, non-intrusiveness and low resource overhead. As smartphones proliferate, typing behavior on smartphone presents an equally powerful modality for emotion detection. It has the added advantage to run in-situ experiments with better coverage than the experiments using desktop computer keyboards. This work explores the efficacy of smartphone typing to detect multiple affective states. We use a qualitative and experimental approach to answer the question. We conduct an online survey among 120 participants to understand the typing habits in smartphones and collect feedback on multiple measurable parameters that affect their emotion while typing. The findings lead us to design and implement an Android based emotion detection system, TapSense, which can identify four different emotion states (happy, sad, stressed, relaxed) with an average accuracy (AUCROC) of 73% (maximum of 94%) based on typing features only. The analysis also reveals that among different features, typing speed is the most discriminative one.

Experience Sampling Method (ESM) is widely used in idiographic approaches to collect within-person patterns. Planning a suitable survey schedule while designing an ESM based experiment is challenging as it must balance between survey fatigue of users, and the timeliness and accuracy of the responses provided by users. Even with the proliferation of ESM experiments, survey scheduling typically remains confined to use of fixed schedules, where periodic probes are sent to user, or event-based schedules, where depending on the number of events, large number of probes may interrupt user frequently. We propose a novel survey scheduling scheme, Low-Interference High-fidelity (LIHF) ESM schedule, which is designed to reduce interference while retaining fidelity of user response. We integrated LIHF into an ESM application, called TapSense, that is used to infer user's emotion from typing characteristics on smartphone keypad. Conducting a 2-week field study involving 9 users, using proposed metrics we show that using LIHF there is 26% reduction in survey fatigue, 50% improvement in triggering survey probes in timely manner, and 8% improvement in predicting emotion states based on typing patterns compared to typical ESM scheduling techniques.

City transit maps are one of the important resources for public navigation in today's digital world. However, the availability of transit maps for many developing countries is very limited, primarily due to the various socio-economic factors that drive the private operated and partially regulated transport services. Public transports at these cities are marred with many factors such as uncoordinated waiting time at bus stoppages, crowding in the bus, sporadic road conditions etc., which also need to be annotated so that commuters can take informed decision. Interestingly, many of these factors are spatio-temporal in nature. In this paper, we develop CityMap, a system to automatically extract transit routes along with their eccentricities from spatio-temporal crowdsensed data collected via commuters' smart-phones. We apply a learning based methodology coupled with a feature selection mechanism to filter out the necessary information from raw smart-phone sensor data with minimal user engagement and drain of battery power. A thorough evaluation of CityMap, conducted for more than two years over 11 different routes in 3 different cities in India, show that the system effectively annotates bus routes along with other route and road features with more than 90% of accuracy.

Typing based communication applications on smartphones, like WhatsApp, can induce emotional exchanges. The effects of an emotion in one session of communication can persist across sessions. In this work, we attempt automatic emotion detection by jointly modeling the typing characteristics, and the persistence of emotion. Typing characteristics, like speed, number of mistakes, special characters used, are inferred from typing sessions. Self reports recording emotion states after typing sessions capture persistence of emotion. We use this data to train a personalized machine learning model for multi-state emotion classification. We implemented an Android based smartphone application, called TapSense, that records typing related metadata, and uses a carefully designed Experience Sampling Method (ESM) to collect emotion self reports. We are able to classify four emotion states - happy, sad, stressed, and relaxed, with an average accuracy (AUCROC) of 84% for a group of 22 participants who installed and used TapSense for 3 weeks.

Public bus services in many cities in countries like India are controlled by private owners, hence, building up a database for all the bus routes is non-trivial. In this paper, we leverage smart-phone based sensing to crowdsource and populate the information repository for bus routes in a city. We have developed an intelligent data logging module for smart-phones and a server side processing mechanism to extract roads and bus routes information. From a 3 month long study involving more than 30 volunteers in 3 different cities in India, we found that the developed system, CrowdMap, can annotate bus routes with a mean error of 10m, while consuming 80% less energy compared to a continuous GPS based system.