The Experience Sampling Method (ESM) is widely used to collect emotion self-reports to train machine learning models for emotion inference. However, as ESM studies are time-consuming and burdensome, participants often withdraw in between. This unplanned withdrawal compels the researchers to discard the dropout participants’ data, significantly impacting the quality and quantity of the self-reports. To address this problem, we leverage only the self-reporting similarity across participants (unlike prior works that apply different machine learning approaches on additional modalities) for missing self-report estimation. In specific, we propose a Multi-task Learning (MTL) framework, MUSE, that constructs the missing self-reports of the dropout participants. We evaluate MUSE in two in-the-wild studies (N1=24, N2=30) of 6-week and 8-week duration, during which the participants reported four emotions (happy, sad, stressed, relaxed) using a smartphone application. The evaluation reveals that MUSE estimates the missing emotion self-reports with an average AUCROC of 84% (Study I) and 82% (Study II). A follow-up evaluation of MUSE for an emotion inference (downstream) task reveals no significant difference in emotion inference performance when estimated self-reports are used. These findings underscore the utility of MUSE in estimating missing self-reports in ESM studies and the applicability of MUSE for downstream tasks (e.g., emotion inference).

Keyboard interaction patterns on a smartphone is the input for many intelligent emotion-aware applications, such as adaptive interface, optimized keyboard layout, automatic emoji recommendation in IM applications. The simplest approach, called the Experience Sampling Method (ESM), is to systematically gather self-reported emotion labels from users, which act as the ground truth labels, and build a supervised prediction model for emotion inference. However, as manual self-reporting is fatigue-inducing and attention-demanding, the self-report requests are to be scheduled at favorable moments to ensure high fidelity response. We, in this paper, perform fine-grain keyboard interaction analysis to determine suitable probing moments. Keyboard interaction patterns, both cadence, and latency between strokes, nicely translate to frequency and time domain analysis of the patterns. In this paper, we perform a 3-week in-the-wild study (N = 22) to log keyboard interaction patterns and self-report details indicating (in)opportune probing moments. Analysis of the dataset reveals that time-domain features (e.g., session length, session duration) and frequency-domain features (e.g., number of peak amplitudes, value of peak amplitude) vary significantly between opportune and inopportune probing moments. Driven by these analyses, we develop a generalized (all-user) Random Forest based model, which can identify the opportune probing moments with an average F-score of 93%. We also carry out the explainability analysis of the model using SHAP (SHapley Additive exPlanations), which reveals that the session length and peak amplitude have strongest influence to determine the probing moments.