Mental health challenges are increasingly prevalent among university students, yet often go undetected due to reliance on traditional assessments that are subjective, infrequent, and lack behavioral context. Digital phenotyping through passively collected smartphone data offers a scalable alternative, but existing approaches often fail to integrate predictive accuracy with narrative-based insights. To overcome these limitations, we present NarrativeSense, a novel framework that combines machine learning models with narrative-based descriptions of daily life events inferred from smartphone sensing data to predict weekly affective states. The system incorporates language model components to transform behavioral patterns into contextualized, human-readable narratives that ground affective predictions in everyday experiences. This narrative layer complements structured prediction by offering intuitive, user-centered insights. Applied to longitudinal data from 58 university students over 119 days, NarrativeSense outperforms baseline machine learning models, standalone LLMs, and ensemble methods, while providing richer insights. Our findings demonstrate the potential of narrative-enhanced digital phenotyping for scalable and explainable mental health monitoring in educational and clinical settings.

Smartphones are essential to daily life, and their rich data streams have been used to study how people use their phones, and more broadly human behaviour. While previous research has largely focused on app usage and keystroke dynamics to predict smartphone use, these analyses are typically limited to making predictions rather than providing explanations or reasoning for observed behaviours. In this exploratory study, we investigate the potential of leveraging screen text and large language models (LLMs) to uncover insights and reasoning about user behaviour. Using a dataset of over 100 million on-screen words collected from 21 participants over two weeks, we explore multiple ways to use screen text and LLMs for three tasks: predicting the next app a user will open, inferring what real-world activities they are engaged in, and understanding how they interact within apps. Orthogonally, we demonstrate the interpretive capabilities of LLMs, highlighting their potential to explain the reasoning behind observed user actions. Our findings suggest that screen text holds promise for providing deeper insights into both digital and real-world human behaviour. We discuss the broader implications of our findings, including enhancing user experience and enabling privacy-preserving, on-device analysis, while proposing future research directions in screen text analysis.

Context sensing on smartphones is often used to understand user behaviour. Amongst the many available sensors, the collection of text is crucial due to its richness. However, previous work has been limited to collecting text only from keyboard input, or intermittently collecting screen text indirectly by taking screenshots and applying optical character recognition. Here, we present a novel software sensor that unobtrusively and continuously captures all screen text on smartphones. We conducted a validation study with 21 participants over a two-week period, where they used our software on their personal smartphones. Our findings demonstrate how data from our sensor can be used to understand user behaviour and categorise mobile apps. We also show how smartphone sensing can be enhanced by using our sensor in conjunction with other sensors. We discuss the strengths and limitations of our sensor, highlighting potential areas for improvement and providing recommendations for its use.

Journaling offers significant benefits, including fostering self-reflection, enhancing writing skills, and aiding in mood monitoring. However, many people abandon the practice because traditional journaling is time-consuming, and detailed life events may be overlooked if not recorded promptly. Given that smartphones are the most widely used devices for entertainment, work, and socialization, they present an ideal platform for innovative approaches to journaling. Despite their ubiquity, the potential of using digital phenotyping, a method of unobtrusively collecting data from digital devices to gain insights into psychological and behavioral patterns, for automated journal generation has been largely underexplored. In this study, we propose AutoJournaling, the first-of-its-kind system that automatically generates journals by collecting and analyzing screenshots from smartphones. This system captures life events and corresponding emotions, offering a novel approach to digital phenotyping. We evaluated AutoJournaling by collecting screenshots every 3 seconds from three students over five days, demonstrating its feasibility and accuracy. AutoJournaling is the first framework to utilize seamlessly collected screenshots for journal generation, providing new insights into psychological states through digital phenotyping.

Daniel Lozoya, Alejandro Berazaluce, Juan Perches, Eloy Lúa, Mike Conway, Simon D’Alfonso. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

In this article, we show how a technology-mediated mental health therapy involving psycho-education, therapist moderators, and social networking can provide support for carers of young people with mental illness. This multi-faceted tool provides opportunities for users to adapt the system to their needs, leading us to refocus the goal of treatment adherence toward a relatively new phenomenon in HCI, concordance, which has not previously been examined in the HCI literature in relation to online mental-health tools. Concordance shares important links with the development of therapeutic alliance, which is centrally important to mental health therapy, and to Self-Determination Theory (SDT), which informed our approach to design. We present a three-month user study, which provides initial encouraging support for both the suitability of concordance as a lens for viewing user engagement and the idea that users can develop a therapeutic alliance with an online support system. This latter result is surprising as the phenomenon of therapeutic alliance generally describes a relationship between client and (human) clinician. Therapeutic alliance has previously been explored for face-to-face groups, and between individuals and online systems, but not for online groups. We show how even automated system behavior can encourage engagement from users and contribute to alliance formation, if the non-human parts of an online system are interactive. We argue that a design approach involving peer/moderator support as well as automated feedback, and which takes account of SDT, can provide support for therapeutic alliance.