Social media text data are often used to train Machine Learning (ML) models to identify users exhibiting high-risk mental health behaviors.However, sharing this sensitive data poses privacy risks and limits the growth of benchmark datasets.We comprehensively evaluate whether privacy-preserving ML techniques can enable safer data sharing while preserving performance.Specifically, we apply federated learning (FL) and Differentially Private FL for two widely-studied mental health prediction tasks: depression detection on X (Twitter) and suicide crisis detection on Reddit.We simulate realistic data-sharing scenarios by treating each user as a client in a non-IID setting, evaluating across different client fractions, aggregation strategies, and privacy budgets.While FL achieves comparable performance to centralized training (centralized 𝐹 1 = 85.63; best FL model 𝐹 1 = 83.16) on depression identification, we find that Differentially Private FL has a large performance-privacy trade-off (up to 𝐹 1 = 27.01 drop) even with low levels of noise (𝜖 = 50).This is due to the distortion of highly informative yet sparse mental health linguistic markers related to mental health, like health topics and emotion words.This research empirically demonstrates the potential and limitations of current privacy preservation techniques for mental health inference tasks.

Content moderators review disturbing content to protect social media users, often at significant cost to their mental health. Recent reports document the mental health conditions of African moderators as notably problematic. Beyond the content itself, what factors contribute to the deteriorating mental health of these workers? We surveyed 134 moderators across Africa to understand their mental health and interviewed 15 moderators to contextualize their experiences. We found that African moderators suffer from high psychological distress and lower well-being compared to moderators in other areas. Former moderators showed significantly higher distress levels, demonstrating long-term impact that extends beyond their moderation work. Our interviews showed that systemic and structural labor conditions contribute to moderators’ severe psychological distress and diminished mental well-being. Corporate wellness programs promoted by platforms were found ineffective and inadequate. We discuss how this requires holistic attention and structural solutions by all involved parties to improve moderators’ mental health.

Inclusion is important for meeting effectiveness, which is in turn central to organizational functioning. One way of improving inclusion in meetings is through feedback, but social dynamics make giving feedback difficult. We propose that AI agents can facilitate feedback exchange by being psychologically safer recipients, and we test this through a meeting system with an AI agent feedback mediator. When delivering feedback, the agent uses the Induced Hypocrisy Procedure, a social psychological technique that prompts behavior change by highlighting value-behavior inconsistencies. In a within-subjects lab study (n = 28), the agent made speaking times more balanced and improved meeting quality. However, a field study at a small consulting firm (n = 10) revealed organizational barriers that led to its use for personal reflection rather than feedback exchange. We contribute a novel sociotechnical system for feedback exchange in groups, and empirical findings demonstrating the importance of considering organizational barriers in designing AI tools for organizations.

Social media platforms have rapidly adopted algorithmic curation with little consideration for the potential harm to users' mental well-being. We present findings from design workshops with 21 participants diagnosed with mental illness about their interactions with social media platforms. We find that users develop cause-and-effect explanations, or folk theories, to understand their experiences with algorithmic curation. These folk theories highlight a breakdown in algorithmic design that we explain using the framework of entanglement, a phenomenon where there is a disconnect between users' actions and platform outcomes on an emotional level. Participants' designs to address entanglement and mitigate harms centered on contextualizing their engagement and restoring explicit user control on social media. The conceptualization of entanglement and the resulting design recommendations have implications for social computing and recommender systems research, particularly in evaluating and designing social media platforms that support users' mental well-being.

Video conferencing meetings are more effective when they are inclusive, but inclusion often hinges on meeting leaders' and/or co-facilitators' practices. AI systems can be designed to improve meeting inclusion at scale by moderating negative meeting behaviors and supporting meeting leaders. We explored this design space by conducting $9$ user-centered ideation sessions, instantiating design insights in a prototype ``virtual co-host'' system, and testing the system in a formative exploratory lab study ($n=68$ across $12$ groups, $18$ interviews). We found that ideation session participants wanted AI agents to ask questions before intervening, which we formalized as the ``Observe, Ask, Intervene'' (OAI) framework. Participants who used our prototype preferred OAI over fully autonomous intervention, but rationalized away the virtual co-host's critical feedback. From these findings, we derive guidelines for designing AI agents to influence behavior and mediate group work. We also contribute methodological and design guidelines specific to mitigating inequitable meeting participation.

Data scientists often formulate predictive modeling tasks involving fuzzy, hard-to-define concepts, such as the ''authenticity'' of student writing or the ''healthcare need'' of a patient. Yet the process by which data scientists translate fuzzy concepts into a concrete, proxy target variable remains poorly understood. We interview fifteen data scientists in education (N=8) and healthcare (N=7) to understand how they construct target variables for predictive modeling tasks. Our findings suggest that data scientists construct target variables through a bricolage process, in which they use creative and pragmatic approaches to make do with the limited data at hand. Data scientists attempt to satisfy five major criteria for a target variable through bricolage: validity, simplicity, predictability, portability, and resource requirements. To achieve this, data scientists adaptively apply problem (re)formulation strategies, such as swapping out one candidate target variable for another when the first fails to meet certain criteria (e.g., predictability), or composing multiple outcomes into a single target variable to capture a more holistic set of modeling objectives. Based on our findings, we present opportunities for future HCI, CSCW, and ML research to better support the art and science of target variable construction.

Data annotation underpins the success of modern AI, but the aggregation of crowd-collected datasets can harm the preservation of diverse perspectives in data. Difficult and ambiguous tasks cannot easily be collapsed into unitary labels. Prior work has shown that deliberation and discussion improve data quality and preserve diverse perspectives--however, synchronous deliberation through crowdsourcing platforms is time-intensive and costly. In this work, we create a Socratic dialog system using Large Language Models (LLMs) to act as a deliberation partner in place of other crowdworkers. Against a benchmark of synchronous deliberation on two tasks (Sarcasm and Relation detection), our Socratic LLM encouraged participants to consider alternate annotation perspectives, update their labels as needed (with higher confidence), and resulted in higher annotation accuracy (for the Relation task where ground truth is available). Qualitative findings show that our agent's Socratic approach was effective at encouraging reasoned arguments from our participants, and that the intervention was well-received. Our methodology lays the groundwork for building scalable systems that preserve individual perspectives in generating more representative datasets.

Online(-only) friendships have become increasingly common in daily lives post-COVID despite debates around their mental health benefits and equivalence to ''real'' relationships. Previous research has reflected a need to understand how online friends engage beyond individual platforms, and the lack of platform-agnostic inquiry limits our ability to fully understand the dynamics of online friendship. We employed an activity-grounded analysis of 25 interviews on lived experiences of close online friendship spanning multiple years. Our findings present unique challenges and strategies in online friendships, such as stigma from real-life circles, an ambivalent relationship with online communities, and counter-theoretical reappropriations of communication technology. This study contributes to HCI research in online communities and social interface design by refocusing prior impressions of strong vs. weak-ties in online social spaces and foregrounding time-stable interactions in design for relationship maintenance through technology. Our work also promotes critical reflection on biased perspectives towards technology-mediated practices and consideration of online friends as an invisible marginalized community.

Online community research routinely poses minimal risk to individuals, but does the same hold true for online communities? In response to high-profile breaches of online community trust and increased debate in the social computing research community on the ethics of online community research, this paper investigates community-level harms and benefits of research. Through 9 participatory-inspired workshops with four critical online communities (Wikipedia, InTheRooms, CaringBridge, and r/AskHistorians), we found researchers should engage more directly with communities' primary purpose by rationalizing their methods and contributions in the context of community goals to equalize the beneficiaries of community research. To facilitate deeper alignment of these expectations, we present the FACTORS (Functions for Action with Communities: Teaching, Overseeing, Reciprocating, and Sustaining) framework for ethical online community research. Finally, we reflect on our findings by providing implications for researchers and online communities to identify and implement functions for navigating community-level harms and benefits.

Given the scale at which online harassment occurs, researchers and practitioners alike have turned to computationally driven approaches to address it. However, because harassment is highly contextual and personal, designing effective solutions to this problem can be extremely challenging. This paper examines how harassment-mitigation systems studied in human-computer interaction (HCI) consider victim-centered principles in their design. Through a scoping literature review and close reading of 17 papers, we contribute—(1) a characterization of how novel and existing systems consider victims’ identity characteristics, definitions of harassment, and preferred strategies for dealing with harassment; (2) challenges faced by the systems along these dimensions to surface limitations, gaps, and tensions; (3) practical recommendations for researchers, designers, and practitioners to overcome these challenges. In doing so, we offer potential new directions to positively design computational approaches to addressing online harassment with victim-centered principles in mind.

Information is crucial to how people understand their mental health and well-being, and many turn to online sources found through search engines and social media. We present an interview study (n = 17) of participants who use online platforms to seek information about their mental illnesses. Participants use their personal information ecosystems in a cyclical process to find information. This cycle is driven by the adoption of new information and questioning the credibility of information. Privacy concerns fueled by perceptions of stigma and platform design also influence their information-seeking decisions. Our work proposes theoretical implications for social computing and information retrieval on information seeking in users’ personal information ecosystems. We offer design implications to support users in navigating personal information ecosystems to find mental health information.

Online platforms provide support for many kinds of distress, including suicidal thoughts and behaviors. However, because many platforms restrict suicidal talk, volunteers on these platforms struggle with how to help suicidal people who come for support. We interviewed 11 volunteer counselors in a large online support platform, including after they role-played conversations with varying severities of suicidality, to explore practices and challenges when identifying and responding to suicidality. We then presented Speed Dating design concepts around emotional preparation and support, real-time guidance, training, and suicide detection. Participants wanted more support and preparation for conversations with suicidal people, but were conflicted about AI-based technologies, including trade-offs between potential benefits of conversational agents for training and limitations of prediction or real-time response suggestions, due to the sensitive, context-dependent decisions that volunteers must make. Our work has important implications for nuanced considerations and design choices around developing digital mental health technologies.

The sudden popularity communities gain via algorithmically-curated "trending'" or "hot" social media feeds can be beneficial or disruptive. On one hand, increased attention often brings new users and promotes community growth. On the other hand, the unexpected influx of newcomers can burden already overworked moderation teams. To examine the impact of sudden popularity, we studied 6,306 posts that reached Reddit's front page---a feed called r/popular that millions of users browse daily---and the effects of sudden popularity within 1,320 subreddits. We find that on average, r/popular posts have 45 times the comments, 42 times the removed comments, and 70 times the number of newcomers compared to posts from the same community that did not reach r/popular. Additionally, r/popular posts led to a peak 85% median increase in the subreddit's comment rate, and these effects lingered for about 12 hours. Our regression analysis shows that stricter moderation and previous r/popular appearances were associated with shorter and less intense effects on the community. By quantifying the differential effects of sudden popularity, we provide recommendations for moderators to promote stability and community resilience in the face of unexpected disruptions.

Social media platforms are a place where people look for information and social support for mental health, resulting in both positive and negative effects on users. TikTok has gained notoriety for an abundance of mental health content and discourse. We present findings from a semi-structured interview study with 16 participants about mental health content and participants’ perceptions of community on TikTok. We find that TikTok’s community structure is permeable, allowing for self-discovery and understanding not found in traditional online communities. However, participants are wary of mental health information due to conflicts between a creator’s vulnerability and credibility. Our interviews suggest that the “For You Page" is a runaway train that encourages diverse community and content engagement but also displays harmful content that participants feel they cannot escape. We propose design implications to support better mental health, as well as implications for social computing research on community in algorithmic landscapes.

Peer production systems have frictions-mechanisms that make contributing more effortful-to prevent vandalism and protect information quality. Page protection on Wikipedia is a mechanism where the platform's core values conflict, but there is little quantitative work to ground deliberation. In this paper, we empirically explore the consequences of page protection on Internet Culture articles on Wikipedia (6,264 articles, 108 edit-protected). We first qualitatively analyzed 150 requests for page protection, finding that page protection is motivated by an article's (1) activity, (2) topic area, and (3) visibility. These findings informed a matching approach to compare protected pages and similar unprotected articles. We quantitatively evaluate the differences between protected and unprotected pages across two dimensions: editor engagement and contributor concentration. Protected articles show different trends in editor engagement and equity amongst contributors, affecting the overall disparity in the population. We discuss the role of friction in online platforms, new ways to measure it, and future work.

Getting training data for machine learning (ML) prediction of mental illness on social media data is labor intensive. To work around this, ML teams will extrapolate proxy signals, or alternative signs from data to evaluate illness status and create training datasets. However, these signals' validity has not been determined, whether signals align with important contextual factors, and how proxy quality impacts downstream model integrity. We use ML and qualitative methods to evaluate whether a popular proxy signal, diagnostic self-disclosure, produces a conceptually sound ML model of mental illness. Our findings identify major conceptual errors only seen through a qualitative investigation -- training data built from diagnostic disclosures encodes a narrow vision of diagnosis experiences that propagates into paradoxes in the downstream ML model. This gap is obscured by strong performance of the ML classifier (F1 = 0.91). We discuss the implications of conceptual gaps in creating training data for human-centered models, and make suggestions for improving research methods.

Workplace bias creates negative psychological outcomes for employees, permeating the larger organization. Workplace meetings are frequent, making them a key context where bias may occur. Video conferencing (VC) is an increasingly common medium for workplace meetings; we therefore investigated how VC tools contribute to increasing or reducing bias in meetings. Through a semi-structured interview study with 22 professionals, we found that VC features push meeting leaders to exercise control over various meeting parameters, giving leaders an outsized role in affecting bias. We demonstrate this with respect to four core VC features---user tiles, raise hand, text-based chat, and meeting recording---and recommend employing at least one of two mechanisms for mitigating bias in VC meetings---1) transferring control from meeting leaders to technical systems or other attendees and 2) helping meeting leaders better exercise the control they do wield.

The post-college transition is a critical period where individuals experience unique challenges and stress before, during, and after graduation. Individuals often use social media to discuss and share information, advice, and support related to post-college challenges in online communities. These communities are important as they fill gaps in institutional support between college and post-college plans. We empirically study the challenges and stress expressed on social media around this transition as students graduate college and move into emerging adulthood. We assembled a dataset of about 299,000 Reddit posts between 2008 and 2020 about the post-college transition from 10 subreddits. We extracted top concerns, challenges, and conversation points using unsupervised Latent Dirichlet Allocation (LDA). Then, we combined the results of LDA with binary transfer learning to identify stress expressions in the dataset (classifier performance at F1=0.94). Finally, we explore temporal patterns in stress expressions, and the variance of per-topic stress levels throughout the year. Our work highlights more deliberate and focused understanding of the post-college transition, as well as useful research and design impacts to study transient cohorts in need of support.

Opioid use disorder (OUD) poses substantial risks to personal well-being and public health. In online communities, users support those seeking recovery, in part by promoting clinically grounded treatments. However, some communities also promote clinically unverified OUD treatments, such as unregulated and untested drugs. Little research exists on which alternative treatments people use, whether these treatments are effective for recovery, or if they cause negative side effects. We provide the first large-scale social media study of clinically unverified, alternative treatments in OUD recovery on Reddit, partnering with an addiction research scientist. We adopt transfer learning across 63 subreddits to precisely identify posts related to opioid recovery. Then, we quantitatively discover potential alternative treatments and contextualize their effectiveness. Our work benefits health research and practice by identifying undiscovered recovery strategies. We also discuss the impacts to online communities dealing with stigmatized behavior and research ethics.

"Human-centered machine learning" (HCML) combines human insights and domain expertise with data-driven predictions to answer societal questions. This area's inherent interdisciplinarity causes tensions in the obligations researchers have to the humans whose data they use. This paper studies how scientific papers represent human research subjects in HCML. Using mental health status prediction on social media as a case study, we conduct thematic discourse analysis on 55 papers to examine these representations. We identify five discourses that weave a complex narrative of who the human subject is in this research: Disorder/Patient, Social Media, Scientific, Data/Machine Learning, and Person. We show how these five discourses create paradoxical subject and object representations of the human, which may inadvertently risk dehumanization. We also discuss the tensions and impacts of interdisciplinary research; the risks of this work to scientific rigor, online communities, and mental health; and guidelines for stronger HCML research in this nascent area.

We are in a transitional economic period emphasizing automation of physical jobs and the shift towards intellectual labor. How can we measure and understand human behaviors of job search, and how communities are adapting to these changes? We use internet search data to estimate employment demand in the United States. Starting with 225 million raw job search queries in 2015 and 2016 from a popular search engine, we classify queries into one of 15 fields of employment with accuracy and F-1 of 97%, and use the resulting query volumes to estimate per-sector employment demand in U.S. counties. We validate against Bureau of Labor Statistics measures, and then demonstrate benefits for communities, showing significant differences in the types of jobs searched for across socio-economic dimensions like poverty and education level. We discuss implications for macroeconomic measurement, as well as how community leaders, policy makers, and the field of HCI can benefit.

Online health communities (OHCs) provide support across conditions; for weight loss, OHCs offer support to foster positive behavior change. However, weight loss behaviors can also be subverted on OHCs to promote disordered eating practices. Using comments as proxies for support, we use computational linguistic methods to juxtapose similarities and differences in two Reddit weight loss communities, r/proED and r/loseit. We employ language modeling and find that word use in both communities is largely similar. Then, by building a word embedding model, specifically a deep neural network on comment words, we contrast the context of word use and find differences that imply different behavior change goals in these OHCs. Finally, these content and context norms predict whether a comment comes from r/proED or r/loseit. We show that norms matter in understanding how different OHCs provision support to promote behavior change and discuss the implications for design and moderation of OHCs.

Distinctive linguistic practices help communities build solidarity and differentiate themselves from outsiders. In an online community, one such practice is variation in orthography, which includes spelling, punctuation, and capitalization. Using a dataset of over two million Instagram posts, we investigate orthographic variation in a community that shares pro-eating disorder (pro-ED) content. We find that not only does orthographic variation grow more frequent over time, it also becomes more profound or “deep,” with variants becoming increasingly distant from the original: as, for example, #anarexyia is more distant than #anarexia from the original spelling #anorexia. We find that the these changes are driven by newcomers, who adopt the most extreme linguistic practices as they enter the community. Moreover, this behavior correlates with engagement with the community: the newcomers that adopt deeper variant orthography tend to remain active for longer in the community, and posts with deeper variation receive more positive feedback in the form of “likes.” Previous work has linked community membership change with language change, and our work casts this connection in a new light, with newcomers driving an evolving practice rather than adapting to it. We also demonstrate the utility of orthographic variation as a new lens to study sociolinguistic change in online communities, particularly when the change results from an exogenous force such as a content ban.

Social media sites are challenged by both the scale and variety of deviant behavior online. While algorithms can detect spam and obscenity, behaviors that break community guidelines on some sites are difficult because they have multimodal subtleties (images and/or text). Identifying these posts is often regulated to a few moderators. In this paper, we develop a deep learning classifier that jointly models textual and visual characteristics of pro-eating disorder content that violates community guidelines. Using a million Tumblr photo posts, our classifier discovers deviant content efficiently while also maintaining high recall (85%). Our approach uses human sensitivity throughout to guide the creation, curation, and understanding of this approach to challenging, deviant content. We discuss how automation might impact community moderation, and the ethical and social obligations of this area.

Social media sites like Facebook and Instagram remove content that is against community guidelines or is perceived to be deviant behavior. Users also delete their own content that they feel is not appropriate within personal or community norms. In this paper, we examine characteristics of over 30,000 pro-eating disorder (pro-ED) posts that were at one point public on Instagram but have since been removed. Our work shows that straightforward signals can be found in deleted content that distinguish them from other posts, and that the implications of such classification are immense. We build a classifier that compares public pro-ED posts with this removed content that achieves moderate accuracy of 69%. We also analyze the characteristics in content in each of these post categories and find that removed content reflects more dangerous actions, self-harm tendencies, and vulnerability than posts that remain public. Our work provides early insights into content removal in a sensitive community and addresses the future research implications of the findings.

Online communities can promote illness recovery and improve well-being in the cases of many kinds of illnesses. However, for challenging mental health condition like anorexia, social media harbor both recovery communities as well as those that encourage dangerous behaviors. The effectiveness of such platforms in promoting recovery despite housing both communities is underexplored. Our work begins to fill this gap by developing a statistical framework using survival analysis and situating our results within the cognitive behavioral theory of anorexia. This model identifies content and participation measures that predict the likelihood of recovery. From our dataset of over 68M posts and 10K users that self-identify with anorexia, we find that recovery on Tumblr is protracted - only half of the population is estimated to exhibit signs of recovery after four years. We discuss the effectiveness of social media in improving well-being around anorexia, a unique health challenge, and emergent questions from this line of work.

Social media sites have struggled with the presence of emotional and physical self-injury content. Individuals who share such content are often challenged with severe mental illnesses like eating disorders. We present the first study quantifying levels of mental illness severity (MIS) in social media. We examine a set of users on Instagram who post content on pro-eating disorder tags (26M posts from 100K users). Our novel statistical methodology combines topic modeling and novice/clinician annotations to infer MIS in a user's content. Alarmingly, we find that proportion of users whose content expresses high MIS have been on the rise since 2012 (13%/year increase). Previous MIS in a user's content over seven months can predict future risk with 81% accuracy. Our model can also forecast MIS levels up to eight months in the future with performance better than baseline. We discuss the health outcomes and design implications as well as ethical considerations of this line of research.

Pro-eating disorder (pro-ED) communities on social media encourage the adoption and maintenance of disordered eating habits as acceptable alternative lifestyles rather than threats to health. In particular, the social networking site Instagram has reacted by banning searches on several pro-ED tags and issuing content advisories on others. We pre-sent the first large-scale quantitative study investigating pro-ED communities on Instagram in the aftermath of moderation -- our dataset contains 2.5M posts between 2011 and 2014. We find that the pro-ED community has adopted non-standard lexical variations of moderated tags to circumvent these restrictions. In fact, increasingly complex lexical variants have emerged over time. Communities that use lexical variants show increased participation and support of pro-ED (15-30%). Finally, the tags associated with content on these variants express more toxic, self-harm, and vulnerable content. Despite Instagram's moderation strategies, pro-ED communities are active and thriving. We discuss the effectiveness of content moderation as an intervention for communities of deviant behavior.

Computing is an increasingly important component of many jobs and demand for computing skills is far outpacing the number of computationally literate workers available. Non-majors, adult learners, and other non-traditional students can potentially fill some of these positions. However, traditional CS education pathways do not currently address the unique needs of these students. New approaches to CS education that fit with the goals and lifestyles of non-traditional CS students are needed. In line with Computational Media approaches known to be successful with non-majors, we designed and implemented two graduate- level courses, one using a Pixelsense and the other using Arduino, to teach computational thinking, programming, and design skills. We compare findings from these two courses with specific focus on non-major graduate students, but including topics relevant for traditional CS educators, such as, the importance of choice of platform, structure of assignments, maintaining student motivation, and the impact of self-guided final projects.