Counterfactual reasoning is widely recognized as one of the most challenging and intricate aspects of causality in artificial intelligence. In this paper, we evaluate the performance of large language models (LLMs) in counterfactual reasoning. In contrast to previous studies that primarily focus on commonsense causal reasoning, where LLMs often rely on prior knowledge for inference, we specifically assess their ability to perform counterfactual inference using a set of formal rules. To support this evaluation, we introduce a new benchmark dataset, CounterBench, comprising 1.2K counterfactual reasoning questions. The dataset is designed with varying levels of difficulty, diverse causal graph structures, distinct types of counterfactual questions, and multiple nonsensical name variants. Our experiments demonstrate that counterfactual reasoning poses a significant challenge for LLMs, with most models performing at levels comparable to random guessing. To enhance LLM's counterfactual reasoning ability, we propose a novel reasoning paradigm, CoIn, which guides LLMs through iterative reasoning and backtracking to systematically explore counterfactual solutions. Experimental results show that our method significantly improves LLM performance on counterfactual reasoning tasks and consistently enhances performance across different LLMs.

OBJECTIVES: Skin cancer is the most common malignancy in the United States, with more than five million cases diagnosed annually among 3.3 million individuals. Melanoma, the deadliest form of skin cancer, accounts for roughly 200 000 new diagnoses each year and nearly 10 000 deaths. AI-based skin cancer detection is being developed and tested in laboratory and academic settings as a promising approach to improve access and reduce disparities. However, current models often underperform on darker skin tones (Fitzpatrick Types V and VI), creating fairness concerns that must be addressed prior to clinical deployment. Existing fairness-aware methods focus on algorithmic adjustments while neglecting data quality and representation. We introduce FAIR-SCAN (Fairness and Accuracy through Ranking-Based Subset Selection for Skin Cancer Detection), a data-centric framework that enhances fairness through subset selection guided by marginal contribution score (MCS) estimation. MATERIALS AND METHODS: FAIR-SCAN ranks data points by their contribution to both accuracy and fairness, then selects an optimal subset for training. We evaluated its effectiveness using images from Diverse Dermatology Images (DDI) and Fitzpatrick 17K. RESULTS: FAIR-SCAN improved balance in accuracy, True Positive Rate, and False Positive Rate across skin tones while reducing the training dataset by 50%, outperforming algorithm-focused fairness methods. DISCUSSION: These findings highlight the importance of strategic data selection in mitigating bias in AI-driven diagnostics. FAIR-SCAN's data-centric approach enhances both precision and equity in skin cancer detection. CONCLUSION: Strategic data selection is critical for equitable AI-driven diagnostics. FAIR-SCAN advances fairness and accuracy in skin cancer detection, supporting development of trustworthy clinical AI systems.

OBJECTIVE: Stigmatizing language (SL) in Electronic Health Records (EHRs) can perpetuate biases and negatively impact patient care. This study introduces a novel method for automatically detecting such language to improve healthcare documentation practices. MATERIALS AND METHODS: We developed a multi-stage transfer learning framework integrating semantic, syntactic, and task adaptation using three datasets: hate speech, clinical phenotypes, and stigmatizing language. Experiments were conducted on stigmatizing language dataset which consists of 4,129 de-identified EHR notes (72.7% stigmatizing, 27.3% non-stigmatizing), split 80/20 for training and testing. Longformer, BERT, and ClinicalBERT models were evaluated, and model performance was assessed on 35 randomized subsets of the test set (each comprising 70% of test data). The Wilcoxon-Mann-Whitney test was used to evaluate statistical significance, with Bonferroni correction applied to control for multiple hypothesis testing. Baseline models included zero-shot and few-shot GPT-4o, Support Vector Machine, Random Forest, Logistic Regression, and Multinomial Naive Bayes. RESULTS: The proposed framework achieved the highest accuracy, with fully adapted Longformer reaching 89.83%. Performance improvements remained statistically significant after Bonferroni correction compared to all baselines (p < .05). The framework demonstrated robust gains across different stigmatizing language types. DISCUSSION: This study underscores the value of domain-adaptive NLP for detecting stigmatizing language in EHRs. The multi-stage transfer learning framework effectively captures subtle biases often missed by conventional models, enabling more objective and respectful clinical documentation. CONCLUSION: This framework offers a statistically validated, high-performing framework for detecting stigmatizing language in EHRs, supporting responsible AI and promoting equity in clinical care.

Social media platforms face escalating harm from concentrated waves of toxic interactions, which we term Neg Storms. Unlike isolated abusive remarks, these storms emerge through rapid, correlated actions that amplify negativity and create severe risks for targets and communities. Existing moderation approaches largely focus on piecemeal detection of individual comments, missing the situational dynamics that drive escalation. This paper introduces a proactive framework for forecasting neg storms using early conversational signals. We formalize Comment Storm Severity (CSS) as a time-aware metric of thread-level toxicity, propose models that predict CSS from only the first$k$comments, and evaluate feature sets combining timing and content cues. Experiments on Reddit and Instagram show that timing features alone outperform content-only features, and that combining both yields the best performance (ROC-AUC$\approx 79.7 \%; R^{2} \approx 0.24$). While predictive scores are modest, these results validate the feasibility of anticipating harmful situations before they fully unfold. We discuss practical implications for platforms, including early checkpoints to prioritize high-risk threads, apply reversible friction, and route uncertain cases for human review. This work establishes an important starting point for research on situationlevel modeling of toxicity and proactive moderation in online communities. Note: This paper deals with a sensitive topic and includes examples of negative online comments.

Mobility services are becoming increasingly reliant on new technologies and mobile apps to manage and enable rides. In recent years, we have witnessed a rapid growth of technology-mediated mobility services (e.g., ridesharing and carsharing) with the ubiquity of smartphones. As an important technology-mediated mobility service involving interactions between passengers, drivers, and platforms, ridesharing has attracted great interest from the research community. Even though many existing studies have focused on the ridesharing experience of passengers, few of them have conducted a comprehensive study of passenger wait times in ridesharing systems. Prior research has shown that wait time is highly related to user experience. Understanding wait times in technology-mediated mobility systems and identifying factors that may impact them is of great importance for better user experience and the design of next-generation interactive mobility systems. Hence, in this paper, we adopt a mixed-methods approach to comprehensively examine two wait times in one of the largest technology-mediated mobility systems-DiDi, i.e., (i) the promised wait time shown on its mobile app after entering the origin and destination and (ii) the actual wait time. We first interviewed 102 individuals (including 52 passengers and 50 drivers) to understand people's perceptions of the two wait times in the DiDi ridesharing system. Our findings reveal that wait time discrepancy causes problems and negative emotions for passengers, and there are multiple potential factors that impact the discrepancy. To further verify some of these findings from a quantitative perspective, we performed a data-driven analysis based on large-scale ridesharing log data from over 36.6 million rides. Based on these findings, we share some design implications for ridesharing systems including those on minimizing expectation mismatch, supporting algorithmic transparency & fairness, and contextual factors consideration.

Our vision is to achieve societally responsible secure and trustworthy cyberspace that puts algorithmic and technological checks and balances on the indiscriminate sharing and analysis of data. We achieve this vision in a holistic manner by framing research directions with four major considerations: (i) Expanding knowledge and understanding of security and privacy perceptions and expectations in vulnerable groups, which significantly contribute to their unwillingness to share data, and use that knowledge to drive research in (a) mitigating missing/imbalanced data problems, (b) understanding and modeling security and privacy risks of data sharing, and (c) modeling utility of data sharing. (ii) Developing a risk-adaptive, policy model capable of capturing and articulating security and privacy expectations of users that are relevant in a particular context and develops associated technology to ensure provenance and accountability. (iii) Developing robust AI/ML algorithms that are transparent and explainable with respect to fairness and bias to reduce/eliminate discrimination, misuse, privacy violations, or other cyber-crimes. (iv) Developing models and techniques for a nuanced, contextually adaptive, and graded privacy paradigm that allows trade-offs between privacy and utility. Towards this, in this paper we present the SAFE-PASS framework to provide Stewardship, Advocacy, Fairness and Empowerment in Privacy, Accountability, Security, and Safety for Vulnerable Groups.

Abstract The popularity of location‐based features in social networks has been increasing over the past few years. Location information gathered from social networks can threaten users' information privacy through granular tracking and exposure of their preferences, behaviors, and identity. In this 6‐week study ( N = 35), we investigate the effect of “audience‐group”‐based interventions on Facebook check‐in behavior of participants. These “audience‐group”‐based nudges help close the gap between the users' perceived audiences and those that are permitted to view their check‐ins. The nudges remind users that their real‐time location information may be visible to a larger group of friends than they expect. Based on both quantitative and qualitative data analyses, we report that reminding users of the unexpected audiences that have access to their location check‐ins could be a promising way to help users manage their privacy in online location sharing. These findings motivate several recommendations for app designers as well as information privacy researchers to better design and evaluate location sharing in online social networks.

Despite the potential of online spaces for the democratic development of public discourse, concerns over aggressive and uncivil interactions in those spaces are rising. Online incivility is the term to define the features of a discussion that convey an unnecessarily disrespectful tone toward the participants or the topic. Concerning the adverse impact of online incivility on users, we sought to explore ways to reduce online incivility to promote secure and trustworthy online debate culture. Meanwhile, the web is becoming increasingly multimodal and images are known to be an effective way of improving emotions. The broaden-and-build theory of positive emotions suggests that positive emotions generated by positive images can impact incivility levels. Hence, with the goal to reduce online incivility, we conducted a one factorial between-subject online experiment with three conditions (N = 105). We compared the three conditions (with a positive background image in color, a positive background image in grayscale, and no background image) to identify an efficient way of reducing online incivility. The data gathered from surveys and participants' online comments were qualitatively and quantitatively analyzed to answer the research questions. The results showed that not only positive backgrounds in color but also positive backgrounds in grayscale may be effective in reducing online incivility. The results will pave way for designing more civil discussion platforms in online settings.

Abstract Filtering, vetting, and verifying digital information is an area of core interest in information science. Online fake news is a specific type of digital misinformation that poses serious threats to democratic institutions, misguides the public, and can lead to radicalization and violence. Hence, fake news detection is an important problem for information science research. While there have been multiple attempts to identify fake news, most of such efforts have focused on a single modality (e.g., only text‐based or only visual features). However, news articles are increasingly framed as multimodal news stories, and hence, in this work, we propose a multimodal approach combining text and visual analysis of online news stories to automatically detect fake news. Drawing on key theories of information processing and presentation, we identify multiple text and visual features that are associated with fake or credible news articles. We then perform a predictive analysis to detect features most strongly associated with fake news. Next, we combine these features in predictive models using multiple machine‐learning techniques. The experimental results indicate that a multimodal approach outperforms single‐modality approaches, allowing for better fake news detection.

Altruism, i.e., an act that one does at their own expense that tends to enhance others well-being, is a fundamental human behavior with implications for personal and societal welfare. Hence, modeling altruism is an important building block in designing human-centered computing systems. Traditional methods for understanding an individual's altruistic propensities have been surveys and lab experiments. However, the emerging “personal big data” coming from mobile and ubiquitous devices allows for creation of lower-cost, quicker, automated methods for modeling human behaviors and propensities. We propose a new methodology to model altruism using phone data. Based on analysis of data from a 10-week field study (N = 55 N = 55participants), we report that: (1) multiple phone-based features are associated with users' altruistic propensities; (2) phone features-based altruism prediction model yielded significantly better performance than a demography-based model. The results pave way for utilizing “personal big data” to model altruism in multiple commercial and social applications.

Abstract Media platforms, technological systems, and search engines act as conduits and gatekeepers for all kinds of information. They often influence, reflect, and reinforce gender stereotypes, including those that represent occupations. This study examines the prevalence of gender stereotypes on digital media platforms and considers how human efforts to create and curate messages directly may impact these stereotypes. While gender stereotyping in social media and algorithms has received some examination in the recent literature, its prevalence in different types of platforms (for example, wiki vs. news vs. social network) and under differing conditions (for example, degrees of human‐ and machine‐led content creation and curation) has yet to be studied. This research explores the extent to which stereotypes of certain strongly gendered professions (librarian, nurse, computer programmer, civil engineer) persist and may vary across digital platforms (Twitter, the New York Times online, Wikipedia, and Shutterstock). The results suggest that gender stereotypes are most likely to be challenged when human beings act directly to create and curate content in digital platforms, and that highly algorithmic approaches for curation showed little inclination towards breaking stereotypes. Implications for the more inclusive design and use of digital media platforms, particularly with regard to mediated occupational messaging, are discussed.

Cyberbullying, which often has a deeply negative impact on the victim, has grown as a serious issue in online social networks. Recently, researchers have created automated machine learning algorithms to detect Cyberbullying using social and textual features. However, the very algorithms that are intended to fight off one threat (cyberbullying) may inadvertently be falling prey to another important threat (bias of the automatic detection algorithms). This is exacerbated by the fact that while the current literature on algorithmic fairness has multiple empirical results, metrics, and algorithms for countering bias across immediately observable demographic characteristics (e.g. age, race, gender), there have been no efforts at empirically quantifying the variation in algorithmic performance based on the network role or position of individuals. We audit an existing cyberbullying algorithm using Twitter data for disparity in detection performance based on the network centrality of the potential victim and then demonstrate how this disparity can be countered using an Equalized Odds postprocessing technique. The results pave the way for more accurate and fair cyberbullying detection algorithms.

Multimedia research has now moved beyond laboratory experiments and is rapidly being deployed in real-life applications including advertisements, social interaction, search, security, automated driving, and healthcare. Hence, the developed algorithms now have a direct impact on the individuals using the abovementioned services and the society as a whole. While there is a huge potential to benefit the society using such technologies, there is also an urgent need to identify the checks and balances to ensure that the impact of such technologies is ethical and positive. This panel will bring together an array of experts who have experience collecting large-scale datasets, building multimedia algorithms, and deploying them in practical applications, as well as, a lawyer whose eyes have been on the fundamental rights at stake. They will lead a discussion on the ethics and lawfulness of dataset creation, licensing, privacy of individuals represented in the datasets, algorithmic transparency, algorithmic bias, explainability, and the implications of application deployment. Through an interactive process engaging the audience, the panel hopes to: increase the awareness of such concepts in the multimedia research community; initiate a discussion on community guidelines all for setting the future direction of conducting multimedia research in a lawful and ethical manner.

An individual's trust propensity - i.e., a dispositional willingness to rely on others" - mediates multiple socio-technical systems and has implications for their personal, and societal, well-being. Hence, understanding and modeling an individual's trust propensity is important for human-centered computing research. Conventional methods for understanding trust propensities have been surveys and lab experiments. We propose a new approach to model trust propensity based on long-term phone use metadata that aims to complement typical survey approaches with a lower-cost, faster, and scalable alternative. Based on analysis of data from a 10-week field study (mobile phone logs) and "ground truth" survey involving 50 participants, we: (1) identify multiple associations between phone-based social behavior and trust propensity; (2) define a machine learning model that automatically infers a person's trust propensity. The results pave way for understanding trust at a societal scale and have implications for personalized applications in the emerging social internet of things.

Emerging multimedia communication apps are allowing for more natural communication and richer user engagement. At the same time, they can be abused to engage in cyberbullying, which can cause significant psychological harm to those affected. Thus, with the growth in multimodal communication platforms, there is an urgent need to devise multimodal methods for cyberbullying detection and prevention. However, there are no existing approaches that use automated audio and video analysis to complement textual analysis. Based on the analysis of a human-labeled cyberbullying data-set of Vine "media sessions' (six-second videos, with audio, and corresponding text comments), we report that: 1) multiple audio and visual features are significantly associated with the occurrence of cyberbullying, and 2) audio and video features complement textual features for more accurate and earlier cyberbullying detection. These results pave the way for more effective cyberbullying detection in emerging multimodal (audio, visual, virtual reality) social interaction spaces.

The understanding and modeling of human purchase behavior in city environment can have important implications in the study of urban economy and in the design and organization of cities. In this article, we study human purchase behavior at the community level and argue that people who live in different communities but work at close-by locations could act as “social bridges” between the respective communities and that they are correlated with similarity in community purchase behavior. We provide empirical evidence by studying millions of credit card transaction records for tens of thousands of individuals in a city environment during a period of three months. More specifically, we show that the number of social bridges between communities is a much stronger indicator of similarity in their purchase behavior than traditionally considered factors such as income and sociodemographic variables. Our findings also suggest that such an effect varies across different merchant categories, that the presence of female customers in social bridges is a stronger indicator compared to that of their male counterparts, and that there seems to be a geographical constraint for this effect, all of which may have implications in the studies of urban economy and data-driven urban planning.

While self-reporting remains the most common method to understand human behavior, recent advances in social networks, mobile technologies, and other computer-mediated communication technologies are allowing researchers to obtain detailed logs of human behavior with ease. While the logged data is very useful (and accurate) at capturing the structure of the user's social network, the self-reported data provides an insight into the user's cognitive map of her social network. Based on a field study involving 47 users for a period of ten weeks we report that combining the two sets of data (self-reported and logged) gives higher predictive power than using either one of them individually. Further, the difference between the two types of values captures the level of dissonance between a user's actual and perceived social behavior and is found to be an important predictor of the person's social outcomes including social capital, social support and trust.

This exploratory work studies the effects of emerging app features on the cyberbullying practices in high school settings. These include the increasing prevalence of image/video content, perceived ephemerality, anonymity, and hyperlocal communication. Based on qualitative analysis of focus groups and follow-up individual interviews with high school students, these features were found to influence the practice of cyberbullying, as well as creating negative socio-psychological effects. For example, visual data was found to be used in cyberbullying settings as evidence of contentious events, a repeated reminder, and caused a graphic impact on recipients. Similarly, perceived ephemerality of content was found to be associated with "broken expectations" with respect to the apps and severe bullying outcomes for those affected. Results shed light on an important technology-mediated social phenomenon of cyberbullying, improve understanding of app use (and abuse) by the teenage user population, and pave the way for future research on countering appcentric cyberbullying.

Photographs are one of the most fundamental ways for human beings to capture their social experiences and smiling is one of the most common actions associated with photo-taking. Photos, thus provide a unique opportunity to study the phenomena of mixing of different people and also the smiles expressed by individuals in these social settings. In this work, we study whether a social media-based computational framework can be employed to obtain smile and diversity scores at very fine, individual relationship resolution, and study their associations. We analyze two data sets from different social networks, Twitter and Instagram, over different time periods. Primarily looking at photographs, using computer vision APIs, we capture the diversity of social interactions in terms of age, gender, and race of those present, and smile levels. Analysis of both data sets suggest similar and significant findings: (a) people, in general, tend to smile more in the presence of others; and (b) people tend to smile more in a more diverse company. The results can help scale, test, and validate multiple theories related to affect and diversity in sociology, psychology, biology, and urban planning, and inform future mechanisms for encouraging people to smile more often in everyday settings

Social capital is one of the most fundamental concepts in social computing. Individual social capital is often connected with one's happiness levels, well-being, and propensity to cooperate with others. The dominant approach for quantifying individual social capital remains self-reported surveys and generator-methods, which are costly, attention-consuming, and fraught with biases. Given the important role played by mobile phones in mediating human social lives, this study explores the use of phone metadata (call and SMS logs) to automatically infer an individual's social capital. Based on Williams' Social Capital survey as ground truth and ten-week phone data collection for 55 participants, we report that (1) multiple phone-based social features are intrinsically associated with social capital; and (2) analytics algorithms utilizing phone data can achieve high accuracy at automatically inferring an individual's bridging, bonding, and overall social capital scores. Results pave way for studying social capital and its temporal dynamics at an unprecedented scale.

No abstract available.

Cyberbullying is an important socio-technical challenge in Online Social Networks (OSN). With the growth trends of heterogeneous data in OSN, better network characterization, and textual feature sophistication, recent efforts have realized the value of looking at heterogeneous modes of information including textual features, social features, and image-based features for better cyberbullying detection. These approaches, however, still use these features either individually or combine them `naively' without considering the different confidence levels associated with each feature or the interdependencies between features. We propose a novel probabilistic information fusion framework that utilizes confidence score and interdependencies associated with different social and textual features and uses those to build better predictors for cyberbullying. The performance of the proposed approach was compared to a recent approach in literature which used a similar dataset and features and the proposed approach resulted in significant improvements in terms of cyberbullying detection.

As increasingly diverse facets of human life - including socializing, exercising, and information-seeking - are mediated by ubiquitous technology, they open the doors for the study of the hitherto under-explored interconnections between them. This work motivates and grounds the use of geo-exploration data to predict the information exploration behavior of users and to support their search. Based on a two-week field study involving 35 participants, we have identified multiple geo-exploration features that have significant associations with a user's information exploration behavior. We also found that the same geo-exploration features could be combined to build predictive models for various facets of an individual's information exploration behavior, and these models performed significantly better than comparable personality-based models.

Cooperation is a fundamental human concept studied across multiple social and biological disciplines. Traditional methods for eliciting an individual's propensity to cooperate have included surveys and laboratory experiments and multiple such studies have connected an individual's cooperation level with her social behavior. We describe a novel approach to model an individual's cooperation level based on her phoneotype i.e. a composite of an individual's traits as observable via a mobile phone. This phone sensing-based method can potentially complement surveys, thus providing a cheaper, faster, automated method for generating insights into cooperation levels of users. Based on a 10-week field study involving 54 participants, we report that: (1) multiple phone-based signals were significantly associated with participant's cooperation attitudes; and (2) combining phone-based signals yielded a predictive model with AUCROC of 0.945 that performed significantly better than a comparable demography-based model at predicting individual cooperation propensities. The results pave the way for individuals and organizations to identify more cooperative peers in personal, social, and commerce related settings.

Recent years have seen a significant increase in the number of sensors and resulting event related sensor data, allowing for a better monitoring and understanding of real-world events and situations. Event-related data come from not only physical sensors (e.g., CCTV cameras, webcams) but also from social or microblogging platforms (e.g., Twitter). Given the wide-spread availability of sensors, we observe that sensors of different modalities often independently observe the same events. We argue that fusing multimodal data about an event can be helpful for more accurate detection, localization and detailed description of events of interest. However, multimodal data often include noisy observations, varying information densities and heterogeneous representations, which makes the fusion a challenging task. In this paper, we propose a hybrid fusion approach that takes the spatial and semantic characteristics of sensor signals about events into account. For this, we first adopt the concept of an image-based representation that expresses the situation of particular visual concepts (e.g. "crowdedness", "people marching") called Cmage for both physical and social sensor data. Based on this Cmage representation, we model sparse sensor information using a Gaussian process, fuse multimodal event signals with a Bayesian approach, and incorporate spatial relations between the sensor and social observations. We demonstrate the effectiveness of our approach as a proof-of-concept over real-world data. Our early results show that the proposed approach can reliably reduce the sensor-related noise, locate the event place, improve event detection reliability, and add semantic context so that the fused data provides a better picture of the observed events.

Situation recognition is the problem of deriving actionable insights from heterogeneous, real-time, big multimedia data to benefit human lives and resources in different applications. This tutorial will discuss the recent developments towards converting multitudes of data streams including weather patterns, stock prices, social media, traffic information, and disease incidents into actionable insights.

The organisation of personal data is receiving increasing research attention due to the challenges we face in gathering, enriching, searching, and visualising such data. Given the increasing ease with which personal data being gathered by individuals, the concept of a lifelog digital library of rich multimedia and sensory content for every individual is fast becoming a reality. The LTA~2016 workshop aims to bring together academics and practitioners to discuss approaches to lifelog data analytics and applications; and to debate the opportunities and challenges for researchers in this new and challenging area.

No abstract available.

Social persuasion to influence the actions, beliefs, and behaviors of individuals, embedded in a social network, has been widely studied. It has been applied to marketing, healthcare, sustainability, political campaigns, and public policy. Traditionally, there has been a separation between physical (offline) and cyber (online) worlds. While persuasion methods in the physical world focused on strong interpersonal trust and design principles, persuasion methods in the online world were rich on data-driven analysis and algorithms. Recent trends including Internet of Things, “big data,” and smartphone adoption point to the blurring divide between the cyber world and the physical world in the following ways. Fine grained data about each individual's location, situation, social ties, and actions are collected and merged from different sources. The messages for persuasion can be transmitted through both worlds at suitable times and places. The impact of persuasion on each individual is measurable. Hence, we posit that the social persuasion will soon be able to span seamlessly across these worlds and will be able to employ computationally and empirically rigorous methods to understand and intervene in both cyber and physical worlds. Several early examples indicate that this will impact the fundamental facets of persuasion including who, how, where, and when, and pave way for multiple opportunities as well as research challenges.

The ability to understand social systems through the aid of computational tools is central to the emerging field of computational social systems. Such understanding can answer epistemological questions on human behavior in a data-driven manner, and provide prescriptive guidelines for persuading humans to undertake certain actions in real-world social scenarios. The growing number of works in this subfield has the potential to impact multiple walks of human life including health, wellness, productivity, mobility, transportation, education, shopping, and sustenance. The contribution of this paper is twofold. First, we provide a functional survey of recent advances in sensing, understanding, and shaping human behavior, focusing on real-world behavior of users as measured using passive sensors. Second, we present a case study on how trust, which is an important building block of computational social systems, can be quantified, sensed, and applied to shape human behavior. Our findings suggest that:1) trust can be operationalized and predicted via computational methods (passive sensing and network analysis) and 2) trust has a significant impact on social persuasion; in fact, it was found to be significantly more effective than the closeness of ties in determining the amount of behavior change.

Multimedia data are now created at a macro, public scale as well as individual personal scale. While distributed multimedia streams (e.g. images, microblogs, and sensor readings) have recently been combined to understand multiple spatio-temporal phenomena like epidemic spreads, seasonal patterns, and political situations; personal data (via mobile sensors, quantified-self technologies) are now being used to identify user behavior, intent, affect, social connections, health, gaze, and interest level in real time. An effective combination of the two types of data can revolutionize multiple applications ranging from healthcare, to mobility, to product recommendation, to content delivery. Building systems at this intersection can lead to better orchestrated media systems that may also improve users' social, emotional and physical well-being. For example, users trapped in risky hurricane situations can receive personalized evacuation instructions based on their health, mobility parameters, and distance to nearest shelter. This workshop bring together researchers interested in exploring novel techniques that combine multiple streams at different scales (macro and micro) to understand and react to each user's needs.

With the growth in social media, internet of things, and planetary-scale sensing there is an unprecedented need to assimilate spatio-temporally distributed multimedia streams into actionable information. Consequently the concepts like objects, scenes, and events, need to be extended to recognize situations (e.g. epidemics, traffic jams, seasons, flash mobs). This paper motivates and computationally grounds the problem of situation recognition. It describes a systematic approach for combining multimodal real-time big data into actionable situations. Specifically it presents a generic approach for modeling and recognizing situations. A set of generic building blocks and guidelines help the domain experts model their situations of interest. The created models can be tested, refined, and deployed into practice using a developed system (EventShop). Results of applying this approach to create multiple situation-aware applications by combining heterogeneous streams (e.g. Twitter, Google Insights, Satellite imagery, Census) are presented.

We are witnessing a phenomenal increase in multimodal human and device sensing to measure and report parameters such as temperature, vehicle speed, visual experiences, flu cases, and people happiness. Soon we expect these heterogeneous datasets (e.g. images, videos, weather sensors, check-ins and tweets) to become available in real-time in the Cloud for reasoning and decision making.

We demonstrate a system to generate dynamic media shows that are significantly richer than static slide shows, which are currently the most popular form of photo playback. The goal is to enable media reliving experiences that are aesthetically pleasing, interactive, and semantically drivable as they center on people, locations, time, and events discovered in a media collection. Dynamic shows allow for better sharing of one's media collections in diverse social networks because people have different time availabilities and perspectives, and hence may want to interact, customize, and reroute the media flow per their individual needs.

Billions of people worldwide use images and videos to capture various events in their lives. The primary purpose of the proposed media sharing application is digital re-living of those events by the photographers and their families and friends. The most popular tools for achieving this today are still static slide-shows (SSS) which primarily focus on visual effects rather than understanding the semantics of the media assets being used, or allowing different viewers (e.g. friends, family, who have different relationships, interests, time availabilities, and familiarities) any control over the flow of the show. We present a novel system that generates an aesthetically appealing and semantically drivable audio-visual media show based on several reliving dimensions of events, people, locations, and time. We allow each viewer to interact with the default presentation to 'on-the-fly' redirect the flow of reliving as desired from their individual perspectives. Moreover, each reliving session is logged and can be shared with other people over a wide array of platforms and devices, allowing sharing experience to go beyond the sharing of the media assets themselves. From a detailed analysis of the logged sessions across different user categories, we have obtained many interesting findings on the reliving needs, behaviors and patterns, which in turn validate our design motivations and principles.

Huge amounts of social multimedia is being created daily by a combination of globally distributed disparate sensors, including human-sensors (e.g. tweets) and video cameras. Taken together, this represents information about multiple aspects of the evolving world. Understanding the various events, patterns and situations emerging in such data has applications in multiple domains. We develop abstractions and tools to decipher various spatio-temporal phenomena which manifest themselves across such social media data. We describe an approach for aggregating social interest of users about any particular theme from any particular location into 'social pixels'. Aggregating such pixels spatio-temporally allows creation of social versions of images and videos, which then become amenable to various media processing techniques (like segmentation, convolution) to derive semantic situation information. We define a declarative set of operators upon such data to allow for users to formulate queries to visualize, characterize, and analyze such data. Results of applying these operations over an evolving corpus of millions of Twitter and Flickr posts, to answer situation-based queries in multiple application domains are promising.

Large volumes of spatio-temporal-thematic data being created using sites like Twitter and Jaiku, can potentially be combined to detect events, and understand various 'situations' as they are evolving at different spatio-temporal granularity across the world. Taking inspiration from traditional image pixels which represent aggregation of photon energies at a location, we consider aggregation of user interest levels at different geo-locations as social pixels. Combining such pixels spatio-temporally allows for creation of social images and video. Here, we describe how the use of relevant (media processing inspired) situation detection operators upon such 'images', and domain based rules can be used to decide relevant control actions. The ideas are showcased using a Swine flu monitoring application which uses Twitter data.

Events are the fundamental abstractions to study the dynamic world. We believe that the next generation of web (i.e. event-web), will focus on interconnections between events as they occur across space and time [3]. In fact we argue that the real value of large volumes of microblog data being created daily lies in its inherent spatio-temporality, and its correlation with the real-world events. In this context, we studied the structural properties of a corpus of 5,835,237 Twitter microblogs, and found it to exhibit Power laws across space and time, much like those exhibited by events in multiple domains. The properties studied over microblogs on different topics can be applied to study relationships between related events, as well as data organization for event-based, real-time, and location-aware applications.

We consider surveillance problems to be a set of system-adversary interaction problems in which an adversary can be modeled as a rational (selfish) agent trying to maximize his utility. We feel that appropriate adversary modeling can provide deep insights into the system performance and also clues for optimizing the system's performance against the adversary. Further, we propose that system designers should exploit the fact that they can impose certain restrictions on the intruders and the way they interact with the system. The system designers can analyze the scenario to determine conditions under which system outperforms the adversaries, and then suitably reengineer the environment under a "scenario engineering" approach to help the system outperform the adversary. We study the proposed enhancements using a game theoretic framework and present results of their adaptation to two significantly different surveillance scenarios. While the precise enforcements for the studied zero-sum ATM lobby monitoring scenario and the nonzero-sum traffic monitoring scenario were different, they lead to some useful generic guidelines for surveillance system designers.

In recent years, we have seen a significant research interest in a number of multimodal sensing applications like surveillance, video ethnography, tele-presence, assisted living, life blogging etc. However, these applications are currently evolving as separate silos with no interconnection. Further, the individual application-centric architectures typically tend to focus on specific sensors, specific (hardwired) queries and deal with specific environments. We present a generic sensing architecture 'Observation System', which allows multiple users to undertake different applications through abstracted interaction with a common set of sensors. The observation system observes behavior of various objects in an environment and keeps a record of important events and activities in an eventbase. In this system, multifarious data collected from disparate sensors and other sources are correlated to understand and gain insights in the environment. The observation system has applications in many areas including but not limited to surveillance, traffic monitoring, ethnography, marketing, and healthcare. In this paper, we present the architecture and functionality of such a system and present details of activity detection using multiple sensor streams in a distributed sensing environment. We also present results of such an approach and potential extensions to the analysis of more complex activities and events.

We consider surveillance problems to be a set of system-adversary interaction problems in which an adversary can be modeled as a rational (selfish) agent trying to maximize his utility. We feel that appropriate adversary modeling can provide deep insights into the system performance and also clues for optimizing the system's performance against the adversary. Further, we propose that system designers should exploit the fact that they can impose certain restrictions on the intruders and the way they interact with the system. The system designers can find the assumptions under which the surveillance system shall out-perform the intruder and then enforce those assumptions over the system-intruder interaction as part of a 'scenario engineering' approach. We study both these aspects using a game theoretic framework and undertake practical experiments to verify the proposed enhancements.