In the era of large-scale visual data, understanding collections of images is a challenging yet important task. To this end, we introduce ImageSet2Text, a novel method to automatically generate natural language descriptions of image sets. Based on large language models, visual-question answering chains, an external lexical graph, and CLIP-based verification, ImageSet2Text iteratively extracts key concepts from image subsets and organizes them into a structured concept graph. We conduct extensive experiments evaluating the quality of the generated descriptions in terms of accuracy, completeness, and user satisfaction. We also examine the method's behavior through ablation studies, scalability assessments, and failure analyses. Results demonstrate that ImageSet2Text combines data-driven AI and symbolic representations to reliably summarize large image collections for a wide range of applications.

Ensembles of Deep Neural Networks, Deep Ensembles, are widely used as a simple way to boost predictive performance. However, their impact on algorithmic fairness is not well understood yet. Algorithmic fairness examines how a model’s performance varies across socially relevant groups defined by protected attributes such as age, gender, or race. In this work, we explore the interplay between the performance gains from Deep Ensembles and fairness. Our analysis reveals that they unevenly favor different groups, a phenomenon that we term the disparate benefits effect. We empirically investigate this effect using popular facial analysis and medical imaging datasets with protected group attributes and find that it affects multiple established group fairness metrics, including statistical parity and equal opportunity. Furthermore, we identify that the per-group differences in predictive diversity of ensemble members can explain this effect. Finally, we demonstrate that the classical Hardt post-processing method is particularly effective at mitigating the disparate benefits effect of Deep Ensembles by leveraging their better-calibrated predictive distributions.

Social networks contribute to the distribution of social capital, defined as the relationships, norms of trust and reciprocity within a community or society that facilitate cooperation and collective action. Therefore, better positioned members in a social network benefit from faster access to diverse information and higher influence on information dissemination. A variety of methods have been proposed in the literature to measure social capital at an individual level. However, there is a lack of methods to quantify social capital at a group level, which is particularly important when the groups are defined on the grounds of protected attributes. To fill this gap, we propose to measure the social capital of a group of nodes by means of the effective resistance and emphasize the importance of considering the entire network topology. Grounded in spectral graph theory, we introduce three effective resistance-based measures of group social capital, namely group isolation, group diameter and group control, where the groups are defined according to the value of a protected attribute. We denote the social capital disparity among different groups in a network as structural group unfairness, and propose to mitigate it by means of a budgeted edge augmentation heuristic that systematically increases the social capital of the most disadvantaged group. In experiments on real-world networks, we uncover significant levels of structural group unfairness when using gender as the protected attribute, with females being the most disadvantaged group in comparison to males. We also illustrate how our proposed edge augmentation approach is able to not only effectively mitigate the structural group unfairness but also increase the social capital of all groups in the network.

In this talk, I first present the mission and research focus of ELLIS Alicante, a pioneering non-profit research unit within the European Laboratory for Learning and Intelligent Systems (ELLIS). ELLIS 1 is a pan-European AI research network dedicated to advancing excellence in machine learning and Artificial Intelligence. Founded in 2018 in Germany, ELLIS was established in response to the growing global competition in AI, particularly from the United States and China, with the aim of retaining top talent within Europe and fostering a collaborative, high-impact AI research ecosystem. ELLIS promotes scientific excellence through a network of research units, programs, and fellowships that support fundamental research while also addressing key societal and industrial challenges.

The intersection between art and technology poses new challenges for creative expression in the digital space. This paper investigates the algorithmic censorship of artistic nudity in social platforms by means of a qualitative study via semi-structured interviews with 14 visual artists who have experienced censorship online. We explore the professional, emotional, financial and artistic consequences of content removal or shadow-banning. Focusing on the concept of artistic nudity, our findings emphasize the significant impact on artists of the algorithmic censorship of art, the need to consider art as a special case to safeguard the freedom of expression, the importance of education, the limitations of today’s content moderation algorithms and the pressing need for transparency and recourse mechanisms. We advocate for a multi-stakeholder governance model conducive to a more supportive, safer and inclusive online environment that respects and nurtures human creativity.

We describe the deep learning-based COVID-19 cases predictor and the Pareto-optimal Non-Pharmaceutical Intervention (NPI) prescriptor developed by the winning team of the 500k XPRIZE Pandemic Response Challenge. The competition aimed at developing data-driven AI models to predict COVID-19 infection rates and to prescribe NPI Plans that governments, business leaders and organizations could implement to minimize harm when reopening their economies. In addition to the validation performed by XPRIZE with real data, our models were validated in a real-world scenario thanks to an ongoing collaboration with the Valencian Government in Spain. Our experience contributes to a necessary transition to more evidence-driven policy-making during a pandemic.

This invited talk describes the work that a multi-disciplinary team of 20+ volunteer scientists did between March of 2020 and April of 2022, working very closely with the Presidency of the Valencian Government to support their decision-making during the COVID-19 pandemic in Spain. This team was known as the Data Science against COVID-19 taskforce. The team's work was structured in 4 areas: (1) large-scale human mobility modeling; (2) development of computational epidemiological models (metapopulation, individual and LSTM-based models); (3) development of predictive models of hospital and intensive care units' occupancy; and (4) a large-scale, online citizen surveys called the COVID19impactsurvey (https://covid19impactsurvey.org) with over 720,000 answers worldwide. This survey enabled us to shed light on the impact that the pandemic had on people's lives during the period of study [3,4,5]. In the talk, I will present the results obtained in each of these four areas, including winning the 500K XPRIZE Pandemic Response Challenge [1] and obtaining a best paper award at ECML-PKDD 2021 [2]. I will share the lessons learned in this very special initiative of collaboration between the civil society at large (through the citizen survey), the scientific community (through the Data Science against COVID-19 taskforce) and a public administration (through our collaboration with the Presidency of the Valencian Government). For those interested in knowing more about this initiative, WIRED magazine published an extensive article describing the story of this effort: https://www.wired.co.uk/article/valencia-ai-covid-data

Augmented Reality or AR filters on selfies have become very popular on social media platforms for a variety of applications, including marketing, entertainment and aesthetics. Given the wide adoption of AR face filters and the importance of faces in our social structures and relations, there is increased interest by the scientific community to analyze the impact of such filters from a psychological, artistic and sociological perspective. However, there are few quantitative analyses in this area mainly due to a lack of publicly available datasets of facial images with applied AR filters. The proprietary, close nature of most social media platforms does not allow users, scientists and practitioners to access the code and the details of the available AR face filters. Scraping faces from these platforms to collect data is ethically unacceptable and should, therefore, be avoided in research. In this paper, we present OpenFilter, a flexible framework to apply AR filters available in social media platforms on existing large collections of human faces. Moreover, we share FairBeauty and B-LFW, two beautified versions of the publicly available FairFace and LFW datasets and we outline insights derived from the analysis of these beautified datasets.

Human mobility prediction is a core functionality in many location-based services and applications. However, due to the sparsity of mobility data, it is not an easy task to predict future POIs (place-of-interests) that are going to be visited. In this paper, we propose MobTCast, a Transformer-based context-aware network for mobility prediction. Specifically, we explore the influence of four types of context in mobility prediction: temporal, semantic, social, and geographical contexts. We first design a base mobility feature extractor using the Transformer architecture, which takes both the history POI sequence and the semantic information as input. It handles both the temporal and semantic contexts. Based on the base extractor and the social connections of a user, we employ a self-attention module to model the influence of the social context. Furthermore, unlike existing methods, we introduce a location prediction branch in MobTCast as an auxiliary task to model the geographical context and predict the next location. Intuitively, the geographical distance between the location of the predicted POI and the predicted location from the auxiliary branch should be as close as possible. To reflect this relation, we design a consistency loss to further improve the POI prediction performance. In our experimental results, MobTCast outperforms other state-of-the-art next POI prediction methods. Our approach illustrates the value of including different types of context in next POI prediction.

Malaria is one of the primary causes of death in Mozambique, responsible for an estimated 15,000 deaths in 2017. Human-mediated parasite movement combined with spatial and seasonal changes in transmission threatens the success of malaria interventions by reintroducing parasites to areas targeted for elimination. In this study we used pseudonymized and aggregated call detail records from a sample of over 8.5 million subscribers of a mobile phone operator in Mozambique to quantify human movements for four months in 2017. We calculated parasite importation and exportation rates to identify potential net sinks and sources of malaria across districts. We used a community structure algorithm to explore the connectedness of provinces in Mozambique, and we calculated parasite mobility between urban and rural areas. This work will help provide evidence to inform malaria elimination strategies regionally.

As "Big Data" has become pervasive, an increasing amount of research has connected the dots between human behaviour in the offline and online worlds. Consequently, researchers have exploited these new findings to create models that better predict different aspects of human life and recommend future behaviour. To date, however, we do not yet fully understand the similarities and differences of human behaviour in these virtual and physical worlds. Here, we analyse and discuss the mobility and application usage of 400,000 individuals over eight months. We find an astonishing similarity between people's mobility in the physical space and how they move from app to app in smartphones. Our data shows that individuals use and visit a finite number of apps and places, but they keep exploring over time. In particular, two distinct profiles of individuals emerge: those that keep changing places and services, and those that are stable over time, named as "explorers" and "keepers". We see these findings as crucial to enrich a discussion for the potentials and the challenges of building human-centric AI systems, which might leverage recent results in Computational Social Science.

Collective urban mobility embodies the residents’ local insights on the city. Mobility practices of the residents are produced from their spatial choices , which involve various considerations such as the atmosphere of destinations, distance, past experiences, and preferences. The advances in mobile computing and the rise of geo-social platforms have provided the means for capturing the mobility practices; however, interpreting the residents’ insights is challenging due to the scale and complexity of an urban environment and its unique context. In this article, we present MobInsight, a framework for making localized interpretations of urban mobility that reflect various aspects of the urbanism. MobInsight extracts a rich set of neighborhood features through holistic semantic aggregation , and models the mobility between all-pairs of neighborhoods . We evaluate MobInsight with the mobility data of Barcelona and demonstrate diverse localized and semantically rich interpretations.

The exponential growth in smartphone adoption is contributing to the availability of vast amounts of human behavioral data. This data enables the development of increasingly accurate data-driven user models that facilitate the delivery of personalized services that are often free in exchange for the use of its customers’ data. Although such usage conventions have raised many privacy concerns, the increasing value of personal data is motivating diverse entities to aggressively collect and exploit the data. In this article, we unfold profiling scenarios around mobile HTTP(S) traffic, focusing on those that have limited but meaningful segments of the data. The capability of the scenarios to profile personal information is examined with real user data, collected in the wild from 61 mobile phone users for a minimum of 30 days. Our study attempts to model heterogeneous user traits and interests, including personality, boredom proneness, demographics, and shopping interests. Based on our modeling results, we discuss various implications to personalization, privacy, and personal data rights.

Mobile technology is becoming a loyal companion in our lives. It is used for increasing amounts of time during the day and night, enabling the development of intelligent user interfaces that characterize their users' traits and adapt to them. In this paper, we show how an individual's tendency to experience boredom, i.e. the personal trait called boredom proneness, affects the use of technology -- specifically a smartphone. We develop machine learning models to automatically classify individuals into high/low boredom proneness from their typical daily patterns of smartphone use. We thus propose boredom proneness as a trait with high potential to enable the design of personalized mobile services that are more meaningful to their users.

Boredom is a common human emotion which may lead to an active search for stimulation. People often turn to their mobile phones to seek that stimulation. In this paper, we tackle the challenge of automatically inferring boredom from mobile phone usage. In a two-week in-the-wild study, we collected over 40,000,000 usage logs and 4398 boredom self-reports of 54 mobile phone users. We show that a user-independent machine-learning model of boredom --leveraging features related to recency of communication, usage intensity, time of day, and demographics-- can infer boredom with an accuracy (AUCROC) of up to 82.9%. Results from a second field study with 16 participants suggest that people are more likely to engage with recommended content when they are bored, as inferred by our boredom-detection model. These findings enable boredom-triggered proactive recommender systems that attune their users' level of attention and need for stimulation.

Customer center call data is typically collected by organiza-tions and corporations in order to improve customer experience through the analysis of such call data. In this paper, we report our findings when analysing more than 26 thousand calls to the call centers of a large corporation in a Latin American country. We focus on the impact of gender and call duration on self-reported customer satisfaction. Speech-based gender detection technology is employed to automatically detect the gender of the customer and the agent involved in the calls. A significant correlation is found between self-reported customer satisfaction at the end of the call and gender homophily between the cus-tomer and the call center’s agent. Interestingly, we do not find any significant effect of call duration on satisfaction. Index Terms: conversation, gender recognition, dialogue anal-ysis interaction, speaker trait, computational paralinguistic

Mobile instant messaging (e.g., via SMS or WhatsApp) often goes along with an expectation of high attentiveness, i.e., that the receiver will notice and read the message within a few minutes. Hence, existing instant messaging services for mobile phones share indicators of availability, such as the last time the user has been online. However, in this paper we not only provide evidence that these cues create social pressure, but that they are also weak predictors of attentiveness. As remedy, we propose to share a machine-computed prediction of whether the user will view a message within the next few minutes or not. For two weeks, we collected behavioral data from 24 users of mobile instant messaging services. By the means of machine-learning techniques, we identified that simple features extracted from the phone, such as the user's interaction with the notification center, the screen activity, the proximity sensor, and the ringer mode, are strong predictors of how quickly the user will attend to the messages. With seven automatically selected features our model predicts whether a phone user will view a message within a few minutes with 70.6% accuracy and a precision for fast attendance of 81.2%

In the context of a myriad of mobile apps which collect personally identifiable information (PII) and a prospective market place of personal data, we investigate a user-centric monetary valuation of mobile PII. During a 6-week long user study in a living lab deployment with 60 participants, we collected their daily valuations of 4 categories of mobile PII (communication, e.g. phonecalls made/received, applications, e.g. time spent on different apps, location and media, e.g. photos taken) at three levels of complexity (individual data points, aggregated statistics and processed, i.e. meaningful interpretations of the data). In order to obtain honest valuations, we employ a reverse second price auction mechanism. Our findings show that the most sensitive and valued category of personal information is location. We report statistically significant associations between actual mobile usage, personal dispositions, and bidding behavior. Finally, we outline key implications for the design of mobile services and future markets of personal data.

In this paper, we present a novel approach to predict crime in a geographic space from multiple data sources, in particular mobile phone and demographic data. The main contribution of the proposed approach lies in using aggregated and anonymized human behavioral data derived from mobile network activity to tackle the crime prediction problem. While previous research efforts have used either background historical knowledge or offenders' profiling, our findings support the hypothesis that aggregated human behavioral data captured from the mobile network infrastructure, in combination with basic demographic information, can be used to predict crime. In our experimental results with real crime data from London we obtain an accuracy of almost 70% when predicting whether a specific area in the city will be a crime hotspot or not. Moreover, we provide a discussion of the implications of our findings for data-driven crime analysis.

Optimizing the use of available resources is one of the key challenges in activities that consist of interactions with a large number of “target individuals,” with the ultimate goal of “winning” as many of them as possible, such as in marketing, service provision, political campaigns, or homeland security. Typically, the cost of interactions is monotonically increasing such that a method for maximizing the performance of these campaigns iPs required. In this paper, we propose a mathematical model to compute an optimized campaign by automatically determining the number of interacting units and their type, and how they should be allocated to different geographical regions in order to maximize the campaign's performance. We validate our proposed model using real world mobility data.

The goal of this work is to provide a fundamental understanding of the daily information needs of people through a large-scale, in-depth, quantitative investigation. To this end, we have conducted one of the most comprehensive studies of information needs to date, spanning a 3-month period and involving more than 100 users. The study employed a contextual experience sampling method, a snippet-based diary technique using SMS technology, and an online Web diary to gather in situ insights into the types of needs that occur from day to day, how those needs are addressed, and how contextual, technological, and demographic factors impact on those needs. Our results not only complement earlier studies but also provide a new understanding of the intricacies of people’s daily information needs.

We propose a conceptual model that explains the relationship between the users' personality profile and their satisfaction with basic mobile phone services (calls, messages, and simple GPRS/3G services). The model captures direct and indirect effects on satisfaction by means of two variables: actual mobile phone usage and perceived usability of the related services. We empirically validate the model with data gathered from 603 customers of a telecommunication operator, and find that: (1) extroversion , conscientiousness , and intellect have a significant impact on customer satisfaction—positively for the first two traits and negatively for the latter; (2) extroversion positively influences mobile phone usage; and (3) extroversion and conscientiousness positively influence the users' perceived usability of mobile services. Interestingly, usability has the strongest positive impact on satisfaction, whereas mobile phone usage has a negative impact on satisfaction. We discuss key findings of this model and propose several implications for the design of mobile phone services.

Recent work in mobile computing has highlighted that conversations and social interactions have a significant impact on mobile Web and mobile search behaviours. To date, however, this social element has not been explored fully and little is known about why and how mobile users search for information in social settings. The goal of this work is to provide a deeper understanding of social mobile search behaviours so that we may improve future mobile search experiences that involve a social component. To this end we present the results of two studies: a survey involving almost 200 users and a two-week diary and follow-up interview study of 20 users. Our results extend past research in the mobile search space, by exploring the motivations, circumstances and experiences of using mobile search in social settings to satisfy group information needs. Our findings point to a number of open research challenges and implications for enriching the search experiences of mobile users.

In this paper we tackle the problem of recommendation in the scenarios with binary relevance data, when only a few (k) items are recommended to individual users. Past work on Collaborative Filtering (CF) has either not addressed the ranking problem for binary relevance datasets, or not specifically focused on improving top-k recommendations. To solve the problem we propose a new CF approach, Collaborative Less-is-More Filtering (CLiMF). In CLiMF the model parameters are learned by directly maximizing the Mean Reciprocal Rank (MRR), which is a well-known information retrieval metric for measuring the performance of top-k recommendations. We achieve linear computational complexity by introducing a lower bound of the smoothed reciprocal rank metric. Experiments on two social network datasets demonstrate the effectiveness and the scalability of CLiMF, and show that CLiMF significantly outperforms a naive baseline and two state-of-the-art CF methods.

In this paper, we tackle the problem of top-N context-aware recommendation for implicit feedback scenarios. We frame this challenge as a ranking problem in collaborative filtering (CF). Much of the past work on CF has not focused on evaluation metrics that lead to good top-N recommendation lists in designing recommendation models. In addition, previous work on context-aware recommendation has mainly focused on explicit feedback data, i.e., ratings. We propose TFMAP, a model that directly maximizes Mean Average Precision with the aim of creating an optimally ranked list of items for individual users under a given context. TFMAP uses tensor factorization to model implicit feedback data (e.g., purchases, clicks) with contextual information.

The increasing number of images available online has created a growing need for efficient ways to search for relevant content. Text-based query search is the most common approach to retrieve images from the Web. In this approach, the similarity between the input query and the metadata of images is used to find relevant information. However, as the amount of available images grows, the number of relevant images also increases, all of them sharing very similar metadata but differing in other visual characteristics. This paper studies the influence of visual aesthetic quality in search results as a complementary attribute to relevance. By considering aesthetics, a new ranking parameter is introduced aimed at improving the quality at the top ranks when large amounts of relevant results exist. Two strategies for aesthetic rating inference are proposed: one based on visual content, another based on the analysis of user comments to detect opinions about the quality of images. The results of a user study with $58$ participants show that the comment-based aesthetic predictor outperforms the visual content-based strategy, and reveals that aesthetic-aware rankings are preferred by users searching for photographs on the Web.

The e-book industry is starting to flourish due, in part, to the availability of affordable and user-friendly e-book readers. As users are increasingly moving from traditional paper books to e-books, there is an opportunity to reinvent and enhance their reading experience, for example, by leveraging the multimedia capabilities of these devices in order to turn the act of reading into a real multimedia experience. In this paper, we focus on the augmentation of the written text with its associated audiobook, so that users can listen to the book they are (currently) reading. We propose an audiobook-to-ebook alignment system by applying a Text-to-Speech(TTS)-based text to audio alignment algorithm, and enhance it with a silence filtering algorithm to cope with the difference on reading style between the TTS output and the speakers in the ebook environment. Experiments done using 12 five-minute excerpts of 6 different audio-books (read by men and women) yield usable word alignment errors below 120ms for 90% of the words. Finally, we also show a user interface implementation in the Ipad for synchronized e-book reading while listening to the associated audiobook.

With the constant improvements in the technical capabilities and bandwidth available to mobile phones, mobile audio and video streaming services are booming. This allows music enthusiasts to watch their favourite music videos instead of only listening to the audio tracks, thus augmenting their listening experience to be multimodal. Usually the highly compressed audio tracks in these videos result in poorer quality music, compared to music the user might already have locally on their phone or playing through another sound source. In this paper we present MuViSync Mobile, a mobile phone application that synchronises real-time high quality music, either stored locally or input through the microphone, with the corresponding streaming music video. We extend previous work on music to music video synchronisation by proposing an alternative algorithm for higher efficiency with similar alignment accuracy, which we tested with music video examples including simulated noise. This algorithm correctly aligns 90% of the audio frames to within 100 ms of the known alignment. We also describe its implementation on an iPhone.

This paper presents ethnographic observations, a diary study and a large-scale quantitative questionnaire (n=395) designed to study the reasons for adoption and refusal of context-aware mobile applications. Through a qualitative study we identify 24 user needs that these applications fulfill and 9 barriers for adoption. We found that for many of the identified needs the end-goal is not that of receiving information, thus complementing work on mobile information needs. Also, this work offers an actionable list of obstacles that prevent contextual services to reach a larger audience. Finally, our findings suggest the opportunity to develop novel mobile applications that fulfill needs in the activity and personal contextual dimensions, and that of developing an application store for feature phones.

The term mobile Web is changing. Mobile is traditionally associated with on-the-move, portable and dynamic. However, with the advent of smartphones, an increasing number of users are accessing the mobile Internet via their phone while in more stationary and familiar settings, like at home or at work. This shift in the meaning of mobile is having a significant effect on mobile Web behavior. Designing great mobile Web experiences requires a deeper understanding of the information needs, behaviors and underlying motivations of mobile users. As such, the goal of this work is to study this shift and its impact on mobile Internet access, with a view to determining what this means for the future of the mobile Web and in particular mobile search. In this paper we present the results of an online diary and interview study of 18 active mobile Web users over a 4-week period focusing on how, why, where and in what situations people use the mobile Internet and mobile search. Our findings raise a new set of open research questions and point to a number of implications for enriching the experiences of mobile Web users.

Content-based video copy detection algorithms (CBCD) focus on detecting video segments that are identical or transformed versions of segments in a known video. In recent years some systems have proposed the combination of orthogonal modalities (e.g. derived from audio and video) to improve detection performance, although not always achieving consistent results. In this paper we propose a fusion algorithm that is able to combine as many modalities as available at the decision level. The algorithm is based on the weighted sum of the normalized scores, which are modified depending on how well they rank in each modality. This leads to a virtually parameter-free fusion algorithm. We performed several tests using 2010 TRECVID VCD datasets and obtain up to 46% relative improvement in min-NDCR while also improving the F1 metric on the fused results in comparison to just using the best single modality.

People share pictures online to increase their social presence. However, recent studies have shown that most of the content shared in social networks is not looked at by peers. Proper metadata can be generated and used to improve the retrieval of this content. In spite of this, we still lack solutions for collecting valid descriptors of content that can be used effectively in the context of social information navigation. In this paper, we propose a mechanism based on persuasive techniques to support peers in providing metadata for multimedia content that can be used for a person's self-promotion. Through an iterative design and experimentation process, we demonstrate how this methodology can be used effectively to increase one's social presence thus building more enjoyable, rich, and creative content that is shared in the social network. In addition, we highlight implications that inform the design of social games with a purpose.

Social networks mediate not only the relations between entities, but also the patterns of information propagation among them and their communication behavior. In this paper, we extensively study the temporal annotations (e.g., time stamps and duration) of historical communications in social networks and propose two novel tools -- communication motifs and maximum-flow communication motifs -- for characterizations of the patterns of information propagation in social networks. Using these motifs, we verify the following hypothesis in social communication network: 1) the functional behavioral patterns of information propagation within both social networks are stable over time; 2) the patterns of information propagation in synchronous and asynchronous social networks are different and sensitive to the cost of communication; and 3) the speed and the amount of information that is propagated through a network are correlated and dependent on individual profiles.

Medication compliance is a critical component in the success of any medical treatment. However, only 50% of patients correctly adhere to their prescription regimens. Mobile and ubiquitous technologies have been proposed to tackle this challenge, mainly in the form of memory aid solutions that remind patients to take their pills. However, most of these methods do not engage patients in shifting their behavior towards better compliance. In this paper, we propose and evaluate a mobile phone-based game called MoviPill that persuades patients to be more adherent to their medication prescription by means of social competition. In a 6-week user study conducted with 18 elders, the use of MoviPill improved both their compliance to take the daily medication and also the accuracy of the drug intake time according to the prescribed time. Moreover, the improvement in the latter increased from 43% to 56% when we considered only participants that had any interest in games, which reveals the importance of applying persuasive techniques in a personalized manner. We conclude with a set of implications for the design of persuasive mobile solutions in this domain.

Before the advent of Hidden Markov Models(HMM)-based speech recognition, many speech applications were built using pattern matching algorithms like the Dynamic Time Warping (DTW) algorithm, which are generally robust to noise and easy to implement. The standard DTW algorithm usually suffers from lack of flexibility on start-end matching points and has high computational costs. Although some DTW-based algorithms have been proposed over the years to solve either one of these problems, none is able to discover multiple alignment paths with low computational costs. In this paper, we present an “unbounded” version on the DTW (U-DTW in short) that is computationally lightweight and allows for total flexibility on where the matching segment occurs. Results on a word matching database show very competitive performances both in accuracy and processing time compared to existing alternatives.

In this paper, we explore the role that image composition plays in image aesthetic appeal classification. We propose low-level image composition features that approximate traditional photography composition guidelines, such as simplicity and visual balance (e.g. golden mean, golden triangles). We then use these features to build an image aesthetics classifier that we test with a baseline dataset. Interestingly, our approach, that only takes into account image composition features, yields close to state-of-the-art image aesthetic-based classification accuracies, which highlights the importance of image composition in image aesthetic appeal assessment.

In recent years, the popularity of compressed music files and online music downloads has increased dramatically. Today's users own large digital collections of high quality music on their computers and portable devices to be played in their homes or on the go. In addition, music videos are being offered online both for free and through monthly subscriptions, opening up the opportunity to turn the music listening activity into a multimedia experience. The work presented in this paper addresses the challenge of music audio and music video synchronisation. In particular, we have developed a prototype, named MuViSync, to automatically synchronise music videos to the songs that users are listening to in real-time. At the core of the MuViSync prototype are novel audio synchronization algorithms to tackle the differences in tempo, pitch, sampling rates, structure, and introductions and endings that are common in the various digital recordings of the same song in modern music. The music and the music video are initially aligned and then kept in sync within the limits of human perception. In experiments with 320 matching pairs of audio files and music videos, the proposed algorithms successfully synchronise music and video to within 100 milliseconds of each other in over 90% of the cases.

The mobile Internet offers anytime, anywhere access to a wealth of information to billions of users across the globe. However, the mobile Internet represents a challenging information access platform due to the inherent limitations of mobile environments, limitations that go beyond simple screen size and network issues. Mobile users often have information needs which are impacted by contexts such as location and time. Furthermore, human beings are social creatures that often seek out new strategies for sharing knowledge and information in mobile settings. To investigate the social aspect of mobile search, we have developed SocialSearchBrowser (SSB), a novel proof-of-concept interface that incorporates social networking capabilities with key mobile contexts to improve the search and information discovery experience of mobile users. In this paper, we present the results of an exploratory field study of SSB and outline key implications for the design of next generation mobile information access services.

Information overload is one of today's major concerns. As high-resolution digital cameras become increasingly pervasive, unprecedented amounts of social media are being uploaded to online social networks on a daily basis. In order to support users on selecting the best photos to create an online photo album, attention has been devoted to the development of automatic approaches for photo storytelling. In this paper, we present a novel photo collection summarization system that learns some of the users' social context by analyzing their online photo albums, and includes storytelling principles and face and image aesthetic ranking in order to assist users in creating new photo albums to be shared online. In an in-depth user study conducted with 12 subjects, the proposed system was validated as a first step in the photo album creation process, helping users reduce workload to accomplish such a task. Our findings suggest that a human audio/video professional with cinematographic skills does not perform better than our proposed system.

Context has been recognized as an important factor to consider in personalized Recommender Systems. However, most model-based Collaborative Filtering approaches such as Matrix Factorization do not provide a straightforward way of integrating context information into the model. In this work, we introduce a Collaborative Filtering method based on Tensor Factorization, a generalization of Matrix Factorization that allows for a flexible and generic integration of contextual information by modeling the data as a User-Item-Context N-dimensional tensor instead of the traditional 2D User-Item matrix. In the proposed model, called Multiverse Recommendation, different types of context are considered as additional dimensions in the representation of the data as a tensor. The factorization of this tensor leads to a compact model of the data which can be used to provide context-aware recommendations.

As the mobile Internet continues to grow, there is an increasing need to provide users with effective search and information access services. In order to build more effective mobile search services, we must first understand the impact that various interface choices have on mobile users. For example, the majority of mobile location-based search services are built on top of a map visualization, but is this intuitive design-decision the optimal interface choice from a human centric perspective? In order to tackle this fundamental design question, we have developed two proactive mobile search interfaces (one map-based and the other text-based) that utilize key mobile contexts to improve the search and information discovery experience of mobile users. In this paper, we present the results of an exploratory field study of these two interfaces - involving 34 users over a 1 month period - where we focus in particular on the impact that the type of user interface (e.g. map vs text) has on the search and information discovery experience of mobile users. We highlight the main usage results - including that maps are not the interface of choice for certain information access tasks - and outline key implications for the design of next generation mobile search services.

Popular content in video sharing websites (e.g., YouTube) is usually replicated via identical copies or near-duplicates. These duplicates are usually studied because they pose a threat to site owners in terms of wasted disk space, or privacy infringements. Furthermore, this content might potentially hinder the users' experience in these websites. The research presented in this article focuses around the central argument that there is no agreement on the technical definition of what these near-duplicates are, and, more importantly, there is no strong evidence that users of video sharing websites would like this content to be removed. Most scholars define near-duplicate video clips (NDVC) by means of non-semantic features (e.g., different image/audio quality), while a few also include semantic features (i.e., different videos of similar content). However, it is unclear what features contribute to the human perception of near-duplicate videos. The findings of four large scale online surveys that were carried out in the context of our research confirm the relevance of both types of features. Some of our findings confirm the adopted definitions of NDVC whereas other findings are surprising: Near-duplicate videos with different image quality, audio quality, or with/without overlays were perceived as NDVC. However, the same could not be verified when videos differed by more than one of these features at the same time. With respect to semantics, it is yet unclear the exact role that it plays in relation to the features that make videos alike. From a user's perspective, participants preferred in most cases to see only one of the NDVC in the search results of a video search query and they were more tolerant to changes in the audio than in the video tracks. Based on all these findings, we propose a new user-centric NDVC definition and present implications for how duplicate content should be dealt with by video sharing Web sites.

Speech and typed text are two common input modalities for mobile phones. However, little research has compared them in their ability to support annotation and retrieval of digital pictures on mobile devices. In this paper, we report the results of a month-long field study in which participants took pictures with their camera phones and had the choice of adding annotations using speech, typed text, or both. Subsequently, the same subjects participated in a controlled experiment where they were asked to retrieve images based on annotations as well as retrieve annotations based on images in order to study the ability of each modality to effectively support users' recall of the previously captured pictures. Results demonstrate that each modality has advantages and shortcomings for the production of tags and retrieval of pictures. Several guidelines are suggested when designing tagging applications for portable devices.

Popular content in video sharing web sites (e.g., YouTube) is usually duplicated. Most scholars define near-duplicate video clips (NDVC) based on non-semantic features (e.g., different image/audio quality), while a few also include semantic features (different videos of similar content). However, it is unclear what features contribute to the human perception of similar videos. Findings of two large scale online surveys (N = 1003) confirm the relevance of both types of features. While some of our findings confirm the adopted definitions of NDVC, other findings are surprising. For example, videos that vary in visual content - by overlaying or inserting additional information - may not be perceived as near-duplicate versions of the original videos. Conversely, two different videos with distinct sounds, people, and scenarios were considered to be NDVC because they shared the same semantics (none of the pairs had additional information). Furthermore, the exact role played by semantics in relation to the features that make videos alike is still an open question. In most cases, participants preferred to see only one of the NDVC in the search results of a video search query and they were more tolerant to changes in the audio than in the video tracks. Finally, we propose a user-centric NDVC definition and present implications for how duplicate content should be dealt with by video sharing websites.

A common approach to designing Recommender Systems (RS) consists of asking users to explicitly rate items in order to collect feedback about their preferences. However, users have been shown to be inconsistent and to introduce a non-negligible amount of natural noise in their ratings that affects the accuracy of the predictions. In this paper, we present a novel approach to improve RS accuracy by reducing the natural noise in the input data via a preprocessing step. In order to quantitatively understand the impact of natural noise, we first analyze the response of common recommendation algorithms to this noise. Next, we propose a novel algorithm to denoise existing datasets by means of re-rating: i.e. by asking users to rate previously rated items again. This denoising step yields very significant accuracy improvements. However, re-rating all items in the original dataset is unpractical. Therefore, we study the accuracy gains obtained when re-rating only some of the ratings.In particular, we propose two partial denoising strategies: data and user-dependent denoising. Finally, we compare the value of adding a rating of an unseen item vs. re-rating an item. We conclude with a proposal for RS to improve the quality of their user data and hence their accuracy: asking users to re-rate items might, in some circumstances, be more beneficial than asking users to rate unseen items.

Nearest-neighbor collaborative filtering provides a successful means of generating recommendations for web users. However, this approach suffers from several shortcomings, including data sparsity and noise, the cold-start problem, and scalability. In this work, we present a novel method for recommending items to users based on expert opinions. Our method is a variation of traditional collaborative filtering: rather than applying a nearest neighbor algorithm to the user-rating data, predictions are computed using a set of expert neighbors from an independent dataset, whose opinions are weighted according to their similarity to the user. This method promises to address some of the weaknesses in traditional collaborative filtering, while maintaining comparable accuracy. We validate our approach by predicting a subset of the Netflix data set. We use ratings crawled from a web portal of expert reviews, measuring results both in terms of prediction accuracy and recommendation list precision. Finally, we explore the ability of our method to generate useful recommendations, by reporting the results of a user-study where users prefer the recommendations generated by our approach.

We present RelAltTab, an enhanced ALT+TAB prototype that assists users in switching windows. Our approach uses semantic and temporal information to create a list of related windows to the window that the user is currently engaged in. The main assumption is that the user is more likely to switch to a related window than to any other window in the system. We propose two different user interfaces that present the related window list to the user. We describe in detail the techniques and user interfaces of the RelAltTab system, and present the results of one user study comparing our approach to the standard Windows ALT+TAB program.

We present MAMI (i.e. Multimodal Automatic Mobile Indexing), a mobile-phone prototype that allows users to annotate and search for digital photos on their camera phone via speech input. MAMI is implemented as a mobile application that runs in real-time on the phone. Users can add speech annotations at the time of capturing photos or at a later time. Additional metadata is also stored with the photos, such as location, user identification, date and time of capture and image-based features. Users can search for photos in their personal repository by means of speech. MAMI does not need connectivity to a server. Hence, instead of full-fledged speech recognition, we propose using a Dynamic Time Warping-based metric to determine the distance between the speech input and all other existing speech annotations. We present our preliminary results with the MAMI prototype and outline our future directions of research, including the integration of additional metadata in the search.

We present TripleBeat, a mobile phone based system that assists runners in achieving predefined exercise goals via musical feedback and two persuasive techniques: a glanceable interface for increased personal awareness and a virtual competition. TripleBeat is based on a previous system named MPTrain. First, we describe TripleBeat's hardware and software, emphasizing how it differs from its predecessor MPTrain. Then, we present the results of a runner study with 10 runners. The study compared the runners efficacy and enjoyment in achieving predefined workout goals when running with MPTrain and TripleBeat. The conclusions from the study include: (1) significantly higher efficacy and enjoyment with TripleBeat, and (2) a unanimous preference for TripleBeat over MPTrain. The glanceable interface and the virtual competition are the two main reasons for the improvements in the running experience. We believe that systems like TripleBeat will play an important role in enhancing the exercise experience and in assisting users towards more active lifestyles.

Information workers are often involved in multiple tasks and activities that they must perform in parallel or in rapid succession. In consequence, task management itself becomes yet another task that information workers need to perform in order to get the rest of their work done. Recognition of this problem has led to research on task management systems, which can help by allowing fast task switching, fast task resumption, and automatic task identification. In this paper we focus on the latter: we tackle the problem of automatically detecting the tasks that the user is involved in, by identifying which of the windows on the user's desktop are related to each other. The underlying assumption is that windows that belong to the same task share some common properties with one another that we can detect from data. We will refer to this problem as the task assignment problem.To address this problem, we have built a prototype named Swish that: (1) constantly monitors users' desktop activities using a stream of windows events; (2) logs and processes this raw event stream, and (3) implements two criteria of window "relatedness", namely the semantic similarity of their titles, and the temporal closeness in their access patterns.In addition to describing the Swish prototype in detail, we validate it with 4 hours of user data, obtaining task classification accuracies of about 70%. We also discuss our plans on including Swish in a number of intelligent user interfaces and future lines of research.

We present MPTrain, a mobile phone based system that takes advantage of the influence of music in exercise performance, enabling users to more easily achieve their exercise goals. MPTrain is designed as a mobile and personal system (hardware and software) that users wear while exercising (walking, jogging or running). MPTrain's hardware includes a set of physiological sensors wirelessly connected to a mobile phone carried by the user. MPTrain's software allows the user to enter a desired exercise pattern (in terms of desired heart-rate over time) and assists the user in achieving his/her exercising goals by: (1) constantly monitoring the user's physiology (heart-rate in number of beats per minute) and movement (speed in number of steps per minute); and (2) selecting and playing music with specific features that will encourage the user to speed up, slow down or keep the pace to be on track with his/her exercise goals.We describe the hardware and software components of the MPTrain system, and present some preliminary results when using MPTrain while jogging.

Intensive computations required for sensing and processing perceptual information can impose significant burdens on personal computer systems. We explore several policies for selective perception in SEER, a multimodal system for recognizing office activity that relies on a layered Hidden Markov Model representation. We review our efforts to employ expected-value-of-information (EVI) computations to limit sensing and analysis in a context-sensitive manner. We discuss an implementation of a one-step myopic EVI analysis and compare the results of using the myopic EVI with a heuristic sensing policy that makes observations at different frequencies. Both policies are then compared to a random perception policy, where sensors are selected at random. Finally, we discuss the sensitivity of ideal perceptual actions to preferences encoded in utility models about information value and the cost of sensing.

Perceptual user interfaces promise modes of fluid computer-human interaction that complement the mouse and keyboard, and have been especially motivated in non-desktop scenarios, such as kiosks or smart rooms. Such interfaces, however, have been slow to see use for a variety of reasons, including the computational burden they impose, a lack of robustness outside the laboratory, unreasonable calibration demands, and a shortage of sufficiently compelling applications. We address these difficulties by using a fast stereo vision algorithm for recognizing hand positions and gestures. Our system uses two inexpensive video cameras to extract depth information. This depth information enhances automatic object detection and tracking robustness, and may also be used in applications. We demonstrate the algorithm in combination with speech recognition to perform several basic window management tasks, report on a user study probing the ease of using the system, and discuss the implications of such a system for future user interfaces.

We present the use of layered probabilistic representations using hidden Markov models for performing sensing, learning, and inference at multiple levels of temporal granularity We describe the use of representation in a system that diagnoses states of a user's activity based on real-time streams of evidence from video, acoustic, and computer interactions. We review the representation, present an implementation, and report on experiments with the layered representation in an office-awareness application.

This paper describes a new framework for processing images by example, called “image analogies.” The framework involves two stages: a design phase, in which a pair of images, with one image purported to be a “filtered” version of the other, is presented as “training data”; and an application phase, in which the learned filter is applied to some new target image in order to create an “analogous” filtered result. Image analogies are based on a simple multi-scale autoregression, inspired primarily by recent results in texture synthesis. By choosing different types of source image pairs as input, the framework supports a wide variety of “image filter” effects, including traditional image filters, such as blurring or embossing; improved texture synthesis, in which some textures are synthesized with higher quality than by previous approaches; super-resolution, in which a higher-resolution image is inferred from a low-resolution source; texture transfer, in which images are “texturized” with some arbitrary source texture; artistic filters, in which various drawing and painting styles are synthesized based on scanned real-world examples; and texture-by-numbers, in which realistic scenes, composed of a variety of textures, are created using a simple painting interface.

We describe a real-time computer vision and machine learning system for modeling and recognizing human behaviors in a visual surveillance task. The system deals in particularly with detecting when interactions between people occur and classifying the type of interaction. Examples of interesting interaction behaviors include following another person, altering one's path to meet another, and so forth. Our system combines top-down with bottom-up information in a closed feedback loop, with both components employing a statistical Bayesian approach. We propose and compare two different state-based learning architectures, namely, HMMs and CHMMs for modeling behaviors and interactions. Finally, a synthetic "Alife-style" training system is used to develop flexible prior models for recognizing human interactions. We demonstrate the ability to use these a priori models to accurately classify real human behaviors and interactions with no additional tuning or training.

We address the problem of tracking and reconstructing 3D human lip motions from a 2D view. This problem is challenging due both to the complex nature of lip motions and the minimal data available from a raw video stream of the face. We counter both of these difficulties with statistical approaches. We first build a physically-based 3D model of lips and train it to cover only the subspace of lip motions. We then track this model in video by finding the shape within the subspace that maximizes the posterior probability of the model given the observed features. In this study, the features are the likelihoods of the lip and non-lip color classes: we iteratively derive forces from these values to apply to the physical model and converge to the final solution. Because of the full 3D nature of the model, this framework allows us to track the lips from any head pose. In addition, because of the constraints imposed by the learned subspace of the model, we are able to accurately estimate the full 3D lip shape from the 2D view.

We present algorithms for coupling and training hidden Markov models (HMMs) to model interacting processes, and demonstrate their superiority to conventional HMMs in a vision task classifying two-handed actions. HMMs are perhaps the most successful framework in perceptual computing for modeling and classifying dynamic behaviors, popular because they offer dynamic time warping, a training algorithm and a clear Bayesian semantics. However the Markovian framework makes strong restrictive assumptions about the system generating the signal-that it is a single process having a small number of states and an extremely limited state memory. The single-process model is often inappropriate for vision (and speech) applications, resulting in low ceilings on model performance. Coupled HMMs provide an efficient way to resolve many of these problems, and offer superior training speeds, model likelihoods, and robustness to initial conditions.

This paper describes an active-camera real-time system for tracking, shape description, and classification of the human face and mouth using only an SGI Indy computer. The system is based on use of 2-D blob features, which are spatially-compact clusters of pixels that are similar in terms of low-level image properties. Patterns of behavior (e.g., facial expressions and head movements) can be classified in real-time using Hidden Markov Model (HMM) methods. The system has been tested on hundreds of users and has demonstrated extremely reliable and accurate performance. Typical classification accuracies are near 100%.