YouTube vlogging, as a popular genre of ubiquitous social video, engages people in entertainment, civic, and social activities. Although several aspects of vlogging have been studied in media studies and multimedia analysis, the longitudinal angle of vlogging regarding recognition of personal state and trait impressions from behavior has not been yet analyzed. We present a study using behavioral data of vloggers who posted vlogs on YouTube for a period between three and six years. We use online crowdsourcing to collect a rich set of 21 impression variables for each video, including perceived personality, mood, skills, and expertise. Acoustic and motion features are extracted to characterize basic nonverbal behavior. The analysis shows that only a couple of perceived variables, including perceived expertise and perceived quality of audio and video, display weak temporal patterns. Furthermore, we show that the use of longitudinal data helps to improve the automatic inference of impressions for several of the impression variables.

Alcohol consumption is the number one risk factor for morbidity and mortality among young people. In late adolescence and early adulthood, excessive drinking and intoxication are more common than in any other life period, increasing the risk of adverse physical and psychological health consequences. In this paper, we examine the feasibility of using smartphone sensor data and machine learning to automatically characterize and classify drinking behavior of young adults in an urban, ecologically valid nightlife setting. Our work has two contributions. First, we use previously unexplored data from a large-scale mobile crowdsensing study involving 241 young participants in two urban areas in a European country, which includes phone data (location. accelerometer, Wifi, Bluetooth, battery, screen, and app usage) along with self-reported, fine-grain data on individual alcoholic drinks consumed on Friday and Saturday nights over a three-month period. Second,we build a machine learning methodology to infer whether an individual consumed alcohol on a given weekend night, based on her/his smartphone data contributed between 8 PM and 4 AM. We found that accelerometer data is the most informative single cue, and that a combination of features results in an overall accuracy of 76.6 percent.

The location tracking functionality of modern mobile devices provides unprecedented opportunity to the understanding of individual mobility in daily life. Instead of studying raw geographic coordinates, we are interested in understanding human mobility patterns based on sequences of place visits which encode, at a coarse resolution, most daily activities. This paper presents a study on place characterization in people's everyday life based on data recorded continuously by smartphones. First, we study human mobility from sequences of place visits, including visiting patterns on different place categories. Second, we address the problem of automatic place labeling from smartphone data without using any geo-location information. Our study on a large-scale data collected from 114 smartphone users over 18 months confirm many intuitions, and also reveals findings regarding both regularly and novelty trends in visiting patterns. Considering the problem of place labeling with 10 place categories, we show that frequently visited places can be recognized reliably (over 80 percent) while it is much more challenging to recognize infrequent places.

While our daily activities usually involve interactions with others, the current methods on activity recognition do not often exploit the relationship between social interactions and human activity. This paper addresses the problem of interpreting social activity from human interactions captured by mobile sensing networks. Our first goal is to discover different social activities such as chatting with friends from interaction logs and then characterize them by the set of people involved, and the time and location of the occurring event. Our second goal is to perform automatic labeling of the discovered activities using predefined semantic labels such as coffee breaks, weekly meetings, or random discussions. Our analysis was conducted on a real-life interaction network sensed with Bluetooth and infrared sensors of about fifty subjects who carried sociometric badges over 6 weeks. We show that the proposed system reliably recognized coffee breaks with 99% accuracy, while weekly meetings were recognized with 88% accuracy.

Human behavior is often complex and context-dependent. This paper presents a general technique to exploit this "multidimensional" contextual variable for human mobility prediction. We use an ensemble method, in which we extract different mobility patterns with multiple models and then combine these models under a probabilistic framework. The key idea lies in the assumption that human mobility can be explained by several mobility patterns that depend on a sub-set of the contextual variables and these can be learned by a simple model. We showed how this idea can be applied to two specific online prediction tasks: what is the next place a user will visit? and how long will he stay in the current place?. Using smartphone data collected from 153 users during 17 months, we show the potential of our method in predicting human mobility in real life.

Studies on human mobility are built on two fundamentally different data sources: manual check-in data that originates from location-based social networks and automatic check-in data that can be automatically collected through various smartphone sensors. In this paper, we analyze the differences and similarities of manual check-ins from Foursquare and automatic check-ins from Nokia's Mobile Data Challenge. Several new findings follow from our analysis: (1) While automatic checking-in overall results in more visits than manual checking-in, the check-in levels are comparable when visiting new places. (2) Daily and weekly check-in activity patterns are similar for both systems except for Saturdays -- when manual check-ins are relatively more probable. (3) A recently proposed rank distribution to describe human mobility, so far validated on manual check-in data, also holds for automatic check-in data given a slight modification to the definition of rank. (4) The patterns described by automatic check-ins are in general more predictable. We also address the question of whether it is possible to find matching places across the two check-in systems. Our analysis shows that while this is challenging in areas such as city centers, our method achieves an accuracy of 51% for places that are not homes of phone users.

We describe an hybrid model that combines deep neural networks (DNN) for nonlinear feature extraction and hidden conditional random fields (HCRF), i.e. conditional random fields with hidden states. The model is globally trained though joint optimization of HCRF and DNN parameters. To deal with this highly non convex optimization criterion, we propose a multi-step training which aims at providing a good initialization before the final joint optimization of all parameters. We investigate then the discriminative power of these models with respect to the architecture of the DNN, and compare our models to HMM and HCRF based algorithms on the IAM database.

This paper presents a large-scale analysis of contextualized smartphone usage in real life. We introduce two contextual variables that condition the use of smartphone applications, namely places and social context. Our study shows strong dependencies between phone usage and the two contextual cues, which are automatically extracted based on multiple built-in sensors available on the phone. By analyzing continuous data collected on a set of 77 participants from a European country over 9 months of actual usage, our framework automatically reveals key patterns of phone application usage that would traditionally be obtained through manual logging or questionnaire. Our findings contribute to the large-scale understanding of applications and context, bringing out design implications for interfaces on smartphones.

By exploiting built-in sensors, mobile smart phone have become attractive options for large-scale sensing of human behavior as well as social interaction. In this paper, we present a new probabilistic model to analyze longitudinal dynamic social networks created by the physical proximity of people sensed continuously by the phone Bluetooth sensors. A new probabilistic model is proposed in order to jointly infer emergent grouping modes of the community together with their temporal context. We present experimental results on a Bluetooth proximity network sensed with mobile smart-phones over 9 months of continuous real-life, and show the effectiveness of our method.

Mobile phones are becoming more and more widely used nowadays, and people do not use the phone only for communication: there is a wide variety of phone applications allowing users to select those that fit their needs. Aggregated over time, application usage patterns exhibit not only what people are consistently interested in but also the way in which they use their phones, and can help improving phone design and personalized services. This work aims at mining automatically usage patterns from apps data recorded continuously with smartphones. A new probabilistic framework for mining usage patterns is proposed. Our methodology involves the design of a bag-of-apps model that robustly represents level of phone usage over specific times of the day, and the use of a probabilistic topic model that jointly discovers patterns of usage over multiple applications and describes users as mixtures of such patterns. Our framework is evaluated using 230 000+ hours of real-life app phone log data, demonstrates that relevant patterns of usage can be extracted, and is objectively validated on a user retrieval task with competitive performance.

Recent works for learning Hidden Markov Models in a discriminant way have focused on maximum margin training, which remains an open problem due to the lack of efficient optimization algorithms. We developed a new algorithm that is based on non convex optimization ideas and that may solve maximum margin learning of GHMMs within the standard setting of partially labeled training sets. We provide experimental results on both on-line handwriting and off-line handwriting recognition.

Large margin learning of Continuous Density HMMs with a partially labeled dataset has been extensively studied in the speech and handwriting recognition fields. Yet due to the non-convexity of the optimization problem, previous works usually rely on severe approximations so that it is still an open problem. We propose a new learning algorithm that relies on non-convex optimization and bundle methods and allows tackling the original optimization problem as is. It is proved to converge to a solution with accuracy ε with a rate O (1/ε). We provide experimental results gained on speech and handwriting recognition that demonstrate the potential of the method.