An individual’s health trajectory is most influenced by personal lifestyle choices made regularly and frequently. It is now possible to measure, store, and analyze both multimodal lifestyle signals as well as multimodal physiological and behavioral health signals perpetually. Moreover, it is possible to model the effects of lifestyle precisely on health signals by collecting a variety of longitudinal data. These actions provide the inputs that affect changes in the health state of an individual based on their lifestyle decisions. With the advent of modern relatively inexpensive and common multi-modal data streams and sensing technologies, individuals now have large amounts of data and information about themselves that have the potential to transform decision-making in day-to-day life for health improvement. Multimodal analytics and contextual prediction and retrieval allow this data to make the best decisions to keep the health state optimal while making good lifestyle choices. This critical problem requires making a large variety of data constantly useful, contextually relevant, and most importantly useful in personalized decision-making.

Heterogeneous data fusion from disparate geospatial sensors has drawn increasing attention in multimedia. Unfortunately, environmental sensors are usually sparsely and preferentially located, which restricts situation recognition of geographical regions and results in uncertainty in derived inferences. Spatial interpolation is an effective way to solve the problem of data sparsity, which demands the availability of related data sources. However, these data sources are usually in different resolutions, distributions, scales, and densities, which poses a major challenge in data integration. To address this problem, we present a novel spatial interpolation framework to incorporate diverse data sources and model the spatial processes explicitly at multiple resolutions. Spectral analysis is deployed to generate features at multiple spatial resolutions and to improve the interpolation accuracy at unobserved locations. A statistical operator based on the spatial Gaussian process is implemented and integrated into a geospatial situation recognition system, which can analyze heterogeneous spatio-temporal data streams derived from sensors. To verify the effectiveness and efficiency of the proposed framework, this framework is applied to the PM2.5 air pollution application. Experiments conducted in California, USA, demonstrate that the proposed method outperforms state-of-the-art approaches.

Recent multimedia research has increasingly focused on large scale multimodal data from disparate geospatial sensors. In addition to the volume of the data, the diversity and granularity of the data poses a major challenge in extracting meaningful and actionable information. To address this, we present a novel spatial interpolation framework, capable of incorporating multimodal data sources and modeling the spatial processes comprehensively at multiple resolutions. The framework transforms the spatial interpolation problem into a graph structure learning problem, based on the latent structure of the data. This enables more efficient and accurate predictions at unobserved locations. We demonstrate the effectiveness of our approach by testing it on air pollution interpolation.

Photos serve dual role. Photos are important for capturing, saving, sharing, and reminiscing memories of events and people. Modern photos, however, are becoming more spontaneous, objective, compelling, and universal reports of a moment in an event also. In this paper our focus is on millions of photos being captured as informative reports and using them for emerging applications including situation recognition, trend analysis, and cultural dynamics. EventShop is an open source platform for situation recognition. Utilizing this platform and using a stream of photo reports from various sources as one of the data streams in this platform, we build a visual analytics system to understand the information that could be gleaned from such photo report streams. Our early experiments are based on the Yahoo Flickr Creative Commons 100 Million photos set released recently. We are also using other sources to import and understand the efficacy of these reports for various important applications.

Many emerging problems increasingly rely on integrating complex heterogeneous sensor streams, ranging from photos, texts, environmental data streams and other participating human sensors. The diversity of data types from disparate sensors poses a major challenge in data aggregation and assimilation. EventShop was originally designed for situation recognition using diverse data sources. We propose and develop new interpolation and prediction models on top of EventShop, allowing for effective ingesting and combining appropriate data streams to improve data quality and predict specific situations. We also incorporate data from participatory sensing into the system. The synergy of data gives powerful insight into better understanding of evolving situations, in which participatory sensing is integrated with the surrounding environment. Furthermore, the enhanced system is used for two real-world problems: asthma risk management and smart city.

Photos are a predominant source of information on a global scale. Cluster analysis of photos can be applied to situation recognition and understanding cultural dynamics. Graph-based learning provides a current approach for modeling data in clustering problems. However, the performance of this framework depends heavily on initial graph construction by input data. Data outliers degrade graph quality, leading to poor clustering results. We designed a new capped lp-norm graph-based model to reduce the impact of outliers. This is accomplished by allowing the data graph to self adjust as part of the graph embedding. Furthermore, we derive an iterative algorithm to solve the objective function optimization problem. Experiments on four real-world benchmark data sets and Yahoo Flickr Creative Commons data set show the effectiveness of this new graph-based capped lp-norm clustering method.

With an increasing amount of diverse heterogeneous data and information, the methodology of multimedia analysis has become increasingly relevant in solving challenging societal problems such as managing emergency situations during disasters. Using cybernetic principles combined with multimedia technology, researchers can develop effective frameworks for using diverse multimedia (including traditional multimedia as well as diverse multimodal) data for situation recognition, and determining and communicating appropriate actions to people stranded during disasters. We present known issues in disaster management and then focus on emergency situations. We show that an emergency management problem is fundamentally a multimedia information assimilation problem for situation recognition and for connecting people's needs to available resources effectively, efficiently, and promptly. Major research challenges for managing emergency situations are identified and discussed. We also present a intelligently detecting evolving environmental situations, and discuss the role of multimedia micro-reports as spontaneous participatory sensing data streams in emergency responses. Given enormous progress in concept recognition using machine learning in the last few years, situation recognition may be the next major challenge for learning approaches in multimedia contextual big data. The data needed for developing such approaches is now easily available on the Web and many challenging research problems in this area are ripe for exploration in order to positively impact our society during its most difficult times.

EventShop is an open-source software which provides a generic infrastructure for the analysis of heterogeneous spatio-temporal data streams. Efficient interpolation of data from spatially sparse sources is critical but currently missing in EventShop. To address this challenge, we implement a Spatial Gaussian Process based statistical operator into the EventShop framework. Spectral analysis is employed to generate features at higher spatial resolution and to improve interpolation accuracy at unsampled locations. Further, we test this operator by interpolating air pollution levels in California. The evaluations of multiple metrics demonstrate that our operators outperform earlier EventShop operators, chemical transportation models, and state-of-the-art methods.