Abstract The escalating prevalence of violent crimes and accidents underscores the urgent need for efficient and timely monitoring systems. Traditional methods reliant on administrative reports often suffer from significant delays. This paper proposes CRIMSON, a novel framework that leverages large-scale online news to provide real-time insights into crime and accident trends. CRIMSON utilizes a multi-label classification technique that leverages a fine-tuned, pre-trained, cross-lingual language model to accurately categorize news articles. Our experimental results, conducted on a substantial dataset of Thai news articles, demonstrate superior performance, achieving an average F1 score of 86%. Beyond classification, CRIMSON aggregates categorized news into real-time statistics, revealing strong correlations between news-reported incidents and official crime data. This study pioneers online news as a reliable and timely crime and accident monitoring source, offering valuable insights for law enforcement, policymakers, and researchers.

With the development of social network services, various phenomena can be shared easily and rapidly through human natural language, including not only natural, but also social-cultural phenomena. Consequently, analyses of social media have appreciated in value for understanding human behaviors to grasp public interests or sentiments, as both the medium and outcome of human experiences. From the state of the art psychology and neuroscience, human behaviors, regarding both physical and linguistic aspects, are mostly dependent on sensory perceptions under the realm of the subconscious. Even though sensation is the most fundamental element to understand human behaviors, the rack of background resources make it hard to study the social sensation comparing with the sentimental or opinion mining. This paper focuses on building sensation knowledges to obtain useful human perceptual experiences in natural language expressions, as a requisite for the social sensation analysis. We try to approach the constructing lexicons as a sensation knowledge from two viewpoints, such as a deep learning and lexicon based methods. Then we classify social media text based on the lexicons with considering a part of speech as well as semantic meanings of each word. Finally, we identify which knowledge has a good performance to distinguish sensation expressions from social media data in terms of accuracy and and F-score.

In the recent years, digitalization of disaster management has gained much importance in aspects of response, recovery and mitigation. Often, issues identified in past disaster response and relief efforts include lack of communication, delayed ordering of actions (e.g., evacuations), and low levels of preparedness by authorities during disasters. A rapid response in the event of a disaster requires good situational awareness and fast sharing of that knowledge across the community. We believe that a digital ecosystem based on multimedia computing can address this major challenge in the society. The role of technology in disaster management is to connect, inform and ultimately save the lives of those impacted by disasters. Multimedia analysis must expand to take advantage of this fact to benefit society, in particular, help during disasters. Data driven tools as part of new ecosystem can link resources and need to build an effective platform for use during disasters. To accomplish ecosystem coevolution, creating a collaborative system with crowdsourced data, spatial data, and historical data is essential. EventShop is one such tool used to detect and observe changing real-world situations in real-time. Multimedia micro-reports are global, powerful, impactful and spontaneous data streams which are obtained from applications like Krumbs, Twitter, Flickr, etc. These micro-reports assist EventShop in building a rescue system that provides rescue responses to victims during disasters. In this paper, we present a framework that uses EventShop and multimedia micro-reports to correspond with victims during one such disaster, the flood.

Heterogeneous data fusion from disparate geospatial sensors has drawn increasing attention in multimedia. Unfortunately, environmental sensors are usually sparsely and preferentially located, which restricts situation recognition of geographical regions and results in uncertainty in derived inferences. Spatial interpolation is an effective way to solve the problem of data sparsity, which demands the availability of related data sources. However, these data sources are usually in different resolutions, distributions, scales, and densities, which poses a major challenge in data integration. To address this problem, we present a novel spatial interpolation framework to incorporate diverse data sources and model the spatial processes explicitly at multiple resolutions. Spectral analysis is deployed to generate features at multiple spatial resolutions and to improve the interpolation accuracy at unobserved locations. A statistical operator based on the spatial Gaussian process is implemented and integrated into a geospatial situation recognition system, which can analyze heterogeneous spatio-temporal data streams derived from sensors. To verify the effectiveness and efficiency of the proposed framework, this framework is applied to the PM2.5 air pollution application. Experiments conducted in California, USA, demonstrate that the proposed method outperforms state-of-the-art approaches.

Recent multimedia research has increasingly focused on large scale multimodal data from disparate geospatial sensors. In addition to the volume of the data, the diversity and granularity of the data poses a major challenge in extracting meaningful and actionable information. To address this, we present a novel spatial interpolation framework, capable of incorporating multimodal data sources and modeling the spatial processes comprehensively at multiple resolutions. The framework transforms the spatial interpolation problem into a graph structure learning problem, based on the latent structure of the data. This enables more efficient and accurate predictions at unobserved locations. We demonstrate the effectiveness of our approach by testing it on air pollution interpolation.

Photos serve dual role. Photos are important for capturing, saving, sharing, and reminiscing memories of events and people. Modern photos, however, are becoming more spontaneous, objective, compelling, and universal reports of a moment in an event also. In this paper our focus is on millions of photos being captured as informative reports and using them for emerging applications including situation recognition, trend analysis, and cultural dynamics. EventShop is an open source platform for situation recognition. Utilizing this platform and using a stream of photo reports from various sources as one of the data streams in this platform, we build a visual analytics system to understand the information that could be gleaned from such photo report streams. Our early experiments are based on the Yahoo Flickr Creative Commons 100 Million photos set released recently. We are also using other sources to import and understand the efficacy of these reports for various important applications.

Situation recognition is the problem of deriving actionable insights from heterogeneous, real-time, big multimedia data to benefit human lives and resources in different applications. This tutorial will discuss the recent developments towards converting multitudes of data streams including weather patterns, stock prices, social media, traffic information, and disease incidents into actionable insights.

No abstract available.

With an increasing amount of diverse heterogeneous data and information, the methodology of multimedia analysis has become increasingly relevant in solving challenging societal problems such as managing emergency situations during disasters. Using cybernetic principles combined with multimedia technology, researchers can develop effective frameworks for using diverse multimedia (including traditional multimedia as well as diverse multimodal) data for situation recognition, and determining and communicating appropriate actions to people stranded during disasters. We present known issues in disaster management and then focus on emergency situations. We show that an emergency management problem is fundamentally a multimedia information assimilation problem for situation recognition and for connecting people's needs to available resources effectively, efficiently, and promptly. Major research challenges for managing emergency situations are identified and discussed. We also present a intelligently detecting evolving environmental situations, and discuss the role of multimedia micro-reports as spontaneous participatory sensing data streams in emergency responses. Given enormous progress in concept recognition using machine learning in the last few years, situation recognition may be the next major challenge for learning approaches in multimedia contextual big data. The data needed for developing such approaches is now easily available on the Web and many challenging research problems in this area are ripe for exploration in order to positively impact our society during its most difficult times.

In this paper, we introduce a new method that explores spatio-temporal-theme correlations between physical and social streaming data for event detection and pattern interpretation from heterogeneous sensors. Particularly, we employ a basic two-phase framework in pattern recognition (i.e. feature extraction and detection) with the novel improvement that concerns the use of semantic information acquired from social sensors to automatically label the low-level features extracted from physical sensors. Moreover, by symbolizing the trend component of time-series data, the proposed method has an ability to interpret event's patterns to help users get insights of how events happen. Differentiating from conventional supervised learning methods whose training data are labeled manually and in an off-line mode, the proposed method can collect labels for training data automatically and in an on-line mode. Moreover, after running for a certain time, a training stage can run parallel with the detecting stage when an event model is totally built. After that, the training stage continues learning to increase the accuracy of the event model by nonstop collecting new samples with labels from streaming data. The problem of environmental factors and particularly air pollution impacts on asthma exacerbation is considered for evaluating the proposed method. The experimental results show that the proposed method can probably detect the prevalence of asthma risks in a specific spatio-temporal context as well as help users understand how a change in the surrounding environment (e.g. weather condition and air pollution) can influence their health (e.g. asthma attack) by interpreting detected event's patterns.

EventShop is an open-source software which provides a generic infrastructure for the analysis of heterogeneous spatio-temporal data streams. Efficient interpolation of data from spatially sparse sources is critical but currently missing in EventShop. To address this challenge, we implement a Spatial Gaussian Process based statistical operator into the EventShop framework. Spectral analysis is employed to generate features at higher spatial resolution and to improve interpolation accuracy at unsampled locations. Further, we test this operator by interpolating air pollution levels in California. The evaluations of multiple metrics demonstrate that our operators outperform earlier EventShop operators, chemical transportation models, and state-of-the-art methods.

We are living in the Internet of Things (IoT) era where all the (smart) objects around us are connected and communicated with each other to serve our life better without the need of explicit instruction. Soon we have to cope with trillions of heterogeneous data streams coming from IoT. Since data is not information, methods for discovering useful and correlative information from data and utilising them for the better life, in real-time mode, are the utmost requirements.