Multimedia Event Processing (MEP) systems play a critical role in various Internet of Things (IoT) applications, including Smart Cities and Health and Safety, by processing large amounts of multimedia data streams. These systems often leverage state-of-the-art Deep Neural Network (DNN) models to enhance their capabilities. However, the growth of Cloud and Edge Big Data applications has imposed a significant environ-mental burden, further intensified by the substantial energy consumption associated with certain DNN model operations.This study addresses the environmental impact of growing Big Data stream applications and focuses on optimising MEP systems to mitigate this issue. We tackle uncertainties arising from user-defined Quality of Service (QoS) interpretations and service worker measurement imprecisions by using uncertainty-aware solutions for the service selection problem in order to improve the QoS within MEP systems.Our results reveal substantial advantages in employing uncertainty-aware strategies. These approaches consistently enhance QoS metrics, outperforming their uncertainty-oblivious counterparts. Specifically, we report improvements in more than 67%, 69%, and 20% of the scenarios, on average, for energy consumption, latency, and accuracy, respectively. These enhancements become evident within just three hours of processing, resulting in energy savings of up to 1.2 kilowatt-hours and latency reductions of 213 seconds, with a 0.29% average loss in query accuracy. These strategies improve system efficiency and ecological sustainability while incurring a small accuracy trade-off. When extrapolated over a year, the environmental benefits become even more noticeable, surpassing the energy requirements for a 1000 Km electric vehicle round-trip from Amsterdam to Paris and back.

With the Internet of Multimedia Things (IoMT) becoming a reality, new approaches are needed to process real-time multimodal event streams. Existing approaches to event processing have limited consideration for the challenges of multimodal events, including the need for complex content extraction, and increased computational and memory costs. This article explores event processing as a basis for processing real-time IoMT data. This article introduces the multimodal event processing (MEP) paradigm, which provides a formal basis for native approaches to neural multimodal content analysis (i.e., computer vision, linguistics, and audio) with symbolic event processing rules to support real-time queries over multimodal data streams using the multimodal event processing language to express single, primitive multimodal, and complex multimodal event patterns. The content of multimodal streams is represented using multimodal event knowledge graphs to capture the semantic, spatial, and temporal content of the multimodal streams. The approach is implemented and evaluated within a MEP engine using single and multimodal queries achieving near real-time performance with a throughput of ~30 frames processed per second (fps) and subsecond latency of 0.075–0.30 s for video streams of 30 fps input rate. Support for high input stream rates (45 fps) is achieved through content-aware load-shedding techniques with a ~127X latency improvement resulting in only a minor decrease in accuracy.

Advances in Deep Neural Network (DNN) techniques have revolutionized video analytics and unlocked the potential for querying and mining video event patterns. This paper details GNOSIS, an event processing platform to perform near-real-time video event detection in a distributed setting. GNOSIS follows a serverless approach where its component acts as independent microservices and can be deployed at multiple nodes. GNOSIS uses a declarative query-driven approach where users can write customize queries for spatiotemporal video event reasoning. The system converts the incoming video streams into a continuous evolving graph stream using machine learning (ML) and DNN models pipeline and applies graph matching for video event pattern detection. GNOSIS can perform both stateful and stateless video event matching. To improve Quality of Service (QoS), recent work in GNOSIS incorporates optimization techniques like adaptive scheduling, energy efficiency, and content-driven windows. This paper demonstrates the Occupational Health and Safety query use cases to show the GNOSIS efficacy.