With increasing public attention to health and wellness, treadmill exercise in gyms has become increasingly popular. However, the risk of falling due to loss of balance during treadmill use remains a significant safety concern. To address this issue, we propose a gait stability prediction framework that leverages a deep learning neural network BiLSTM-KAN trained on multimodal data. Specifically, we use MediaPipe to extract 3D skeletal key points and smart insoles to capture essential plantar features and then perform data preprocessing to synchronize the postural features with the plantar data. The trained model is capable of predicting 15 biomechanical indicators related to foot stability based on pose data obtained from a camera, thereby enabling indirect assessment of fall risk without requiring the user to wear any devices during use. Experimental results demonstrate that the proposed system achieves high accuracy in predicting key stability-related parameters such as Gait-Line CPE and Impulse. This study validates the feasibility of a non-contact, continuous monitoring approach based on multimodal fusion for gait stability assessment, providing a practical solution for safer exercise and intelligent health monitoring.

Abstract Previous studies have shown that about 90% of traffic accidents are due to human error, which means that human factors may affect a driver's braking behaviors and thus their driving safety, especially when the driver makes a braking motion. However, most studies have mounted sensors on the brake pad, ignoring to some extent an analysis of the driver's behavior before the brake pad is pressed (pre-braking). Therefore, to determine the effect of different human factors on drivers' pre-braking behaviors, this study focused on analyzing drivers' local joints (knee, ankle, and toe) by a motion capture device. A Hilbert–Huang Transform (HHT)-based local human body movement analysis method was used to decompose the realistic complex pre-braking actions into sub-actions such as intrinsic mode functions (IMF1, IMF2, etc.). Analysis of the results showed that IMF1 is a common and necessary action when pre-braking for all drivers, and IMF2 may be the safety assurance action that allows right-foot transverse movement at the beginning part of the pre-braking process. We also found that the experienced, male, and Phys.50 groups may have consistent characteristics in the HHT scheme, which could mean that such drivers would have better performance and efficiency during the pre-braking process. The results of this study will be useful in decomposing and discerning the specific actions that lead to accidents, providing insights into driver training for novice drivers, and guiding the construction of daily automated driver assistance or accident prevention systems (advanced driver assistance systems, ADASs).

The usage of surveillance cameras for video understanding, raises concerns about privacy intrusion recently. This motivates the research community to seek potential alternatives of cameras for emerging multimedia applications. Stepping to this goal, a few researchers have explored the usage of Wi-Fi or Bluetooth sensors to handle action recognition. However, the practical ability of these sensors is limited by their frequency band and deployment inconvenience because of the separate transmitter/receiver architecture. Motivated by the same purpose of reducing privacy issues, we introduce a latest microwave sensor for multi-person action recognition in this paper. The microwave sensor works at 77GHz ~ 80GHz band, and is implemented with both transmitter and receiver inside itself, thus can be easily deployed for action recognition. Although with its advantages, two main challenging issues still remain. One is the difficulty of labelling the invisible signal data with embedding actions. The other is the difficulty of cancelling the environment noise for high-accurate action recognition. To address the challenges, we propose a novel learning framework by designed original loss functions with the considerations on weakly-supervised multi-label learning and attention mechanism to improve the accuracy for action recognition. We build a new microwave sensor data set, and conduct comprehensive experiments to evaluate the recognition accuracy of our proposed framework, and the effectiveness of parameters in each component. The experiment results show that our framework outperforms the state-of-the-art methods up to 14% in terms of mAP.

The driver's driving experience is one of the important factors affecting his or her behaviors. Prior studies have noted the effect of driving experiences on drivers' eye-movements when right-turning. However, related studies usually focused on the accident scenarios, for drivers' eye-movements on daily driving when facing right-turn, the effects of drivers' experience remain unknown. Therefore, according to design and apply a set of experiments, this paper focused on the analysis of driver's eye-movements during the daily right-turn task to compare the differences between Experienced and Novice drivers. A total of 10 drivers were invited and be classified into two groups (Experienced and Novice) to participate in the experiments. All of the drivers are driving on the right-hand side of the road, and the steering wheel is on the left side of the vehicle. The aimed data were collected by a set of glasses type eye-tracker and be further classified into four driving vision-based AOI (Areas of Interest). The results of Mann-Whitney U-tests showed that Novice drivers have a disordered line of sight and tend to spend more attention on their right view and switch their line of sight back and forth between the AOIs. Moreover, Experienced drivers more tend to keep their view directly in front of their heads instead of using the peripheral vision. The results of this study may provide guidelines to prevent accidents in Advanced Driver Assistance Systems (ADAS) and offers useful insights for the training of new drivers.

The recent development of commodity 360° cameras have enabled a single video to capture an entire scene, which endows promising potentials in surveillance scenarios. However, research in omnidirectional video analysis has lagged behind the hardware advances. In this work, we address the important problem of action recognition in topview 360° videos. Due to the wide filed-of-view, 360° videos usually capture multiple people performing actions at the same time. Furthermore, the appearance of people are deformed. The proposed framework first transforms top-view omnidirectional videos into panoramic videos using a calibrationfree method. Then spatial-temporal features are extracted using region-based 3D CNNs for action recognition. We propose a weakly-supervised method based on multiinstance multi-label learning, which trains the model to recognize and localize multiple actions in a video using only video-level action labels as supervision. We perform experiments to quantitatively validate the efficacy of the proposed method over state-of-the-art baselines and variants of our model, and qualitatively demonstrate action localization results. To enable research in this direction, we introduce the 360Action dataset. It is the first omnidirectional video dataset for multi-person action recognition with a diverse set of scenes, actors and actions. The dataset is available at https://github.com/ryukenzen/360action.

This paper presents a method of refining image search results using multiple attributes. The explosive growth of multimedia contents such as images and videos increases the difficulty of finding pieces of information from such a huge amount of contents. Users who only have a few or imprecise search conditions will be overwhelmed by the large number of search results with many of them are similar and redundant, especially when searching video frames. In this work, we utilize multiple attributes to refine search results and reduce their redundancy. We propose a technique of attribute ranking and selection in order to refine search results efficiently and effectively. In this way, we need to only use a small number of attributes that will contribute to the refinement. We demonstrate the performance of our method using real-world surveillance videos. Our proposed method can reduce up to 60% of result computation time compared to the existing method. Furthermore, our method constantly outperforms the existing method in terms of both MAP (Mean Average Precision) and precision versus recall.

Notwithstanding recent advances in machine vision, video activity recognition from multiple cameras still remains a challenging task as many real-world interactions cannot be automatically recognised for many reasons, such as partial occlusion or coverage black-spots. In this paper we propose a new technique that infers the unseen relationship between two individuals captured by different cameras and use it to retrieve relevant video clips if there is a likely interaction between the two individuals. We introduce a human object interaction (HOI) model integrating the causal relationship between the humans and the objects. For this we first extract the key frames and generate the labels or annotations using the state-of-the-art image captioning models. Next, we extract SVO (subject, verb, object) triples and encode the descriptions into a vector form for HOI inference using the Stanford CoreNLP parser. In order to calculate the HOI co-existence and the possible causality score we use transfer entropy. From our experimentation, we found that integrating casual relations into the content indexing process and using transfer entropy to calculate the causality score leads to improvement in retrieval performance.

The amount of 360-degree panoramas shared online has been rapidly increasing due to the availability of affordable and compact omnidirectional cameras, which offers huge amount of new information unavailable before. In this paper, we present the first work to exploit unlabeled 360-degree data for image representation learning. We propose middle-out, a new self-supervised learning task, which leverages the spatial configuration of normal field-of-view images sampled from a 360-degree image as supervisory signal. We train a Siamese ConvNet model to identify the middle image among three shuffled images sampled from a panorama by perspective projection. Compared to previous self-supervised methods that train models using image patches or video frames with limited field-of-view, our method leverages the rich semantic information contained in 360-degree images and enforces the model to not only learn about objects, but also develop a higher-level understanding about object relationships and scene structures. We quantitatively demonstrate that the feature representation learned using the proposed task is useful for a wide range of vision tasks including object classification, object detection, scene classification, semantic segmentation, and geometry estimation. We also qualitatively show that the proposed method can enforce the ConvNet to extract high-level semantic concepts, an ability which previous self-supervised learning methods have not acquired.

Loitering is a suspicious behavior that often leads to criminal actions, such as pickpocketing and illegal entry. Tracking methods can determine suspicious behavior based on trajectory, but require continuous appearance and are difficult to scale up to multi-camera systems. Using the duration of appearance of features works on multiple cameras, but does not consider major aspects of loitering behavior, such as repeated appearance and trajectory of candidates. We introduce an entropy model that maps the location of a person's features on a heatmap. It can be used as an abstraction of trajectory tracking across multiple surveillance cameras. We evaluate our method over several datasets and compare it to other loitering detection methods. The results show that our approach has similar results to state of the art, but can provide additional interesting candidates.

It is very difficult to fully automate the detection of loitering behavior in video surveillance, therefore humans are often required for monitoring. Alternatively, we could provide a list of potential loiterer candidates for a final yes/no judgment of a human operator. Our system, VisLoiter+, realizes this idea with a unique, user-friendly interface and by employing an entropy model for improved loitering analysis. Rather than using only frequency of appearance, we expand the loiter analysis with new methods measuring the amount of person movements across multiple camera views. The interface gives an overview of loiterer candidates to show their behavior at a glance, complemented by a lightweight video playback for further details about why a candidate was selected. We demonstrate that our system outperforms state-of-the-art solutions using real-life data sets.

This paper presents a novel video search system that can provide users with diversified and summarized results automatically. Users frequently confront the dilemmas of specifying conditions for video search because too many or few conditions would lead to either insufficient or redundant results. We solve this problem by introducing novel definitions to explore hidden knowledge inherent in underlying datasets. We demonstrate our methods by implementing a prototype system with real-world surveillance videos.

Recently, massive data management plays an increasingly important role in data analytics because data access is a major bottleneck. Data skipping is a promising technique to reduce the number of data accesses. Data skipping partitions data into pages and accesses only pages that contain data to be retrieved by a query. Therefore, effective data partitioning is required to minimize the number of page accesses. However, it is an NP-hard problem to obtain optimal data partitioning given query pattern and data distribution.

The enormous increase of data in RDF format calls for efficient storage and retrieval approaches. Being a highly connected data, RDF generates massive amounts of intermediate results during query processing. Many of the current RDF storage approaches involve large amounts of inter-node data movement even for simple selective query patterns. We propose JARS, a join-aware distributed RDF storage system with a dual-hash partitioning strategy coupled with two layered distributed clustered indexing and a rule-based query-execution approach. JARS eliminates the inter-node communication for star patterns and mitigates the communication cost for chain pattern SPARQL queries. Our experiments indicate that JARS achieves significant performance enhancement over the state-of-the-art RDF storage systems.

Since loitering is a suspicious behavior that often leads to abnormal situations, such as pickpocketing and terrorist attacks, its analysis attracts increasing research attention. In this paper, we present AntiLoiter, a practical loitering discovery system for surveillance. AntiLoiter can efficiently discover loitering candidates from longtime videos across multiple cameras. Most of existing systems mainly focus on how to detect or identify loiterers by behavior tracking techniques. However, the difficulties of tracking-based methods are known as that their analysis results are heavily influenced by occlusions, overlaps, and shadows. Moreover, tracking-based methods need to track the human appearance continuously. Therefore, they are not readily applied to longtime videos across multiple cameras due to the appearance discontinuity of criminal loitering. For instance, a suspect would probably come to the same place several times on different dates for preliminary inspections before a terrorist attack happens, where the loitering behavior is discontinuous. To solve this problem, we abandon the tracking method, instead, design a novel approach to efficiently discover loiterers based on their frequent appearance patterns for longtime videos across multiple cameras. Our approach simply yet effectively adopts existing face recognition techniques for loitering discovery. We conducted extensive experiments on both synthetic and real surveillance videos to evaluate the efficiency and efficacy of our approach. The experimental results show that our system can find out loitering candidates correctly and outperforms existing method by 100 times in terms of runtime.

In this demo, we present Wally, a scalable distributed automated video surveillance system. Wally provides users with rich search functionalities for surveillance, such as instant and historical search based on face or clothing feature by specifying multiple conditions. Unlike most existing systems that focus on video analysis but do not well consider system scalability and search functionality, Wally takes into account both of them, and is developed as a commercial level system for realistic surveillance. To achieve this, we 1) employ relational database, in-memory cache, key-value store, and indexing techniques to ensure the system scalability when it meets Big Data, and 2) develop user-friendly interfaces to enable users to search key frames by rich functionalities. It is estimated by our extensive evaluations that Wally can complete a search task within 50 minutes from the videos captured one day by 1000 real-time cameras, while human monitoring needs 1000x24 man-hours.

The ubiquity of location enabled devices has resulted in a wide proliferation of location based applications and services. To handle the growing scale, database management systems driving such location based services (LBS) must cope with high insert rates for location updates of millions of devices, while supporting efficient real-time analysis on latest location. Traditional DBMSs, equipped with multi-dimensional index structures, can efficiently handle spatio-temporal data. However, popular open source relational database systems are overwhelmed by the high insertion rates, real-time querying requirements, and terabytes of data that these systems must handle. On the other hand, Key-value stores can effectively support large scale operation, but do not natively support multi-attribute accesses needed to support the rich querying functionality essential for the LBSs. We present MD-HBase, a scalable data management system for LBSs that bridges this gap between scale and functionality. Our approach leverages a multi-dimensional index structure layered over a Key-value store. The underlying Key-value store allows the system to sustain high insert throughput and large data volumes, while ensuring fault-tolerance, and high availability. On the other hand, the index layer allows efficient multi-dimensional query processing. We present the design of MD-HBase that builds two standard index structuresâ€"the K-d tree and the Quad treeâ€"over a range partitioned Key-value store. Our prototype implementation using HBase, a standard open-source Key-value store, can handle hundreds of thousands of inserts per second using a modest 16 node cluster, while efficiently processing multidimensional range queries and nearest neighbor queries in real-time with response times as low as hundreds of milliseconds.

Database systems serving cloud platforms must serve large numbers of applications (or tenants ). In addition to managing tenants with small data footprints, different schemas, and variable load patterns, such multitenant data platforms must minimize their operating costs by efficient resource sharing. When deployed over a pay-per-use infrastructure, elastic scaling and load balancing, enabled by low cost live migration of tenant databases, is critical to tolerate load variations while minimizing operating cost. However, existing databases---relational databases and Key-Value stores alike---lack low cost live migration techniques, thus resulting in heavy performance impact during elastic scaling. We present Albatross , a technique for live migration in a multitenant database serving OLTP style workloads where the persistent database image is stored in a network attached storage. Albatross migrates the database cache and the state of active transactions to ensure minimal impact on transaction execution while allowing transactions active during migration to continue execution. It also guarantees serializability while ensuring correctness during failures. Our evaluation using two OLTP benchmarks shows that Albatross can migrate a live tenant database with no aborted transactions, negligible impact on transaction latency and throughput both during and after migration, and an unavailability window as low as 300 ms.

We have developed an application service hosting middleware for enterprise information systems, "Polimatica", upon which a policy-customizable private virtual organization (pVO) is built. The pVO is a collection of customizable application services in the global grid computing environment. "Polimatica" is distinctive in terms of abstraction capabilities in (1) policy instance description, (2) view synthesis of an extensive resource data model, (3) operation aggregation on the OGSI extension interfaces. The performance to enforce the policy instances is measured in a laboratory testbed for an enterprise ASP, providing B2B e-commerce (BizEngine/WS) and interactive media communication service (LiveComm). We have verified that, in 60 seconds, the feedback loop in a pVO adjusts the application services as customized given the changing environment.