The prevalence of check fraud, particularly with stolen checks sold on platforms such as Telegram, creates significant challenges for both individuals and financial institutions. This underscores the urgent need for innovative solutions to detecting and preventing such fraud on social media platforms. While deep learning techniques show great promise in detecting objects and extracting information from images, their effectiveness in addressing check fraud is hindered by the lack of comprehensive, open-source, large training datasets specifically for check information extraction. To bridge this gap, this paper introduces "CheckGuard," a large labeled image-to-text cross-modal dataset designed for check information extraction. CheckGuard comprises over 7,000 real-world stolen check image segments from more than 15 financial institutions, featuring a variety of check styles and layouts. These segments have been manually labeled, resulting in over 50,000 samples across seven key elements: Drawer, Payee, Amount, Date, Drawee, Routing Number, and Check Number. This dataset supports various tasks such as visual question answering (VQA) on checks and check image captioning. Our paper details the rigorous data collecting, cleaning, and annotation processes that make CheckGuard a valuable resource for researchers in check fraud detection, machine learning, and multimodal large language models (MLLMs). We not only benchmark state-of-the-art (SOTA) methods on this dataset to assess their performance but also explore potential enhancements. Our application of parameter-efficient fine-tuning (PEFT) techniques on the SOTA MLLMs demonstrates significant performance improvements, providing valuable insights and practical approaches for enhancing model efficacy on this task. As an evolving project, CheckGuard will continue to be updated with new data, enhancing its utility and driving further advancements in the field. Our PEFT-based MLLM code is available at: https://github.com/feizhao19/CheckGuard. For data access, researchers are required to contact the authors directly.

Harmful Algal Blooms (HABs) present significant environmental and public health threats. Recent machine learning-based HABs monitoring methods often rely solely on unimodal data, e.g., satellite imagery, overlooking crucial environmental factors such as temperature. Moreover, existing multi-modal approaches grapple with real-time applicability and generalizability challenges due to the use of ensemble methodologies and hard-coded geolocation clusters. Addressing these gaps, this paper presents a novel deep learning model using a single-model-based multi-task framework. This framework is designed to segment water bodies and predict HABs severity levels concurrently, enabling the model to focus on areas of interest, thereby enhancing prediction accuracy. Our model integrates multimodal inputs, i.e., satellite imagery, elevation data, temperature readings, and geolocation details, via a dual-branch architecture: the Satellite-Elevation (SE) branch and the Temperature-Geolocation (TG) branch. Satellite and elevation data in the SE branch, being spatially coherent, assist in water area detection and feature extraction. Meanwhile, the TG branch, using sequential temperature data and geolocation information, captures temporal algal growth patterns and adjusts for temperature variations influenced by regional climatic differences, ensuring the model's adaptability across different geographic regions. Additionally, we propose a geometric multimodal focal loss to further enhance representation learning. On the Tick-Tick Bloom (TTB) dataset, our approach outperforms the SOTA methods by 15.65%.

The k-core of a graph is the largest induced sub-graph with minimum degree k. The problem of k-core decomposition finds the k-cores of a graph for all valid values of k, and it has many applications such as network analysis, computational biology and graph visualization. Currently, there are two types of parallel algorithms for k-core decomposition: (1) degree-based vertex peeling, and (2) iterative h-index refinement. There is, however, few studies on accelerating k-core decomposition using GPU. In this paper, we propose a highly optimized peeling algorithm on a GPU, and compare it with possible implementations on top of think-like-a-vertex graph-parallel GPU systems as well as existing serial and parallel k-core decomposition algorithms on CPUs. Extensive experiments show that our GPU algorithm is the overall winner in both time and space. Our source code is released at https://github.com/akhlaqueak/KCoreGPU.

Recently, the security problem of wireless networked control systems (WNCSs) has become critical. In this article, we focus on strategies designing problem for sensor and denial-of-service (DoS) attacker in WNCS based on linear quadratic Gaussian (LQG) cost. In this scenario, a sensor measures the output of the system and transmits its local state estimate to the remote estimator via an unreliable channel which may suffer interference from an intelligent DoS attacker. In each step, the sensor demands to determine the power to transmit its packet data, at the same time, the attacker needs to determine the interference power to degrade the performance of the WNCS. To analyze this interactive decision-making process, we construct a two-player zero-sum stochastic game and propose an improved LQG cost considering energy-efficient factors. Then, for continuous power-level setting, we discuss the existence and the structure of equilibrium strategies of the sensor–attacker game. For a discrete power set, an algorithm is developed to solve equilibrium strategies for both players. Finally, we present an example to illustrate our results.

With the explosion of social media networks, many modern applications are concerning about people's connections, which leads to the so-called social computing. An elusive question is to study how opinion communities form and evolve in real-world networks with great individual diversity and complex human connections. In this scenario, the classic K-means technique and its extended versions could not be directly applied, as they largely ignore the relationship among interactive objects. On the other side, traditional community detection approaches in statistical physics would be neither adequate nor fair: they only consider the network topological structure but ignore the heterogeneous-objects' attributive information. To this end, we attempt to model a realistic social media network as a discrete-time dynamical system, where the opinion matrix and the community structure could mutually affect each other. In this paper, community detection in social media networks is naturally formulated as a multi-objective optimization problem (MOOP), i.e., finding a set of densely connected components with similar opinion vectors. We propose a novel and powerful graph K-means framework, which is composed of three coupled phases in each discrete-time period. Specifically, the first phase uses a fast heuristic approach to identify those opinion leaders who have relatively high local reputation; the second phase adopts a novel dynamic game model to find the locally Pareto-optimal community structure; and the final phase employs a robust opinion dynamics model to simulate the evolution of the opinion matrix. We conduct a series of comprehensive experiments on real-world benchmark networks to validate the performance of GK-means through comparisons with the state-of-the-art graph clustering technologies.

Based on the notion of “emergence”, Xu et al. (Usenix Security 2012; TDSC 2013) developed the first concrete instantiation of emerging-image moving-object (EIMO) CAPTCHAs using 2D hollow objects (codewords), shown to be usable and believed to be secure. In this paper, we highlight the hidden security weaknesses of such a 2D EIMO CAPTCHA design. A key vulnerability is that the camera projection on 2D objects is constant (unlike 3D objects), making it possible to reconstruct the underlying codewords by superimposing and aggregating the temporally scattered parts of the object extracted from consecutive frames. We design and implement an automated attack framework to defeat this design using image processing techniques, and show that its accuracy in recognizing moving codewords is up to 89.2 percent, under different parameterizations. Our framework can be broadly used to undermine the security of different instances of 2D EIMO CAPTCHAs (not just the current state-of-the-art by Xu et al.), given the generalized and robust back-end theories in our attack, namely the methods to locate a codeword, reduce noises and accumulate objects' contour information from consecutive frames corresponding to multiple time periods. As a countermeasure, we propose a fundamentally different design of EIMO CAPTCHAs based on pseudo 3D objects, and examine its security as well as usability. We argue that this design can resist our attack against 2D EIMO CAPTCHAs, although at the cost of reduced usability compared to the - now insecure - 2D EIMO CAPTCHAs.

We propose GraphD, an out-of-core Pregel-like system targeting efficient big graph processing with a small cluster of commodity PCs connected by Gigabit Ethernet, an environment affordable to most users. This is in contrast to some recent efforts for out-of-core graph computation with specialized hardware. In our setting, a vertex-centric program is often data-intensive, since the CPU cost of calculating a message value is negligible compared with the network cost of transmitting that message. As a result, network bandwidth is usually the bottleneck, and out-of-core execution would not sacrifice performance if disk IO overhead can be hidden by message transmission, which is achieved by GraphD through the parallelism of computation and communication. GraphD streams edge and message data on local disks, and thus consumes negligible memory space. For a broad class of Pregel algorithms where message combiner is applicable, GraphD completely eliminates the need of any expensive external-memory join or group-by, which is required by existing systems such as Pregelix and Chaos. Extensive experiments show that GraphD beats existing out-of-core systems by orders of magnitude, and achieves comparable performance to in-memory systems running with adequate memory.

Detecting logo frequency and duration in sports videos provides sponsors an effective way to evaluate their advertising efforts. However, general-purposed object detection methods cannot address all the challenges in sports videos. In this paper, we propose a mutual-enhanced approach that can improve the detection of a logo through the information obtained from other simultaneously occurred logos. In a Fast-RCNN-based framework, we first introduce a homogeneity-enhanced re-ranking method by analyzing the characteristics of homogeneous logos in each frame, including type repetition, color consistency, and mutual exclusion. Different from conventional enhance mechanism that improves the weak proposals with the dominant proposals, our mutual method can also enhance the relatively significant proposals with weak proposals. Mutual enhancement is also included in our frame propagation mechanism that improves logo detection by utilizing the continuity of logos across frames. We use a tennis video dataset and an associated logo collection for detection evaluation. Experiments show that the proposed method outperforms existing methods with a higher accuracy.

Contour detection is a fundamental problem in computer vision. However, there is still a considerable disparity between detection results and actual contours. To detect object-level contours on the basis of comprehensive analysis of potential edges, we present a deep-learning-based approach with a conditional random fields (CRF) model. We obtain the initial edgemap with a VGGNet-based model, and establish a contour correlation graph (CCG) to describe the potential edges and their relationships. Then a CRF model is adopted to fulfill the optimization on the basis of the analysis of object-level contour characteristics and to predict the validity of candidate contours. The experiment results on various datasets demonstrate that the proposed method outperforms the existing methods by a noticeable margin.

Existing CAPTCHA solutions are a major source of user frustration on the Internet today, frequently forcing companies to lose customers and business. Game CAPTCHAs are a promising approach which may make CAPTCHA solving a fun activity for the user. One category of such CAPTCHAs – called Dynamic Cognitive Game (DCG) CAPTCHA – challenges the user to perform a game-like cognitive (or recognition) task interacting with a series of dynamic images. Specifically, it takes the form of many objects floating around within the images, and the user’s task is to match the objects corresponding to specific target(s), and drag/drop them to the target region(s). In this paper, we pursue a comprehensive analysis of DCG CAPTCHAs. We design and implement such CAPTCHAs, and dissect them across four broad but overlapping dimensions: (1) usability, (2) fully automated attacks, (3) human-solving relay attacks, and (4) hybrid attacks that combine the strengths of automated and relay attacks. Our study shows that DCG CAPTCHAs are highly usable, even on mobile devices and offer some resilience to relay attacks, but they are vulnerable to our proposed automated and hybrid attacks.

Retrieval of images with object-of-interest from a vast pool of social media images has been a research interest in cyber crime research community for detecting criminal behaviors in social media. Due to inherent diversity and the low duplicate property of images on social media, it brings forth many challenges in image retrieval, especially in identifying distinct features for a given object-of-interest. Previous literature approached this problem with extended General Hough Transform, where Hough space is analyzed for each specific object-of-interest. Different objects of interest produce different types of patterns in Hough space and no unified framework can be easily established to incorporate all those patterns. In this paper, we propose a unified framework based on convolutional neural network (CNN) for classifying the social media images and retrieving the images based on the probability score from the softmax classifier. In our framework, a reduced size CNN model is trained by distilling the knowledge from a pretrained full size CNN model, which is suitable for applications with limited training data such as ours and results in a higher accuracy in image retrieval as well as better performance in execution speed in comparison with the full size CNN model. Experiments on three image datasets relating to suspicious e-crime and terrorist involvement - Guy Fawkes masks, credit card logos, and ISIS logos show that our framework outperforms extended General Hough Transform and the full size CNN model.

CAPTCHAs represent an important pillar in the web security domain. Yet, current CAPTCHAs do not fully meet the web security requirements. Many existing CAPTCHAs can be broken using automated attacks based on image processing and machine learning techniques. Moreover, most existing CAPTCHAs are completely vulnerable to human-solver relay attacks, whereby CAPTCHA challenges are simply outsourced to a remote human solver.

Body volume and body shape have been used in the estimation of body composition in clinical research. However, the determination of body volume typically requires sophisticated and expensive equipment. Similarly, the use of body shape to predict body composition is limited by rater biases as well as reproducibility. In this paper, we aim to introduce simple yet relatively accurate techniques for body volume and body shape representation that reduce limitations of traditional approaches. We propose an automated method to construct a 3D model of the body by accumulating ellipse-like slices formed by using the length and width features sampled from the back and side profile images. Body volume is represented in pixels by adding up the areas of the slices. Apart from representing body volume in pixels, we also aim to extract shape features from the 2D images and to create clusters of individuals according to their body shape. The body volume representation and the proposed shape features together with other meta-information including age, sex, race, height, and weight, could be effectively used in body composition prediction. Our study results indicate that the body volume calculated by the proposed method is reasonably accurate and the extracted shape clusters provide important information when estimating body composition.

The snake algorithm has been proposed to solve many remote sensing and computer vision problems such as object segmentation, surface reconstruction, and object tracking. This paper introduces a framework for 3-D building model construction from LIDAR data based on the snake algorithm. It consists of nonterrain object identification, building and tree separation, building topology extraction, and adjustment by the snake algorithm. The challenging task in applying the snake algorithm to building topology adjustment is to find the global minima of energy functions derived for 2-D building topology. The traditional snake algorithm uses dynamic programming for computing the global minima of energy functions which is limited to snake problems with 1-D topology (i.e., a contour) and cannot handle problems with 2-D topology. In this paper, we have extended the dynamic programming method to address the snake problems with a 2-D planar topology using a novel graph reduction technique. Given a planar snake, a set of reduction operations is defined and used to simplify the graph of the planar snake into a set of isolated vertices while retaining the minimal energy of the graph. Another challenging task for 3-D building model reconstruction is how to enforce different kinds of geometric constraints during building topology refinement. This framework proposed two energy functions, deviation and direction energy functions, to enforce multiple geometric constraints on 2-D topology refinement naturally and efficiently. To examine the effectiveness of the framework, the framework has been applied on different data sets to construct 3-D building models from airborne LIDAR data. The results demonstrate that the proposed snake algorithm successfully found the global optima in polynomial time for all of the building topologies and generated satisfactory 3-D models for most of the buildings in the study areas.

Existing captcha solutions on the Internet are a major source of user frustration. Game captchas are an interesting and, to date, little-studied approach claiming to make captcha solving a fun activity for the users. One broad form of such captchas -- called Dynamic Cognitive Game (DCG) captchas -- challenge the user to perform a game-like cognitive task interacting with a series of dynamic images. We pursue a comprehensive analysis of a representative category of DCG captchas. We formalize, design and implement such captchas, and dissect them across: (1) fully automated attacks, (2) human-solver relay attacks, and (3) usability. Our results suggest that the studied DCG captchas exhibit high usability and, unlike other known captchas, offer some resistance to relay attacks, but they are also vulnerable to our novel dictionary-based automated attack.

Dynamic Cognitive Game (DCG) CAPTCHAs are a promising new generation of interactive CAPTCHAs aiming to provide improved security against automated and human-solver relay attacks. Unlike existing CAPTCHAs, defeating DCG CAPTCHAs using pure automated attacks or pure relay attacks may be challenging in practice due to the fundamental limitations of computer algorithms (semantic gap) and synchronization issues with solvers. To overcome this barrier, we propose two hybrid attack frameworks. which carefully combine the strengths of an automated program and offline/online human intelligence. These hybrid attacks require maintaining the synchronization only between the game and the bot similar to a pure automated attack, while solving the static AI problem (i.e., bridging the semantic gap) behind the game challenge similar to a pure relay attack. As a crucial component of our framework, we design a new DCG object tracking algorithm, based on color code histogram, and show that it is simpler, more efficient and more robust compared to several known tracking approaches. We demonstrate that both frameworks can effectively defeat a wide range of DCG CAPTCHAs.

Inaccurate image segmentation often has a negative impact on object-based image retrieval. Researchers have attempted to alleviate this problem by using hierarchical image representation. However, these attempts suffer from the inefficiency in building the hierarchical image representation and the high computational complexity in matching two hierarchically represented images. Existing approaches construct the hierarchical image representation in two steps. The first step is to perform segmentation at different image resolutions, and the second step is to construct a hierarchical representation of the image by associating segments from different resolutions. In this research, an innovative all-in-one run approach is proposed that concurrently performs image segmentation and hierarchical tree construction, producing a hierarchical region tree to represent the image. In addition, an efficient hierarchical region tree matching algorithm is proposed with a reasonably low time complexity and used in multiple object image retrieval. The experimental results demonstrate the efficacy and efficiency of the proposed approach.

In this paper we present a two-phase spam image clustering framework. The proposed framework performs a histogram based projective clustering on visual features in the first phase, followed by a text-based clustering in the second phase. There are several contributions in this study. First, we address the complex nature of spam image obfuscation techniques. Second, a multi-clue framework is developed to profile spam images of common spamming sources which provide evidence for tracking spam gangs. Third, projective clustering eliminates the need to choose among distance metrics for clustering analysis, while systematically exploring subspaces that correspond to clusters.

We study the problem of place recognition. Given a photo, we estimate its location by scene matching to a large database of internet photos of known locations. Traditional strategies, which involve a linear scan of the database to find matching scenes, fail to scale. On the other hand, internet photos contain a massive amount of noise and redundancy, which is of little help for place recognition. By exploiting the scene distribution of photos, we summarize the database by a set of canonical views. The set of canonical views eliminates the noise and redundancy in internet photos, and provides a compact representation for the database. By restricting scene matching to the set of canonical views, we observe a good tradeoff between efficiency and recall: the average processing time for a query photo is reduced by 97%, while the recall rate for place recognition remains at 75%.

With the rapid growth of information, efficient and robust information retrieval techniques have become increasingly more important. Multiple object retrieval remains challenging due to the complex nature of this problem. The proposed research, unlike most existing works that are designed for single object retrieval or adopt heuristic multiple object matching scheme, aims at contributing to this field through the development of an image retrieval system that adopts a hierarchical region-tree representation of image, and enables effective and efficient multiple object retrieval and automatic discovery of the objects of interest through users' relevance feedback. We believe this is the first systematic attempt to formulate a comprehensive, intelligent, and interactive framework for multiple object retrieval in image databases that makes use of a hierarchical region-tree representation.

In this abstract, we introduce the idea of our new multimodality approach towards video genre detection. The two novelties of this algorithm are that firstly it is based on identifying relationships between semantic concepts and genres in videos, which can later help in distinguishing videos in different genres. Secondly, we apply topic level genre detection such as a 'talk show' genre can be further categorized as 'political talk show' or 'economic talk show', etc., which is a more precise categorization.

Text mining tools and algorithms are being successfully used for information extraction especially on large corpus like biomedical publications. These tools not only aid in information extraction but also in forming new theories and relationships between various fields of biomedical research. Extraction of gene-gene or gene-disease relationship is one such application. In this paper, we introduce a method to detect coexpressed genes from text, using the grammatical dependencies among the words within sentences and Dynamic Conditional Random Fields (DCRFs). Determining the coexpression relationship between and among genes can help in identifying important concepts such as the functionality of gene(s) involved, their pathogenic mechanism, and in deciphering protein-protein interactions. This work attempts to extract relevant sentences by labeling the genes involved as well as the word representing the relationship, from full-length papers collected from PubMed. The results obtained were compared with that of Support Vector Machine (SVM) and Nearest Neighbor with generalization (NNge), and have been found to outperform both.

Most approaches to vehicle tracking have adopted a single calibrated camera for the task, which leads to an under-conditioned problem. We present a surveillance system for on-line vehicle tracking based on two cameras and structure from motion (SfM). Our surveillance system starts by tracking feature points. A novel matching scheme is proposed that allows a subset of feature points to be corresponded across disparate views. Based on the reconstructed subset, the full set of feature points are reconstructed in 3D and segmented into different vehicles by solving a multiple labeling problem.

Most existing object-based image retrieval systems are based on single object matching, with its main limitation being that one individual image region (object) can hardly represent the user's retrieval target especially when more than one object of interest is involved in the retrieval. In this paper, we present a Feedback-based Image Clustering and Retrieval Framework (FIRM) using a novel image clustering algorithm and integrating it with Integrated Region Matching (IRM) and Relevance Feedback (RF). The performance of the system is evaluated on a large image database, demonstrating the effectiveness of our framework in reflecting users' retrieval interests in object-based image retrieval.

This paper proposes a human-centered interactive framework for automatically mining and retrieving semantic events in videos. After preprocessing, the object trajectories and event models are fed into the core components of the framework for learning and retrieval. As trajectories are spatiotemporal in nature, the learning component is designed to analyze time series data. The human feedback to the retrieval results provides progressive guidance for the retrieval component in the framework. The retrieval results are in the form of video sequences instead of contained trajectories for user convenience. Thus, the trajectories are not directly labeled by the feedback as required by the training algorithm. A mapping between semantic video retrieval and multiple instance learning (MIL) is established in order to solve this problem. The effectiveness of the algorithm is demonstrated by experiments on real-life transportation surveillance videos.

Bacteria colony classification and enumeration is an essential step in clonogenic assay, which is often performed manually in many clinical research laboratories. It is known to be time consuming and labor intensive, but also error-prone since the manual counting tends to have more subjective interpretation and largely relies on persistent practice, especially when a large amount of colonies exist in the plate. In this demonstration, we introduce a fully automatic counter for bacteria colony enumeration and classification. The proposed counter accepts digital camera images as its input and performs colony enumeration and classification based on image mining. The proposed counter shows a robust performance in terms of both precision and recall, and is efficient in terms of labor- and time-savings.

This paper proposes a framework for retrieving semantic video events from indoor surveillance video databases. The goal is to locate video sequences containing events of interest to the user. This framework starts by tracking objects and segmenting videos into Common Appearance Intervals (CAIs). The spatiotemporal trajectories are obtained, based on which features are extracted for the construction of semantic event models. In the retrieval, the database user interacts with the machine and provides "feedbacks" to the retrieval result. The learning component learns from the spatiotemporal data, the semantic event model as well as the "feedback" and returns the refined result to the user. Specifically, the learning algorithm is developed based on a Coupled Hidden Markov Model (CHMM), which models the interactions of objects in CAIs and recognizes hidden patterns among them. This iterative learning and retrieval process contributes to the bridging of the "semantic gap", and the experimental results show the effectiveness of the proposed framework.

Energy-minimizing active contour models (snakes) have been proposed for solving many computer vision problems such as object segmentation, surface reconstruction, and object tracking. Dynamic programming which allows natural enforcement of constraints is an effective method for computing the global minima of energy functions. However, this method is only limited to snake problems with one dimensional (ID) topology (i.e., a contour) and cannot handle problems with two-dimensional (2D) topology. In this paper, we have extended the dynamic programming method to address the snake problems with 2D topology using a novel graph reduction algorithm. Given a 2D snake with first order energy terms, a set of reduction operations are defined and used to simplify the graph of the 2D snake into one single vertex while retaining the minimal energy of the snake. The proposed algorithm has a polynomial-time complexity bound and the optimality of the solution for a reducible 2D snake is guaranteed. However, not all types of 2D snakes can be reduced into one single vertex using the proposed algorithm. The reduction of general planar snakes is an NP-complete problem. The proposed method has been applied to optimize 2D building topology extracted from airborne LIDAR data to examine the effectiveness of the algorithm. The results demonstrate that the proposed approach successfully found the global optima for over 98% of building topology in a polynomial time.

Detecting and retrieving incidents from traffic surveillance videos is an important research topic in designing an Intelligent Transportation System (ITS). Most existing video analysis techniques focus on low level features of video data. A "semantic gap" exists between the machine-readable low level features and the high level human understanding of the video content. Aiming at this problem, we propose an interactive framework for semantic video retrieval. This framework is based on the spatio-temporal modeling of vehicle trajectories. With Relevance Feedback (RF), human interaction is involved in the learning and retrieval process. The retrieval mechanism is thus guided by the user's response to the retrieved results. Experiments show the effectiveness of the framework.

Gridding and spot segmentation are two critical steps in microarray gene expression data analysis. However, the problems of noise contamination and donut-shaped spots often make signal extraction process a laborintensive task. In this paper, we propose a three-step method for automatic gridding and spot segmentation. The method starts with a background removal and noise eliminating step, and then proceeds in two steps. The first step applies a fully unsupervised method to extract blocks and grids from the cleaned data. The second step applies a simple, progressive spot segmentation method to deal with inner holes and noise in spots. We tested its performance on real microarray images against a widely used software GenePix. Our results show that the proposed method deals effectively with poor-conditioned microarray images in both gridding and spot segmentation.

Understanding and retrieving videos based on their semantic contents is an important research topic in multimedia data mining and has found various real- world applications. Most existing video analysis techniques focus on the low level visual features of video data. However, there is a "semantic gap" between the machine-readable features and the high level human concepts i.e. human understanding of the video content. In this paper, an interactive platform for semantic video mining and retrieval is proposed using relevance feedback (RF), a popular technique in the area of content-based image retrieval (CBIR). By tracking semantic objects in a video and then modeling spatio-temporal events based on object trajectories and object interactions, the proposed interactive learning algorithm in the platform is able to mine the spatio-temporal data extracted from the video. An iterative learning process is involved in the proposed platform, which is guided by the user's response to the retrieved results. Although the proposed video retrieval platform is intended for general use and can be tailored to many applications, we focus on its application in traffic surveillance video database retrieval to demonstrate the design details. The effectiveness of the algorithm is demonstrated by our experiments on real-life traffic surveillance videos.

Content-based image retrieval has become an important part of information retrieval technology. Images can be viewed as high dimensional data and are usually represented by their low-level features. How to effectively find the semantic meanings of images is a central challenge in the area. In this paper, we propose an interactive platform for region-based image clustering and retrieval. A genetic algorithm is used to perform the initial clustering. In order to further refine the clustering results, we adopt the maximum flow/minimum cut theorem from graph theory to do outlier/outlier group detection. Outlier detection can help identify misclustered image segments and is used to improve the quality of clusters in this paper. In the interactive retrieval phase, user feedback is used to dynamically locate candidate images from clusters and outliers/outlier groups. Through relevance feedback, more information is gathered and fed to the learning algorithm-one-class SVM. Experiments show the effectiveness of the proposed platform

In this paper, we present an automatic spatio-temporal mining system of rolling and adherent leukocytes for intravital videos. The magnitude of leukocyte adhesion and the decrease in rolling velocity are common interests for inflammation response studies. Currently, there is no existing system which is perfect for such purposes. Our approach starts with locating moving leukocytes by probabilistic learning of temporal features. It then removes noises through median and location-based filtering, and finally performs motion correspondence through centroid trackers. By extracting the information about moving leukocytes first, we are able to extract adherent leukocytes in a more robust way with an adaptive threshold method. The effectiveness and the efficiency of the proposed method are demonstrated by the experimental results

Event detection is of great importance in high-level semantic indexing and selective browsing of video clips. However, the use of low-level visual-audio feature descriptors alone generally fails to yield satisfactory results in event identification due to the semantic gap issue. In this paper, we propose an advanced approach for exciting event detection in soccer video with the aid of multi-level descriptors and classification algorithm. Specifically, a set of algorithms are developed for efficient extraction of meaningful mid-level descriptors to bridge the semantic gap and to facilitate the comprehensive video content analysis. The data classification algorithm is then performed upon the combination of multimodal mid-level descriptors and low-level feature descriptors for event detection. The effectiveness and efficiency of the proposed framework are demonstrated over a large collection of soccer video data with different styles produced by different broadcasters.

A rapid increase in the amount of image data and the inefficiency of traditional text-based image retrieval systems have served to make content-based image retrieval an active research field. It is crucial to effectively discover users' concept patterns through an acquired understanding of the subjective role played by humans in the retrieval process for such systems. A learning and retrieval framework is used to achieve this. It seamlessly incorporates multiple instance learning for relevant feedback to discover users concept patterns-especially in the region of greatest user interest. It also maps the local feature vector of that region to the high-level concept pattern. This underlying mapping can be progressively discovered through feedback and learning. The user guides the retrieval systems learning process using his/her focus of attention. Retrieval performance is tested to establish the feasibility and effectiveness of the proposed learning and retrieval framework

Multiple Instance Learning (MIL) is a special kind of supervised learning problem that has been studied actively in recent years. In this paper, we propose an approach based on One-Class Support Vector Machine (SVM) to solve MIL problem in the region-based Content Based Image Retrieval (CBIR). Relevance Feedback technique is incorporated to provide progressive guidance to the learning process. Performance is evaluated and the effectiveness of our retrieval algorithm has been shown through comparative studies.

Relevance feedback (RF) is a widely used technique in incorporating user's knowledge with the learning process for content-based image retrieval (CBIR). As a supervised learning technique, it has been shown to significantly increase the retrieval accuracy. However, as a CBIR system continues to receive user queries and user feedbacks, the information of user preferences across query sessions are often lost at the end of search, thus requiring the feedback process to be restarted for each new query. A few works targeting long-term learning have been done in general CBIR domain to alleviate this problem. However, none of them address the needs and long-term similarity learning techniques for region-based image retrieval. This paper proposes a latent semantic indexing (LSI) based method to utilize users' relevance feedback information. The proposed region-based image retrieval system is constructed on a multiple instance learning (MIL) framework with one-class support vector machine (SVM) as its core. Experiments show that the proposed method can better utilize users' feedbacks of previous sessions, thus improving the performance of the learning algorithm (one-class SVM).

We propose a new multimedia data mining framework for the extraction of soccer goal events in soccer videos by using combined multimodal analysis and decision tree logic. The extracted events can be used to index the soccer videos. We first adopt an advanced video shot detection method to produce shot boundaries and some important visual features. Then, the visual/audio features are extracted for each shot at different granularities. This rich multimodal feature set is filtered by a pre-filtering step to clean the noise as well as to reduce the irrelevant data. A decision tree model is built upon the cleaned data set and is used to classify the goal shots. Finally, the experimental results demonstrate the effectiveness of our framework for soccer goal extraction.

The paper proposes a method to discover effectively users' concept patterns when multiple objects of interest (e.g., foreground and background objects) are involved in content-based image retrieval. The proposed method incorporates multiple instance learning into the user relevance feedback in a seamless way to discover where the user's objects/regions of most interest are and how to map the local features of that(those) region(s) to the user's high-level concepts. A three-layer neural network is used to model the underlying mapping progressively through the feedback and learning procedure.

In this paper, we propose a unified framework, called Markov Model Mediator (MMM), to facilitate image database clustering and to improve the query performance. The structure of the MMM framework consists of two hierarchical levels: local MMMs and integrated MMMs, which model the affinity relations among the images within a single image database and within a set of image databases, respectively, via an effective data mining process. The effectiveness and efficiency of the MMM framework for database clustering and image retrieval are demonstrated over a set of image databases which contain various numbers of images with different dimensions and concept categories.

Abstract As an environmental phenomenon, hurricanes cause significant property damage and loss of life in coastal areas almost every year. Research concerning hurricanes and their aftermath is gaining more and more attention nowadays. This paper presents our work in designing and building a Web‐based distributed software system that can be used for the statistical analysis and projection of hurricane occurrences. Firstly, our system is a large‐scale system and can handle the huge amount of hurricane data and intensive computations in hurricane data analysis and projection. Secondly, it is a distributed system, which allows multiple users at different locations to access the system simultaneously and to share and exchange the data and data model. Thirdly, our system is a database‐centered system where the Oracle database is employed to store and manage the large amount of hurricane data, the hurricane model and the projection results. Finally, a three‐tier architecture has been adopted to make our system robust and resistant to the potential change in the lifetime of the system. This paper focuses on the three‐tier system architecture, describing the design and implementation of the components at each layer. Copyright © 2004 John Wiley & Sons, Ltd.

In this demonstration, we present an image retrieval system to support multimedia authoring and presentation. An affinity-based mechanism, Markov Model Mediator (MMM), is used as the search engine for the system, which utilizes both the low-level image features and the learned high-level concepts via user access patterns and access frequencies. This system is one of the major components of MediaManager, a distributed multimedia management system developed by us. Both retrieval and learning facilities are supported in this system. The retrieval system also provides input information to the Multimedia Augmented Transition Network (MATN) environment for multimedia authoring and presentation.

Recent advances in airborne light detection and ranging (LIDAR) technology allow rapid and inexpensive measurements of topography over large areas. This technology is becoming a primary method for generating high-resolution digital terrain models (DTMs) that are essential to numerous applications such as flood modeling and landslide prediction. Airborne LIDAR systems usually return a three-dimensional cloud of point measurements from reflective objects scanned by the laser beneath the flight path. In order to generate a DTM, measurements from nonground features such as buildings, vehicles, and vegetation have to be classified and removed. In this paper, a progressive morphological filter was developed to detect nonground LIDAR measurements. By gradually increasing the window size of the filter and using elevation difference thresholds, the measurements of vehicles, vegetation, and buildings are removed, while ground data are preserved. Datasets from mountainous and flat urbanized areas were selected to test the progressive morphological filter. The results show that the filter can remove most of the nonground points effectively.

One key technology of intelligent transportation systems is the use of advanced sensor systems for on-line surveillance to gather detailed information on traffic conditions. Traffic video analysis can provide a wide range of useful information to traffic planners. In this context, the object-level indexing of video data can enable vehicle classification, traffic flow analysis, incident detection and analysis at intersections, vehicle tracking for traffic operations, and update of design warrants. In this paper, a learning-based automatic framework is proposed to support the multimedia data indexing and querying of spatio-temporal relationships of vehicle objects in a traffic video sequence. The spatio-temporal relationships of vehicle objects are captured via the proposed unsupervised image/video segmentation method and object tracking algorithm, and modeled using a multimedia augmented transition network model and multimedia input strings. An efficient and effective background learning and subtraction technique is employed to eliminate the complex background details in the traffic video frames. It substantially enhances the efficiency of the segmentation process and the accuracy of the segmentation results to enable more accurate video indexing and annotation. The paper uses four real-life traffic video sequences from several road intersections under different weather conditions in the study experiments. The results show that the proposed framework is effective in automating data collection and access for complex traffic situations.

Automatic video scene change detection is a challenging task. Using audio or visual information alone often cannot provide a satisfactory solution. However, how to combine audio and visual information efficiently still remains a difficult issue since there are various cases in their relationship due to the versatility of videos. We present an effective scene change detection method that adopts the joint evaluation of the audio and visual features. First, video information is used to find the shot boundaries. Second, the audio features for each video shot can be extracted. Lastly, an audio-video combination schema is proposed to detect the video scene boundaries.

In this paper a novel unsupervised segmentation framework for texture image queries is presented. The proposed framework consists of an unsupervised segmentation method for texture images, and a multi-filter query strategy. By applying the unsupervised segmentation method on each texture image, a set of texture feature parameters for that texture image can be extracted automatically. Based upon these parameters, an effective multi-filter query strategy which allows the users to issue texture-based image queries is developed The test results of the proposed framework on 318 texture images obtained from the MIT VisTex and Brodatz database are presented to show its effectiveness.

In order to manage the growing amount of video information efficiently, a video scene change detection method is necessary. Many advanced video applications such as video on demand (VOD) and digital library also require the scene change detection to organize the video content. In this paper, we present an effective scene change detection method using an unsupervised segmentation algorithm and the technique of object tracking based on the results of the segmentation. Our results have shown that this method can perform not only accurate scene change detection, but also obtain object level information of the video frames, which is very useful for video content indexing and analysis. 1.

In today's fast-growing information age, multimedia data is becoming more and more common in daily applications, however this data is nearly useless if there is no computer-aided browsing, searching, and retrieving mechanism to obtain the desired contents. In this demonstration, we present a multimedia query and presentation management system for multimedia databases. It uses an unsupervised segmentation method for image and video feature extraction. A presentation design interface systems is provided that allows query results to be used to create multimedia presentations.

S.C. Chen et al. (1999) proposed a multimedia augmented transition network (ATN) model, together with its multimedia input strings, to model and structure video data. This multimedia ATN model was based on an ATN model that had been used within the artificial intelligence (AI) arena for natural-language understanding systems, and its inputs were modeled by multimedia input strings. The temporal and spatial relations of semantic objects were captured by an unsupervised video segmentation method called the SPCPE (simultaneous partitioning and class parameter estimation) algorithm, and they were modeled by the multimedia input strings. However, the segmentation method used was not able to identify objects that are overlapped together within video frames. The identification of overlapped objects is a great challenge. For this purpose, a backtrack-chain-update-split algorithm is developed in this paper that identifies the split segment (object) and uses this information in the current frame to update the previous frames in a backtrack-chain manner. The proposed split algorithm provides more accurate temporal and spatial information of the semantic objects for video indexing.