Multimodal Large Language Models (MLLMs) are advancing the ability to reason about complex sports scenarios by integrating textual and visual information. To comprehensively evaluate their capabilities, we introduce SPORTU, a benchmark designed to assess MLLMs across multi-level sports reasoning tasks. SPORTU comprises two key components: SPORTU-text, featuring 900 multiple-choice questions with human-annotated explanations for rule comprehension and strategy understanding. This component focuses on testing models' ability to reason about sports solely through question-answering (QA), without requiring visual inputs; SPORTU-video, consisting of 1,701 slow-motion video clips across 7 different sports and 12,048 QA pairs, designed to assess multi-level reasoning, from simple sports recognition to complex tasks like foul detection and rule application. We evaluated four prevalent LLMs mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting on the SPORTU-text part. GPT-4o achieves the highest accuracy of 71\%, but still falls short of human-level performance, highlighting room for improvement in rule comprehension and reasoning. The evaluation for the SPORTU-video part includes 6 proprietary and 8 open-source MLLMs. Experiments show that models fall short on hard tasks that require deep reasoning and rule-based understanding. GPT-4o performs the best with only 57.8\% accuracy on the hard task, showing large room for improvement. We hope that SPORTU will serve as a critical step toward evaluating models' capabilities in sports understanding and reasoning. The dataset is available at [https://github.com/chili-lab/SPORTU](https://github.com/chili-lab/SPORTU).

In recent years, the application of autonomous aerial vehicle (AAV) devices in military, industrial, and civilian sectors has become increasingly widespread. Consequently, research on multi-AAV formation cooperative control strategies has garnered significant attention. However, current multiagent reinforcement learning algorithms often struggle with unguided exploration, making it challenging for agents to develop efficient action strategies for complex collaborative tasks. To address this issue, this article introduces a multicritic deep deterministic policy gradient (MCDDPG) algorithm. This algorithm designs a multicritic (MC) structure based on the DDPG algorithm. This structure guides AAVs using physical models for tracking and obstacle avoidance, while deep learning models are employed to facilitate cooperative coordination among AAVs. Furthermore, to address the weight allocation issue among different Critic modules in the MC structure, a dynamic difficulty priority weight optimization algorithm is implemented. This enhances the algorithm’s collaborative capabilities. To validate the collaborative planning capability of the proposed algorithm, a simulation scenario involving multicoupled tasks is designed in the multiagent particle environment (MPE). In this scenario, the MCDDPG algorithm demonstrates the fastest convergence speed and the optimal collaborative strategy, outperforming other state-of-the-art multiagent deep reinforcement learning (MADRL) algorithms currently in use.

Abnormal behavior detection in surveillance video, as one of the essential functions in the intelligent surveillance system, plays a vital role in anti-terrorism, maintaining stability, and ensuring social security. Aiming at the problem of extremely imbalance between normal behavior data and abnormal behavior data, the probabilistic memory model-based network is designed to learn from the distribution of normal behaviors and guide the detection of abnormal behavior. An auto-encoding model is employed as the backbone network, and the gap between the predicted future frame and the real frame is used to measure the degree of abnormality. An autoregressive conditional probability estimation model and a normal distribution memory model are employed as auxiliary modules, to achieve the prediction of normal frames. When extracting temporal and spatial features in the backbone network, the causal three-dimensional convolution and time-dimension shared fully connected layers are used to avoid future information leakage and ensure the timing of information. In addition, from the perspective of probability entropy and behavioral modality diversity, autoregressive probability model is proposed to fit the distribution of input normal frame, so the network converges to the low entropy state of the normal behavior distribution. The memory module stores the feature of normal behavior in historical data, and injects the current input data. The memory vector and the encoding vector are concatenated along the time dimension and input to the decoder, realizing normal frame prediction. Using public datasets, ablation and comparison experiments show that the proposed algorithm has significant advantages in anomaly detection.

Personal belongings information (e.g., backpacks and reticules) and attribute descriptions (e.g., gender and age) provide critical discriminative cues for person re-identification (Re-ID) tasks. However, existing Re-ID algorithms leveraging additional semantic models often fail to accurately recognize personal belongings and suffer from noisy attribute predictions derived from global or local features, as they inadequately exploit attribute correlations. To address these challenges, we propose a novel person re-identification network, ReID-FSAI, which fuses personal belongings information and attribute descriptions from isolated semantic regions. ReID-FSAI integrates personal belongings areas identified through feature clustering with semantic parsing results from an auxiliary semantic model. By treating the generated semantic regions as body labels, our network refines global features into precise semantic features and accurately predicts attribute information from these regions. Furthermore, ReID-FSAI employs a reweighting model to enhance the confidence in specific attributes, improving attribute prediction accuracy. By combining predictions of attributes and personal belongings with global features, our approach significantly improves the representation ability of pedestrians. Experimental evaluations on the Market-1501 and DukeMTMC-reID datasets demonstrate that ReID-FSAI achieves superior performance in both person re-ID and attribute prediction, surpassing state-of-the-art methods.

3D object detection has aroused increasing interest as a crucial component of autonomous driving systems. While recent works have explored various multi-modal fusion methods to enhance accuracy and robustness, fusing multi-view images and high-definition (HD) maps remains uncharted. Inspired by our previous work, we endeavor to introduce HD maps to camera-based detection, prompting the design of a new framework. To address this, we first analyze the function of HD maps in object detection to understand their benefits and the rationale for their fusion. From this analysis, we identify key disparities in view, semantics, and scale, leading to the development of MIM, a framework for HD Maps Incorporated Multi-view 3D object detection. HD maps are enriched in semantics by sampling unlabeled areas and encoding them into map features. Simultaneously, multi-view images are transformed into features in bird’s-eye view (BEV) using the adopted baseline. These features are then fused using attention mechanisms to align scales. Experiments conducted on the nuScenes dataset demonstrate that MIM outperforms camera-based methods. Moreover, an in-depth analysis investigates how HD maps impact object detection regarding each semantic layer. The results underscore the operational intricacies of HD maps in perception, setting the stage for future research. Code is available athttps://github.com/WHU-xjs/MIM-3D-Det.

Haotian Xia, Zhengbang Yang, Yuqing Wang, Rhys Tracy, Yun Zhao, Dongdong Huang, Zezhi Chen, Yan Zhu, Yuan-fang Wang, Weining Shen. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments. In this paper, we study how to address three critical challenges for this task: the cross-modal grounding, the ill-posed feedback, and the generalization problems. First, we propose a novel Reinforced Cross-Modal Matching (RCM) approach that enforces cross-modal grounding both locally and globally via reinforcement learning (RL). Particularly, a matching critic is used to provide an intrinsic reward to encourage global matching between instructions and trajectories, and a reasoning navigator is employed to perform cross-modal grounding in the local visual scene. Evaluation on a VLN benchmark dataset shows that our RCM model significantly outperforms baseline methods by 10 percent on Success Rate weighted by Path Length (SPL) and achieves the state-of-the-art performance. To improve the generalizability of the learned policy, we further introduce a Self-Supervised Imitation Learning (SIL) method to explore and adapt to unseen environments by imitating its own past, good decisions. We demonstrate that SIL can approximate a better and more efficient policy, which tremendously minimizes the success rate performance gap between seen and unseen environments (from 30.7 to 11.7 percent).

Existing Temporal Activity Localization (TAL) methods largely adopt strong supervision for model training, which requires (1) vast amounts of untrimmed videos per each activity category and (2) accurate segment-level boundary annotations (start time and end time) for every instance. This poses a critical restriction to the current methods in practical scenarios where not only segment-level annotations are expensive to obtain, but many activity categories are also rare and unobserved during training. Therefore, Can we learn a TAL model under weak supervision that can localize unseen activity classes? To address this scenario, we define a novel example-based TAL problem called Minimum Effort Temporal Activity Localization (METAL): Given only a few examples, the goal is to find the occurrences of semantically-related segments in an untrimmed video sequence while model training is only supervised by the video-level annotation. Towards this objective, we propose a novel Similarity Pyramid Network (SPN) that adopts the few-shot learning technique of Relation Network and directly encodes hierarchical multi-scale correlations, which we learn by optimizing two complimentary loss functions in an end-to-end manner. We evaluate the SPN on the THUMOS'14 and ActivityNet datasets, of which we rearrange the videos to fit the METAL setup. Results show that our SPN achieves performance superior or competitive to state-of-the-art approaches with stronger supervision.

Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments. In this paper, we study how to address three critical challenges for this task: the cross-modal grounding, the ill-posed feedback, and the generalization problems. First, we propose a novel Reinforced Cross-Modal Matching (RCM) approach that enforces cross-modal grounding both locally and globally via reinforcement learning (RL). Particularly, a matching critic is used to provide an intrinsic reward to encourage global matching between instructions and trajectories, and a reasoning navigator is employed to perform cross-modal grounding in the local visual scene. Evaluation on a VLN benchmark dataset shows that our RCM model significantly outperforms previous methods by 10% on SPL and achieves the new state-of-the-art performance. To improve the generalizability of the learned policy, we further introduce a Self-Supervised Imitation Learning (SIL) method to explore unseen environments by imitating its own past, good decisions. We demonstrate that SIL can approximate a better and more efficient policy, which tremendously minimizes the success rate performance gap between seen and unseen environments (from 30.7% to 11.7%).

This research strives for natural language moment retrieval in long, untrimmed video streams. The problem is not trivial especially when a video contains multiple moments of interests and the language describes complex temporal dependencies, which often happens in real scenarios. We identify two crucial challenges: semantic misalignment and structural misalignment. However, existing approaches treat different moments separately and do not explicitly model complex moment-wise temporal relations. In this paper, we present Moment Alignment Network (MAN), a novel framework that unifies the candidate moment encoding and temporal structural reasoning in a single-shot feed-forward network. MAN naturally assigns candidate moment representations aligned with language semantics over different temporal locations and scales. Most importantly, we propose to explicitly model moment-wise temporal relations as a structured graph and devise an iterative graph adjustment network to jointly learn the best structure in an end-to-end manner. We evaluate the proposed approach on two challenging public benchmarks DiDeMo and Charades-STA, where our MAN significantly outperforms the state-of-the-art by a large margin.

We present a new large-scale multilingual video description dataset, VATEX1, which contains over 41,250 videos and 825, 000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSRVTT dataset [64], VATEX is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on VATEX: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the VATEX dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using VATEXfor other video-and-language research.

Though impressive results have been achieved in visual captioning, the task of generating abstract stories from photo streams is still a little-tapped problem.Different from captions, stories have more expressive language styles and contain many imaginary concepts that do not appear in the images.Thus it poses challenges to behavioral cloning algorithms.Furthermore, due to the limitations of automatic metrics on evaluating story quality, reinforcement learning methods with hand-crafted rewards also face difficulties in gaining an overall performance boost.Therefore, we propose an Adversarial REward Learning (AREL) framework to learn an implicit reward function from human demonstrations, and then optimize policy search with the learned reward function.Though automatic evaluation indicates slight performance boost over state-of-the-art (SOTA) methods in cloning expert behaviors, human evaluation shows that our approach achieves significant improvement in generating more human-like stories than SOTA systems.Code will be made available here 1 .

Video captioning is the task of automatically generating a textual description of the actions in a video. Although previous work (e.g. sequence-to-sequence model) has shown promising results in abstracting a coarse description of a short video, it is still very challenging to caption a video containing multiple fine-grained actions with a detailed description. This paper aims to address the challenge by proposing a novel hierarchical reinforcement learning framework for video captioning, where a high-level Manager module learns to design sub-goals and a low-level Worker module recognizes the primitive actions to fulfill the sub-goal. With this compositional framework to reinforce video captioning at different levels, our approach significantly outperforms all the baseline methods on a newly introduced large-scale dataset for fine-grained video captioning. Furthermore, our non-ensemble model has already achieved the state-of-the-art results on the widely-used MSR-VTT dataset.

This study introduces an algorithm for video denoising based on improved dual‐domain filtering and 3D block matching. The wavelet thresholding based on 3D block matching is introduced to make full use of the correlation of video sequence in order to apply dual‐domain filtering to the video. A layered approach is used that attempts denoising in both a base layer and a detail layer. The result of wavelet thresholding based on 3D block matching is used as a guide image to make the base layer smoother. Shrinkage of short‐time Fourier transform coefficients further decreases the noise in the detail layer. Experimental results show that the authors’ algorithm generates a better base layer and detail layer than the traditional dual‐domain filtering algorithms. The subjective and objective comparisons of different algorithms also prove that the proposed algorithm performs better for video denoising.

Transferring artistic styles onto everyday photographs has become an extremely popular task in both academia and industry. Recently, offline training has replaced online iterative optimization, enabling nearly real-time stylization. When those stylization networks are applied directly to high-resolution images, however, the style of localized regions often appears less similar to the desired artistic style. This is because the transfer process fails to capture small, intricate textures and maintain correct texture scales of the artworks. Here we propose a multimodal convolutional neural network that takes into consideration faithful representations of both color and luminance channels, and performs stylization hierarchically with multiple losses of increasing scales. Compared to state-of-the-art networks, our network can also perform style transfer in nearly real-time by performing much more sophisticated training offline. By properly handling style and texture cues at multiple scales using several modalities, we can transfer not just large-scale, obvious style cues but also subtle, exquisite ones. That is, our scheme can generate results that are visually pleasing and more similar to multiple desired artistic styles with color and texture cues at multiple scales.

Advances in video technology have enabled its wide adoption in the auto industry. Today, many vehicles are equipped with backup, front-looking, and side-looking cameras that allow the driver to easily monitor traffic around the vehicle for enhancing safety. One difficulty with performing automated image analysis using a vehicle's onboard video has to do with the significant lens distortion of these sensors to cover a large field of view around the vehicle. This paper reports our research on proposing a tracking scheme that improves the accuracy and denseness of object tracking in the presence of large lens distortion. The contribution of our research is 4-fold: (1) We evaluated a large collection of state-of-the-art trackers to understand their deficiency when applied to videos with large lens distortion, (2) we showed how to derive useful evaluation metrics from public-domain, real-world driving videos that do not come with ground-truth information on pixel tracking, (3) we identified many enhancement techniques that can potentially help improve the poor performance of current trackers on videos of large lens distortion, and (4) we performed a systematic study to validate the efficacy of these enhancement techniques and proposed a new tracker design that achieved substantial improvement over the state-of-the- art, in terms of both accuracy and density, based on a rigorous precision vs. recall analysis.

Multivariate time series (MTS) datasets broadly exist in numerous fields, including health care, multimedia, finance, and biometrics. How to classify MTS accurately has become a hot research topic since it is an important element in many computer vision and pattern recognition applications. In this paper, we propose a Mahalanobis distance-based dynamic time warping (DTW) measure for MTS classification. The Mahalanobis distance builds an accurate relationship between each variable and its corresponding category. It is utilized to calculate the local distance between vectors in MTS. Then we use DTW to align those MTS which are out of synchronization or with different lengths. After that, how to learn an accurate Mahalanobis distance function becomes another key problem. This paper establishes a LogDet divergence-based metric learning with triplet constraint model which can learn Mahalanobis matrix with high precision and robustness. Furthermore, the proposed method is applied on nine MTS datasets selected from the University of California, Irvine machine learning repository and Robert T. Olszewski's homepage, and the results demonstrate the improved performance of the proposed approach.

Advances in video technology have enabled its wide adoption in the auto industry. Today, many vehicles are equipped with backup, front-looking, and side-looking cameras that allow the driver to easily monitor the traffic around the vehicle for enhanced safety. This paper reports our research on evaluating many existing object tracking techniques, and proposes a new tracker design and its application for 3D environmental mapping in vehicular technology applications. The contribution of our research is 4-fold: (1) We evaluate a large collection of state-of-the-art trackers using multiple criteria relevant to vehicular technology applications, (2) we show how to derive useful evaluation metrics from public-domain, real-world driving videos that do not come with ground-truth information on pixel tracking, (3) we propose a new tracker that is geared specifically for vehicular technology application and show that it achieves tracking accuracy that outperforms SIFT and is on-par with state-of-the-art optical-flow tracking algorithm, which has the best accuracy in our evaluation. Furthermore, we show that our tracker is 600 times more efficient than optical flow and 7 times more efficient than SIFT, and (4) we validated our new tracker design for 3D environmental map building application and showed that the new tracker can obtain comparable results as SIFT but with a significant saving in runtime.

The goal of the project is to design intelligent and robust image-processing and augmented-reality algorithms for driver assistance and enhanced vehicular safety. In particular, the focuses were two-fold: (1) realizing the abilities to identify and localize in a vehicle''s on-board video the sweeping windshield wipers during raining days and (2) designing and implementing an in-painting technique to remove the image of the windshield wipers and replace it with the corresponding pixels (not blocked by the wipers) from an adjacent video frame.

This paper proposes a new vehicle blind spot elimination system which utilizes the on-board videos captured from other vehicles and the host vehicle. Such information is exchanged by WAVE/DSRC devices to achieve collaborative safety. The preceding vehicle which fully or partially blocks the field of view of the host vehicle could be translucentized in the video captured by the host vehicle and the driving environment of the front vehicle could be then visually checked by the host driver.

In this paper, we present a prototype video surveillance system that uses stationary-dynamic (or master-slave) camera assemblies to achieve wide-area surveillance and selective focus-of-attention. We address two critical issues in deploying such camera assemblies in real-world applications: off-line camera calibration and on-line selective focus-ofattention. Our contributions over existing techniques are twofold: (1) in terms of camera calibration, our technique calibrates all degrees-of-freedom (DOFs) of both stationary and dynamic cameras, using a closed-form solution that is both efficient and accurate, and (2) in terms of selective focus-of-attention, our technique correctly handles dynamic changes in the scene and varying object depths. This is a significant improvement over existing techniques that use an expensive and non-adaptable table-look-up process.

Identifying or matching the surface color of a moving object in surveillance video is critical for achieving reliable object-tracking and searching. Traditional color models provide little help, since the surface of an object is usually not flat, the object’s motion can alter the surface’s orientation, and the lighting conditions can vary when the object moves. To tackle this research problem, we conduct extensive data mining on video clips collected under various lighting conditions and distances from several video-cameras. We observe how each of the eleven culture colors can drift in the color space when an object’s surface is in motion. In the color space, we then learn the drift pattern of each culture color for classifying unseen surface colors. Finally, we devise a distance function taking color drift into consideration to perform color identification and matching. Empirical studies show our approach to be very promising: achieving over 95% color-prediction accuracy.

We present EXTENT, an image annotation system that combines the context and content information to annotate images with metadata that cannot be reliably inferred from either the context or the content alone. EXTENT first applies contextual information for restricting the search scope in an image database and reducing the complexity of ensuing content analysis. It can then afford to use more expensive (hence more robust) algorithms for performing content analysis within the restricted database scope. Our experiments show that effectively combining content with context information can infer metadata with high accuracy

With the proliferation of inexpensive cameras, availability of large-capacity disk storage, and the ubiquitous presence of high-speed, broad-band communication networks, it is now economically and technically feasible to construct and deploy multi-camera video surveillance systems. In line with this is the need of intelligent, robust, (semi-)automated video analysis paradigms to assist the operators in scene analysis and event classification. In this paper, we summarize our current research work toward realizing such a multi-camera video surveillance system. Interested readers can visit our web sites http://www.ece.ucsb.edu//sup /spl sim//echang and http://www.cs.ucsb.edu//sup /spl sim//yfwang for more information.

We present a framework for detecting and recognizing human activities for outdoor video surveillance applications. Our research makes the following contributions: For activity detection and tracking, we improve robustness by providing intelligent control and fail-over mechanisms, built on top of low-level motion detection algorithms such as frame differencing and feature correlation. For activity recognition, we propose an efficient representation of human activities that enables recognition of different interaction patterns among a group of people based on simple statistics computed on the tracked trajectories, without building complicated Markov chain, hidden Markov models (HMM), or coupled hidden Markov models (CHMM). We demonstrate our techniques using real-world video data to automatically distinguish normal behaviors from suspicious ones in a parking lot setting, which can aid security surveillance

To answer user queries efficiently, a stream management system must handle continuous, high-volume, possibly noisy, and time-varying data streams. One major research area in stream management seeks to allocate resources (such as network bandwidth and memory) to query plans, either to minimize resource usage under a precision requirement, or to maximize precision of results under resource constraints. To date, many solutions have been proposed; however, most solutions are ad hoc with hard-coded heuristics to generate query plans. In contrast, we perceive stream resource management as fundamentally a filtering problem, in which the objective is to filter out as much data as possible to conserve resources, provided that the precision standards can be met. We select the Kalman Filter as a general and adaptive filtering solution for conserving resources. The Kalman Filter has the ability to adapt to various stream characteristics, sensor noise, and time variance. Furthermore, we realize a significant performance boost by switching from traditional methods of caching static data (which can soon become stale) to our method of caching dynamic procedures that can predict data reliably at the server without the clients' involvement. In this work we focus on minimization of communication overhead for both synthetic and real-world streams. Through examples and empirical studies, we demonstrate the flexibility and effectiveness of using the Kalman Filter as a solution for managing trade-offs between precision of results and resources in satisfying stream queries.

Using cameras for detecting hazardous or suspicious events has spurred new research for security concerns. To make such detection reliable, researchers trust overcome difficulties such as variation in camera capabilities, environmental factors. imbalances of positive and negative training data, and asymmetric costs of misclassifying events of different classes. Following up on the event-detection framework (Wu et al. (2003)) that we have proposed, we present in this paper the framework's two major components: invariant feature extraction and biased statistical inference. We report results of our experiments using the framework for detecting suspicious motion events in a parking lot.

We present a framework for multi-camera video surveillance. The framework consists of three phases: detection, representation, and recognition. The detection phase handles multi-source spatio-temporal data fusion for efficiently and reliably extracting motion trajectories from video. The representation phase summarizes raw trajectory data to construct hierarchical, invariant, and content-rich descriptions of the motion events. Finally, the recognition phase deals with event classification and identification on the data descriptors. Because of space limits, we describe only briefly how we detect and represent events, but we provide in-depth treatment on the third phase: event recognition. For effective recognition, we devise a sequence-alignment kernel function to perform sequence data learning for identifying suspicious events. We show that when the positive training instances (i.e., suspicious events) are significantly outnumbered by the negative training instances (benign events), then SVMs (or any other learning methods) can suffer a high incidence of errors. To remedy this problem, we propose the kernel boundary alignment (KBA) algorithm to work with the sequence-alignment kernel. Through empirical study in a parking-lot surveillance setting, we show that our spatio-temporal fusion scheme and biased sequence-data learning method are highly effective in identifying suspicious events.

MOTIVATION: Many tools have been developed to visualize protein structures. Tools that have been based on Java 3D((TM)) are compatible among different systems and they can be run remotely through web browsers. However, using Java 3D for visualization has some performance issues with it. The primary concerns about molecular visualization tools based on Java 3D are in their being slow in terms of interaction speed and in their inability to load large molecules. This behavior is especially apparent when the number of atoms to be displayed is huge, or when several proteins are to be displayed simultaneously for comparison. RESULTS: In this paper we present techniques for organizing a Java 3D scene graph to tackle these problems. We have developed a protein visualization system based on Java 3D and these techniques. We demonstrate the effectiveness of the proposed method by comparing the visualization component of our system with two other Java 3D based molecular visualization tools. In particular, for van der Waals display mode, with the efficient organization of the scene graph, we could achieve up to eight times improvement in rendering speed and could load molecules three times as large as the previous systems could. AVAILABILITY: EPV is freely available with source code at the following URL: http://www.cs.ucsb.edu/~tcan/fpv/

We propose a new method of retrieving multi-part, articulate objects from images and video. The scheme is particularly well suited for analyzing images and video for objects that can pose differently with possible shape deformation and articulated motion. The scheme involves computing an invariant signature for each segmented region in the image, in a manner that is insensitive to translation, rotation, scale, and shear. Using circular cross-correlation, these signatures can then be efficiently compared with that of user-defined regions of interest. Ambiguities between individual region matches are then resolved through relaxation labeling techniques. A final match is established when a collection of segmented regions conform to the query object, both in terms of local shape description and global structural relation. The scheme thus allows for articulated movement of object parts within the scene. The procedure is easy to implement, yet shows promising results in its ability to isolate interesting regions in images and video, to account for structural and relational constraints among regions, and to integrate both local shape and global structural information for a detailed examination of the scene in a way that is invariant to many visual variations.

Video-endoscopy, a mode of minimally invasive surgery, has proven to be significantly less invasive to the patient. However, it creates a much more complex operation environment that requires the surgeon to operate through a video interface. Visual feedback control and image interpretation can be difficult. Poor visual feedback in video-endoscopy prolongs the operation time, increases the risk to the patient, and drives up the cost of health care. It is a major roadblock in replacing the traditional, highly traumatic open surgical procedures with the much less invasive, more patient friendly video-endoscopy, and in training the surgeons to master this new made of operation. Our research objective is thus to design, code, and validate on real images novel image analysis and rectification algorithms to enhance the visual feedback to the surgeon in video-endoscopy.

We propose invariant formulations that can potentially be combined into a single system. In particular, we describe a framework for computing invariant features which are insensitive to rigid motion, affine transform, changes of parameterization and scene illumination, perspective transform, and view point change. This is unlike most current research on image invariants which concentrates on either geometric or illumination invariants exclusively. The formulations are widely applicable to many popular basis representations, such as wavelets, short-time Fourier analysis, and splines. Exploiting formulations that examine information about shape and color at different resolution levels, the new approach is neither strictly global nor local. It enables a quasi-localized, hierarchical shape analysis which is rarely found in other known invariant techniques, such as global invariants. Furthermore, it does not require estimating high-order derivatives in computing invariants (unlike local invariants), whence is more robust. We provide results of numerous experiments on both synthetic and real data to demonstrate the validity and flexibility of the proposed framework.

This paper addresses 3D shape recovery and motion estimation using a realistic camera model with an aperture and a shutter. The spatial blur and temporal smear effects induced by the camera's finite aperture and shutter speed are used for inferring both the shape and motion of the imaged objects.

This paper presents our research at bringing the state-of-the-art in vision and robotics technologies to enhance the emerging laparoscopic surgical procedure. In particular, a framework utilizing intelligent visual modeling, recognition, and serving capabilities for assisting the surgeon in manoeuvring the scope (camera) in laparoscopy is proposed. The proposed framework integrates top-down model guidance, bottom-up image analysis, and surgeon-in-the-loop monitoring for added patient safety. For the top-down directives, high-level models are used to represent the abdominal anatomy and to encode choreographed scope movement sequences based on the surgeon's knowledge. For the bottom-up analysis, vision algorithms are designed for image analysis, modeling, and matching in a flexible, deformable environment (the abdominal cavity). For reconciling the top-down and bottom-up activities, robot serving mechanisms are realized for executing choreographed scope movements with active vision guidance.

This paper describes a practical and reliable image analysis and tracking algorithm to achieve automated instrument localization and scope manoeuvring in endoscopic surgery.

In this paper, we propose a scheme for 3D model construction by fusing heterogeneous sensor data. The Poposed scheme is intended for use in an environment w K ere multiple, heterogeneous sensors operate asynchronously. Surface de th, orientation, and curvature measurements obtained $0, multiple sensors and vantage points are incorporated to construct a computer description of the ima ed object. The proposed scheme uses Kalman filter surface as the recording data structure. Kalman filter is used to obtain statistically optimal estimates of the imaged surface structure based on potentially noisy sensor measurements. Imaged object is then recorded using hierarchical spline surfaces in a quadtree data structure which are storage efficient and allow adaptive refinement. We show in this paper how these mathematical tools can be used in designing a modeling scheme to fuse heterogeneous sensor data. as t E e sensor data integration tool and hierarchical spline

The authors introduce a technique for 3D surface reconstruction using elastic deformable-models. The model used is an imaginary elastic grid, which is made of membranous, thin-plate-type material. The elastic grid can bent, twisted, compressed, and stretched into any desired 3D shape, which is specified by the shape constraints derived automatically from images of a real 3D object. Shape reconstruction is guided by a set of imaginary springs that enforce the consistency in the position, orientation, and/or curvature measurements of the elastic grid and the desired shape. The dynamics of a surface reconstruction process is regulated by Hamilton's principle or the principle of the least action. Furthermore, a 1D deformable template that borders the elastic grid may be used. This companion boundary template is attracted/repelled by image forces to conform with the silhouette of the imaged object. Implementation results using simple analytic shapes and images of real objects are presented.>

A new technique for computing intrinsic surface properties is presented. Intrinsic surface properties are those properties of a surface that are not affected by the choice of the coordinate system, the position of the viewer relative to the surface, and the particular parametric representation used to describe the imaged surface. Since intrinsic properties are characteristics of a surface, they are ideal for the purposes of representation and recognition. The intrinsic properties of interest are the principal curvatures, the Gaussian curvatures, and the lines of curvature. It is proposed that a structured-light sensing configuration where a grid pattern is projected to encode the imaged surfaces for analysis be adopted. At each stripe junction, the curvatures of the projected stripes on the imaged surface are computed and related to those of the normal sections that share the same tangential directional as the projected curves. The principal curvatures and their directions at the stripe junction under consideration are then recovered using Euler's theorem. Results obtained using both synthetic and real images are presented.>

A technique is introduced for 3-D surface reconstruction and graphic animation using elastic, deformable models. The basic structure used is an imaginary elastic grid, which is made of membranous, thin-plate type material. This elastic grid is bent, twisted, compressed, and stretched into any desirable 3-D shape, or from one flexible state to another. The desired shape can be specified by the shape constraints derived automatically from images of a real 3-D object, or by an analytic surface function. Shape reconstruction is guided by a set of imaginary springs that enforce the consistency in the position, orientation, and/or curvature measurements of the elastic grid and the desired shape. The dynamics of a reconstruction process is regulated by the Hamilton principle or the principle of the least action. Implementation results using simple analytic shapes and images of real free-form objects are presented. The authors believe that their model is widely applicable in many surface reconstruction and graphic animation processes.>

An algorithm is introduced for inferring the structure of three-dimensional (3-D) objects from multiple viewing directions using an integration of active and passive sensing. The structural description of a 3-D object is constructed in two stages: (i) the surface orientation and partial structure are first inferred from a set of single views, and (ii) the visible surface structures inferred from different viewpoints are integrated to complete the description of the 3-D object. In the first stage, an active stripe coding technique is used for recovering visible surface orientation and partial structure. In the second stage, an iterative construction/refinement scheme is used which exploits both passive and active sensing for representing the object surfaces. The final surface structure is recorded in a data structure where the surface contours in a set of parallel planar cross sections are stored. The system construction is inexpensive, and the algorithms introduced are adaptive, versatile, and suitable for applications in dynamic environments.>

In this paper, we introduce a new algorithm for modelling 3-D objects using information gathered from both active and passive sensing mechanisms. Construction of the structural description of a 3-D object is composed of two stages: (i) The visible surface orientation and partial structure are first inferred from a set of single views, and (ii) the partial surface structures inferred from different viewpoints are integrated to complete the 3-D structural description of the object. In the first stage, an active stripe coding technique is used which projects spatially modulated patterns to encode the objects surfaces for analysis. The visible surface orientation is inferred using a constraint satisfaction process based upon the observed orientation of the projected patterns. The visible surface structure is recovered through integrating a dense orientation map. In the second stage, an iterative construction/refinement scheme is used which exploits both passive and active sensing for representing the object surfaces. The bounding volume description of the object is first constructed using multiple occluding contours which are acquired through passive sensing. The bounding volume is then refined using the partial surface structures inferred from active sensing. The final surface structure is recorded in a memory efficient data structure where the surface contours in a set of parallel planar cross sections are recorded. We expect this approach to be widely applicable in the field of robotics, geometric modeling and factory automation.

In this correspondence, algorithms are introduced to infer surface orientation and structure of visible object surfaces using grid coding. We adopt the active lighting technique to spatially ``encode'' the scene for analysis. The observed objects, which can have surfaces of arbitrary shape, are assumed to rest on a plane (base plane) in a scene which is ``encoded'' with light cast through a grid plane. Two orthogonal grid patterns are used, where each pattern is obtained with a set of equally spaced stripes marked on a glass pane. The scene is observed through a camera and the object surface orientation is determined using the projected patterns on the object surface. If the surfaces under consideration obey certain smoothness constraints, a dense orientation map can be obtained through proper interpolation. The surface structure can then be recovered given this dense orientation map. Both planar and curved surfaces can be handled in a uniform manner. The algorithms we propose yield reasonably accurate results and are relatively tolerant to noise, especially when compared to shape-from-shading techniques. In contrast to other grid coding techniques reported which match the grid junctions for depth reconstruction under the stereopsis principle, our techniques use the direction of the projected stripes to infer local surface orientation and do not require any correspondence relationship between either the grid lines or the grid junctions to be specified. The algorithm has the ability to register images and can therefore be embedded in a system which integrates knowledge from multiple views.

A method for matching three-dimensional objects against a library of models from an observed sequence of silhouettes is presented in this correspondence. Based upon the observed silhouettes, the three-dimensional structure of the object is constructed and refined. The principal moments and three primary silhouettes are computed for the constructed three-dimensional objects to represent the aggregate and detailed structure parameters. The adaptive matching technique requires that sufficient silhouettes be added to modify the structure of the unknown object until consistent and steady matching results are obtained. The library for matching is based on three primary silhouettes of the model objects. Experiments conducted show a fast convergence to a consistent result may be achieved provided that a reasonable choice of silhouettes is made.