We introduce LLaMMo (Large Language and Multi-Person Motion Assistant), the first instruction-tuning multimodal framework tailored for multi-human motion analysis. LLaMMo incorporates a novel human-centric and social-temporal learner that models and fuses both intra-person dynamics and inter-person dependencies, yielding robust, context-aware representations of complex group behaviors while maintaining low computational overhead. To support LLaMMo, we construct LLaVerse, a large-scale dataset with fine-grained manual annotations covering diverse multi-person activities spanning daily social interaction and professional team sports. Built on top of LLaVerse, we also propose LLaMI-Bench, a dedicated benchmark for evaluating multi-human behavior understanding across motion and video modalities. Extensive experiments demonstrate that LLaMMo consistently outperforms baselines in understanding multi-person interactions under low-latency settings, with notable gains in both social and sport-specific contexts.

Millimeter-wave radar offers a privacy-preserving and lighting-invariant alternative to RGB sensors for Human Pose Estimation (HPE) task. However, the radar signals are often sparse due to specular reflection, making the extraction of robust features from radar signals highly challenging. To address this, we present milliMamba, a radar-based 2D human pose estimation framework that jointly models spatio-temporal dependencies across both the feature extraction and decoding stages. Specifically, given the high dimensionality of radar inputs, we adopt a Cross-View Fusion Mamba encoder to efficiently extract spatio-temporal features from longer sequences with linear complexity. A Spatio-Temporal-Cross Attention decoder then predicts joint coordinates across multiple frames. Together, this spatio-temporal modeling pipeline enables the model to leverage contextual cues from neighboring frames and joints to infer missing joints caused by specular reflections. To reinforce motion smoothness, we incorporate a velocity loss alongside the standard keypoint loss during training. Experiments on the TransHuPR and HuPR datasets demonstrate that our method achieves significant performance improvements, exceeding the baselines by 11.0 AP and 14.6 AP, respectively, while maintaining reasonable complexity. Code: https://github.com/NYCU-MAPL/milliMamba

The creation of 3D scenes has traditionally been both labor-intensive and costly, requiring designers to meticulously configure 3D assets and environments. Recent advancements in generative AI, including text-to-3D and image-to-3D methods, have dramatically reduced the complexity and cost of this process. However, current techniques for editing complex 3D scenes continue to rely on generally interactive multi-step, 2D-to-3D projection methods and diffusion-based techniques, which often lack precision in control and hamper interactive-rate performance. In this work, we propose 3DSceneEditor, a fully 3D-based paradigm for interactive-rate, precise editing of intricate 3D scenes using Gaussian Splatting. Unlike conventional methods, 3DSceneEditor operates through a streamlined 3D pipeline, enabling direct Gaussian-based manipulation for efficient, high-quality edits based on input prompts. The proposed framework (i) integrates a pre-trained instance segmentation model for semantic labeling; (ii) employs a zero-shot grounding approach with CLIP to align target objects with user prompts; and (iii) applies scene modifications, such as object addition, repositioning, recoloring, replacing, and removal—directly on Gaussians. Extensive experimental results show that 3DSceneEditor surpasses existing state-of-the-art techniques in terms of both editing precision and efficiency, establishing a new benchmark for efficient and interactive 3D scene customization.

Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific vision tasks. Yet, existing methods either employ complex spatial-temporal modules or rely heavily on additional perception models to extract temporal features for video understanding, performing well only on short videos. For long videos, the computational complexity and memory costs associated with long-term temporal connections are significantly increased, posing additional challenges. Leveraging the hierarchical memory structure of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination, we propose MovieChat within a training-free memory consolidation mechanism to overcome these challenges, which transfers dense frames from short-term memory into sparse tokens in long-term memory by temporally merging adjacent frames. We lift pre-trained large multi-modal models for understanding long videos without additional trainable modules, employing a zero-shot approach. Additionally, in our new version, MovieChat+, we design an enhanced training-free vision-question matching-based memory consolidation mechanism to better anchor predictions to relevant visual content. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1 K benchmark with 1 K long video, 2 K temporal grounding labels, and 14 K manual annotations.

In recent years, integrating multimodal information, particularly visual and echo data, has shown great promise for improving depth estimation performance. While existing works demonstrate that combining binaural echo features with image attributes can enhance depth estimation, they often use rudimentary feature alignment and fusion methods, failing to fully exploit the complementary nature of cross-modal information and limiting integration effectiveness. To address these challenges, this paper introduces an innovative multimodal fusion framework. First, the framework incorporates a combination of multi-scale self-attention and cross-attention mechanisms, establishing correlations between features and facilitating cohesive interactions between the visual and echo domains. Furthermore, we propose an incremental feature updating mechanism based on Convolutional Gated Recurrent Units (ConvGRU), which implements cascaded iterative optimization, integrating contextual features with the multi-scale fused features from both echo and image modalities. In each iteration, the framework preserves contextual information from previous steps while employing a multi-level loss function to guide result updates. This approach effectively captures spatial structural information and progressively enhances depth estimation accuracy. Comprehensive experimental evaluations on the Replica, Matterport3D and BatVision (BV1) datasets validate the effectiveness of the proposed method. Comparative analyses with state-of-the-art monocular plus echo methods underscore the superior performance achievable through this novel framework.

The Segment Anything Model 2 (SAM 2) has demonstrated exceptional performance in object segmentation tasks but encounters challenges in visual object tracking, particularly in handling crowded scenes with fast-moving or self-occluding objects. Additionally, its fixed-window memory mechanism indiscriminately retains past frames, leading to error accumulation. This issue results in incorrect memory retention during occlusions, causing the model to condition future predictions on unreliable features and leading to identity switches or drift in crowded scenes. This paper introduces SAMURAI, an enhanced adaptation of SAM 2 that integrates temporal motion cues with a novel motion-aware memory selection strategy. SAMURAI effectively predicts object motion and refines mask selection, achieving robust and precise tracking without requiring retraining or fine-tuning. It demonstrates strong training-free performance across multiple VOT benchmark datasets, underscoring its generalization capability. SAMURAI achieves state-of-the-art performance on LaSOText, GOT-10k, and TrackingNet, while also delivering competitive results on LaSOT, VOT2020-ST, VOT2022-ST, and VOS benchmarks such as SA-V. These results highlight SAMURAI's robustness in complex tracking scenarios and its potential for real-world applications in dynamic environments with an optimized memory selection mechanism. Code and results are available at https://github.com/yangchris11/samurai.

Vision Transformers (ViTs) mark a revolutionary advance in neural networks with their token mixer's powerful global context capability. However, the pairwise token affinity and complex matrix operations limit its deployment on resource-constrained scenarios and real-time applications, such as mobile devices, although considerable efforts have been made in previous works. In this paper, we introduce CAS-ViT: Convolutional Additive Self-attention Vision Transformers, to achieve a balance between efficiency and performance in mobile applications. Firstly, we argue that the capability of token mixers to obtain global contextual information hinges on multiple information interactions, such as spatial and channel domains. Subsequently, we propose Convolutional Additive Token Mixer (CATM) employing underlying spatial and channel attention as novel interaction forms. This module eliminates troublesome complex operations such as matrix multiplication and Softmax. We introduce Convolutional Additive Self-attention(CAS) block hybrid architecture and utilize CATM for each block. And further, we build a family of lightweight networks, which can be easily extended to various downstream tasks. Finally, we evaluate CAS-ViT across a variety of vision tasks, including image classification, object detection, instance segmentation, and semantic segmentation. Our M and T model achieves 83.0%/84.1% top-1 with only 12M/21M parameters on ImageNet-1K. Meanwhile, throughput evaluations on GPUs, ONNX, and iPhones also demonstrate superior results compared to other state-of-the-art backbones. Extensive experiments demonstrate that our approach achieves a better balance of performance, efficient inference and easy-to-deploy. Our code and model are available at: https://github.com/Tianfang-Zhang/CAS-ViT.

Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, Lei Li. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Occlusion remains a major challenge in online Multi-Object Tracking (MOT), where existing multi-stage association methods often rely on detection confidence scores despite their weak correlation with occlusion, leading to frequent errors. We propose DARUMA, a depth-aware MOT framework that prioritizes non-occluded objects using occlusion-aware association by re-prioritizing the matching order and refines association with a depth-weighted cost metric for improved robustness in occluded and depth-varying environments. Additionally, we introduce Generic Observation-Centric Momentum (GOCM), which integrates depth-aware velocity estimation and confidence-weighted historical observations to enhance motion modeling. Our method can integrates into existing MOT frameworks, improving association robustness without additional supervision.Extensive evaluations on DanceTrack demonstrate that DARUMA achieves state-of-the-art performance, particularly in complex, occlusion-heavy scenarios.

This paper presents LLaMo (Large Language and Human Motion Assistant), a multimodal framework for human motion instruction tuning. In contrast to conventional instruction-tuning approaches that convert non-linguistic inputs, such as video or motion sequences, into language tokens, LLaMo retains motion in its native form for instruction tuning. This method preserves motion-specific details that are often diminished in tokenization, thereby improving the model’s ability to interpret complex human behaviors. By processing both video and motion data alongside textual inputs, LLaMo enables a flexible, human-centric analysis. Experimental evaluations across high-complexity domains, including human behaviors and professional activities, indicate that LLaMo effectively captures domain-specific knowledge, enhancing comprehension and prediction in motion-intensive scenarios. We hope LLaMo offers a foundation for future multimodal AI systems with broad applications, from sports analytics to behavioral prediction.

Recent advancements in 3D Large Multi-modal Models (3D-LMMs) have driven significant progress in 3D question answering. However, recent multi-frame Vision-Language Models (VLMs) demonstrate superior performance compared to 3D-LMMs on 3D question answering tasks, largely due to the greater scale and diversity of available 2D image data in contrast to the more limited 3D data. Multi-frame VLMs, although achieving superior performance, suffer from the difficulty of retaining all the detailed visual information in the 3D scene while limiting the number of visual tokens. Common methods such as token pooling, reduce visual token usage but often lead to information loss, impairing the model’s ability to preserve visual details essential for 3D question answering tasks. To address this, we propose voxel-based Dynamic Token Compression (DTC), which combines 3D spatial priors and visual semantics to achieve over 90% reduction in visual tokens usage for current multi-frame VLMs. Our method maintains performance comparable to state-of-the-art models on 3D question answering benchmarks including OpenEQA and ScanQA, demonstrating its effectiveness.

Monocular 3D object detection (Mono3D) holds noteworthy promise for autonomous driving applications owing to the cost-effectiveness and rich visual context of monocular camera sensors. However, depth ambiguity poses a significant challenge, as it requires extracting precise 3D scene geometry from a single image, resulting in suboptimal performance when transferring knowledge from a LiDARbased teacher model to a camera-based student model. To facilitate effective distillation, we introduce Monocular Teaching Assistant Knowledge Distillation (MonoTAKD), which proposes a camera-based teaching assistant (TA) model to transfer robust 3D visual knowledge to the student model, leveraging the smaller feature representation gap. Additionally, we define 3D spatial cues as residual features that capture the differences between the teacher and the TA models. We then leverage these cues to improve the student model's 3D perception capabilities. Experimental results show that our MonoTAKD achieves state-of-the-art performance on the KITTI3D dataset. Furthermore, we evaluate the performance on nuScenes and KITTI raw datasets to demonstrate the generalization of our model to multi-view 3D and unsupervised data settings. Our code is available at https://github.com/hoiliu-0801/MonoTAKD.

Recent advances in generative modeling enable neural networks to generate weights without relying on gradient-based optimization. However, current methods are limited by issues of over-coupling and long-horizon. The former tightly binds weight generation with task-specific objectives, thereby limiting the flexibility of the learned optimizer. The latter leads to inefficiency and low accuracy during inference, caused by the lack of local constraints. In this paper, we propose Lo-Hp, a decoupled two-stage weight generation framework that enhances flexibility through learning various optimization policies. It adopts a hybrid-policy sub-trajectory balance objective, which integrates on-policy and off-policy learning to capture local optimization policies. Theoretically, we demonstrate that learning solely local optimization policies can address the long-horizon issue while enhancing the generation of global optimal weights. In addition, we validate Lo-Hp’s superior accuracy and inference efficiency in tasks that require frequent weight updates, such as transfer learning, few-shot learning, domain generalization, and large language model adaptation.

In the field of multi-object tracking (MOT), traditional methods often rely on the Kalman filter for motion prediction, leveraging its strengths in linear motion scenarios. However, the inherent limitations of these methods become evident when confronted with complex, nonlinear motions and occlusions prevalent in dynamic environments like sports and dance. This paper explores the possibilities of replacing the Kalman filter with a learning-based motion model that effectively enhances tracking accuracy and adaptability beyond the constraints of Kalman filter-based tracker. In this paper, our proposed method MambaMOT and MambaMOT+, demonstrate advanced performance on challenging MOT datasets such as DanceTrack and SportsMOT, showcasing their ability to handle intricate, nonlinear motion patterns and frequent occlusions more effectively than traditional methods.

Meta-learning is a powerful paradigm for tackling few-shot tasks. However, recent studies indicate that models trained with the whole-class training strategy can achieve comparable performance to those trained with meta-learning in few-shot classification tasks. To demonstrate the value of meta-learning, we establish an entropy-limited supervised setting for fair comparisons. Through both theoretical analysis and experimental validation, we establish that meta-learning has a tighter generalization bound compared to whole-class training. We unravel that meta-learning is more efficient with limited entropy and is more robust to label noise and heterogeneous tasks, making it well-suited for unsupervised tasks. Based on these insights, We propose MINO, a meta-learning framework designed to enhance unsupervised performance. MINO utilizes the adaptive clustering algorithm DBSCAN with a dynamic head for unsupervised task construction and a stability-based meta-scaler for robustness against label noise. Extensive experiments confirm its effectiveness in multiple unsupervised few-shot and zero-shot tasks.

Video detailed captioning is a key task which aims to generate comprehensive and coherent textual descriptions of video content, benefiting both video understanding and generation. In this paper, we propose AuroraCap, a video captioner based on a large multimodal model. We follow the simplest architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, we implement the token merging strategy, reducing the number of input visual tokens. Surprisingly, we found that this strategy results in little performance loss. AuroraCap shows superior performance on various video and image captioning benchmarks, for example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include simple descriptions, consisting of a few dozen words, which limits research in this field. Therefore, we develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. In addition, we propose a new LLM-assisted metric VDCscore for bettering evaluation, which adopts a divide-and-conquer strategy to transform long caption evaluation into multiple short question-answer pairs. With the help of human Elo ranking, our experiments show that this benchmark better correlates with human judgments of video detailed captioning quality.

Although significant advancements have been made in Unsupervised Domain Adaptation (UDA), existing methods are typically validated on curated public datasets that do not adequately represent the complexities of real-world applications, such as imbalanced (long-tailed) label distributions and simultaneous domain and label shifts. To bridge this gap, we introduce the Target Distribution Agnostic Adaptation Network (TDAAN), a novel framework designed to facilitate robust adaptation from source to target domains under significant label shifts. Additionally, we present a new domain adaptation dataset, MS-DA, which focuses on marine species and incorporates natural domain and label shifts, a feature sorely lacking in current domain adaptation research. Our experiments show that TDAAN not only significantly outperforms the baseline UDA method but also surpasses the performance of leading UDA methods on the MS-DA dataset. Remarkably, TDAAN maintains competitive performance on standard UDA benchmarks, proving its efficacy even in scenarios with minimal label shifts. These results position TDAAN as a superior method for UDA, particularly in real-world applications characterized by complex and diverse data distributions. Our code is available at https://github.com/SEFSC/FATES-ATI-DomainAdaptationLabelShift.

Token merging has emerged as an effective strategy to accelerate Vision Transformers (ViT) by reducing computational costs. However, existing methods primarily rely on the visual token’s feature similarity for token merging, overlooking the potential of integrating spatial information, which can serve as a reliable criterion for token merging in the early layers of ViT, where the visual tokens only possess weak visual information. In this paper, we propose ToSA, a novel token merging method that combines both semantic and spatial awareness to guide the token merging process. ToSA leverages the depth image as input to generate pseudo spatial tokens, which serve as auxiliary spatial information for the visual token merging process. With the introduced spatial awareness, ToSA achieves a more informed merging strategy that better preserves critical scene structure. Experimental results demonstrate that ToSA outperforms previous token merging methods across multiple benchmarks on visual and embodied question answering while largely reducing the runtime of the ViT, making it an efficient solution for ViT acceleration. The code will be available at: https://github.com/hsiangwei0903/ToSA.

Existing action counting methods typically rely on pixel-based changes within videos, leading to high computational redundancy and low accuracy due to the limited spatial sensitivity. To address these challenges, we introduce MoCount, the first framework that leverages 3D motion representations for counting tasks. MoCount significantly reduces computational overhead and improves counting accuracy, benefiting from the simplicity of motion representation and strong spatial sensitivity. Specifically, we utilize a motion estimator to convert video subjects into 3D motion data. A motion encoder, combined with a Sparse Spatial-Temporal module, is then applied to extract robust human body representations, yielding precise counting results. Extensive experiments on the RepCount and UCFRep datasets show that MoCount achieves state-of-the-art performance, reducing inference latency by approximately 2-3 times compared to existing video counting models. These advantages position MoCount as a leading solution for real-world action counting applications.

Spatial intelligence is fundamental to AI systems that interact with the physical world, particularly in 3D scene generation and spatial comprehension. Current layout generation in 3D scene synthesis remains highly complex, often constrained by predefined datasets and limited dynamic adaptation to changing spatial relationships. In this paper, we propose GraphCanvas3D, a flexible, query-driven framework for controllable 3D scene generation. Unlike traditional methods that require retraining and predefined input masks for modifications, GraphCanvas3D provides a training-free solution supporting the generation of diverse scenes-both indoor and outdoor-through free manipulation of objects and scene elements. Our framework employs hierarchical, graph-driven scene descriptions, representing spatial elements as graph nodes and establishing coherent relationships among objects in 3D environments. The decoupled object representation enables flexible, on-the-fly scene adjustments and dynamic, customizable scene creation. Experimental results and user studies demonstrate that GraphCanvas3D improves usability, adaptability, and generalization across various 3D scene generation tasks, offering a powerful tool for scalable and diverse scene synthesis.

Visual Autoregressive (VAR) modeling has garnered significant attention for its innovative next-scale prediction approach, which yields substantial improvements in efficiency, scalability, and zero-shot generalization. Nevertheless, the coarse-to-fine methodology inherent in VAR results in exponential growth of the KV cache during inference, causing considerable memory consumption and computational redundancy. To address these bottlenecks, we introduce ScaleKV, a novel KV cache compression framework tailored for VAR architectures. ScaleKV leverages two critical observations: varying cache demands across transformer layers and distinct attention patterns at different scales. Based on these insights, ScaleKV categorizes transformer layers into two functional groups: drafters and refiners. Drafters exhibit dispersed attention across multiple scales, thereby requiring greater cache capacity. Conversely, refiners focus attention on the current token map to process local details, consequently necessitating substantially reduced cache capacity. ScaleKV optimizes the multi-scale inference pipeline by identifying scale-specific drafters and refiners, facilitating differentiated cache management tailored to each scale. Evaluation on the state-of-the-art text-to-image VAR model family, Infinity, demonstrates that our approach effectively reduces the required KV cache memory to 10% while preserving pixel-level fidelity.

Throughput prediction in current adaptive bitrate (ABR) schemes often neglects streaming-aware characteristics, such as sequence irregularity and prediction smoothness, resulting in inaccurate predictions and suboptimal performance. To address these challenges, we propose Kairos, an MPC-based ABR scheme that integrates an attention-based throughput predictor with buffer-aware uncertainty control to enhancing both prediction accuracy and adaptability to dynamic network conditions. Specifically, Kairos employs a multi-time attention network (mTAN) to process irregularly sampled streaming data, producing uniformly spaced latent representations. Based on these, we introduce a percentile prediction network to estimate future throughput percentiles, along with a buffer-aware uncertainty control module that selects the optimal percentile based on the current buffer status. As smoothness is another key component of QoE, we incorporate a smoothness regularizer to ensure consistent throughput predictions, thereby facilitating smoother ABR decisions. Our Kairos design integrates sampling irregularity, prediction uncertainty, and smoothness into the throughput prediction, significantly enhancing bitrate decision making within the MPC framework. Extensive trace-driven and real-world experiments demonstrate that Kairos outperforms state-of-the-art ABR schemes, achieving a QoE improvement ranging from 6.42% to 29.45% across diverse network conditions.

Extending image-based Large Multimodal Models (LMMs) to video-based LMMs always requires temporal modeling in the pre-training. However, training the temporal modules gradually erases the knowledge of visual features learned from various image-text-based scenarios, leading to degradation in some downstream tasks. % Adapting pre-trained video-based large language models (LLMs) to downstream fine-grained video understanding tasks always requires modeling on temporal modules. However, training the temporal modules during video pretraining gradually erases the knowledge of visual features learned from various image-text-based scenarios, leading to degradation in some downstream tasks. % Instead of tuning video-based LLMs to downstream tasks, To address this issue, in this paper, we introduce a novel, efficient transfer approach termed MTransLLAMA, which employs transfer learning from pre-trained image LMMs for fine-grained video tasks with only small-scale training sets. Our method enablesfewer trainable parametersand achievesfaster adaptationandhigher accuracythan pre-training video-based LMM models. Specifically, our method adopts early fusion between textual and visual features to capture fine-grained information, reuses spatial attention weights in temporal attentions for cyclical spatial-temporal reasoning, and introduces dynamic attention routing to capture both global and local information in spatial-temporal attentions. Experiments demonstrate that across multiple datasets and tasks, without relying on video pre-training, our model achieves state-of-the-art performance, enabling lightweight and efficient transfer from image-based LMMs to fine-grained video tasks.

Using attention mechanisms in saliency detection networks enables effective feature extraction, and using linear methods can promote proper feature fusion, as verified in numerous existing models. Current networks usually combine depth maps with red-green-blue (RGB) images for salient object detection (SOD). However, fully leveraging depth information complementary to RGB information by accurately highlighting salient objects deserves further study. We combine a gated attention mechanism and a linear fusion method to construct a dual-stream interactive recursive feature-reshaping network (IRFR-Net). The streams for RGB and depth data communicate through a backbone encoder to thoroughly extract complementary information. First, we design a context extraction module (CEM) to obtain low-level depth foreground information. Subsequently, the gated attention fusion module (GAFM) is applied to the RGB depth (RGB-D) information to obtain advantageous structural and spatial fusion features. Then, adjacent depth information is globally integrated to obtain complementary context features. We also introduce a weighted atrous spatial pyramid pooling (WASPP) module to extract the multiscale local information of depth features. Finally, global and local features are fused in a bottom-up scheme to effectively highlight salient objects. Comprehensive experiments on eight representative datasets demonstrate that the proposed IRFR-Net outperforms 11 state-of-the-art (SOTA) RGB-D approaches in various evaluation indicators.

Deep learning has the potential to revolutionize sports performance, with applications ranging from perception and comprehension to decision. This article presents a comprehensive survey of deep learning in sports performance, focusing on three main aspects: algorithms, datasets and virtual environments, and challenges. First, we discuss the hierarchical structure of deep learning algorithms in sports performance which includes perception, comprehension and decision while comparing their strengths and weaknesses. Second, we list widely used existing datasets in sports and highlight their characteristics and limitations. Finally, we summarize current challenges and point out future trends of deep learning in sports. Our survey provides valuable reference material for researchers interested in deep learning in sports applications.

Animal visual perception is an important technique for automatically monitoring animal health, understanding animal behaviors, and assisting animal-related research. However, it is challenging to design a deep learning-based perception model that can freely adapt to different animals across various perception tasks, due to the varying poses of a large diversity of animals, lacking data on rare species, and the semantic inconsistency of different tasks. We introduce UniAP, a novel Universal Animal Perception model that leverages few-shot learning to enable cross-species perception among various visual tasks. Our proposed model takes support images and labels as prompt guidance for a query image. Images and labels are processed through a Transformer-based encoder and a lightweight label encoder, respectively. Then a matching module is designed for aggregating information between prompt guidance and the query image, followed by a multi-head label decoder to generate outputs for various tasks. By capitalizing on the shared visual characteristics among different animals and tasks, UniAP enables the transfer of knowledge from well-studied species to those with limited labeled data or even unseen species. We demonstrate the effectiveness of UniAP through comprehensive experiments in pose estimation, segmentation, and classification tasks on diverse animal species, showcasing its ability to generalize and adapt to new classes with minimal labeled examples.

No-reference point cloud quality assessment (NR-PCQA) aims to automatically evaluate the perceptual quality of distorted point clouds without available reference, which have achieved tremendous improvements due to the utilization of deep neural networks. However, learning-based NR-PCQA methods suffer from the scarcity of labeled data and usually perform suboptimally in terms of generalization. To solve the problem, we propose a novel contrastive pre-training framework tailored for PCQA (CoPA), which enables the pre-trained model to learn quality-aware representations from unlabeled data. To obtain anchors in the representation space, we project point clouds with different distortions into images and randomly mix their local patches to form mixed images with multiple distortions. Utilizing the generated anchors, we constrain the pretraining process via a quality-aware contrastive loss following the philosophy that perceptual quality is closely related to both content and distortion. Furthermore, in the model fine-tuning stage, we propose a semantic-guided multi-view fusion module to effectively integrate the features of projected images from multiple perspectives. Extensive experiments show that our method outperforms the state-of-the-art PCQA methods on popular benchmarks. Further investigations demonstrate that CoPA can also benefit existing learning-based PCQA models.

Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing systems can only handle videos with very few frames. For long videos, the computation complexity, memory cost, and long-term temporal connection impose additional challenges. Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose the MovieChat to overcome these challenges. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video and 14K manual annotations for validation of the effectiveness of our method. The code, models and data can be found in https://reself.github.io/MovieChat.

The confidence scores of 2D pose estimation are widely utilized in various fields, including multi-view 3D human pose estimation, skeleton-based human tracking, human action recognition, human re-identification, etc. Despite widespread use, confidence scores from 2D pose estimation methods are unreliable in indicating the accuracy of estimation results, particularly in occlusion situations, i.e., keypoints with high confidence scores may have low accuracy and vice versa. To address this issue, we propose a new 2D human pose estimation calibration method in this paper. Our method not only enhances the accuracy of 2D pose estimation but also aligns the confidence scores with the quality and visibility of keypoints. We achieve 77.6 mAP in the COCO val dataset, compared with 76.5 mAP of the original HRNet. For key-point visibility prediction, we can reach 89.4% accuracy, 87.6% precision, and 97.0% recall in the COCO val dataset.

Most works in class incremental learning (CIL) assume disjoint sets of classes as tasks. Although a few works deal with overlapped sets of classes, they either assume a balanced data distribution or assume a mild imbalanced distribution. Instead, in this paper, we explore one of the understudied real-world CIL settings where (1) different tasks can share some classes but with new data samples, and (2) the training data of each task follows a long-tail distribution. We call this setting CIL-LT. We hypothesize that previously trained classification heads possess prototype knowledge of seen classes and thus could help learn the new model. Therefore, we propose a method with the multi-expert idea and a dynamic weighting technique to deal with the exacerbated forgetting introduced by the long-tail distribution. Experiments show that the proposed method effectively improves the accuracy in the CIL-LT setup on MNIST, CIFAR10, and CIFAR100. Code and data splits will be released.

Dense object counting or crowd counting has come a long way thanks to the recent development in the vision community. However, indiscernible object counting, which aims to count the number of targets that are blended with respect to their surroundings, has been a challenge. Image-based object counting datasets have been the mainstream of the current publicly available datasets. Therefore, we propose a large-scale dataset called YoutubeFish-35, which contains a total of 35 sequences of high-definition videos with high frame-per-second and more than 159,000 annotated center points across a selected variety of scenes. For bench-marking purposes, we select three mainstream methods for dense object counting and carefully evaluate them on the newly collected dataset. We propose TransVidCount, a new strong baseline that combines density and regression branches along the temporal domain in a unified framework and can effectively tackle indiscernible object counting with state-of-the-art performance on YoutubeFish-35 dataset.

Robust perception is a vital component for ensuring safe autonomous driving. Automotive radar (77 to 81 GHz) offering weather-resilient sensing provides a complementary capability to the vision-or LiDAR-based autonomous driving systems. Raw radio-frequency (RF) radar tensors contain rich spatiotemporal semantics besides 3D location information. Most previous methods take in 3D (Doppler-range-azimuth) RF radar tensors, allowing prediction of an object’s location, heading angle, and size in bird’s-eye-view (BEV). However, they lack the ability to simultaneously infer objects’ size, orientation, and identity in the 3D space. To overcome this limitation, we propose a joint architecture, called CenterRadarNet, designed to facilitate high-resolution representation learning from 4D (Doppler-range-azimuth-elevation) radar data for 3D object detection and re-identification (reID) tasks. Moreover, we build an online tracker utilizing the learned appearance embedding for re-ID. CenterRadarNet achieves the state-of-the-art result on the K-Radar 3D object detection benchmark. In addition, we present the first 3D object-tracking result on the K-Radar dataset. CenterRadarNet shows consistent, robust performance in diverse driving scenarios, emphasizing its wide applicability. Code is available at: https://github.com/Andy-Cheng/CenterRadarNet

Adaptive bitrate (ABR) using conventional codecs cannot further modify the bitrate once a decision has been made, exhibiting limited adaptation capability. This may result in either overly conservative or overly aggressive bitrate selection, which could cause either inefficient utilization of the network bandwidth or frequent re-buffering, respectively. Neural representation for video (NeRV), which embeds the video content into neural network weights, allows video reconstruction with incomplete models. Specifically, the recovery of one frame can be achieved without relying on the decoding of adjacent frames. NeRV has the potential to provide high video reconstruction quality and, more importantly, pave the way for developing more flexible ABR strategies for video transmission. In this work, a new framework, named Evolutional Video streaming Adaptation via Neural representation (EVAN), which can adaptively transmit NeRV models based on soft actor-critic (SAC) reinforcement learning, is proposed. EVAN is trained with a more exploitative strategy and utilizes progressive playback to avoid re-buffering. Experiments showed that EVAN can outperform existing ABRs with 50% reduction in re-buffering and achieve nearly 20% improvement in users’ quality of experience (QoE).

Sensor fusion is crucial for an accurate and robust perception system on autonomous vehicles. Most existing datasets and perception solutions focus on fusing cameras and LiDAR. However, the collaboration between camera and radar is significantly under-exploited. Incorporating rich semantic information from the camera and reliable 3D information from the radar can achieve an efficient, cheap, and portable solution for 3D perception tasks. It can also be robust to different lighting or all-weather driving scenarios due to the capability of mmWave radars. In this paper, we introduce the CRUW3D dataset, including 66K synchronized and well-calibrated camera, radar, and LiDAR frames in various driving scenarios. Unlike other large-scale autonomous driving datasets, our radar data is in the format of radio frequency (RF) tensors that contain not only 3D location information but also spatio-temporal semantic information. This kind of radar format can enable machine learning models to generate more reliable object perception results after interacting and fusing the information or features between the camera and radar. We run several camera- and radar-based baseline methods for 3D object detection and multi-object tracking on our dataset. We hope the CRUW3D dataset will foster radar and multi-modal 3D perception research. CRUW3D is available at https://huggingface.co/datasets/uwipl/CRUW3D

In the domain of autonomous driving, the integration of multi-modal perception techniques based on data from diverse sensors has demonstrated substantial progress. Effectively surpassing the capabilities of state-of-the-art single-modality detectors through sensor fusion remains an active challenge. This work leverages the respective advantages of cameras in perspective view and radars in Bird’s Eye View (BEV) to greatly enhance overall detection and tracking performance. Our approach, Camera-Radar Associated Fusion Tracking Booster (CRAFTBooster) represents a pioneering effort to enhance radar-camera fusion in the tracking stage, contributing to improved 3D MOT accuracy. The superior experimental results on K-Radaar dataset, which exhibit 5-6% on IDF1 tracking performance gain, validate the potential of effective sensor fusion in advancing autonomous driving.

With the rapid advancement of deep learning technology, feature extraction backbones that are effectively trained have found increasing use in various traffic perception tasks, such as vehicle recognition and roadway user detection and classification. However, given the naturally imbalanced distribution of objects in the real world, deep learning networks can inadvertently act as bias amplifiers, leading to unfair detection and classification outcomes. Addressing and quantifying this bias in traffic applications has thus become a pressing challenge. In response, this research introduces the first comprehensive traffic imbalance object recognition dataset tailored for autonomous vehicles, called the Autonomous-vehicle Long-tail Image Dataset (ALIDA). This dataset reflects real-world sample distribution and includes four categories—motorized users, non-motorized users, roadway facilities, and traffic signs—spanning 87 classes and totaling 37,558 images. Our experimental results confirm that these backbones may struggle to accurately recognize less common objects with limited training data, such as children and wheelchair users. To mitigate such biases and improve traffic perception equality, we introduce a DEbiased Traffic Object Recognition (DETOR) scheme. This scheme leverages both few-shot and representation learning techniques. Employing DETOR, the residual neural network achieved a 290% increase in accuracy for recognizing minority classes, such as children, motorcyclists, deer, and bears. This not only enhances the effectiveness but also significantly improves the fairness and scalability of traffic perception using deep neural networks.

Learning-based methods have dominated the 3D human pose estimation (HPE) tasks with significantly better performance in most benchmarks than traditional optimization-based methods. Nonetheless, 3D HPE in the wild is still the biggest challenge for learning-based models, whether with 2D-3D lifting, image-to-3D, or diffusion-based methods, since the trained networks implicitly learn camera intrinsic parameters and domain-based 3D human pose distributions and estimate poses by statistical average. On the other hand, the optimization-based methods estimate results case-by-case, which can predict more diverse and sophisticated human poses in the wild. By combining the advantages of optimization-based and learning-based methods, we propose the Zero-shot Diffusion-based Optimization (ZeDO) pipeline for 3D HPE to solve the problem of cross-domain and in-the-wild 3D HPE. Our multi-hypothesis ZeDO achieves state-of-the-art (SOTA) performance on Human3.6M, with minMPJPE 51.4mm, without training with any 2D-3D or image-3D pairs. Moreover, our single-hypothesis ZeDO achieves SOTA performance on 3DPW dataset with PA-MPJPE 40.3mm on cross-dataset evaluation, which even outperforms learning-based methods trained on 3DPW. Our code is available here: https://github.com/ipl-uw/ZeDO-Release.

Image-Text Matching (ITM) aims to establish the correspondence between images and sentences. ITM is fundamental to various vision and language understanding tasks. However, there are limitations in the way existing ITM benchmarks are constructed. The ITM benchmark collects pairs of images and sentences during construction. Therefore, only samples that are paired at collection are annotated as positive. All other samples are annotated as negative. Many correlations are missed in these samples that are annotated as negative. For example, a sentence matches only one image at the time of collection. Only this image is annotated as positive for the sentence. All other images are annotated as negative. However, these negative images may contain images that correspond to the sentences. These mislabeled samples are calledfalse negatives. Existing ITM models are optimized based on annotations containing mislabels, which can introduce noise during training. In this paper, we propose an ITM framework integrating Language Guidance (LG) for correcting false negatives. A language pre-training model is introduced into the ITM framework to identify false negatives. To correct false negatives, we propose language guidance loss, which adaptively corrects the locations of false negatives in the visual-semantic embedding space. Extensive experiments on two ITM benchmarks show that our method can improve the performance of existing ITM models. To verify the performance of correcting false negatives, we conduct further experiments on ECCV Caption. ECCV Caption is a verified dataset where false negatives in annotations have been corrected. The experimental results show that our method can recall more relevant false negatives. The code is available athttps://github.com/AAA-Zheng/LG_ITM.

Three-dimensional perception in intelligent virtual and augmented reality (VR/AR) and autonomous vehicles (AV) applications is critical and attracting significant attention. The self-supervised monocular depth and ego-motion estimation serves as a more intelligent learning approach that provides the required scene depth and location for 3D perception. However, the existing self-supervised learning methods suffer from scale ambiguity, boundary blur, and imbalanced depth distribution, limiting the practical applications of VR/AR and AV. In this article, we propose a new self-supervised learning framework based on superpixel and normal constraints to address these problems. Specifically, we formulate a novel 3D edge structure consistency loss to alleviate the boundary blur of depth estimation. To address the scale ambiguity of estimated depth and ego-motion, we propose a novel surface normal network for efficient camera height estimation. The surface normal network is composed of a deep fusion module and a full-scale hierarchical feature aggregation module. Meanwhile, to realize the global smoothing and boundary discriminability of the predicted normal map, we introduce a novel fusion loss which is based on the consistency constraints of the normal in edge domains and superpixel regions. Experiments are conducted on several benchmarks, and the results illustrate that the proposed approach outperforms the state-of-the-art methods in depth, ego-motion, and surface normal estimation.

Current convolution neural network (CNN)-based multiperson pose estimators have achieved great progress, however, they pay no or less attention to “hard” samples, such as occluded keypoints, small and nearly invisible keypoints, and ambiguous keypoints. In this article, we explicitly deal with these “hard” samples by proposing a novel multiscale coarse-to-fine human pose estimation network (HM2PN), which includes two sequential subnetworks: CoarseNet and FineNet. CoarseNet conducts a coarse prediction to locate “simple” keypoints like hands and ankles with a multiscale fusion module, which is integrated with bottleneck, resulting in a novel module called multiscale bottleneck. The new module improves the multiscale representation ability of the network in a fine-grained level, while marginally reducing the computation cost because of group convolution. FineNet further infers “hard” keypoints and refines “simple” keypoints simultaneously with a hard keypoint mining loss. Distinct from the previous works, the proposed loss deals with “hard” keypoints differentially and prevents “simple” keypoints from dominating the computed gradients during training. Experiments on the COCO keypoint benchmark show that our approach achieves superior pose estimation performance compared with other state-of-the-art methods. Source code is available for further research:https://github.com/sues-vision/C2F-HumanPoseEstimation.

When applying a pre-trained 2D-to-3D human pose lifting model to a target unseen dataset, large performance degradation is commonly encountered due to domain shift issues. We observe that the degradation is caused by two factors: 1) the large distribution gap over global positions of poses between the source and target datasets due to variant camera parameters and settings, and 2) the deficient diversity of local structures of poses in training. To this end, we combine global adaptation and local generalization in PoseDA, a simple yet effective framework of unsupervised domain adaptation for 3D human pose estimation. Specifically, global adaptation aims to align global positions of poses from the source domain to the target domain with a proposed global position alignment (GPA) module. And local generalization is designed to enhance the diversity of 2D-3D pose mapping with a local pose augmentation (LPA) module. These modules bring significant performance improvement without introducing additional learnable parameters. In addition, we propose local pose augmentation (LPA) to enhance the diversity of 3D poses following an adversarial training scheme consisting of 1) a augmentation generator that generates the parameters of pre-defined pose transformations and 2) an anchor discriminator to ensure the reality and quality of the augmented data. Our approach can be applicable to almost all 2D-3D lifting models. PoseDA achieves 61.3 mm of MPJPE on MPI-INF-3DHP under a cross-dataset evaluation setup, improving upon the previous state-of-the-art method by 10.2%.

Existing studies in Human-Object Interaction (HOI) classification rely on costly human-annotated labels. The goal of this paper is to study a new zero-shot setup to remove the dependency on ground-truth labels. We propose a novel Heterogenous Teacher-Student (HTS) framework and a new loss function. HTS employs a generative pretrained image captioner as the teacher and a contrastive pre-trained classifier as the student. HTS combines the discriminability from generative pre-training and efficiency from contrastive pre-training. To facilitate learning of HOI in this setup, we introduce pseudo-label filtering which aggregates HOI probabilities from multiple regional captions to supervise the student. To enhance the multi-label learning of the student on few-shot classes, we design LogSumExp (LSE)-Sign loss which features a dynamic gradient re-weighting mechanism. Eventually, the student achieves 49.6 mAP on the HICO dataset without using ground truth, becoming a new state-of-the-art method that outperforms supervised approaches. Code is available.

The aim of in-trawl catch monitoring for use in fishing operations is to detect, track and classify fish targets in real-time from video footage. Information gathered could be used to release unwanted bycatch in real-time. However, traditional multi-object tracking (MOT) methods have limitations, as they are developed for tracking vehicles or pedestrians with linear motions and diverse appearances, which are different from the scenarios such as livestock monitoring. Therefore, we propose a novel MOT method, built upon an existing observation-centric tracking algorithm, by adopting a new iterative association step to significantly boost the performance of tracking targets with a uniform appearance. The iterative association module is an extendable component that can be merged into most existing tracking methods. Our method offers improved performance in tracking targets with uniform appearance and outperforms state-of-the-art techniques on our underwater fish datasets as well as the MOT17 dataset, without increasing latency nor sacrificing accuracy as measured by HOTA, MOTA, and IDF1 performance metrics.

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to an unlabeled target domain, mostly through learning a domain invariant feature representation. Currently, the best performing UDA methods use category level domain alignment to capture fine-grained information, resulting in significantly improved performance over global alignment. While successful, category level UDA methods suffer from the unreliable pseudo-labels for target data. In this paper, we propose an UDA approach with teacher-student learning where the teacher network is used to provide more reliable target pseudo-labels for the student during training. Furthermore, we use a progressive mixup augmentation strategy which generates intermediate samples that become increasingly target-dominant as training progresses. Aligning the source and intermediate domains allows the model to gradually transfer fine-grained domain knowledge from the source to the target domain while minimizing the negative impact of noisy target pseudo-labels. This progressive mixup augmented teacher-student (PMATS) training strategy achieves state-of-the-art performance on two public UDA benchmark datasets: Office-31 and Office-Home.

3D dynamic point cloud (DPC) compression relies on mining its temporal context, which faces significant challenges due to DPC's sparsity and non-uniform structure. Existing methods are limited in capturing sufficient temporal dependencies. Therefore, this paper proposes a learning-based DPC compression framework via hierarchical block-matching-based inter-prediction module to compensate and compress the DPC geometry in latent space. Specifically, we propose a hierarchical motion estimation and motion compensation (Hie-ME/MC) framework for flexible inter-prediction, which dynamically selects the granularity of optical flow to encapsulate the motion information accurately. To improve the motion estimation efficiency of the proposed inter-prediction module, we further design a KNN-attention block matching (KABM) network that determines the impact of potential corresponding points based on the geometry and feature correlation. Finally, we compress the residual and the multi-scale optical flow with a fully-factorized deep entropy model. The experiment result on the MPEG-specified Owlii Dynamic Human Dynamic Point Cloud (Owlii) dataset shows that our framework outperforms the previous state-of-the-art methods and the MPEG standard V-PCC v18 in inter-frame low-delay mode.

This paper introduces a novel human pose estimation benchmark, Human Pose with Millimeter Wave Radar (HuPR), that includes synchronized vision and radio signal components. This dataset is created using cross-calibrated mmWave radar sensors and a monocular RGB camera for cross-modality training of radar-based human pose estimation. There are two advantages of using mmWave radar to perform human pose estimation. First, it is robust to dark and low-light conditions. Second, it is not visually perceivable by humans and thus, can be widely applied to applications with privacy concerns, e.g., surveillance systems in patient rooms. In addition to the benchmark, we propose a cross-modality training framework that leverages the ground-truth 2D keypoints representing human body joints for training, which are systematically generated from the pre-trained 2D pose estimation network based on a monocular camera input image, avoiding laborious manual label annotation efforts. The framework consists of a new radar pre-processing method that better extracts the velocity information from radar data, Cross- and Self-Attention Module (CSAM), to fuse multi-scale radar features, and Pose Refinement Graph Convolutional Networks (PRGCN), to refine the predicted keypoint confidence heatmaps. Our intensive experiments on the HuPR benchmark show that the proposed scheme achieves better human pose estimation performance with only radar data, as compared to traditional pre-processing solutions and previous radiofrequency-based methods. Our code is available at here1

To improve the generalization of 3D human pose estimators, many existing deep learning based models focus on adding different augmentations to training poses. However, data augmentation techniques are limited to the "seen" pose combinations and hard to infer poses with rare "unseen" joint positions. To address this problem, we present CameraPose, a weakly-supervised framework for 3D human pose estimation from a single image, which can not only be applied on 2D-3D pose pairs but also on 2D alone annotations. By adding a camera parameter branch, any in-the-wild 2D annotations can be fed into our pipeline to boost the training diversity and the 3D poses can be implicitly learned by reprojecting back to 2D. Moreover, CameraPose introduces a refinement network module with confidence-guided loss to further improve the quality of noisy 2D keypoints extracted by 2D pose estimators. Experimental results demonstrate that the CameraPose brings in clear improvements on cross-scenario datasets. Notably, it outperforms the baseline method by 3mm on the most challenging dataset 3DPW. In addition, by combining our proposed refinement network module with existing 3D pose estimators, their performance can be improved in cross-scenario evaluation.

Deep learning has become a popular method for studying the semantic segmentation of high-resolution remote sensing images (HRRSIs). Existing methods have adopted convolutional neural networks to achieve better segmentation accuracy of HRRSIs, and the success of these models often depends on the model complexity and parameter quantity. However, the deployment of these models on equipment with limited resources is a significant challenge. To solve this problem, a lightweight student network framework—a graph attention guidance network (GAGNet) with knowledge distillation, called GAGNet-S*—is proposed in this study, which distills knowledge from pretrained large teacher network (GAGNet-T) and builds reliable weak labels to optimize untrained student network (GAGNet-S). Inspired by the graph convolution network, this study designs a graph convolution module called the attention-graph decoder, which combines attention mechanisms with graph convolution to optimize image features and improve segmentation accuracy in the semantic segmentation task of HRRSIs. In addition, a dense cross-decoder was designed for multiscale dense fusion, which utilizes rich semantic information in the high-level features to guide and refine the low-level features from the bottom up. Extensive experiments showed that GAGNet-S* (GAGNet-S with knowledge distillation) achieved excellent segmentation performance on two widely used datasets: Potsdam and Vaihingen. The code and models are available at https://github.com/F8AoMn/GAGNet-KD.

In recent years, various neural network architectures for computer vision have been devised, such as the visual transformer and multilayer perceptron (MLP). A transformer based on an attention mechanism can outperform a traditional convolutional neural network. Compared with the convolutional neural network and transformer, the MLP introduces less inductive bias and achieves stronger generalization. In addition, a transformer shows an exponential increase in the inference, training, and debugging times. Considering a wave function representation, we propose the WaveNet architecture that adopts a novel vision task-oriented wavelet-based MLP for feature extraction to perform salient object detection in RGB (red-green-blue)-thermal infrared images. In addition, we apply knowledge distillation to a transformer as an advanced teacher network to acquire rich semantic and geometric information and guide WaveNet learning with this information. Following the shortest-path concept, we adopt the Kullback-Leibler distance as a regularization term for the RGB features to be as similar to the thermal infrared features as possible. The discrete wavelet transform allows for the examination of frequency-domain features in a local time domain and time-domain features in a local frequency domain. We apply this representation ability to perform cross-modality feature fusion. Specifically, we introduce a progressively cascaded sine-cosine module for cross-layer feature fusion and use low-level features to obtain clear boundaries of salient objects through the MLP. Results from extensive experiments indicate that the proposed WaveNet achieves impressive performance on benchmark RGB-thermal infrared datasets. The results and code are publicly available at https://github.com/nowander/WaveNet.

Multi-object tracking (MOT) is an essential task in the computer vision field. With the fast development of deep learning technology in recent years, MOT has achieved great improvement. However, some challenges still remain, such as sensitiveness to occlusion, instability under different lighting conditions, and non-robustness to deformable objects, causing incorrect temporal associations. To address such common challenges in most of the existing trackers, in this paper, a tracklet booster (TBooster) algorithm is proposed to correct the association errors resulting from existing trackers. The correction of the association error from TBooster has two folds: split tracklets on potential ID-change positions and then connect multiple tracklets into one if they are from the same object. To achieve this goal, the TBooster consists of two components,i.e., Splitter and Connector. In Splitter, an architecture with stacked temporal dilated convolution blocks is employed for the splitting position prediction via label smoothing strategy with adaptive Gaussian kernels. In Connector, a multi-head self-attention-based encoder is exploited for the tracklet embedding, which is further used to connect tracklets into full tracks. We conduct sufficient experiments on MOT17 and MOT20 benchmark datasets and achieve promising results. Combined with the proposed tracklet booster, existing trackers can achieve large improvements on the IDF1 score, which shows the effectiveness of the proposed TBooster.

The predefined artificially-balanced training classes in object recognition have limited capability in modeling real-world scenarios where objects are imbalanced-distributed with unknown classes. In this paper, we discuss a promising solution to the Open-set Long-Tailed Recognition (OLTR) task utilizing metric learning. Firstly, we propose a distribution-sensitive loss, which weighs more on the tail classes to decrease the intra-class distance in the feature space. Building upon these concentrated feature clusters, a local-density-based metric is introduced, called Localizing Unfamiliarity Near Acquaintance (LUNA), to measure the novelty of a testing sample. LUNA is flexible with different cluster sizes and is reliable on the cluster boundary by considering neighbors of different properties. Moreover, contrary to most of the existing works that alleviate the open-set detection as a simple binary decision, LUNA is a quantitative measurement with interpretable meanings. Our proposed method exceeds the state-of-the-art algorithm by 4-6% in the closed-set recognition accuracy and 4% in F-measure under the open-set on the public benchmark datasets, including our own newly introduced fine-grained OLTR dataset about marine species (MS-LT), which is the first naturally-distributed OLTR dataset revealing the genuine genetic relationships of the classes.

This paper presents a grounded language-image pretraining (GLIP) model for learning object-level, language-aware, and semantic-rich visual representations. GLIP unifies object detection and phrase grounding for pre-training. The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representations semantic-rich. In our experiments, we pre-train GLIP on 27M grounding data, including 3M human-annotated and 24M web-crawled image-text pairs. The learned representations demonstrate strong zero-shot and few-shot transferability to various object-level recognition tasks. 1) When directly evaluated on COCO and LVIS (without seeing any images in COCO during pre-training), GLIP achieves 49.8 AP and 26.9 AP, respectively, surpassing many supervised baselines.11Supervised baselines on COCO object detection: Faster-RCNN w/ ResNet50 (40.2) or ResNet101 (42.0), and DyHead w/ Swin-Tiny (49.7). 2) After fine-tuned on COCO, GLIP achieves 60.8 AP on val and 61.5 AP on test-dev, surpassing prior SoTA. 3) When transferred to 13 downstream object detection tasks, a 1-shot GLIP rivals with a fully-supervised Dynamic Head. Code will be released at https://github.com/microsoft/GLIP.

Gait recognition, which refers to the recognition or identification of a person based on their body shape and walking styles, derived from video data captured from a distance, is widely used in crime prevention, forensic identification, and social security. However, to the best of our knowledge, most of the existing methods use appearance, posture and temporal feautures without considering a learned temporal attention mechanism for global and local information fusion. In this paper, we propose a novel gait recognition framework, called Temporal Attention and Keypoint-guided Embedding (GaitTAKE), which effectively fuses temporal-attention-based global and local appearance feature and temporal aggregated human pose feature. Experimental results show that our proposed method achieves a new SOTA in gait recognition with rank-1 accuracy of 98.0% (normal), 97.5% (bag) and 92.2% (coat) on the CASIA-B gait dataset; 90.4% accuracy on the OU-MVLP gait dataset.

The goal of electronic monitoring of longline fishing is to visually monitor the fish catching activities on fishing vessels based on cameras, either for regulatory compliance or catch counting. The previous hierarchical classification method demonstrates efficient fish species identification of catches from longline fishing, where fishes are under severe deformation and self-occlusion during the catching process. Although the hierarchical classification mitigates the laborious efforts of human reviews by providing confidence scores in different hierarchical levels, its performance drops dramatically under the class incremental learning (CIL) scenario. A CIL system should be able to learn about more and more classes over time from a stream of data, i.e., only the training data for a small number of classes have to be present at the beginning and new classes can be added progressively. In this work, we introduce a Hierarchical Class Incremental Learning (HCIL) model, which significantly improves the state-of-the-art hierarchical classification methods under the CIL scenario.

The Alberta Infant Motor Scale (AIMS) is a well-known assessment scheme that evaluates the gross motor development of infants by recording the number of specific poses achieved. With the aid of the image-based pose recognition model, the AIMS evaluation procedure can be shortened and automated, providing early diagnosis or indicator of potential developmental disorder. Due to limited public infant-related datasets, many works use the SMIL-based method to generate synthetic infant images for training. However, this domain mismatch between real and synthetic training samples often leads to performance degradation during inference. In this paper, we present a CNN-based model which takes any infant image as input and predicts the coarse and fine-level pose labels. The model consists of an image branch and a pose branch, which respectively generates the coarse-level logits facilitated by the unsupervised domain adaptation and the 3D keypoints using the HRNet with SMPLify optimization. Then the outputs of these branches will be sent into the hierarchical pose recognition module to estimate the fine-level pose labels. We also collect and label a new AIMS dataset, which co—tains 750 real and 4000 synthetic infants images with AIMS pose labels. Our experimental results show that the proposed method can significantly align the distribution of synthetic and real-world datasets, thus achieving accurate performance on fine-grained infant pose recognition.

Visual Semantic Embedding (VSE) is a dominant method for cross-modal vision-language retrieval. Its purpose is to learn an embedding space so that visual data can be embedded in a position close to the corresponding text description. However, there are large intra-class variations in the vision-language data. For example, multiple texts describing the same image may be described from different views, and the descriptions of different views are often dissimilar. The mainstream VSE method embeds samples from the same class in similar positions, which will suppress intra-class variations and lead to inferior generalization performance. This paper proposes a Multi-View Visual Semantic Embedding (MV-VSE) framework, which learns multiple embeddings for one visual data and explicitly models intra-class variations. To optimize MV-VSE, a multi-view upper bound loss is proposed, and the multi-view embeddings are jointly optimized while retaining intra-class variations. MV-VSE is plug-and-play and can be applied to various VSE models and loss functions without excessively increasing model complexity. Experimental results on the Flickr30K and MS-COCO datasets demonstrate the superior performance of our framework.

We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e.g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e.g., VQA, image captioning). GLIPv2 elegantly unifies localization pre-training and Vision-Language Pre-training (VLP) with three pre-training tasks: phrase grounding as a VL reformulation of the detection task, region-word contrastive learning as a novel region-word level contrastive learning task, and the masked language modeling. This unification not only simplifies the previous multi-stage VLP procedure but also achieves mutual benefits between localization and understanding tasks. Experimental results show that a single GLIPv2 model (all model weights are shared) achieves near SoTA performance on various localization and understanding tasks. The model also shows (1) strong zero-shot and few-shot adaption performance on open-vocabulary object detection tasks and (2) superior grounding capability on VL understanding tasks.

Depth estimation from a camera is an important task for 3D perception. Recently, without using the labeled ground truth of depth map, a self-supervised deep learning network can use relative pose to synthesize the target image from the reference image, and the photometric error between synthesized reference image and real one is used as self-supervisory signal. In this paper, we propose a novel self-supervised depth estimation network, which takes advantage of the quadtree constraint to optimize the depth estimation network. Based on the quadtree constraint, the photometric loss and depth loss of quadtree are proposed. In order to solve the problem that multiple depth values in repeated structures and uniform texture regions can cause relatively low photometric loss, we use quadtree-based photometric loss, which calculates the averaged photometric loss in quadtree blocks instead of the pixel-wise loss. For the problem of imbalanced depth distribution, we use quadtree depth loss, which constrains the depth inconsistency within quadtree blocks. The depth estimation network is composed of deep fusion module and cross-layer feature fusion module, which can better extract the feature information of RGB image and sparse keypoints depths, and makes full use of the detail information of the shallow feature map and the semantic information of the deep feature map to enrich the feature information extraction. Experimental results demonstrate that our method outperforms the state-of-the-art approaches of depth estimation.

Under ideal environmental conditions, RGB-based deep convolutional neural networks can achieve high performance for salient object detection (SOD). In scenes with cluttered backgrounds and many objects, depth maps have been combined with RGB images to better distinguish spatial positions and structures during SOD, achieving high accuracy. However, under low-light and uneven lighting conditions, RGB and depth information may be insufficient for detection. Thermal images are insensitive to lighting and weather conditions, being able to capture important objects even during nighttime. By combining thermal images and RGB images, we propose an effective and consistent feature fusion network (ECFFNet) for RGB-T SOD. In ECFFNet, an effective cross-modality fusion module fully fuses features of corresponding sizes from the RGB and thermal modalities. Then, a bilateral reversal fusion module performs bilateral fusion of foreground and background information, enabling the full extraction of salient object boundaries. Finally, a multilevel consistent fusion module combines features across different levels to obtain complementary information. Comprehensive experiments on three RGB-T SOD datasets show that the proposed ECFFNet outperforms 12 state-of-the-art methods under different evaluation indicators.

Scene parsing of high spatial resolution (HSR) remote sensing images has achieved notable progress in recent years by the adoption of convolutional neural networks. However, for scene parsing of multimodal remote sensing images, effectively integrating complementary information remains challenging. For instance, the decrease in feature map resolution through a neural network causes loss of spatial information, likely leading to blurred object boundaries and misclassification of small objects. In addition, object scales on a remote sensing image vary substantially, undermining the parsing performance. To solve these problems, we propose an end-to-end common extraction and gate fusion network (CEGFNet) to capture both high-level semantic features and low-level spatial details for scene parsing of remote sensing images. Specifically, we introduce a gate fusion module to extract complementary features from spectral data and digital surface model data. A gate mechanism removes redundant features in the data stream and extracts complementary features that improve multimodal feature fusion. In addition, a global context module and a multilayer aggregation decoder handle scale variations between objects and the loss of spatial details due to downsampling, respectively. The proposed CEGFNet was quantitatively evaluated on benchmark scene parsing datasets containing HSR remote sensing images, and it achieved state-of-the-art performance.

Compared with traditional handcrafted features, deep learning has greatly improved the performance of scene parsing. However, it remains challenging under various environmental conditions caused by imaging limitations. Thermal imaging cameras have several advantages over cameras for the visible spectrum, such as operation in total darkness, robustness to shadow effects, insensitivity to illumination variations, and strong ability to penetrate smog and haze. These advantages of thermal imaging cameras make them ideal for the scene parsing of semantic objects in daytime and nighttime. In this paper, we propose a novel multiscale feature fusion and enhancement network (MFFENet) for accurate parsing of RGB–thermal urban road scenes even when the quality of the available RGB data is compromised. The proposed MFFENet consists of two encoders, a feature fusion layer, and a multi-label supervision layer. We concatenate the multi-scale features with the features that contain global semantic information. Furthermore, we explore the cross-modal fusion of RGB and thermal features at multiple stages, rather than fusing them once at the low or high stage. Then, we propose a spatial attention mechanism module that provides a higher weight to (focuses more on) the foreground area, allowing MFFENet to emphasize foreground objects. Finally, multi-label supervision is introduced to optimize parameters of the proposed MFFENet. Experimental results confirm that the proposed MFFENet outperforms similar high-performing methods.

Monocular absolute 3D fish pose estimation allows for efficient fish length measurement in the longline fisheries, where fishes are under severe deformation during the catching process. This task is challenging since it requires locating absolute 3D fish keypoints based on a short monocular video clip. Unlike related works, which either require expensive 3D ground-truth data and/or multiple-view images to provide depth information, or are limited to rigid objects, we propose a novel frame-based method to estimate the absolute 3D fish pose and fish length from a single-view 2D segmentation mask. We first introduce a relative 3D fish template. By minimizing an objective function, our method systematically estimates the relative 3D pose of the target fish and fish 2D keypoints in the image. Finally, with a closed-form solution, the relative 3D fish pose can help locate absolute 3D keypoints, resulting in the frame-based absolute fish length measurement, which is further refined based on the statistical temporal inference for the optimal fish length measurement from the video clip. Our experiments show that this method can accurately estimate the absolute 3D fish pose and further measure the absolute length, even outperforming the state-of-the-art multi-view method.

Knowing the 3D locations of the surrounding vehicles is of vital importance in autonomous driving scenarios. It can be pretty challenging to make an accurate estimation from a monocular moving camera. In this paper, we present an effective vehicle 3D localization method, that utilizes 2D key-points predicted from a trained CNN to model the vehicles’ structure, from which the ground points are further inferred. An adaptive ground plane estimation method is exploited under the monocular camera for 3D geometric back-projection. Benefiting from tracking, we also take into account temporal information of the same object to ensure the trajectory consistency. Viewpoint and size knowledge are also considered for refinement. The evaluation on the KITTI benchmark for on-road vehicles shows the effectiveness of our proposed approach with promising 3D localization results.

Based on Alberta Infant Motor Scale (AIMS), a questionnaire that tracks an infant’s motor function, an infant’s mental development can be evaluated by recording poses a baby can achieve. Therefore, it is meaningful to propose a systematic image-based pose classifier to classify infant actions based on AIMS to provide early diagnosis of a potential develop-mental disorder such as Autism. This paper presents a hierarchical pose classifier, given a baby image frame that com-bines the benefits of 3D human pose estimation and scene context information. Due to privacy policies, we cannot collect enough real infant images/videos for experiments. In-stead, we generate synthetic baby images with the help of the Skinned Multi-Infant Linear (SMIL) model. Images are first fed into a ResNet-50 for coarse-level pose classification. A stacked hourglass CNN and a hierarchical 3D pose estimation scheme are used for 2D/3D pose estimation. Finally, an innovative Hierarchical Infant Pose Classifier (HIPC) takes the estimated 3D keypoints and coarse-level pose classification confidence scores to give the fine-level baby pose classification results. Our experimental results show that our hierarchical pose classifier achieves accurate and stable performance on infant pose recognition.

One-stage long-tailed recognition methods improve the overall performance in a "seesaw" manner, i.e., either sacrifice the head’s accuracy for better tail classification or elevate the head’s accuracy even higher but ignore the tail. Existing algorithms bypass such trade-off by a multi-stage training process: pre-training on imbalanced set and fine-tuning on balanced set. Though achieving promising performance, not only are they sensitive to the generalizability of the pre-trained model, but also not easily integrated into other computer vision tasks like detection and segmentation, where pre-training of classifiers solely is not applicable. In this paper, we propose a one-stage long-tailed recognition scheme, ally complementary experts (ACE), where the expert is the most knowledgeable specialist in a sub-set that dominates its training, and is complementary to other experts in the less-seen categories without being disturbed by what it has never seen. We design a distribution-adaptive optimizer to adjust the learning pace of each expert to avoid over-fitting. Without special bells and whistles, the vanilla ACE outperforms the current one-stage SOTA method by 3 ~ 10% on CIFAR10-LT, CIFAR100-LT, ImageNet-LT and iNaturalist datasets. It is also shown to be the first one to break the "seesaw" trade-off by improving the accuracy of the majority and minority categories simultaneously in only one stage. Code and trained models are at https://github.com/jrcai/ACE.

Vehicle tracking is an essential task in the multi-object tracking (MOT) field. A distinct characteristic in vehicle tracking is that the trajectories of vehicles are fairly smooth in both the world coordinate and the image coordinate. Hence, models that capture motion consistencies are of high necessity. However, tracking with the standalone motion-based trackers is quite challenging because targets could get lost easily due to limited information, detection error and occlusion. Leveraging appearance information to assist object re-identification could resolve this challenge to some extent. However, doing so requires extra computation while appearance information is sensitive to occlusion as well. In this paper, we try to explore the significance of motion patterns for vehicle tracking without appearance information. We propose a novel approach that tackles the association issue for long-term tracking with the exclusive fully-exploited motion information. We address the tracklet embedding issue with the proposed reconstruct-to-embed strategy based on deep graph convolutional neural networks (GCN). Comprehensive experiments on the KITTI-car tracking dataset and UA-Detrac dataset show that the proposed method, though without appearance information, could achieve competitive performance with the state-of-the-art (SOTA) trackers. The source code will be available at https://github.com/GaoangW/LGMTracker.

The Radar Object Detection 2021 (ROD2021) Challenge, held in the ACM International Conference on Multimedia Retrieval (ICMR) 2021, has been introduced to detect and classify objects purely using an FMCW radar for autonomous driving applications. As a robust sensor to all-weather conditions, radar has rich information hidden in the radio frequencies, which can potentially achieve object detection and classification. This insight will provide a new object perception solution for an autonomous vehicle even in adverse driving scenarios. The ROD2021 Challenge is the first public benchmark focusing on this topic, which attracts great attention and participation. There are more than 260 participants among 37 teams from more than 10 countries with different academic and industrial affiliations, contributing about 300 submissions in the first phase and 400 submissions in the second phase. The final performance is evaluated by average precision (AP). Results add strong value and a better understanding of the radar object detection task for the autonomous vehicle community.

The 2021 embedded deep learning object detection model compression competition for traffic in Asian countries held in IEEE ICMR2021 Grand Challenges focuses on the object detection technologies in autonomous driving scenarios. The competition aims to detect objects in traffic with low complexity and small model size in the Asia countries (e.g., Taiwan), which contains several harsh driving environments. The target detected objects include vehicles, pedestrians, bicycles and crowded scooters. There are 89,002 annotated images provided for model training and 1,000 images for validation. Additional 5,400 testing images are used in the contest evaluation process, in which 2,700 of them are used in the qualification stage competition, and the rest are used in the final stage competition. There are in total 308 registered teams joining this competition this year, and the top 15 teams with the highest detection accuracy entering the final stage competition, from which 9 teams submitted the final results. The overall best model belongs to team "as798792", followed by team "Deep Learner" and team "UCBH." Two special awards of best accuracy award best and bicycle detections go to the same team "as798792," and the other special award of scooter detection goes to team "abcda."

Radar is usually more robust than the camera in severe driving scenarios, e.g., weak/strong lighting and bad weather. However, unlike RGB images captured by a camera, the semantic information from the radar signals is noticeably difficult to extract. In this paper, we propose a deep radar object detection network (RODNet), to effectively detect objects purely from the carefully processed radar frequency data in the format of range-azimuth frequency heatmaps (RAMaps). Three different 3D autoencoder based architectures are introduced to predict object confidence distribution from each snippet of the input RAMaps. The final detection results are then calculated using our post-processing method, called location-based non-maximum suppression (L-NMS). Instead of using burdensome human-labeled ground truth, we train the RODNet using the annotations generated automatically by a novel 3D localization method using a camera-radar fusion (CRF) strategy. To train and evaluate our method, we build a new dataset - CRUW, containing synchronized videos and RAMaps in various driving scenarios. After intensive experiments, our RODNet shows favorable object detection performance without the presence of the camera.

Camera calibration is a crucial prerequisite in many applications of computer vision. In this paper, a new geometry-based camera calibration technique is proposed, which resolves two main issues associated with the widely used Zhang's method: (i) the lack of guidelines to avoid outliers in the computation and (ii) the assumption of fixed camera focal length. The proposed approach is based on the closed-form solution of principal lines with their intersection being the principal point while each principal line can concisely represent relative orientation/position (up to one degree of freedom for both) between a special pair of coordinate systems of image plane and calibration pattern. With such analytically tractable image features, computations associated with the calibration are greatly simplified, while the guidelines in (i) can be established intuitively. Experimental results for synthetic and real data show that the proposed approach does compare favorably with Zhang's method, in terms of correctness, robustness, and flexibility, and addresses issues (i) and (ii) satisfactorily.

In this paper, we propose a novel framework for multi-target multi-camera tracking (MTMCT) of vehicles based on metadata-aided re-identification (MA-ReID) and the trajectory-based camera link model (TCLM). Given a video sequence and the corresponding frame-by-frame vehicle detections, we first address the isolated tracklets issue from single camera tracking (SCT) by the proposed traffic-aware single-camera tracking (TSCT). Then, after automatically constructing the TCLM, we solve MTMCT by the MA-ReID. The TCLM is generated from camera topological configuration to obtain the spatial and temporal information to improve the performance of MTMCT by reducing the candidate search of ReID. We also use the temporal attention model to create more discriminative embeddings of trajectories from each camera to achieve robust distance measures for vehicle ReID. Moreover, we train a metadata classifier for MTMCT to obtain the metadata feature, which is concatenated with the temporal attention based embeddings. Finally, the TCLM and hierarchical clustering are jointly applied for global ID assignment. The proposed method is evaluated on the CityFlow dataset, achieving IDF1 76.77%, which outperforms the state-of-the-art MTMCT methods.

Semantic segmentation is a fundamental task in computer vision, and it has various applications in fields such as robotic sensing, video surveillance, and autonomous driving. A major research topic in urban road semantic segmentation is the proper integration and use of cross-modal information for fusion. Here, we attempt to leverage inherent multimodal information and acquire graded features to develop a novel multilabel-learning network for RGB-thermal urban scene semantic segmentation. Specifically, we propose a strategy for graded-feature extraction to split multilevel features into junior, intermediate, and senior levels. Then, we integrate RGB and thermal modalities with two distinct fusion modules, namely a shallow feature fusion module and deep feature fusion module for junior and senior features. Finally, we use multilabel supervision to optimize the network in terms of semantic, binary, and boundary characteristics. Experimental results confirm that the proposed architecture, the graded-feature multilabel-learning network, outperforms state-of-the-art methods for urban scene semantic segmentation, and it can be generalized to depth data.

In recent years, the detection of distinctive objects in stereoscopic 3D images has drawn increasing attention. Unlike 2D salient object detection, salient object detection in stereoscopic 3D images is highly challenging. Hence, we propose a novel Deep Convolutional Residual Autoencoder (DCRA) for end-to-end salient object detection in stereoscopic 3D images. The core trainable architecture of the salient object detection model employs raw stereoscopic 3D images as the inputs and their corresponding ground truth saliency masks as the labels. A convolutional residual module is applied to both the encoder and the decoder as a basic building block in the DCRA, and long-range skip connections are employed to bypass the equal-sized feature maps between the encoder and the decoder. To explore the complex relationships and exploit the complementarity between RGB (photometric) and depth (geometric) information, multiple feature map fusion modules are constructed. These modules integrate texture and structure information between the RGB and depth branches of the encoder and fuse their features over several multiscale layers. Finally, to efficiently optimize DCRA parameters, a supervision pyramid based on boundary loss and background prior loss is adopted, which employs supervised learning over the multiscale layers in the decoder to prevent vanishing gradients and accelerate the training at the fusion stage. We compare the proposed DCRA with state-of-the-art methods on two challenging benchmark datasets. The results of these experiments demonstrate that our proposed DCRA performs favorably against the comparison models.

Traffic sign detection is an important task in assisted safety and autonomous driving. It is important to continuously detect the traffic signs emerged on the road. Currently, most object detection methods make independent detections based on single images. When we apply these methods directly to a video clip to detect traffic signs without taking into account temporal correlations among adjacent frames, missed detections or incorrect detections can frequently occur due to motion blur, size change, partial occlusion, and/or bad pose. In this paper, we fully exploit the temporal consistency of traffic sign detection in videos. More specifically, we incorporate information of adjacent frames with high confidence scores to enhance the discovery of potential objects in the missed or incorrect detected frames by “recovering” the missed RoI proposals or by “improving” the incorrect RoI proposals with low confidence scores. Our method can be regarded as a “detection-by-tracking” strategy, which results in a more robust detection performance in videos.

3D human pose estimation (HPE) is crucial in many fields, such as human behavior analysis, augmented reality/virtual reality (AR/VR) applications, and self-driving industry. Videos that contain multiple potentially occluded people captured from freely moving monocular cameras are very common in realworld scenarios, while 3D HPE for such scenarios is quite challenging, partially because there is a lack of such data with accurate 3D ground truth labels in existing datasets. In this paper, we propose a temporal regression network with a gated convolution module to transform 2D joints to 3D and recover the missing occluded joints in the meantime. A simple yet effective localization approach is further conducted to transform the normalized pose to the global trajectory. To verify the effectiveness of our approach, we also collect a new moving camera multi-human (MMHuman) dataset that includes multiple people with heavy occlusion captured by moving cameras. The 3D ground truth joints are provided by accurate motion capture (MoCap) system. From the experiments on static-camera based Human3.6M data and our own collected moving-camera based data, we show that our proposed method outperforms most state-of-the-art 2D-to-3D pose estimation methods, especially for the scenarios with heavy occlusions.

In this paper, we propose a novel fusion framework based on a dense network with channel and spatial attention (CSpA-DN) for PET and MR images. In our approach, an encoder composed of the densely connected neural network is constructed to extract features from source images, and a decoder network is leveraged to yield the fused image from these features. Simultaneously, a self-attention mechanism is introduced in the encoder and decoder to further integrate local features along with their global dependencies adaptively. The extracted feature of each spatial position is synthesized by a weighted summation of those features at the same row and column with this position via a spatial attention module. Meanwhile, the interdependent relationship of all feature maps is integrated by a channel attention module. The summation of the outputs of these two attention modules is fed into the decoder and the fused image is generated. Experimental results illustrate the superiorities of our proposed CSpA-DN model compared with state-of-the-art methods in PET and MR images fusion according to both visual perception and objective assessment.

Multi-target multi-camera tracking (MTMCT), i.e., tracking multiple targets across multiple cameras, is a crucial technique for smart city applications. In this paper, we propose an effective and reliable MTMCT framework for vehicles, which consists of a traffic-aware single camera tracking (TSCT) algorithm, a trajectory-based camera link model (CLM) for vehicle re-identification (ReID), and a hierarchical clustering algorithm to obtain the cross camera vehicle trajectories. First, the TSCT, which jointly considers vehicle appearance, geometric features, and some common traffic scenarios, is proposed to track the vehicles in each camera separately. Second, the trajectory-based CLM is adopted to facilitate the relationship between each pair of adjacently connected cameras and add spatio-temporal constraints for the subsequent vehicle ReID with temporal attention. Third, the hierarchical clustering algorithm is used to merge the vehicle trajectories among all the cameras to obtain the final MTMCT results. Our proposed MTMCT is evaluated on the CityFlow dataset and achieves a new state-of-the-art performance with IDF1 of 74.93%.

Despite the increasing need of analyzing human poses on the street and in the wild, multi-person 3D pose estimation using monocular static or moving camera in real-world scenarios remains a challenge, either requiring large-scale training data or high computation complexity due to the high degrees of freedom in 3D human poses. We propose a novel scheme to effectively track and hierarchically estimate 3D human poses in natural videos in an efficient fashion. Without the need of using labelled 3D training data, we formulate torso estimation as a Perspective-N-Point (PNP) problem, and limb pose estimation as an optimization problem, and hierarchically structure the high dimensional poses to efficiently address the challenge. Experiments show good performance and high efficiency of multi-person 3D pose estimation on real-world videos, including street scenarios and various human daily activities from fixed and moving cameras, resulting in great new opportunities to understand and predict human behaviors.

In recent years, supervised deep learning methods have shown a great promise in dense depth estimation. However, massive high-quality training data are expensive and impractical to acquire. Alternatively, self-supervised learning-based depth estimators can learn the latent transformation from monocular or binocular video sequences by minimizing the photometric warp error between consecutive frames, but they suffer from the scale ambiguity problem or have difficulty in estimating precise pose changes between frames. In this paper, we propose a joint self-supervised deep learning pipeline for depth and ego-motion estimation by employing the advantages of adversarial learning and joint optimization with spatial-temporal geometrical constraints. The stereo reconstruction error provides the spatial geometric constraint to estimate the absolute scale depth. Meanwhile, the depth map with an absolute scale and a pre-trained pose network serves as a good starting point for direct visual odometry (DVO). DVO optimization based on spatial geometric constraints can result in a fine-grained ego-motion estimation with the additional backpropagation signals provided to the depth estimation network. Finally, the spatial and temporal domain-based reconstructed views are concatenated, and the iterative coupling optimization process is implemented in combination with the adversarial learning for accurate depth and precise ego-motion estimation. The experimental results show superior performance compared with state-of-the-art methods for monocular depth and ego-motion estimation on the KITTI dataset and a great generalization ability of the proposed approach.

Urban traffic optimization using traffic cameras as sensors is driving the need to advance state-of-the-art multi-target multi-camera (MTMC) tracking. This work introduces CityFlow, a city-scale traffic camera dataset consisting of more than 3 hours of synchronized HD videos from 40 cameras across 10 intersections, with the longest distance between two simultaneous cameras being 2.5 km. To the best of our knowledge, CityFlow is the largest-scale dataset in terms of spatial coverage and the number of cameras/videos in an urban environment. The dataset contains more than 200K annotated bounding boxes covering a wide range of scenes, viewing angles, vehicle models, and urban traffic flow conditions. Camera geometry and calibration information are provided to aid spatio-temporal analysis. In addition, a subset of the benchmark is made available for the task of image-based vehicle re-identification (ReID). We conducted an extensive experimental evaluation of baselines/state-of-the-art approaches in MTMC tracking, multi-target single-camera (MTSC) tracking, object detection, and image-based ReID on this dataset, analyzing the impact of different network architectures, loss functions, spatio-temporal models and their combinations on task effectiveness. An evaluation server is launched with the release of our benchmark at the 2019 AI City Challenge (https://www.aicitychallenge.org/) that allows researchers to compare the performance of their newest techniques. We expect this dataset to catalyze research in this field, propel the state-of-the-art forward, and lead to deployed traffic optimization(s) in the real world.

In the paper, we study the problem of inductive embedding learning on attributed heterogeneous networks, and propose a Multi-task sequence-to-sequence learning based Inductive Network Embedding framework (MINE) capturing the attribute similarity, network proximity, and partial label information simultaneously. In particular, MINE trains an encoder function that aggregates information from a node's long-range scope of contexts, with the node attribute sequences generated by the proposed type-guided heterogeneous random walk as inputs. We present an one-to-many multi-task sequence-to-sequence model where the encoder is shared between two related tasks: an unsupervised node identity sequence generation task to learn context-aware embeddings, and a semi-supervised label prediction task to learn semantics-rich embeddings. Extensive experiments on real-world datasets demonstrate that the proposed method significantly outperforms several state-of-the-art methods.

Recognizing fish species captured live on wild sea surface in videos is a challenging task due to the deformation of fish shape, self-occlusion of body parts and similar texture between different fish classes. To address these issues, we propose a fine-grained image classification method based on a deep convolution neural network (CNN) trained by an innovative metric learning scheme with a temporal constraint. By introducing the temporal constraint in metric learning, we help the network to learn a feature embedding which implicitly takes the shape and pose changes of fish into account. Besides, for each class, we learn the representative features discriminatively by introducing an intermediate layer in the CNN before the classifier. In testing stage, we first aggregate the features of a fish from each frame into several clips in the feature space, send the clips to the classifier and then perform weighted majority vote for the final classification. The experimental results show that our approach outperforms the conventional softmax classification on our rail-fishing dataset.

Monocular visual odometry (VO), which is a subset of simultaneous localization and mapping (SLAM) used to determine the position and orientation of a moving object by analyzing the associated monocular camera image sequences, is a critical part in the vision system of autonomous driving. However, based on the frame-by-frame pose estimation, drift error can be incrementally accumulated. Bundle adjustment (BA) is thus introduced to deal with the error-drift problem through correlating several image frames together to optimize camera poses and extracted 3D map points simultaneously. In this paper, we propose a joint BA framework which takes into account additional constraints from the detected road traffic signs. This framework can be effectively integrated into existing VO systems, as evidenced by the improved vehicular localization accuracy in experimental performance when compared with the state-of-the-art baseline VO method.

In this paper, we propose an unsupervised joint deep learning pipeline for depth and ego-motion estimation that explicitly incorporated with traditional spatial-temporal geometric constraints. The stereo reconstruction error provides the spatial geometric constraint to estimate the absolute scale depth. Meanwhile, the depth map with absolute scale and a pre-trained pose network serve as a good starting point for direct visual odometry (DVO), resulting in a fine-grained ego-motion estimation with the additional back-propagation signals provided to the depth estimation network. The proposed joint training pipeline enables an iterative coupling optimization process for accurate depth and precise ego-motion estimation. The experimental results show the state-of-the-art performance for monocular depth and ego-motion estimation on the KITTI dataset and a great generalization ability of the proposed approach.

Uploading Virtual Reality (VR) video over cellular networks is expected to boom in near future, as general consumers could generate high-quality VR videos with portable 360-degree cameras and are willing to share with others. Con-sequently, concerns of uplink bandwidth and delay arose for current popular technology of tile-based VR video streaming, which requires high quality video to transcode into multiple representations for further adaptive streaming. Motivated by this, we proposed a novel scheme for uplink delivery of tile-based VR video over cellular network, in which encoding bit rate of each tile is determined by uplink resource allocation (RA), and quality of content (QoC) contribution of each tile and channel quality of user equipments (UEs) are jointly considered during RA. Moreover, the RA problem is formulated as a frequency and time dependent non-deterministic polynomial(NP)-hard problem, which can be effectively solved by our proposed approximate convex algorithm. Simulation results show that the proposed algorithm can achieve higher utility, that is higher total quality of experience (QoE) for viewers.

3D localization of objects in road scenes is important for autonomous driving and advanced driver-assistance systems (ADAS). However, with common monocular camera setups, 3D information is difficult to obtain. In this paper, we propose a novel and robust method for 3D localization of monocular visual objects in road scenes by joint integration of depth estimation, ground plane estimation, and multi-object tracking techniques. Firstly, an object depth estimation method with depth confidence is proposed by utilizing the monocular depthmap from a CNN. Secondly, an adaptive ground plane estimation using both dense and sparse features is proposed to localize the objects when their depth estimation is not reliable. Thirdly, temporal information is taken into consideration by a new object tracklet smoothing method. Unlike most existing methods which only consider vehicle localization, our method is applicable for common moving objects in the road scenes, including pedestrians, vehicles, cyclists, etc. Moreover, the input depthmap can be replaced by some equivalent depth information from other sensors, like LiDAR, depth camera and Radar, which makes our system much more competitive compared with other object localization methods. As evaluated on KITTI dataset, our method achieves favorable performance on 3D localization of both pedestrians and vehicles when compared with the state-of-the-art vehicle localization methods, though no published performance on pedestrian 3D localization can be compared with, from the best of our knowledge.

Multi-object tracking (MOT) is an important topic and critical task related to both static and moving camera applications, such as traffic flow analysis, autonomous driving and robotic vision. However, due to unreliable detection, occlusion and fast camera motion, tracked targets can be easily lost, which makes MOT very challenging. Most recent works exploit spatial and temporal information for MOT, but how to combine appearance and temporal features is still not well addressed. In this paper, we propose an innovative and effective tracking method called TrackletNet Tracker (TNT) that combines temporal and appearance information together as a unified framework. First, we define a graph model which treats each tracklet as a vertex. The tracklets are generated by associating detection results frame by frame with the help of the appearance similarity and the spatial consistency. To compensate camera movement, epipolar constraints are taken into consideration in the association. Then, for every pair of two tracklets, the similarity, called the connectivity in the paper, is measured by our designed multi-scale TrackletNet. Afterwards, the tracklets are clustered into groups and each group represents a unique object ID. Our proposed TNT has the ability to handle most of the challenges in MOT, and achieves promising results on MOT16 and MOT17 benchmark datasets compared with other state-of-the-art methods.

Drones, or general UAVs, equipped with a single camera have been widely deployed to a broad range of applications, such as aerial photography, fast goods delivery and most importantly, surveillance. Despite the great progress achieved in computer vision algorithms, these algorithms are not usually optimized for dealing with images or video sequences acquired by drones, due to various challenges such as occlusion, fast camera motion and pose variation. In this paper, a drone-based multi-object tracking and 3D localization scheme is proposed based on the deep learning based object detection. We first combine a multi-object tracking method called TrackletNet Tracker (TNT) which utilizes temporal and appearance information to track detected objects located on the ground for UAV applications. Then, we are also able to localize the tracked ground objects based on the group plane estimated from the Multi-View Stereo technique. The system deployed on the drone can not only detect and track the objects in a scene, but can also localize their 3D coordinates in meters with respect to the drone camera. The experiments have proved our tracker can reliably handle most of the detected objects captured by drones and achieve favorable 3D localization performance when compared with the state-of-the-art methods.

Session-based recommendations recently receive much attentions due to no available user data in many cases, e.g., users are not logged-in/tracked. Most session-based methods focus on exploring abundant historical records of anonymous users but ignoring the sparsity problem, where historical data are lacking or are insufficient for items in sessions. In fact, as users' behavior is relevant across domains, information from different domains is correlative, e.g., a user tends to watch related movies in a movie domain, after listening to some movie-themed songs in a music domain (i.e., cross-domain sessions). Therefore, we can learn a complete item description to solve the sparsity problem using complementary information from related domains. In this paper, we propose an innovative method, called Cross-Domain Item Embedding method based on Co-clustering (CDIE-C), to learn cross-domain comprehensive representations of items by collectively leveraging single-domain and cross-domain sessions within a unified framework. We first extract cluster-level correlations across domains using co-clustering and filter out noise. Then, cross-domain items and clusters are embedded into a unified space by jointly capturing item-level sequence information and cluster-level correlative information. Besides, CDIE-C enhances information exchange across domains utilizing three types of relations (i.e., item-to-context-item, item-to-context-co-cluster and co-cluster-to-context-item relations). Finally, we train CDIE-C with two efficient training strategies, i.e., joint training and two-stage training. Empirical results show CDIE-C outperforms the state-of-the-art recommendation methods on three cross-domain datasets and can effectively alleviate the sparsity problem.

Electronic monitoring of fishery activities has drawn increasing attention. Deformable objects, noise from the wild sea surface, and dynamic background, however, make conventional tracking and segmentation methods unreliable. In this paper, we present an online 3D tracking and segmentation system for stereo video-based monitoring of rail fish catching on the wild sea surface. Based on the result of a pre-trained image object (fish) detector, a Kalman filtering-based tracking system overcomes the issues of low detection scores of deformed objects and of unreliable bounding boxes by rescoring multiple object proposals using spatial information in 3D. A clustering-and-scoring strategy is then applied on the depth map so that a plane classification method can effectively segment the objects from the dynamic background without any prior modeling. The object segmentation is further refined using fully connected conditional random fields based on color and geometric features. Using the segmentation results, we can measure the 3D lengths of objects and update the positions of bounding boxes to help tracking. Experimental results show that a reliable tracking and measurement performance under noisy and dynamic sea surface environment can be achieved.

Uncertainty sampling-based active learning has been well studied for selecting informative samples to improve the performance of a classifier. In batch-mode active learning, a batch of samples are selected for a query at the same time. The samples with top uncertainty are encouraged to be selected. However, this selection strategy ignores the relations among the samples, because the selected samples may have much redundant information with each other. This paper addresses this problem by proposing a novel method that combines uncertainty, diversity, and density via sparse modeling in the sample selection. We use sparse linear combination to represent the uncertainty of unlabeled pool data with Gaussian kernels, in which the diversity and density are well incorporated. The selective sampling method is proposed before optimization to reduce the representation error. To deal with ${l}_{0}$ norm constraint in the sparse problem, two approximated approaches are adopted for efficient optimization. Four image classification data sets are used for evaluation. Extensive experiments related to batch size, feature space, seed size, significant analysis, data transform, and time efficiency demonstrate the advantages of the proposed method.

Volumetric media, popularly known as holograms, need to be delivered to users using both on-demand and live streaming, for new augmented reality (AR) and virtual reality (VR) experiences. As in video streaming, hologram streaming must support network adaptivity and fast startup, but must also moderate large bandwidths, multiple simultaneously streaming objects, and frequent user interaction, which requires low delay. In this paper, we introduce the first system designed specifically for streaming volumetric media. The system reduces bandwidth by introducing 3D tiles, and culling them or reducing their level of detail depending on their relation to the user's view frustum and distance to the user. To allocate bits among different tiles across multiple objects, we introduce a simple greedy yet provably optimal algorithm for rate-utility optimization, whose utility measures is based not only on the underlying quality of the representation, but on the level of detail relative to the user's viewpoint and device resolution. Simulation results show that the proposed algorithm provides superior quality compared to existing video-streaming approaches adapted to hologram streaming, in terms of utility and user experience over variable, throughput-constrained networks.

360-degree videos for Virtual Reality (VR) applications are getting more popular because of their diverse applications. However, VR videos usually need more bandwidth than conventional videos to provide the same quality of experience (QoE). Tiled videos can help save bandwidth by selecting lower quality encoding for the tiles with lower probability of viewing. Dynamic Adaptive Streaming over HTTP (DASH) enables the adaptive rate selection of tiles based on the channel conditions. Multicasting also can help save bandwidth, since many users share the spectrum when they request the same video contents. In this paper, we formulate the utility maximization problem to find which tiles should have which video representations to satisfy the most users in multicasting groups using limited resources. A cross-layer optimization framework, which includes user grouping, resource allocation, and the tile-based rate-selection algorithms, is proposed to maximize the total utility among all users. Simulation results show that the proposed cross-layer optimization framework can achieve a higher utility than the broadcasting solution or existing multicast solutions.

This paper presents a new scheme to perform inter-camera human tracking in a surveillance camera network with high resolution cameras by taking advantage of all possible collected visual information. The proposed approach utilizes the tracked trajectory information of pedestrians within a camera to get accurate face positions and poses. To solve varied face pose problem under different cameras, we frontalize random posed face with a generic 2D-to-3D mapping matrix between facial feature points. Texture-based face descriptor is then exploited to extract useful features from facial components and combined with pose-invariant appearance feature, which models dominant color components in two partitioned body regions as GMM. The proposed algorithm shows promising performance by evaluating on the public benchmark Dana36 dataset.

This paper proposes an effective and practical method for self-calibration of traffic surveillance cameras. Based on analyzing multiple moving vehicles across multiple frames, the Canny edge detector and Hough transform are first adopted to obtain orthogonal horizontal vanishing points pairs, from which corresponding vertical vanishing points are derived. Next, mean shift clustering and Laplace linear regression are employed to deal with noise and outliers during estimation of vanishing points. To overcome the unreliable estimation issues of orthogonal vanishing points pairs, we further utilize the projective line segments obtained from 3-D vehicle model to create more reliable pairs and iteratively improve the calibration results. Finally, the estimation of distribution algorithm (EDA) is also applied to relax the assumptions made on camera parameters and the moving trajectories of vehicles during the iterations. Experimental results on different datasets prove the feasibility of our proposed scheme.

The goal of data analytics in surveillance videos is to fully understand and reconstruct the 3D scene, i.e., to recover the trajectory and action of each object. In a surveillance system with camera arrays of overlapping views, we propose a novel video scene reconstruction framework to collaboratively track multiple human objects and estimate their 3D poses. First, tracklets are extracted from each single view following the tracking-by-detection paradigm. We propose an effective integration of visual and semantic object attributes, i.e., appearance models, geometry information and poses/actions, to associate tracklets across different views. Based on the optimum viewing perspectives derived from tracking, a hierarchical estimation of human poses is introduced to generate the 3D skeleton of each object. The estimated body joint points are fed back to the tracking stage to enhance tracklet association. Experiments on benchmarks of multiview tracking and 3D pose estimation validate the effectiveness of the proposed method.

The scarcity of the long term evolution (LTE) bandwidth and the increasing data demand call for more wireless local area networks (WLANs) to help the cellular offloading. A software-defined networking (SDN) based control approach is proposed to effectively utilize the heterogeneous LTE and WLAN radio bandwidth by operating the multi-radio interfaces simultaneously. The use of the deployed Wi-Fi access points (APs) for Internet access has been a common practice for most people, however, the total utility or the quality of experience (QoE) drops when users compete for a single Wi-Fi AP or there is no controller to aid them to allocate rates to each radio access technology (RAT). This problem will even aggravate when more APs are involved in wireless mobile networks. This paper investigates how heterogeneous resources should be coordinated and allocated to multi-user access of the LTE and Wi-Fi aggregation (LWA) network. The proposed application layer scheme can adjust the target users and resources adaptively based on estimated users states so that the total QoE attained by all users is maximized. The developed scheme can help the operator decide the associations of users to appropriate Wi-Fi APs and adaptively adjust rate allocations of each user on the associated Wi-Fi and LTE. The good performance and low complexity of the proposed scheme is validated in a network simulator (NS-3).

Due to the expanding scale of camera networks, Multiple Camera Tracking (MCT) of humans has received increased attention in recent years. In this paper, we present a novel approach to tracking each human within a single camera and across multiple disjoint cameras. Our framework includes a multi-object tracking and segmentation system, a two-phase feature extractor, and an online-learning-based camera link model estimation. For tracking within a single camera, we apply tracking by segmentation and local object detection with multi-kernel feedback to adaptively improve the robustness of the algorithm. In inter-camera tracking, we introduce an effective integration of appearance and context features. Couples are automatically detected, and the couple feature is also integrated with existing features. The proposed algorithm is scalable by a fully unsupervised online-learning framework. In our experiments, the proposed method outperforms all of the state-of-the-art methods in the benchmark National Laboratory of Pattern Recognition (NLPR)_MCT dataset.

Existing learning-based atmospheric particle-removal approaches such as those used for rainy and hazy images are designed with strong assumptions regarding spatial frequency, trajectory, and translucency. However, the removal of snow particles is more complicated because they possess additional attributes of particle size and shape, and these attributes may vary within a single image. Currently, hand-crafted features are still the mainstream for snow removal, making significant generalization difficult to achieve. In response, we have designed a multistage network named DesnowNet to in turn deal with the removal of translucent and opaque snow particles. We also differentiate snow attributes of translucency and chromatic aberration for accurate estimation. Moreover, our approach individually estimates residual complements of the snow-free images to recover details obscured by opaque snow. Additionally, a multi-scale design is utilized throughout the entire network to model the diversity of snow. As demonstrated in the qualitative and quantitative experiments, our approach outperforms state-of-the-art learning-based atmospheric phenomena removal methods and one semantic segmentation baseline on the proposed Snow100K dataset. The results indicate our network would benefit applications involving computer vision and graphics.

Automated segmentation of intracranial arteries on magnetic resonance angiography (MRA) allows for quantification of cerebrovascular features, which provides tools for understanding aging and pathophysiological adaptations of the cerebrovascular system. Using a convolutional autoencoder (CAE) for segmentation is promising as it takes advantage of the autoencoder structure in effective noise reduction and feature extraction by representing high dimensional information with low dimensional latent variables. In this paper, we trained an 8-layer CAE to learn a 3D segmentation model of intracranial arteries from 49 cases of MRA data. After parameter optimization and prediction refinement, our trained model was shown to perform better than the three traditional segmentation methods in both binary classification and visual evaluation.

Dynamic Adaptive Streaming over HTTP (DASH) is a fast growing video streaming platform which enables adaptive rate selection based on channel conditions. File Delivery over Unidirectional Transport (FLUTE) further enables multicasting of the DASH segments over LTE eMBMS systems. In this paper, an optimal DASH-multicasting solution is proposed to allow more DASH clients in an LTE network to receive better videos by optimizing the resource allocation, Forward Error Correction (FEC) code rate and modulation and coding scheme (MCS) of each multicasting group, which corresponds to a FLUTE session. Multiple FLUTE sessions are considered to deliver multiple videos and multiple video rates for enhancing the overall utility. We have applied the convex optimization method to find the optimal resource allocation in terms of utility for multiple FLUTE sessions. We also find the optimal FEC code rates to add redundancies to protect the video segments for each FLUTE session. Moreover, an efficient MCS selection is introduced to reduce the complexity of the algorithm. Simulation results, with realistic LTE parameters, are shown to prove the proposed scheme is optimal, with more DASH clients receiving better video representations within limited resources when compared to other existing algorithms.

In this paper, we present a novel fully automatic approach to track the same human across multiple disjoint cameras. Our framework includes a two-phase feature extractor and an online-learning-based camera link model estimation. We introduce an effective and robust integration of appearance and context features. Couples are detected automatically, and the couple feature is also integrated with appearance features effectively. The proposed algorithm is scalable with the use of a fully unsupervised online learning framework. In the experiments, it outperforms all the state-of-the-art methods on the benchmark NLPR_MCT dataset.

Visual simultaneous localization and mapping (V-SLAM) based tracking method for moving cameras has drawn increasing attention. The unpredictability of road conditions and noise from the camera calibration, however, make conventional ground plane estimation unreliable and adversely affecting the tracking result. In this paper, we propose an adaptive ground plane estimation algorithm in a moving monocular camera. In our algorithm, we use structure from motion (SfM) to estimate the pose of moving camera, then the estimated camera yaw angle is used as a feedback to improve the accuracy of the ground plane estimation. Combining the efficient constrained-multiple-kernel (CMK) tracking of video objects in 3D space and the reliable ground plane estimation, the proposed system not only achieves high effectiveness but also well handles occlusion in the tracking. The proposed system is evaluated on several challenging datasets and the experimental results show the favorable performance.

Uncertainty based active learning has been well studied for selecting informative samples to improve the performance of the classifier. One of the simplest strategy is that we always select samples with top largest uncertainties for a query. However, the selected samples may be very similar to each other, which results in little information added to update the classifier. In other words, we should avoid selecting similar samples for training the classifier. This paper addresses this problem by proposing a novel method using uncertainty based active learning algorithm with diversity constraint by sparse selection. First, uncertainty scores of unlabeled samples are obtained based on the previously trained support vector machine (SVM) classifiers. Then the sample selection is represented as a sparse modeling problem and optimal samples up to the pre-defined batch size are selected for a query. Besides that, two approximated approaches are proposed to solve the sparse problem via greedy search and quadratic programming (QP), respectively. After selection, the SVM classifiers are re-trained with new labeled data and the performance is tested on the testing dataset. We conduct several experiments on three image datasets for image classification task. The experimental results show the proposed method outperforms other four different methods and achieves promising performance.

More than 70% of consumer mobile Internet traffic will be mobile video transmissions by 2019. The development of wireless video transmission technologies has been boosted by the rapidly increasing demand of video streaming applications. Although more and more videos are delivered for video analysis (e.g., object detection/tracking and action recognition), most existing wireless video transmission schemes are developed to optimize human perception quality and are suboptimal for video analysis. In mobile surveillance networks, a cloud server collects videos from multiple moving cameras and detects suspicious persons in all camera views. Camera mobility in smartphones or dash cameras implies that video is to be uploaded through bandwidth-limited and error-prone wireless networks, which may cause quality degradation of the decoded videos and jeopardize the performance of video analyses. In this paper, we propose an effective rate-allocation scheme for multiple moving cameras in order to improve human detection (content) performance. Therefore, the optimization criterion of the proposed rate-allocation scheme is driven by quality of content (QoC). Both video source coding and application layer forward error correction coding rates are jointly optimized. Moreover, the proposed rate-allocation problem is formulated as a convex optimization problem and can be efficiently solved by standard solvers. Many simulations using High Efficiency Video Coding standard compression of video sequences and the deformable part model object detector are carried, and results demonstrate the effectiveness and favorable performance of our proposed QoC-driven scheme under different pedestrian densities and wireless conditions.

We propose a joint rate and power adaptation scheme to maximize the decoding quality for scalable video coding (SVC)-based video transmissions over multi-input multioutput (MIMO) systems. The rate adaptation in our proposed scheme includes selection of the best modulation and coding schemes, set of spatial channels, number of SVC layers (source coding rates), and their corresponding application layer forward error correction (APP-FEC) coding rates. The power adaptation involves the proper allocation of the power to each antenna in the MIMO system. SVC-based video transmissions require unequal error protection (UEP) for different SVC layers due to the inter-layer dependency. In most of the previous works, the bit stream of each particular SVC layer is allocated to one spatial channel and the UEP is achieved by transmitting the more important SVC layers through the spatial channels with higher channel gains. However, in our proposed scheme, the bit stream of each particular SVC layer is distributed to multiple spatial channels so that additional diversity gain can be exploited by applying APP-FEC. The UEP can also be achieved by allocating different APP-FEC coding rates on each video layer. Moreover, transmit power allocation is also effectively and jointly determined to improve the system performance. The effectiveness and favorable performance of our proposed scheme are shown by simulations with H.264 SVC traces of high-definition video clips over MIMO systems.

This paper proposes an ensemble of invariant features (EIFs), which can properly handle the variations of color difference and human poses/viewpoints for matching pedestrian images observed in different cameras with nonoverlapping field of views. Our proposed method is a direct reidentification (re-id) method, which requires no prior domain learning based on prelabeled corresponding training data. The novel features consist of the holistic and region-based features. The holistic features are extracted by using a publicly available pretrained deep convolutional neural network used in generic object classification. In contrast, the region-based features are extracted based on our proposed two-way Gaussian mixture model fitting, which overcomes the self-occlusion and pose variations. To make a better generalization during recognizing identities without additional learning, the ensemble scheme aggregates all the feature distances using the similarity normalization. The proposed framework achieves robustness against partial occlusion, pose, and viewpoint changes. Moreover, the evaluation results show that our method outperforms the state-of-the-art direct re-id methods on the challenging benchmark viewpoint invariant pedestrian recognition and 3D people surveillance data sets.

Fishery surveys that call for the use of single or multiple underwater cameras have been an emerging technology as a nonextractive mean to estimate the abundance of fish stocks. Tracking live fish in an open aquatic environment posts challenges that are different from general pedestrian or vehicle tracking in surveillance applications. In many rough habitats, fish are monitored by cameras installed on moving platforms, where tracking is even more challenging due to inapplicability of background models. In this paper, a novel tracking algorithm based on the deformable multiple kernels is proposed to address these challenges. Inspired by the deformable part model technique, a set of kernels is defined to represent the holistic object and several parts that are arranged in a deformable configuration. Color histogram, texture histogram, and the histogram of oriented gradients (HOGs) are extracted and serve as object features. Kernel motion is efficiently estimated by the mean-shift algorithm on color and texture features to realize tracking. Furthermore, the HOG-feature deformation costs are adopted as soft constraints on kernel positions to maintain the part configuration. Experimental results on practical video set from underwater moving cameras show the reliable performance of the proposed method with much less computational cost comparing with state-of-the-art techniques.

Image processing and analysis techniques have drawn increasing attention since they enable a non-extractive and non-lethal approach to fisheries survey, such as fish size measurement, abundance prediction, catch estimation and compliance, species recognition and population counting. In this work, we present an innovative and effective method for measuring the chute-based fish length based on the morphological midline of the fish. The midline is generated through recursive morphological operations on the segmented fish mask. To conduct reliable measurement, even under harsh environment, we also propose a systematic method for detecting water drop on camera lens. The robust detection, which can be performed either in real-time or in offline processing, is based on a blur measure derived from the gradient of the image and the contour of fish.

In a video surveillance system with static cameras, object segmentation often fails when part of the object has similar color with the background, resulting in poor performance of the subsequent object tracking. Multiple kernels have been utilized in object tracking to deal with occlusion, but the performance still highly depends on segmentation. This paper presents an innovative system, named Multiple-kernel Adaptive Segmentation and Tracking (MAST), which dynamically controls the decision thresholds of background subtraction and shadow removal around the adaptive kernel regions based on the preliminary tracking results. Then the objects are tracked for the second time according to the adaptively segmented foreground. Evaluations of both segmentation and tracking on benchmark datasets and our own recorded video sequences demonstrate that the proposed method can successfully track objects in similar-color background and/or shadow areas with favorable segmentation performance.

In a video surveillance system with a single static camera, tracking results of moving persons can be effectively used for camera self-calibration. However, the current methods need to depend on robustness of both tracking and segmentation procedures. RANSAC has been widely used to remove outliers in finding the vertical vanishing point and the horizon line, but the performance is degraded when the proportion of outliers is high. Last but not least, all of them require excessive simplifications in the algorithmic procedures resulting in increasing reprojection error. In this paper, a robust segmentation and tracking system is applied to provide accurate estimation of head and foot locations of moving persons. The noise in the computation of vanishing points is handled by mean shift clustering and Laplace linear regression through convex optimization. We also propose to use the estimation of distribution algorithm (EDA) to search for the local optimal solution for camera calibration that minimizes average reprojection error on the ground plane, while relaxing the assumptions on camera parameters. Promising evaluations of the performance of our proposed method on real scenes are presented.

Live fish recognition is one of the most crucial elements of fisheries survey applications where the vast amount of data is rapidly acquired. Different from general scenarios, challenges to underwater image recognition are posted by poor image quality, uncontrolled objects and environment, and difficulty in acquiring representative samples. In addition, most existing feature extraction techniques are hindered from automation due to involving human supervision. Toward this end, we propose an underwater fish recognition framework that consists of a fully unsupervised feature learning technique and an error-resilient classifier. Object parts are initialized based on saliency and relaxation labeling to match object parts correctly. A non-rigid part model is then learned based on fitness, separation, and discrimination criteria. For the classifier, an unsupervised clustering approach generates a binary class hierarchy, where each node is a classifier. To exploit information from ambiguous images, the notion of partial classification is introduced to assign coarse labels by optimizing the benefit of indecision made by the classifier. Experiments show that the proposed framework achieves high accuracy on both public and self-collected underwater fish images with high uncertainty and class imbalance.

This paper proposes a robust ground-moving-platform-based human tracking system, which effectively integrates visual simultaneous localization and mapping (V-SLAM), human detection, ground plane estimation, and kernel-based tracking techniques. The proposed system systematically detects humans from recorded video frames of a moving camera and tracks the humans in the V-SLAM-inferred 3-D space via a tracking-by-detection scheme. To efficiently associate the detected human frame by frame, we propose a novel human tracking framework, combining the constrained-multiple-kernel tracking and the estimated 3-D information (depth), to globally optimize the data association between consecutive frames. By taking advantage of the appearance model and 3-D information, the proposed system not only achieves high effectiveness but also well handles occlusion in the tracking. Experimental results show the favorable performance of the proposed system, which efficiently tracks humans in a camera equipped on a ground-moving platform such as a dash camera and an unmanned ground vehicle.

In this paper, an optimal power allocation and rate adaptation scheme is proposed to maximize the user Quality of Experience (QoE) of scalable video streaming over multi-user MIMO (MU-MIMO) networks. The proposed scheme utilizes the wireless resources to accommodate multiple users consuming different video contents more efficiently under limited propagation power and spectrum. The optimal power allocation scheme is derived based on convex optimization and an effective algorithm is proposed to select the best modulation set to control the transmission rate of each scalable video layer. Simulation results show that the proposed scheme can achieve more favorable performance than water- filling algorithm and any other power allocation scheme in terms of the sum-utility measure and QoE.

In this paper, we propose an innovative human tracking algorithm, which efficiently integrates the deformable part model (DPM) into the multiple-kernel based tracking using a moving camera. By representing each part model of a DPM detected human as a kernel, the proposed algorithm iteratively mean-shift the kernels (i.e., part models) based on color appearance and histogram of gradient (HOG) features. More specifically, the color appearance features, in terms of kernel histogram, are used for tracking each body part from one frame to the next, the deformation cost provided by DPM detector is further used to constrain the movement of each body kernel based on the HOG features. The proposed deformable multiple-kernel (DMK) tracking algorithm takes advantage of not only low computation owing to the kernelbased tracking, but also robustness of the DPM detector. Experimental results have shown the favorable performance of the proposed algorithm, which can successfully track human using a moving camera more accurately under different scenarios.

Human tracking across multiple cameras is highly demanded for large scale video surveillance. To successfully track human across multiple uncalibrated cameras that have no overlapping field of views, a system to train more reliable camera link models is proposed in this paper. We employ a novel approach of combining multiple camera links and building bidirectional transition time distribution in the process of estimation. Through the unsupervised scheme, the system builds several camera link models simultaneously for the camera network that has multi-path in presence of the outliers. Our proposed method decreases incorrect correspondences and results in more accurate camera link model for higher tracking accuracy. The proposed algorithm shows the effectiveness by evaluating in the real-world camera network scenarios.

We propose a forward error correction (FEC) coding rate adaptation scheme which maximizes the quality of experience (QoE), for scalable video coding (SVC) based video transmissions over multi-input multi-output (MIMO) systems. The proposed scheme adaptively selects the best set of spatial channels, number of video layers and their corresponding FEC coding rate according to channel state information (CSI) from the receiver. Unlike previous work, our proposed scheme distributes the FEC encoded bit streams to multiple spatial channels so that additional diversity gains can be obtained. Due to the complexity of the optimization, we decompose the original problem into several sub-problems, which can then be solved by a heuristic algorithm. The optimal solution can be found by choosing the best among all the candidate solutions obtained from the sub-problems. The effectiveness and superb performance of our proposed scheme can be demonstrated by many simulations with different videos and channel conditions.

We propose an application (APP) layer scheduling scheme for scalable video transmissions over multiple radio access technologies (multi-RATs). More specifically, the proposed scheme adaptively adjusts the transmission parameters based on estimated network characteristics so that the decoding quality of experience (QoE) at user end is maximized. These parameters include number of transmitted video layers (source coding rate), the APP layer forward error correction (FEC) redundancy for each video layer (channel coding rate), and the transmission data rates in both cellular network and wireless local area network (WLAN). The network conditions are estimated by real-time protocol (RTP) and real-time control protocol (RTCP). Since the proposed scheme is an APP layer design, it can be easily implemented without changing the configurations of lower layers (e.g., transport or MAC layers). Simulations are conducted in network simulator 3 (NS-3), and demonstrate the effectiveness of our proposed scheme.

Nowadays, more and more videos are transmitted for video analytics purposes rather than human perceptions. In mobile surveillance networks, a cloud server collects videos delivered from multiple moving cameras and detects suspicious people in all the camera views. However, all the videos recorded by moving cameras such as phone or dash cameras are uploaded through bandwidth-limited wireless networks. Therefore, videos are required to be encoded with high compression ratio to satisfy the total data rate constraint, which may affect the video analyses (e.g., human detection/tracking and action recognition, etc.) performance due to the degraded video decoding qualities at the server side. In this paper, we propose an effective content-driven video source coding rate allocation scheme, which can improve the human detection success rate in mobile surveillance networks under a total data rate constraint. The proposed scheme allocates appropriate amount of data rate to each moving camera based on the corresponding content information (i.e., human detection results). A model of human detection accuracy based on object area and video quality is provided. The rate allocation problem is formulated as a convex optimization problem and can be solved by standard solvers. Simulations with real video sequences demonstrate the effectiveness of our proposed scheme.

We propose an ensemble of invariant features for person re-identification. The proposed method requires no domain learning and can effectively overcome the issues created by the variations of human poses and viewpoint between a pair of different cameras. Our ensemble model utilizes both holistic and region-based features. To avoid the misalignment problem, the test human object sample is used to generate multiple virtual samples, by applying slight geometric distortion. The holistic features are extracted from a publically available pre-trained deep convolutional neural network. On the other hand, the region-based features are based on our proposed Two-Way Gaussian Mixture Model Fitting and the Completed Local Binary Pattern texture representations. To make better generalization during the matching without additional learning processes for the feature aggregation, the ensemble scheme combines all three feature distances using distances normalization. The proposed framework achieves robustness against partial occlusion, pose and viewpoint changes. In addition, the experimental results show that our method exceeds the state of the art person re-identification performance based on the challenging benchmark 3DPeS.

Nonextractive fish abundance estimation with the aid of visual analysis has drawn increasing attention. Unstable illumination, ubiquitous noise, and low-frame-rate (LFR) video capturing in the underwater environment, however, make conventional tracking methods unreliable. In this paper, we present a multiple fish-tracking system for low-contrast and LFR stereo videos with the use of a trawl-based underwater camera system. An automatic fish segmentation algorithm overcomes the low-contrast issues by adopting a histogram backprojection approach on double local-thresholded images to ensure an accurate segmentation on the fish shape boundaries. Built upon a reliable feature-based object matching method, a multiple-target tracking algorithm via a modified Viterbi data association is proposed to overcome the poor motion continuity and frequent entrance/exit of fish targets under LFR scenarios. In addition, a computationally efficient block-matching approach performs successful stereo matching that enables an automatic fish-body tail compensation to greatly reduce segmentation error and allows for an accurate fish length measurement. Experimental results show that an effective and reliable tracking performance for multiple live fish with underwater stereo cameras is achieved.

In this paper, we propose a novel model-based vehicle localization approach on the basis of surveillance cameras. The proposed approach regards each patch of the 3-D vehicle model as a kernel, and tracks the kernels under certain constrains facilitated by the 3-D geometry of the vehicle model. Meanwhile, a kernel density estimator is designed to well fit the 3-D vehicle model during tracking. With elegant application of the constrained multiple-kernel tracking facilitated with the 3-D vehicle model, the vehicles are able to be tracked efficiently and located precisely. The proposed approach achieves high effectiveness in the tracking and localization by taking advantage of the color similarity and shape fitness. Experimental results have shown the favorable performance of the proposed approach, in several scenarios, which efficiently tracks vehicles while maintaining the knowledge of 3-D geometry of the tracked vehicles.

In this paper, we propose a new framework to track on-road pedestrians across multiple driving recorders. The framework is built upon the results of tracking under a single driving recorder. More specifically, we treat the problem as a multi-label classification task and determine whether a specific pedestrian belongs to one or several cameras' field of views by considering association likelihood of the tracked pedestrians . The likelihood is calculated based on the pedestrians' motion cues and appearance features, which are necessarily transformed via brightness transfer functions obtained by some available spatially overlapping views for compensating diversity of the cameras. When a pedestrian is leaving a camera's field of view, the proposed framework predicts and interpolates its possible moving trajectories, facilitated by open map service which can provide routing information. Experimental results show the robustness and effectiveness of the proposed framework in tracking pedestrians across several recorded driving videos. Moreover, based on the GPS locations, we can also reconstruct a 3-D visualization on a 3-D virtual real-world environment, so as to show the dynamic scenes of the recorded videos.

Abstract This paper presents a channel reservation and preemption (CRP) model using overlapping regions in a cellular network with multiple sectors. To fully exploit and reuse the frequencies, directional antennas are installed on base stations (BSs) to divide the coverage into a number of equal‐sized sectors. When traffic is unevenly distributed across the sectors in a BS, channel utilization in every sector may become very different; low‐traffic sectors may be underutilized while high‐traffic sectors may be overutilized. A CRP scheme is thus proposed to more efficiently utilize free channels among sectors. CRP aims at reducing the dropping probabilities of handoff calls. Specifically, when free channels in a sector are not available, a handoff call, instead of being dropped, is allowed to preempt an ongoing call residing in the overlapping region of two adjacent sectors or two neighbor cells. Under CRP, the preempted ongoing call will not be disconnected, because it can switch over its service to the BS of a neighbor cell or to another directional antenna of an adjacent sector. For the purpose of evaluation, we build an analytical model for the proposed CRP using six‐tuple Markov chains. Analytical results show that the proposed CRP can significantly reduce the dropping probabilities of inter‐sector handoff calls, particularly when traffic between two sectors is not evenly distributed. Copyright © 2013 John Wiley & Sons, Ltd.

Machine-to-Machine (M2M) or machine-type communication technology standardized by ETSI/3GPP has recently gained a great deal of attention, and has been utilized in a variety of M2M applications, which commonly require real-time data reporting. This paper investigates the energy minimized scheduling problem for real-time reporting of data-critical M2M applications. Although many uplink scheduling algorithms have been proposed for different wireless mobile networks, they are mainly designed for human-to-human communication paradigms. This paper proves this energy minimized scheduling problem is NP-hard, and proposes a heuristic energy-efficient algorithm to address it. Our algorithm effectively schedules the transmissions of an M2M node in the same time slots, so that the active time of the M2M node can be minimized. The experiment results show that under limited bandwidth resource, our algorithm can maintain fairness and low data dropping ratios while achieving energy efficiency for a reasonable number of M2M nodes.

Camera networks have been deployed to facilitate human tracking across multi-cameras in the modern surveillance systems. However, human privacy is an important concern on security surveillance. More specifically, in the real world, surveillance cameras are commonly installed by different entities (such as departments or companies), and any recorded video by one entity should not be shared with others to protect the privacy of tracked humans, while maintaining the knowledge of those moving trajectories of tracked humans in a centralized cloud server. This tracking across multi-cameras information can serve as a very powerful analysis tool for locating crime suspects or collecting business statistics. This paper is the first to aim at the importance of privacy-preserving in a multiple-camera tracking system. We address the problems of privacy-preserving human tracking based on Paillier encryption without revealing any recorded video or data, and introduce the secure multiple-camera system which consists of two stages: training and testing stages. Finally, the security analyses and simulations show the effectiveness of the proposed scheme.

In this paper, we propose a near optimal power allocation scheme, which maximizes the quality of experience (QoE), for scalable video coding (SVC) based video transmissions over multi-input multi-output (MIMO) systems. This scheme tries to optimize the received video quality according to video frame-error-rate (FER), which may be caused by either transmission errors in physical (PHY) layer or video coding structures in application (APP) layer. Due to the complexity of the original optimization problem, we decompose it into several sub-problems, which can then be solved by classic convex optimization methods. Detailed algorithms with corresponding theoretical derivations are provided. Simulations with real video traces demonstrate the effectiveness of our proposed scheme.

Live fish recognition in open aquatic habitats suffers from the high uncertainty in many of the data. To alleviate this problem without discarding those data, the system should learn a species hierarchy so that high-level labels can be assigned to ambiguous data. In this paper, a systematic hierarchical partial classification algorithm is therefore proposed for underwater fish species recognition. Partial classification is applied at each level of the species hierarchy so that the coarse-to-fine categorization stops once the decision confidence is low. By defining the exponential benefit function, we formulate the selection of decision threshold as an optimization problem. Also, attributes from important fish anatomical parts are focused to generate discriminative feature descriptors. Experiments show that the proposed method achieves an accuracy up to 94%, with partial decision rate less than 5%, on underwater fish images with high uncertainty and class imbalance.

A multiple-camera tracking system that tracks humans across cameras with nonoverlapping views is proposed in this paper. The systematically estimated camera link model, including transition time distribution, brightness transfer function, region mapping matrix, region matching weights, and feature fusion weights, is utilized to facilitate consistently labeling the tracked humans. The system is divided into two stages: in the training stage, based on an unsupervised scheme, we formulate the estimation of the camera link model as an optimization problem, in which temporal features, holistic color features, region color features, and region texture features are jointly considered. The deterministic annealing is applied to effectively search the optimal model solutions. The unsupervised learning scheme tolerates the presence of outliers in the training data well. In the testing stage, the systematic integration of multiple cues from the above features enables us to perform an effective reidentification. The camera link model can be continuously updated during tracking in the testing stage to adapt the changes of the environment. Several simulations and comparative studies demonstrate the superiority of our proposed estimation method to the others. Moreover, the complete system has been tested in a small-scale real-world camera network scenario.

In wireless multicast services, a base station (BS) requires channel quality indicator (CQI) feedback from subscribers in order to determine a proper modulation and coding scheme (MCS). However, as the group sizes increase, the required feedbacks also increase, which will cause more signaling overheads in uplink. This paper proposes an efficient CQI feedback resource allocation scheme for wireless video multicast applications. More specifically, when a BS is serving several multicast groups simultaneously, our algorithm determines the proper sizes of feedback sets for each multicast group so that the system performance, in terms of average number of multicast service supported subscribers, is maximized. Simulation results not only well demonstrate the advantages of our proposed scheme, but also match with theoretical analyses.

We present a self-organized and scalable multiple-camera tracking system that tracks human across the cameras with nonoverlapping views. Given the GPS locations of uncalibrated cameras, the system automatically detects the existence of camera link within the camera network based on the routing information provided by Google Maps. The connected zones in any pair of directly-connected cameras are identified based on the feature matching between the camera's view and Google Street View. The camera link model is further estimated by an unsupervised learning scheme. Finally, multiple-camera tracking is performed. Thanks to the unsupervised pairwise learning and tracking in our system, the camera network is self-organized, and our proposed system is able to be scaled up efficiently when more cameras are added into the network.

Automation of fishery survey through the aid of visual analysis has received increasing attention. In this paper, a novel algorithm for the aggregated segmentation of fish images taken from conveyor belt videos is proposed. The watershed algorithm driven by an automatic marker generation scheme successfully separates clustered fish images without damaging their boundaries. A target selection based on appearance classification then rejects non-fish objects. By applying histogram backprojection and kernel density estimation, an innovative algorithm for combining object masks of one tracked fish from multiple frames into a refined single one is also proposed. Experimental results show that accurate fish segmentation from conveyor belt videos is achieved.

In this paper, we propose a novel vehicle tracking system under a surveillance camera. The proposed system tracks vehicles by using constrained multiple-kernel, facilitated with Kalman filtering, to continuously update the position and the orientation of the moving vehicles. To further reliably track vehicles under partial occlusion or even total occlusion, our tracking algorithm also systematically builds 3-D vehicle model, from which the license plate region is identified and a self-similarity descriptor is further used for low-resolution license plate matching. Experimental results have shown the favorable performance of the proposed system, which can successfully track vehicles under serious occlusion while maintaining the knowledge of 3-D geometry of the tracked vehicles.

This paper proposes an adaptive mode and modulation coding switching (AMMCS) scheme for multiple-input multiple-output (MIMO) multicasting system. More specifically, in addition to the adaptively chosen modulation and error coding, two types of MIMO modes are adaptively switched, i.e., between spatial multiplexing (SM) mode, which can achieve higher data rate, and spatial diversity (SD) mode, which can provide higher received signal-to-noise ratio (SNR). Analytical equations for scheduling rate and average spectral efficiency are derived for both SD and SM modes in a 2×2 MIMO system. Simulation results well justify our analytical equations and also demonstrate that our proposed scheme can achieve higher scheduling rate when SNR is low and higher spectral efficiency when SNR is high.

Non-extractive fish abundance estimation with the aid of visual analysis has drawn increasing attention. Low frame rate and variable illumination in the underwater environment, however, makes conventional tracking methods unreliable. In this paper, a robust multiple fish tracking system for low-frame-rate underwater stereo cameras is proposed. With the result of fish segmentation, a computationally efficient block-matching method is applied to perform successful stereo matching. A multiple-feature matching cost function is utilized to give a simple but effective metric for finding the temporal match of each target. Built upon reliable stereo matching, a multiple-target tracking algorithm via the Viterbi data association is developed to overcome the poor motion continuity of targets. Experimental results show that an accurate underwater live fish tracking result with stereo cameras is achieved.

In this paper, we propose a novel vehicle tracking system under a surveillance camera. The proposed system tracks vehicles by using constrained multiple-kernel facilitated with Kalman filtering, and then continuously updates the position and the orientation by adopting a systematically built 3-D vehicle model in an evolutionary computing framework. The proposed system can thus successfully track vehicles under occlusion as facilitated by the obtained 3-D geometry of vehicles. Experimental results have shown the favorable performance of the proposed system, which can successfully track vehicles while maintaining the knowledge of 3-D vehicle geometry.

Kernel based trackers have been proven to be a promising approach for video object tracking. The use of a single kernel often suffers from occlusion since the available visual information is not sufficient for kernel usage. In order to provide more robust tracking performance, multiple inter-related kernels have thus been utilized for tracking in complicated scenarios. This paper presents an innovative method, which uses projected gradient to facilitate multiple kernels, in finding the best match during tracking under predefined constraints. The adaptive weights are applied to the kernels in order to efficiently compensate the adverse effect introduced by occlusion. An effective scheme is also incorporated to deal with the scale change issue during the object tracking. Moreover, we embed the multiple-kernel tracking into a Kalman filtering-based tracking system to enable fully automatic tracking. Several simulation results have been done to show the robustness of the proposed multiple-kernel tracking and also demonstrate that the overall system can successfully track the video objects under occlusion.

Distributed camera networks have been deployed in the modern surveillance systems. The camera link model, including transition time distribution and brightness transfer function that represent the space-time relationship and color model between two cameras, is a critical element for tracking objects across the cameras. In this paper, we formulate the estimation of the camera link model as an optimization problem, where the deterministic annealing and the barrier method are applied to effectively extract the model parameters. Through the unsupervised scheme, our method utilizes the time stamps and color features together to establish the camera link model in presence of the outliers. Several simulations and comparative studies show the effectiveness of our approach.

In the human body tracking based on video sequences, the pose estimation of the upper/lower limbs is the most challenging task since the limbs possess most variations of motions and are easily occluded. In this work, we present a sophisticated scheme to track the human limbs. First, the tracking is formulated as a constrained optimization problem with multiple kernels. The color features of the upper/lower limbs are used as the control variables in the objective function. Moreover, the inequality constraints are imposed to control the angle between the arm/forearm or upper/lower legs during tracking. Finally, the gradient projection algorithm is adopted to solve the optimization problem with inequality constraints. The proposed scheme is implemented and experimented on HumanEva dataset and self-recorded video sequences including tracking of arm/forearm and upper/lower legs.

In this paper, a joint uplink (UL) and downlink (DL) framework for wireless mobile camera networks over an OFDMA-based infrastructure is proposed. On the UL, this system collects unicast real-time video streams from mobile wireless camera stations (CSs) and forwards them to a control center. On the DL, the aggregated video streams are multicast to multiple mobile stations (MSs) to facilitate monitoring of the same video scenes. A target bit rate is set for each UL video stream, based on an objective function that includes both the quality of experience (QoE) and popularity of video contents as key criteria. A QoE-driven scheduling and resource allocation policy at the MAC/PHY layers, and a real-time scalable video layer adaptation algorithm in the APP layer are proposed. On the DL, a previously published opportunistic layered multicasting scheduling algorithm is applied. Simulation results demonstrate that this proposed novel architecture can significantly enhance both the spectral efficiency and the QoE of users in both UL and DL directions.

We propose Opportunistic Layered Multicasting (OLM), a joint user scheduling and resource allocation algorithm that provides enhanced quality and efficiency for layered video multicast over Mobile WiMAX. This work is a lead off and complete synergy of layered video multicasting with opportunistic concept. The target application is characterized by groups of users acquiring popular video programs over a fading channel. To accommodate various bandwidth requirements and device capability, video streams are coded into base and enhancement layers using scalable video coding technology. Correspondingly, the optimization problems, which select the best subset of users to receive a specific video layer and assign the most appropriate modulation and coding scheme for this video layer, are specifically formulated for both video layer types. We also design fast and effective algorithms to bridge the gap between theoretical throughput capacity and implementation concerns. Thus, the basic video quality can be efficiently guaranteed to all subscribers while creating most utility out of limited resources on enhancement information. To overcome the inevitable packet loss in a multicast session, an FEC rate adaptation scheme to approach theoretical performance is also presented. Favorable performance of the proposed algorithms is demonstrated by simulations utilizing realistic Mobile WiMAX parameters.

Multicast broadcast service (MBS) is one of the important features supported by Mobile WiMAX to efficiently transmit data common to a group of users. As MBS services are usually delay-sensitive applications, the concept of the MBS zone is also introduced to provide better quality of service (QoS) for mobile users. However, the large inter- MBS zone handover delay and frame offsets between adjacent MBS zones cause large packet loss, but this problem is little studied. In this paper, we propose an improvement on the inter-MBS zone handover procedure to greatly reduce the handover delay. Furthermore, we also propose a data recovery scheme for inter-MBS zone handover by using an additional multicast connection as a recovery channel for each MBS service to minimize the packet loss. Simulation results show that our proposed schemes achieves almost zero packet loss when the number of recovery channels is the same with the number of MBS sessions.

In kernel-based video object tracking, the use of single kernel often suffers from the occlusion. In order to provide more robust tracking performance, multiple inter-related kernels have thus been utilized for tracking in complicated scenarios. This paper presents an innovative method that uses projected gradient to facilitate multiple kernels in finding the best match during tracking under predefined constraints. The adaptive weights are also applied to the kernels in order to efficiently compensate the adverse effect introduced by occlusion. An effective scheme is also incorporated to deal with the scale changing issue during the object tracking. Simulation results demonstrate that the proposed method can successfully track the video object under severe occlusion.

We propose a systematic cross-layer joint scheduling and resource allocation algorithm to provide enhanced quality of service (QoS) for real-time video uplink of camera networks over WiMAX. Under the constraint of limited available OFDMA resource, how to fairly schedule all mobile camera stations (CSs) with varying channel quality and maintain system performance simultaneously is always a critical issue for practical QoS design of a WiMAX base station (BS). The proposed system uses video encoding rate adaptation in application (APP) layer of CSs and effectively incorporates with the medium access control (MAC) QoS mechanisms of the BSs to greatly improve the spectrum efficiency of WiMAX.

This paper describes an automatic segmentation algorithm for fish sampled by a trawl-based underwater camera system. To overcome the problem caused by very low brightness contrast between fish and their underwater background with dynamically changing luminance, our proposed algorithm adopts an innovative histogram backprojection procedure on double local-thresholded images to ensure a reliable segmentation on the fish shape boundaries. The thresholded results are further validated by area and variance criteria to remove unwanted objects. Finally, a post-processing step is applied to refine the segmentation. Promising results, as validated by expert-generated ground truth data, were obtained via our proposed algorithm.

Video forgery, also referred as video falsifying, is a technique for generating fake videos by altering, combining, or creating new video contents. For instance, the outcome of a 100 m race in the olympic game is forged as an example in this paper. We track objects and segment motions using a modified mean shift mechanism. The resulting video layers can be played in different speeds and from different reference points with respect to the original video. In order to obtain a smooth movement of target objects, a motion interpolation mechanism is proposed based on reference stick figures (i.e., a structure of human skeleton) and a video inpainting mechanism. The video inpainting mechanism is performed in a quasi-3-D space via guided 3-D patch matching. Interpolated target objects and background layers are then fused. The objective is to create a forged video, which is almost indistinguishable from the original video. We demonstrate the original and the forged videos in our Web site at http://member.mine.tku.edu.tw/www/TSMC09/. Although video forgery may create moral or legal issues, which is beyond the scope of this paper, our intension is to create special effects in video editing applications.

We present an effective real-time approach for automatically estimating 3D human body poses from monocular video sequences. In this approach, human body is automatically detected from video sequence, then image features such as silhouette, edge and color are extracted and integrated to infer 3D human poses by iteratively minimizing the cost function defined between 2D features derived from the projected 3D model and those extracted from video sequence. In addition, 2D locations of head, hands, and feet are tracked to facilitate 3D tracking. When tracking failure happens, the approach can detect and recover from failures quickly. Finally, the efficiency and robustness of the proposed approach is shown in two real applications: human event detection and video gaming.

In this paper, we propose a unified framework targeting at two important issues in a distributed wireless camera network, i.e., object tracking and network communication, to achieve reliable multi-object tracking over distributed wireless camera networks. In the object tracking part, we propose a fully automated approach for tracking of multiple objects across multiple cameras with overlapping and non-overlapping field of views without initial training. To effectively exchange the tracking information among the distributed cameras, we proposed an idle probability based broadcasting method, iPro, which adaptively adjusts the broadcast probability to improve the broadcast effectiveness in a dense saturated camera network. Experimental results for the multi-object tracking demonstrate the promising performance of our approach on real video sequences for cameras with overlapping and non-overlapping views. The modeling and ns-2 simulation results show that iPro almost approaches the theoretical performance upper bound if cameras are within each other's transmission range. In more general scenarios, e.g., in case of hidden node problems, the simulation results show that iPro significantly outperforms standard IEEE 802.11, especially when the number of competing nodes increases.

Internet Protocol Television (IPTV) is an emerging and killer application for WiMAX networks, which takes advantage of multicasting mechanisms to efficiently deliver the video data to groups of subscribers. In this paper, we propose a Reception-rate-tracking Opportunistic Multicast Scheduling (ROMS) to enhance the performance of scalable IPTV layered multicasting over WiMAX networks. The proposed ROMS jointly considers the subscriber scheduling and resource allocation to achieve resource efficient multicasting. We validate our solution through theoretical analyses and empirical simulations. Our study indicates that the proposed solution outperforms the Slot-throughput-tracking Opportunistic Multicast Scheduling (SOMS) in terms of resource efficiency for 19.6%-33% in different subscriber group sizes.

A wireless resource scheduling problem is formulated in the context of IPTV multicasting streaming services over WiMAX channels. The goal is to maximize quality of services, in terms of number of subscribers served, number of IPTV channels carried, and video qualities of individual channels; subject to constraints of finite amount of multicast channel capacities as well as time and spatially varying wireless channel qualities. While a globally optimal solution is NP-hard, we present an efficient heuristic algorithm that exploits the structure of the problem formulation to yield solutions that are very close to the optimal solution.

In this paper, a receiver driven overlap Forward Error Correction (FEC) scheme, which offers unequal importance protection of video layers, is proposed to protect H.264/SVC video over the Internet. A mathematical model is provided to calculate video quality in terms of subjective Video Quality Metric (VQM). With the provided mathematical model, the video server prepares for receivers a lookup table, which lists all optimal combinations of video and FEC layers under various available downlink bandwidth and packet drop rates. By table lookup, a receiver can determine an optimal combination of video and FEC layers to subscribe to maximize the subjective video quality based on its available downlink bandwidth and packet drop rates.

In this paper, we propose a fully automated approach for tracking of multiple objects across multiple cameras with overlapping and non-overlapping views in a unified framework without initial training. For single camera cases, Kalman filter and adaptive particle sampling are integrated for multiple objects tracking. When extended to multiple cameras cases, the relations between adjacent cameras are learned systematically by using image registration techniques for consistent handoff of tracking-object labels across cameras. In addition, object appearance measurement is employed to validate the labeling results. Experimental results demonstrate the performance of our approach on real video sequences for cameras with overlapping and non-overlapping views.

We present an effective real-time approach for automatically reconstructing 3D human body poses from monocular video sequences. In this approach, human body is automatically detected from video sequence, then image features such as silhouette, edge and color are extracted and integrated to infer 3D human poses in an iterative way by minimizing the cost function defined between 2D features from the projected 3D model and image sequence. After convergence, the reconstruction result is evaluated for detecting tracking failure, which can be quickly recovered by adjusting initial pose to restart the minimization procedure. The results show the efficiency and robustness of the proposed approach.

We propose a resource allocation algorithm that provides enhanced QoS and efficiency for layered video multicast over mobile WiMAX. The application is characterized by groups of users acquiring popular video programs over a fading channel. To accommodate various bandwidth requirements and adaptive perceptual quality to different users, video streams are coded into base and enhancement layers using scalable video coding technology. Correspondingly, the optimization problems, which assign the most appropriate modulation schemes to provide best video quality to as many users, can be formulated using opportunistic multicasting concept. We maximize the minimum effective throughput across all users for mandatory (base) layer delivery through adapting modulation and coding schemes. At the same time, optional (enhancement) layers are allocated to maximize total utility. Thus the basic video quality can be efficiently guaranteed to all subscribers while making most out of limited resources on enhancement information. We further present an FEC rate adaptation scheme to approach theoretical performance. Favorable performance of the proposed algorithms is demonstrated by simulations utilizing realistic Mobile WiMAX parameters.

We propose a distributed cross-layer congestion control algorithm that provides enhanced quality of service QoS and reliable operation for real-time uplink video over WiFi applications. Such applications are characterized by many wireless devices transmitting video at various PHY rates over a relatively congested channel. Unfortunately, today's off-the-shelf 802.11 equipment can be easily demonstrated to suffer catastrophic failure when subject to these conditions-let alone provide acceptable perceptual quality to the user. We show that in order to remedy these issues, it is preferable to apply airtime fairness with a cross-layer approach. The idea is to use a fast frame-by-frame control loop in the carrier sense multiple access/collision avoidance (CSMA/CA)-based medium access control (MAC) layer while simultaneously exploiting the powerful control loop gain attainable by performing source-rate adaptation in the application layer. We support the proposed algorithm through both simulation and experimentation with various channel and PHY rate scenarios.

Image inpainting or image completion is the technique that automatically restores/completes removed areas in an image. When dealing with a similar problem in video, not only should a robust tracking algorithm be used, but the temporal continuity among video frames also needs to be taken into account, especially when the video has camera motions such as zooming and tilting. In this paper, we extend an exemplar-based image inpainting algorithm by incorporating an improved patch matching strategy for video inpainting. In our proposed algorithm, different motion segments with different temporal continuity call for different candidate patches, which are used to inpaint holes after a selected video object is tracked and removed. The proposed new video inpainting algorithm produces very few ldquoghost shadows,rdquo which were produced by most image inpainting algorithms directly applied on video. Our experiments use different types of videos, including cartoon, video from games, and video from digital camera with different camera motions. Our demonstration at http://member.mine.tku.edu.tw/www/T_CSVT/web/shows the promising results.

To eliminate packet loss accumulation and overcome bursty packet loss problems caused by peer departures in peer-to-peer (P2P) streaming systems, a multisource structure combining with a distributed forward error correction (FEC) scheme is proposed. In the proposed structure, each peer connects to multiple parents according to the prespecified FEC packets ensemble and each parent forwards partial streaming packets to the peer. If one or few parents fail, other parents can still provide remaining part of streaming packets that can be used to recover the missing packets by using packet level FEC scheme. Packet loss probability and packet loss accumulation from parent peers to child peers in tree-based P2P streaming systems are investigated. The analysis and NS2 simulation results show that packet loss probability is reduced and furthermore the packet loss accumulation can be eliminated when an appropriate FEC protection is used.

The attention analysis of multimedia data is challenging since different models have to be constructed according to different attention characteristics. This paper analyzes how people are excited about the watched video content and proposes a content-driven attention ranking strategy which enables client users to iteratively browse the video according to their preference. The proposed attention rank (AR) algorithm, which is extended from the Google PageRank algorithm that sorts the websites based on the importance, can effectively measure the user interest (UI) level for each video frame. The degree of attention is derived by integrating the object-based visual attention model (VAM) with the contextual attention model (CAM), which not only can more reliably take advantage of the human perceptual characteristics, but also can effectively identify which video content may attract users' attention. The information of users' feedback is utilized in re-ranking procedure to further improve the retrieving accuracy. The proposed algorithm is specifically evaluated on broadcasted baseball videos.

This paper presents a hierarchical lane detection system with the ability to deal with both structured and unstructured roads. The proposed system classifies the environment first before applying suitable algorithms for different types of roads. For environment classification, pixels with lane- marking colors are extracted as feature points. Eigenvalue decomposition regularized discriminant analysis is utilized in model selection and maximum likelihood estimation of Gaussian parameters in high dimensional feature space. For structured roads, the extracted feature points are reused for lane detection. For unstructured roads, mean-shift segmentation is applied to divide the scene into regions. Possible road boundary candidates are selected, and Bayes rule is used to choose the most probable boundary pairs. The experimental results show that the system is able to robustly find the boundaries of the lanes on different types of roads and various weather conditions.

In this paper, we propose a scheme for peer-to-peer (P2P) live streaming with multi-source multicast and forward error correction. In our scheme, there is a control topology for membership management, and a multi-source multicast tree for data delivery. The control topology facilitates peers to locate multiple sources for media content, and the multisource multicast tree make the system adaptive to node churn and packet loss. Simulation results show that the performance of our proposed method is significantly better than that of BitTorrent-Like (BT-Like) systems.

We propose a distributed cross-layer congestion control algorithm that provides enhanced QoS and reliable operation for real-time uplink video over WiFi applications. Such applications are characterized by many wireless devices transmitting video at various PHY rates over a relatively congested channel. Unfortunately, today's off-the-shelf 802.11 equipment can be easily demonstrated to suffer catastrophic failure when subject to these conditions - let alone provide acceptable perceptual quality to the user. We show that in order to remedy these issues, it is preferred to use a cross-layer approach rather than a single-layer approach. The idea is to use a fast frame-by-frame control loop in the MAC layer while simultaneously exploiting the powerful control-loop gain attainable by performing source-rate adaptation in the APP layer. We support the proposed algorithm through both simulation and experimentation.

In this paper, we propose a hierarchical push-pull scheme for peer-to-peer (P2P) live streaming. In our scheme, a control topology is used for membership management, and a multi-source multicast tree for data delivery. The control topology facilitates peers to locate multiple sources for forwarding media content, and the multi-source multicast tree makes the system adaptive to node churn and packet loss. Simulation results show that our proposed method is significantly better than BT-Like systems.

We present an effective congestion and error control mechanism for scalable video coding (SVC) extension of the H.264/AVC video dissemination over Internet. The congestion control is used to determine the appropriate number of SVC video layers based on bandwidth inference congestion (BIC) control protocol for layered multicast scenarios and the error control is achieved by unequal forward error correction (FEC) layered protection using block erasure coding. Through the real Internet streaming experiments, we demonstrate the effectiveness of the proposed layered SVC delivery, in terms of subscription layer, average packet loss rate and PSNRs, under several layered-definition scalabilities.

In this paper, we propose a multi-source multicast structure combining with a distributed FEC scheme to overcome burst packet loss problems caused by peer departures in P2P live streaming systems. The analytical results show that the burst packet loss can be eliminated by using multi-source structure with appropriate FEC parameters. A prototype system is implemented to verify that the proposed system is practically feasible and effective. The experimental results show the proposed system can effectively overcome the burst packet loss by taking advantage on the multiple source structure with the distributed FEC scheme.

Traditional approaches to receiver-driven layered multicast apply coarse-grain layered congestion control, which enables each receiver to adjust its receiving rate to match the available bandwidth. However, this restricts the scalability of fine-grain scalable video coding. In this paper, we present an approach that facilitates bandwidth inference congestion control in a hybrid coarse/fine layered multicast scheme for multimedia delivery. One-way delay trend detection has proven effective in explicitly or implicitly estimating end-to-end available bandwidth. Using a hierarchical layered probing scheme and a delay trend detection method, we have developed a congestion control protocol for fine-grain layered multicast. Combining this protocol with scalable video coding, we present a framework that achieves efficient scalable video streaming over heterogeneous networks. The performance of our hybrid coarse/fine layered multicast scheme is evaluated by network simulations and the video coding efficiency.

We propose a novel two-stage non-coherent SFH/BFSK detection scheme when the environment is interfered by an uncoordinated FH system. The first stage detects user number and power level of uncoordinated FH signals. Based on knowledge of detected side information from the first stage, the second stage performs SFH/BFSK maximum likelihood detection. When user number and power level of uncoordinated FH signals are unknown, the proposed two-stage detection scheme outperforms existing multilevel FSK conventional detectors and two-user multiuser detectors. In this scenario, simulation results demonstrate a critical role of the first-stage detector in the overall scheme when it provides the above side information. On the other hand, when knowledge of uncoordinated user number and power level is available to the desired user's receiver, simulation results also show superiority of the proposed second- stage detector over equal-gain receivers and self-normalized receivers.

A multiple-target tracking system aimed at analyzing crossroad traffic systematically is proposed in this paper. The proposed mechanism is based on Kalman filtering and modified probabilistic data association. Unlike traditional Kalman filtering tracking, the proposed mechanism constructs candidate measurement lists by matching the sizes of the measurements and the targets first. When the sizes do not match, object matching within a limited area is performed. Also, we modify the classical probabilistic data association method to enhance its performance and make it more suitable for vision-based systems. The proposed mechanism, which can serve as the foundation for automatic traffic event detection, can solve the occlusion problems effectively without incurring too much computational complexity.

In this paper we developed a system for automated human body tracking and modeling based on a monocular camera. In this system, eleven joint points including head, shoulder, hip, elbows, knees, hands and feet are extracted separately to build a 2D human body model. The head is extracted by analyzing negative minimum curvature (NMC) points on a parameterized silhouette. The torso, along with its angle and size, is determined by integrating multiple frame information with connectivity constraint. Hands and feet can be identified correctly based on a modified star skeleton approach and the nearest-neighbor tracking mechanism. The rest of joint points can also be located by taking advantage of the connectivity constraints. A successful construction of the proposed human body modeling will pave a critical foundation for further intelligent analysis in many applications, such as automated video surveillance system or systematic video understanding.

Image in-painting or image completion removes objects from a photo and automatically produces a visually pleasant result. However, to remove objects from a video, the resulting video may have ghost shadows even each individual frame is in-painted properly. We use motion estimation algorithm to separate objects and backgrounds into several layers. Objects in separated layers are in-painted from back to front layers, with a consideration of the temporal continuity of motion segments among different frames. The resulting video is visually more pleasant with most ghost shadows removed. Interested readers are welcome to look at our demonstration Website at http://www.mine.tku.edu.tw/demo.

We present a framework for analyzing the human body 3D motion of golf swing from single-camera video sequences. The system is different from the methods in the literature as it evaluates the 3D model for each major body part separately to derive a more accurate 3D representation. The human body parts used for the analysis are automatically extracted using a video object segmentation technique. This 2D information estimated from the segmented body parts is utilized to obtain 3D body part models, consisting of head, upper arms, lower arms, body trunk, upper legs, lower legs and feet. The objective of the system is to obtain the 3D motion information for performance evaluation in golf swinging and comparing this information for different players.

In this paper, we propose new video attention modeling and content-driven mining strategies which enable client users to browse the video according to their preference. By integrating the object-based visual attention model (V'AM) with the contextual attention model (CAM), the proposed scheme not only can more reliably take advantage of the human perceptual characteristics but also effectively discriminate which video contents may attract users' attention. In addition, extended from the Google PageRank algorithm which sorts the websites based on the importance, we introduce the so-call content-based attention rank (AR) to effectively measure the user interest (UI) level of each video frame. The information of users' feedback is treated as the enhanced query data to further improve the retrieving accuracy. The proposed algorithm is evaluated on commercial baseball game sequences and produces promising results.

Voice over IP (VoIP) has become the fastest growing wireless alternative to conventional telephony service by way of ongoing deployment of WLAN hotspots and even powerful WiMAX coverage. Resulting from the wired/wireless combined best-effort based heterogeneous IP networks which provide more fluctuation in available bandwidth and end-to-end delay, the performance of VoIP quality, especially using the handheld wireless devices, has been greatly degraded due to frequent packet loss and longer delays. This paper proposes a real-time embedded packet train probing scheme for estimating end-to-end available bandwidth so as to accomplish effective congestion control. By trading acceptable delays with adaptive packetization of voice bitstreams, as well as adaptive insertion of forward error correction (FEC) packets, an optimized system driven QoS approach for VoIP can thus be achieved.

We present an algorithm for analyzing the human body 3-D motion of golf swing from single-camera video sequences. As the first step, the human body used for the analysis is automatically extracted using a video object segmentation technique. Once human body is extracted, this two-dimensional information is utilized to obtain a three-dimensional body model consisting of head, upper arms, lower arms, body trunk, upper legs, lower legs and feet using an iterative 3-D fitting algorithm and Dynamic Bayesian Networks. The ultimate objective of the system is to obtain the 3-D motion information for golf swinging and comparing this information for different players regardless of the great variability caused by different camera viewing perspectives. 3-D body motion during the golf swing is computed for all segments of the body. This system will allow the spatial-temporal relationship of each body segment, as they make their transition, be thoroughly studied, and enable the parameters for different players to be compared, as well.

We developed a system for automatic 3-D human body motion in video surveillance sequences. As the first step, the human body that will be used for the analysis is extracted and tracked based on video object (VO) segmentation technique. Once human body is extracted, this 2-D object, which consists of the merging of several regions, is then analyzed for activity and body part interpretation. The specific 2-D human body shape along with the information derived from maximum probable explanation (MPE) of dynamic Bayesian modeling is then effectively utilized to obtain 3-D body model frame-by-frame so that the 3-D motion trajectory of the human object can be derived for any future event analysis, which is independent of the viewing perspective of the 2-D video

In a best-effort networking environment, efficient and fair congestion control is highly desired for every traffic flow to share the bandwidth appropriately. This paper proposes a congestion control algorithm for user datagram protocol rate-based layered streaming of scalable video, e.g., 3-D wavelet based scalable video streaming, which provides a variety of video bit rates. This proposed congestion control mechanism, as an extension of explicit control protocol that is a newly proposed congestion control protocol believed to be superior to transport control protocol, accommodates both window-based and rate-based flows to the heterogeneous network environment which can include wired and wireless channels. This paper further introduces the notion of reserved packet length so that the traffic of layered video can better share the bandwidth of a network by taking account of the max-min fairness with other traffic

In a wireless network environment, common channel errors, due to multipath fading, shadowing and attenuation, may cause bit errors and packet loss quite different from the packet loss caused by network congestion. In congestion control, the packet loss information can serve as an index of network congestion for effective rate adjustment; therefore, wireless packet loss can mistakenly lead to dramatic performance degradation. The paper proposes a packet loss classification algorithm based on detecting the trend in relative one-way trip time (ROTT) when it falls in the ambiguous zone where the packet loss classification is not straightforward. We show that the proposed algorithm greatly benefits rate-based congestion control algorithms for multimedia over IP networks.

In this paper, we present a novel video content analysis system. An innovative 2D to 3D parameter inference algorithm is presented. It is applied to the tennis player body shape modeling, after a coarse-to-fine analysis on real world sports video sequences. As the first step, the video shots are classified in coarse level. Only shots containing appropriate body shape size are retained for the fine-level analysis. The fine-level analysis begins with a video object (VO) segmentation stage to obtain the player body shapes. The VO then undergo training and testing stages. The training VO are classified into serving and non-serving classes by Gaussian mixture modeling (GMM). The VO in serving class are further clustered and the corresponding 3D parameters of a human body model are obtained manually for each cluster center. For a testing VO sequence, the VO that contain servings are found by GMM and the initial 3D parameters are fitted to the closest matches to the cluster centers. Based on the initial guess, an innovative multidimensional optimization procedure is employed to obtain the 3D parameters. Experiments are performed on broadcast tennis games and promising results are obtained.

The variable block size motion estimation (VBSME) technique can improve the coding efficiency greatly and has been adopted into the latest international video coding standard H.264/AVC. However, the VBSME technique introduces excessive computations comparing to fix block size ones. In H.264/AVC, the possible macroblock partition modes can have up to 259 different combinations, which make the design of an efficient VLSI architecture for H.264/AVC a challenging task. Many VLSI designs for the general motion estimation modules were presented in past decades, and some of them were also implemented as commercial products. Thereby, instead of developing a newly motion estimation module for H.264/AVC, this paper proposes an embedded merging scheme to extend the applicability of existing VLSI architectures. More specifically, the proposed embedded merging scheme only adds a few components into the existing VLSI architecture modules but can empower them to deal with the new H.264/AVC applications.

In wireless communication, noise and channel fluctuation often cause bit errors and subsequently the packet loss, which has different characteristic from the loss from network congestion. For most congestion control algorithms, the packet loss information serves as an index of network congestion and is used for effective rate adjustment; therefore wireless packet loss can mistakenly lead to dramatic performance degradation. We discuss the reasons leading to packet loss and propose a packet loss classification (PLC) algorithm that is based on trend detection of relative one-way trip time. With the assistance of PLC, not only can the wireless loss be separated from the congestion loss, the wireless packet error rate can be also estimated. With the combined information, the new adaptive congestion control algorithm is constructed so that a receiver can acquire an appropriate share of bandwidth. The transmitted data is protected by adaptive maximum distance separable erasure codes according to the wireless channel condition and end-to-end available bandwidth

A VideoGIS system combining geo-referenced video information with conventional geographic information (GI) is developed to provide a more comprehensive understanding over a spatial area. In our on-going project, the hypermedia can be transmitted to GPS-guided vehicles in a scalable (layered) fashion while providing highlighting and tracking of landmark objects on video upon drivers' request. Special efforts on GPS error calibration and target objects tracking are reported in this paper.

In this paper, a real-time system to create a talking head from a video sequence without any user intervention is presented. In the proposed system, a probabilistic approach, to decide whether or not extracted facial features are appropriate for creating a three-dimensional (3-D) face model, is presented. Automatically extracted two-dimensional facial features from a video sequence are fed into the proposed probabilistic framework before a corresponding 3-D face model is built to avoid generating an unnatural or nonrealistic 3-D face model. To extract face shape, we also present a face shape extractor based on an ellipse model controlled by three anchor points, which is accurate and computationally cheap. To create a 3-D face model, a least-square approach is presented to find a coefficient vector that is necessary to adapt a generic 3-D model into the extracted facial features. Experimental results show that the proposed system can efficiently build a 3-D face model from a video sequence without any user intervention for various Internet applications including virtual conference and a virtual story teller that do not require much head movements or high-quality facial animation.

In a best-effort networking environment, efficient and fair congestion control is highly desired for every traffic flow, to share the bandwidth appropriately. This paper proposes a congestion control algorithm for UDP based layered video, whose bandwidth resolution in each layer has been predefined. This proposed congestion control mechanism is an extension of XCP, which is a newly proposed protocol believed to be superior to TCP, especially for high bandwidth-delay product networks. This paper also introduces reserved packet length so that the layered video traffic can share the bandwidth of a network better with the consideration of max-min fairness to other traffic.

Digital video recording (DVR) systems have become popular following the rapid development of digital video devices and video coding techniques. A DVR is a computerized surveillance system that supports all the features of a traditional videotape system by using digital cameras, motion detector, pan-and-tilt controls, capture cards and a computer. It can save the captured frames as hard-disk files, which can be played back later. Another advantage of DVR is that it supports realtime remote display of the captured video frames while the digital recording is working at the same time. We present research on the realtime remote display of a DVR system, especially the congestion control algorithm for the multi-channel transmission case. Our proposed congestion control algorithm can achieve efficient and stable realtime transmission of multichannel captured video sequences for a wide range of bandwidth capacity. The algorithm is also TCP-friendly as it follows the additive increase multiplicative decrease (AIMD) scheme. Several trade-offs are made to make the system fit for the special purpose of remote surveillance.

A VideoGIS system aims at combining georeferenced video information with traditional geographic information in order to provide a more comprehensive understanding over a spatial location. Video data have been used with geographic information in some projects to facilitate a better understanding of the spatial objects of interest. We present an on-going VideoGIS project, in which scalable georeferenced video and geographic information (GI) are transmitted to GPS-guided vehicles. The hypermedia, which contains cross-referenced video and GI, are organized in scalable (layered) fashion. The remote user can request, through 3G mobile devices, the abundant information related to the objects of interest, while adapting to heterogeneous network condition and other factors such as display area size, CPU processing power, etc.

We present a novel fully automatic moving video-object extraction algorithm based on graph partitioning and object tracking. A sophisticated graph partitioning algorithm is first used to (intra-frame) segment the moving video objects (VOs) of a specific (key) frame without any prior knowledge of segmentation results of previous frame(s). Once the VOs for this key frame are extracted, the (inter-frame) segmentation of VOs of subsequent frames is accomplished by tracking the segmented objects of consecutive frames by means of motion projection and modified histogram back-projection. To avoid significant error propagation caused by the (inter-frame) object tracking, a new key frame has to be identified periodically, based on a criterion in terms of camera motion parameters. Simulation results illustrate that the proposed algorithm gives comparable results to semi-automatic methods introduced in the literature.

Quality of service has been a great concern to video dissemination over the Internet, especially due to the heterogeneous networking environment. In contrast to the traditional passive networking, active networking by means of active routers/agents in a wide area network shows promise of better video service by offering packet control at finer resolution. This paper proposes an effective architecture based on active router and selective drop queue management as a solution to video unicast and multicast in a variable network environment.

This paper presents an approach of router management that is easy to deploy and can improve the performance of existing layered video schemes. The router is configured to selectively drop a packet of the same application instance from its queue when the network is congested, which may be caused by either network dynamics or failed join experiment. Compared with uniform drop, this application level selective drop (ALSD) can increase the received video quality and provide a more stable subscription level for the narrow-bandwidth receivers competing on the same bottleneck link with high-bandwidth receivers. We evaluate the promising performance of the proposed ALSD algorithm with multiple layered video schemes through network simulations.

A dynamic Bayesian network (DBN) based framework to model video sequences is proposed. The video sequences of interest include single-shot video sequences containing only one event and multishot video sequences containing various events. By taking advantage of the temporal continuity of video sequences and assuming Markovian property between successive image frames, we propose DBNs as the tool to map low-level features to high-level concepts. The feasibility of DBN modeling is tested on single-and multishot video sequences. Specifically, a coarse-grained video interpretation framework based on one kind of DBN, the hierarchical hidden Markov model (HHMM), is proposed for multishot video sequences. For single-shot video sequences, we present a fine-grained object based interpretation and classification system based on another version of DBNs. The preliminary simulations show great promise on the efficiency and flexibility of using DBNs for video sequence modeling.

Layered multicast is a promising technique for disseminating adaptive-quality audio/video to multiple heterogeneous receivers. Congestion control in a layered multicast scheme is very important to support heterogeneity and scalability. Previous works on congestion control use packet loss, delay, and receiving rate to infer whether there is spare capacity along the path, which suffers from slow convergence, lack of inter-session fairness or TCP-fairness, layer oscillations, and loss induced by the join experiments. In this paper, we propose a layered multicast bandwidth inference congestion (BlC) control, which use delay increasing trend detection to infer the spare capacity. The major contribution of this paper is introducing the source probe organization and effective spare capacity inference by delay trend detection algorithm. We evaluate BIC for a large variety of scenarios and show that it converges fast to the optimal link utilization and can adapt to network dynamics effectively. We also show that BIC is stable, inter-session fair and fair to competing TCP traffics.

Congestion control is an important function in a multimedia application. Since multimedia packets are mostly transmitted in UDP, which lacks the congestion control mechanism in TCP, a separate congestion control is necessary to avoid excessive packet losses and to be fair with competing TCP flows. As UDP congestion control is rate-based, unlike the window-based one used in TCP, end-to-end available bandwidth estimation can play a critical role in UDP congestion control. This paper proposes a new method to estimate the available bandwidth based on the delay trend detection, which is suitable for a multimedia application. We deduct the relationship between delay trend and available also show the Internet experiment results. Various practical time measurement errors are analyzed and their countermeasures are proposed.

Wilh the rapid deployment of videoconference, the fast-accumulated personal videoconference archives need to be effectively indexed. This paper proposes a well-designed indexing system - PVCAIS that integrates many multimedia-indexing techniques to manage personal videoconference archives. Firstly, the contents of video, audio, text and whiteboard communications are stored after removing the redundancies. Next, more information, e.g., participants, title, keywords and slides, is extracted by face detection and recognition, speech recognition, OCR, automatic title generation, keyword selection, etc. Then, an XML index file containing the summary of the videoconference is also generated. The whole indexing process is automatic except that the face of a new contact needs to be interactively identified for only once. Finally, we demonstrate a graphical user interface which allows the user to search and browse the indexed videoconference archives conveniently.

This paper presents an algorithm that estimates the spatial distribution and temporal evolution of snow water equivalent and snow depth based on passive remote sensing measurements. It combines the inversion of passive microwave remote sensing measurements via dense media radiative transfer modeling results with snow accumulation and melt model predictions to yield improved estimates of snow depth and snow water equivalent, at a pixel resolution of 5 arc-min. In the inversion, snow grain size evolution is constrained based on pattern matching by using the local snow temperature history. This algorithm is applied to produce spatial snow maps of Upper Rio Grande River basin in Colorado. The simulation results are compared with that of the snow accumulation and melt model and a linear regression method. The quantitative comparison with the ground truth measurements from four Snowpack Telemetry (SNOTEL) sites in the basin shows that this algorithm is able to improve the estimation of snow parameters.

In this paper, a real-time system for automatic creation of 3D face models from a video sequence is presented. The system consists of a novel scheme to extract face shape and a 3D model adaptation method based on a least-square approach. The proposed face shape extractor, which is accurate and computationally cheap, is based on an ellipse controlled by three anchor points. The least-square approach is selected to adapt a generic 3D model to input face, minimizing errors between feature points on a created 3D face and extracted feature points from input video. Experimental results show that the proposed system can efficiently builds 3D face models, which are compatible with MPEG-4 facial objects, from a video sequence without any user intervention.

In this paper, we present a novel scheme for object-based video analysis and interpretation based on automatic video object extraction, video object abstraction, and semantic event modeling. In this scheme video objects (VOs) are first automatically extracted, followed by a video object abstraction algorithm for identifying key frames to reduce data redundancy and provide reliable feature data for next stage of the algorithm. After the semantic objects, which are VOs that dominate the semantics in a video shot, a:re identified (automatically or selected by users), all the other objects in the video shot are considered as background. Semantic feature modeling scheme is based on temporal variation of low-level features in semantic object area. More specifically, the general Dynamic Bayesian Network (DBN) is used to characterize the spatial-temporal nature of the semantic objects. Experimental results that demonstrate the effective performance of the proposed approach are also presented.

We present a probabilistic approach to decide whether or not extracted facial features are appropriate for creating 3D face models. Automatically extracted 2D facial features from a video sequence are fed into the proposed probabilistic framework before a corresponding 3D face model is built to avoid generating unnatural or non-realistic 3D faces. In addition, a new algorithm for audio-to-visual conversion based on constrained optimization is presented to generate visual parameters for driving the mouth movement of the 3D face models from speech. Lagrangian optimization is applied to transform a constrained problem into an unconstrained problem. Experimental results are provided to show the effectiveness and validity of the proposed algorithms for various video sequences and speech.

This paper presents an algorithm that estimates the spatial distribution as well as temporal evolution of snow water equivalent and snow depth based on passive remote sensing measurements. It combines the inversion of passive microwave remote sensing measurements with snow accumulation and melt model predictions to yield improved estimates of snow depth and snow water equivalent, at a pixel resolution of 10 km. In the inversion, snow grain size evolution is constrained based on pattern matching by using the local snow temperature history. This algorithm is applied to produce spatial snow map of Upper Rio Grande basin in Colorado. The simulation result of this algorithm is compared with that of snow hydrology model and linear regression method. The comparison with the ground truth measurements from 4 SNOwpack TELemetry (SNOTEL) sites in the basin shows that this algorithm is able to improve the estimates of snow depth and snow water equivalent.

The new video-coding standard MPEG-4 enables content-based functionality, as well as high coding efficiency, by taking into account shape information of moving objects. A novel algorithm for segmentation of moving objects in video sequences and extraction of video object planes (VOPs) is proposed . For the case of multiple video objects in a scene, the extraction of a specific single video object (VO) based on connected components analysis and smoothness of VO displacement in successive frames is also discussed. Our algorithm begins with a robust double-edge map derived from the difference between two successive frames. After removing edge points which belong to the previous frame, the remaining edge map, moving edge (ME), is used to extract the VOP. The proposed algorithm is evaluated on an indoor sequence captured by a low-end camera as well as MPEG-4 test sequences and produces promising results.

Key frames are the subset of still images which best represent the content of a video sequence in an abstracted manner. In other words, video abstraction transforms an entire video clip to a small number of representative images. We present a scheme for object-based video abstraction facilitated by an efficient video-object segmentation (VOS) system. In such a framework, the concept of a "key frame" is replaced by that of a "key video-object plane (VOP)." In order to achieve an online object-based framework such as an object-based video surveillance system, it becomes essential that semantically meaningful video objects are directly accessed from video sequences. Moreover, the extraction of key VOPs needs to be automated and context dependent so that they maintain the important contents of the video while removing all redundancies. Once a VOP is extracted, the shape of the VOP needs to be well described. To this end, both region-based and contour-based shape descriptors are investigated, and the region-based descriptor is selected for the proposed system. The key VOPs are extracted in a sequential manner by successive comparison with the previously declared key VOP. Experimental results on the proposed online processing scheme combined with efficient VOS show the proposed integrated scheme generates desirable summarizations of surveillance videos.

An automatic system for creating a virtual head that is compatible with MPEG-4 facial object specification is presented. Color classification and a valley detection filter are performed to find face and facial definition points (FDPs) at the initialization stage. Extracted FDPs are tracked by normalized correlation and their trajectories are fed into an extended Kalman filter (EKF) to recover camera geometry, facial orientation, and depth of selected FDPs. Based on a recovered point-wise 3D structure, Dirichlet free-form deformations (DFFD) is applied to deform a generic 3D model. Once a virtual head is created, the head can be used to track FDPs for large out-of-plane rotations and to update the head model continuously based on refined depth information. A complete texture map is created by mixing frontal and rotated faces based on the recovered face orientation.

Among various semantic primitives of video, objects of interest along with their actions and generated events can play an important role in some applications such as an object-based video surveillance system and an object based video indexing/retrieval system. In this paper, we propose an object-based video abstraction algorithm by cluster analysis using the mean shift algorithm. The generated clusters, called segments in this paper, can be used as a small unit in the object-based video indexing/retrieval systems. In the proposed algorithm, Hu's (1962) seven moments are used as shape descriptors for each video object plane (VOP), and shape distance between two VOPs is measured by using weighted Euclidean distance. Promising experimental results on the proposed scheme are presented.

The aims of this study are (1) to propose an image segmentation framework for multiple contrast weighting MR images; and (2) to analyze the agreement between segmentation results and histology sections. The proposed technique is actually based on the mean-shift density estimation algorithm and carefully designed to overcome the drawbacks in other existing methods. First, it has a very reliable and accurate initialization scheme that guarantees all potential clusters are within estimation. Secondly, the proposed method introduces a dynamic sphere mechanism that makes the mean-shift vector more reliable even with poor initialization. It can also enhance the accuracy of cluster center estimation when the searching processing is approaching the mode. Experimental results and comparison with histology sections demonstrate its encouraging performance. Moreover, the proposed approach can be easily extended to other general-purpose low-level image analysis problems.

To prevent the loss of the information embedded in the generated variable bit rate data that is transmitted over a constant bit rate channel, several methods were proposed in MPEG TMN5, H.263 TMN5, and TMN8. In these methods, the quantity of coded data is controlled by adjusting the coding parameters according to the amount of data remaining in the buffer. Because this control is based on the past coded information and does not reflect the nature of the image being coded, there is no assurance that sufficient image quality will be obtained. In this paper, we present a Scene Context Dependent coding scheme to control the generated variable bit rate data over a constant bit rate channel for non real-time video applications and show the improvements over the encoding using the TMN H263 codec. This can be considered as a method to control the bit rate in accordance with the characteristics of human visual perception using the combination of feedforward control, feed-backward control, and model-based approaches.

This paper investigates robust transmission of a coded video sequence over unreliable packet networks. A content-based error-resilient coding (CBERC) algorithm is proposed to prevent severe error propagation. Specifically, the CBERC encoder performs motion-compensated prediction by taking the video content into consideration. A criterion based on the worst-case scenario is also provided to determine appropriate CBERC threshold. The CBERC algorithm does not require a feedback channel and supports both point-to-point and multi-point communications. It is of low complexity and is compatible with all the widely accepted video coding standards such as H.263+, H.261, MPEG-1, MPEG-2 and MPEG-4.

This paper describes a snow parameter retrieval algorithm from passive microwave remote sensing measurements. The three components of the retrieval algorithm include a dense media radiative transfer (DMRT) model, which is based on the quasicrystalline approximation (QCA) with the sticky particle assumption, a physically-based snow hydrology model (SHM) that incorporates meteorological and topographical data, and a neural network (NN) for computational efficient inversions. The DMRT model relates physical snow parameters to brightness temperatures. The SHM simulates the mass and heat balance and provides initial guesses for the neural network. The NN is used to speed up the inversion of parameters. The retrieval algorithm can provide speedy parameter retrievals for desired temporal and spatial resolutions, Four channels of brightness temperature measurements: 19V, 19H, 37V, and 37H are used. The algorithm was applied to stations in the northern hemisphere. Two sets of results are shown. For these cases, the authors use ground-truth precipitation data, and estimates of snow water equivalent (SWE) from SHM give good results. For the second set, a weather forecast model is used to provide precipitation inputs for SHM. Additional constraints in grain size and density are used. They show that inversion results compare favorably with ground truth observations.

A new minimal path active contour model for boundary extraction is presented. Implementing the new approach requires four steps (1) users place some initial end points on or near the desired boundary through an interactive interface; (2) a potential searching window is defined between two end points; (3) a graph search method based on conic curves is used to search the boundary; and (4) a "wriggling" procedure is used to calibrate the contour and reduce sensitivity of the search results on the selected initial end points. The last three steps are performed automatically. In the proposed approach, the potential window systematically provides a new node connection for the later graph search, which is different from the row-by-row and column-by-column methods used in the classical graph search. Furthermore, this graph search also suggests ways to design a "wriggling" procedure to evolve the contour in the direction nearly perpendicular to itself by creating a list of displacement vectors in the potential window. The proposed minimal path active contour model speeds up the search and reduces the "metrication error" frequently encountered in the classical graph search methods e.g., the dynamic programming minimal path (DPMP) method.

In this paper, we present the design and development of a real-time interactive virtual classroom multimedia distance learning system at the University of Washington. There has been rapid progress in digital media compression research, and the delivery of media data on the public Internet is becoming widespread. A real-time interactive virtual classroom allows a remote participant to not only receive a live class feed, but also to interact in a live class by asking questions with audio, video in real-time using an Internet connection. Many instructors use electronically prepared slides during their class. The traditional video coding algorithms are not able to compress this slide data very well at low bit-rates. We propose a Web-based real-time presentation system for the electronic slides. Instructors also write text on a white board or a piece of paper during the class. At low bit-rates, conventional video encoding algorithms cannot encode this handwritten text video with enough fidelity, resulting in an illegible decoded video. We propose an extension of the well known bilevel image encoding algorithms to handle the handwritten text video. Our method results in decoded video frames which can be read very clearly when encoded at low bit-rates. We have developed a set of tools which allows recording the live classroom session and automatic creation of a synchronized multimedia integration language (SMIL) presentation, which can be used for a later viewing.

A video content based coding algorithm is presented to eliminate the error propagation effects caused by transmitting highly compressed video sequences over unreliable packet networks. We study the criteria to determine an appropriate threshold for it and conduct experiments in a bursty packet loss environment. Promising results are obtained in the preliminary simulations.

Fingerprint identification is one of the most popular and reliable personal biometric identification methods. This paper describes an on-line fingerprint identification system consisting of fingerprint image preprocessing, feature extraction and matching. The preprocessing part includes image enhancing steps to acquire binarized and skeletonized ridges, which are needed for feature points extraction. Feature points (minutiae) such as endpoints, bifurcations, and core point are then extracted, followed by false minutiae elimination. The fast and robust matching algorithm is proposed, which is a correlation-based method that runs over the 1/8 sized feature points maps.

Carotid artery vessel tracking and lumen segmentation is an important task for atherosclerotic plaque study, where the active contour model Snake has become one of the most powerful techniques for this purpose. However, its intrinsic weakness in initialization and topology changes handling limits its application in complicated situations. In this research, we focus our work on atherosclerotic blood vessel tracking and lumen contour segmentation of magnetic resource (MR) image sequences, along which the blood vessel bifurcates at indefinite position. To automatically capture this topology change in the processing, a practical solution is proposed by extending our previous research work. The procedure first presegments each MR slice into regions, and then uses a decision tree to track the blood vessel's topology change. Finally, the minimal path Snake (MPS) algorithm is applied to further search optimal lumen contours. Some experimental results in the preliminary study are provided to demonstrate its encouraging performance.

Magnetic resonance (MR) imaging and analysis have become one of the most important tools in medical research and clinical applications. To quantitatively evaluate the quality of MR images, some research work has been done previously. However, most of the research was only based on 2D spin echo images which can hardly be applied to atherosclerosis study. In this paper, we apply Shannon's information theory for optimal MR imaging and lesion index analysis. First, we review the information content (IC) which is used as a subjective criterion of image quality measurement in our work. Then we extend the existing spatial spectrum model to the 3D time-of-flight (ToF) imaging technique and find the optimal imaging resolution with which the maximum image information content can be obtained. A theoretical proof of its uniqueness is also given. At last, a lesion index system is proposed and developed for the blood vessel wall plaque analysis in our study. It is used to determine atherosclerotic lesion complexity and identify lesion changes over time. Some phantom and in vivo MR images are analyzed along with examples to demonstrate the performance.

In this paper, we present a novel scheme for object-based key-frame extraction facilitated by an efficient video object segmentation system. Key-frames are the subset of still images which best represent the content of a video sequence in an abstracted manner. Thus, key-frame based video abstraction transforms an entire video clip to a small number of representative images. The challenge is that the extraction of key-frames needs to be automated and context dependent so that they maintain the important contents of the video while remove all redundancy. Among various semantic primitives of video, objects of interest along with their actions and generated events can play an important role in some applications such as object-based video surveillance system. Furthermore, on-line processing combined with fast and robust video object segmentation is crucial for real-time applications to report unwanted action or event as soon as it happens. Experimental results on the proposed scheme for object-based video abstraction are presented.

This paper investigates the performance of ordered statistics decoding (OSD) of linear block codes with binary differential phase-shift-keying (2DPSK) transmission on frequency nonselective multipath channels. First, the wireless channel is modeled as a frequency nonselective, slow fading environment without intersymbol interference. Second, block interleaving is embedded to decorrelate the successive received symbols so that the performance of OSD, designed essentially for memoryless channels, is retained. Third, we derive the log-likelihood ratio of received symbols based on 2DPSK modulation. Fourth, we derive the statistics of received symbols and the statistics of received symbols after ordering. Further, we analyze the system error performance for sufficient degree of interleaving. Specifically, we derive the asymptotic bit error rate (BER) of OSD at high SNRs using the first-order approximation, finally, we conduct experiments at various channel characteristics and interleaving degrees for (128, 61, 22) extended BCH code and (24, 12, 8) extended Golay code. Results show that the error performance improves tremendously with increased interleaving degree.

This paper investigates the performance of ordered statistics decoding of linear block codes with binary differential phase-shift-keying (2DPSK) transmission on the wide-sense-stationary uncorrelated-scattering (WSSUS) Rayleigh fading channel. For a typical mobile speed of 60 mph, tropospheric scatter radio communication at a carrier frequency of 900 MHz and very low bit rate video communication at a transmission speed of 32 kbit/s, the channel is modeled as a frequency non-selective, slow fading environment without inter-symbol interference (ISI). At a bit error rate (BER) of 10/sup -5/, 34.5 dB and 38 dB gains compared to uncoded 2DPSK are obtained for the decoding of the (24,12,8) extended Golay code and the (128,64,22) extended BCH code with sufficient degree of interleaving.

In this paper, we propose a novel boundary detection approach for three-dimensional shape modeling. Our method is based on finding surfaces of minimal weighted area in a Riemannian metric. In order to take advantage of intensity information of images, we further integrate this intensity information into the boundary detection algorithm. We apply this algorithm to identify the inner and outer boundaries of the blood vessel wall in magnetic resonance images, and assess its accuracy and reproducibility. Our algorithm is reasonably accurate (about 2% difference in comparison with the manual method) and highly reproducible.

We aimed at training a neural network to classify stress test exercise data into one of three classes: normal, heart failure, or lung failure. Good classification accuracy was obtained using a backpropagation neural network architecture with one hidden layer during cross validation on a data set of 110 vectors, when all 17 channels were used. We further aimed at determining which of these channels were critical to the decision making process. This was done through an input sensitivity analysis. Results showed that nine channels formed a critical superset of which possibly any eight could achieve almost perfect classification. We thus show that faster and more accurate classification may be obtained by input channel elimination due to dimension reduction of input space, which makes better generalization.

The new video coding standard MPEG-4 is enabling content-based functionalities as well as high coding efficiency considering shape information of moving objects. A novel algorithm for segmentation of moving objects in video sequences and VOP (video object planes) extraction is presented. This algorithm begins with a robust double edge map from the difference between two successive frames. After removing edges which belong to previous frame, the edge map, named ME (moving edge) is used to extract VOP. The proposed algorithm is evaluated for MPEG-4 test sequences and produces promising results.

This paper investigates the bit-rate conversion and bit re-allocation for video combining in a Multipoint Control Unit (MCU). To combine the pre-encoded multiple video sequences into a single sequence, the total bit-rate may overwhelm the outgoing channel bandwidth. Instead of multiplexing, we apply the transcoding approach to control the bit-rate and reallocate the bits in order to improve the pictures quality. To speed up the operation, a video transcoder usually reuses the decoded motion vectors to reencode the video sequences. Those decoded motion vectors also serve as a good indication of the picture activities. In this paper we uniformly distribute the bit-rate to the combined sequence and then use the motion information and the Lagrange optimization method to reallocate the bits to each sub-picture. The simulation results show that the PSNR distribution of the sub-sequence is more uniform and the video qualities of the active subsequences are much improved.

A knowledge based system has been designed for identifying and measuring fibrous cap (FC) in magnetic resonance images (MRI). The proposed method consists of three basic procedures. First, we apply Markov random field (MRF) to segment the whole image into regions and automatically identify the inner boundary of FC based on prior knowledge of lumen area. An enhanced algorithm, called QHCF, is proposed to find the optimal region segmentation. In the second step, active contour model is employed in finding FC's outer boundary. In this step, we use a new scheme which uses radial searching to find the end points so as to automatically initialize the contour tracking process. Finally, a thickness transformation is designed by extending binary mathematical morphology operation. Based on it, all the thickness related parameters can thus be measured quantitatively and consistently. Through these three integrated steps, FC is identified and measured. Experimental result demonstrates the encouraging performance of this algorithm.

In this paper, a novel audio-to-visual conversion method is presented. Many multimedia applications, such as videophones, videoconferencing, man-machine interface, language dubbing, character animation in virtual reality, etc., require techniques for synchronizing audio and video in a synthesized talking head sequence. For these applications, it is necessary to reliably estimate accurate mouth (visual) movements from the corresponding speech (audio) data. The hidden Markov model inversion (HMMI) technique introduced for robust speech recognition is extended in this paper into the audio-visual feature space. Based on the Baum-Welch HMMI method, reliable visual parameters are extracted given speech data only. Our preliminary simulation results show that the estimated visual parameters from the proposed method match the true visual parameters smoothly as well as accurately. The proposed estimation technique can be combined with video coding and graphics techniques for other multimedia applications.

We present the design and the development of a real-time interactive virtual classroom. There has been rapid progress in digital media compression research and the delivery of media data on the public Internet is becoming widespread. A real-time interactive virtual classroom allows a remote participant not only to receive a live class feed but also to interact in a live class by asking questions. Many instructors use electronically prepared slides during their class. The traditional video coding algorithms are not able to compress this slide data very well at low bit rate. We propose a Web-based real-time presentation system for the electronic slides. Instructors also write text on a white board or a piece of paper during the class. At low bit rate, the conventional video encoding algorithms cannot encode this handwritten text video with enough fidelity, resulting in an illegible decoded video. We propose an extension of the well known bi-level image encoding algorithms to handle the handwritten text video. Our method results in decoded video frames which can be read very clearly when encoded at low bit rate.

With the popularity of MPEG-4 and MPEG-7 standards, video object (VO) segmentation becomes a very challenging research area in video applications. We present a novel method for semi-automatic object segmentation for a video sequence. The proposed approach starts with a rough user input VO definition. It then combines each frame's region segmentation and motion estimation results to construct the objects of interest for temporal tracking this object along the time. An active contour model based algorithm is employed to further fine-tune the object's contour so as to extract accurate object boundary. Some experimental results and future research directions are also discussed.

The MPEG video-compression standard effectively exploits spatial, temporal, and coding redundancies in the algorithm. In its generic form, however, only a minimal amount of scene adaptation is performed. Video can be further compressed by taking advantage of scenes where the temporal statistics allow larger interreference-frame distances. This paper proposes the use of motion analysis (MA) to adapt to scene content. The actual picture type [intracoded (I), predicted (P), or bidirectionally coded (B)] decision is made by examining the accumulation of motion measurements since the last reference frame (either I or P) was labeled. The proposed MA-based adaptive reference frame-placement scheme outperforms the standard fixed-reference frame-placement and adaptive schemes based on histogram of difference. When compared with the standard fixed scheme, depending on the video contents, this proposed algorithm can achieve from 2 to 13.9% savings in bits while maintaining similar quality.

An algorithm to incorporate passive microwave remote-sensing measurements within a spatially distributed snow hydrology model to provide estimates of the spatial distribution of snow water equivalent (SWE) as a function of time is implemented. A priori information provided by the snow hydrology model is used to provide initial estimates of snow parameters from brightness temperature measurements. The algorithm is illustrated by applying it to a mountainous region. The passive microwave remote-sensing measurements are 25-km resolution (grid). However, in mountain regions, the spatial variability of SWE over a 25-km grid is large due to topographic influences. On the other hand, the snow hydrology model has built-in topographic information and the capability to estimate SWE at a 1-km resolution. In their work, the snow hydrology SWE estimates are updated and corrected using Special Sensor Microwave/Imager (SSM/I) passive microwave remote-sensing measurements. The method is applied to the Upper Rio Grande River Basin in the mountains of Colorado. The change in prediction of SWE from hydrology modeling with and without updating is compared with measurements from two SNOwpack TELemetry (SNOTEL) sites in and near the basin. The results indicate that the method incorporating the remote-sensing measurements into the hydrology model is able to more closely estimate the temporal evolution of the measured values of SWE as a function of time.

We present a new fast motion estimation method useful for high speed video encoding. Most of the motion estimation methods for video coding can be classified as block matching (BM) methods or pel recursive (PR) methods. The majority of the current fast motion estimation methods belong to block matching category. These methods try to reduce the number of search locations. Our proposed method is based on the pel recursive formulation. However, in order to achieve fast estimation, we operate on a block of pixels using a total least squares (TLS) based estimation scheme which tries to estimate the true motion vector for each block. The major advantages of the proposed method include very fast estimation, almost constant time for motion estimation for all the video sequences, fractional pel accuracy, and better performance for noisy sequences. We present extensive simulation results to illustrate the performance of the proposed method.

Inverse remote sensing problems are generally ill-posed. In this paper, we propose an approach, which integrates the dense media radiative transfer (DMRT) model, snow hydrology model, neural networks and SSM/I microwave measurements, to infer the snow depth. Four multilayer perceptrons (MLPs) were trained using the data from DMRT model. With the provision of an initial guess from snow hydrology prediction, neural networks effectively invert the snow parameters based on SSM/I measurements. In addition, a prediction neural network is used to achieve adaptive learning rates and a good initial estimate of snow depth for inversion. Result shows that our algorithm can effectively and accurately retrieve snow parameters from these highly nonlinear and many-to-one mappings.

Soft-decision decoding of linear block codes for robust H.263 video transmission in a zero-mean, additive white Gaussian noise (AWGN) channel is investigated. We implement an effective error concealment (EC) scheme at the source decoder to reduce the annoying artifacts caused by decoding a corrupted bit stream. To alleviate the spatial and temporal error propagation, an error prevention (EP) strategy is introduced at the H.263 encoder. Simulation results show that a large portion of the peak signal-to-noise ratio (PSNR) gain is obtained by ordered statistics decoding of the received sequence. Furthermore, the residual channel coding errors are concealed and compensated by realizing the proposed EC and EP schemes.

This paper investigates the dynamic frame skipping strategy in video transcoding. To speed up the operation, a video transcoder usually reuses the decoded motion vectors to reencode the video sequences at a lower bit-rate. When frame skipping is allowed in a transcoder, those motion vectors can not be reused because the motion vectors of the current frame is no longer estimated from the immediate past frame. To reduce the computational complexity of motion vectors reestimation, a bilinear interpolation approach is developed to overcome this problem. Based on these interpolated motion vectors, the search range can be much reduced. Furthermore, we propose a frame rate control scheme which can dynamically adjust the number of skipped frames according to the accumulated magnitude of the motion vectors. As a result, the decoded sequence can present much smoother motion.

This paper reviews key attributes of neural processing essential to intelligent multimedia processing (IMP). The objective is to show why neural networks (NNs) are a core technology for the following multimedia functionalities: (1) efficient representations for audio/visual information, (2) detection and classification techniques, (3) fusion of multimodal signals, and (4) multimodal conversion and synchronization. It also demonstrates how the adaptive NN technology presents a unified solution to a broad spectrum of multimedia applications. As substantiating evidence, representative examples where NNs are successfully applied to IMP applications are highlighted. The examples cover a broad range, including image visualization, tracking of moving objects, image/video segmentation, texture classification, face-object detection/recognition, audio classification, multimodal recognition, and multimodal lip reading.

Inverse problems have been often considered ill-posed, i.e., the statement of the problem does not thoroughly constrain the solution space. In this paper we take advantage of this lack of information by adding informative constraints to the problem solution using Bayesian methodology. Remote sensing problems afford opportunities for inclusion of ground truth information, prior probabilities, noise distributions, and other informative constraints within a Bayesian probabilistic framework. We apply Bayesian methods to a synthetic remote sensing problem, showing that the performance is superior to a previously published method of iterative inversion of neural networks. In addition, we show that the addition of ground truth information, naturally included through Bayesian modeling, provides a significant performance improvement.

This paper presents a maximum likelihood joint-space adaptation technique for robust speech recognition. In the joint-space adaptation process, the N-best hidden Markov model (HMM) inversion frame-by-frame adapts the speech features non-parametrically to compensate the temporal deviation, while the models are transformed parametrically to catch the global characteristics of the mismatch. The proposed joint-space adaptation provides a better compensation to the mismatch than the single-space adaptations. This algorithm operates only on the given testing speech and the models, therefore no adaptation data are required. As verified by the experiments performed under different mismatch environments, the proposed method improves the performance in all the cases without degrading the performance under the match condition.

The MPEG video compression standard effectively exploits spatial, temporal, and coding redundancies in the algorithm. In its generic form, however, only a minimal amount of scene adaptation is performed. Video can be further compressed by taking advantage of scenes where the temporal statistics allow larger inter-reference frame distances. This paper proposes the use of motion analysis (MA) to adapt to scene content. The actual picture type (I, P, or B) decision is made by examining the accumulation of motion measurements since the last reference frame was labeled. Depending on the video content, this proposed algorithm can achieve from 2% to 13.9% savings in bits while maintaining similar quality.

The rapid evolution of communication technologies, contributed by the advanced data networks, which enable the integration of multimedia information processing and have greatly impacted our approach to distance learning. Several new multimedia features are developed and reported in this paper. These features allow students to browse the lecture contents directly on digital video sequences while also providing access to multimedia information like lecture captions, databases or handbooks with additional information or another video sequence to explain the current contents. Using these multimedia features, students can learn the course more effectively as compared to learning from conventional analog media in the distance learning environments.

We present a technique for detecting the endocardium of the left ventricles and estimating the heart motion from ultrasound images acquired using transesophageal echo-cardiography. The endocardial contours during the cardiac cycles are first detected using a 4-D knowledge-based snake (KB-Snake). We then apply the continuous distance transform neural network (CDTNN) to represent the 3-D endocardial contours at end diastole and quantitatively estimate the subsequent motion of the left ventricles of a human heart. The motion compensated contours are compared with the manually traced contours using the centerline method. Our simulations show very comparable results with human observers.

The rapid evolution of communication technologies, contributed by the advanced data networks and the integration of multimedia information processing, has greatly impacted our approach to distance learning. Most existing research work emphasizes the communication aspects of multimedia roles in distance learning [3, 4]. On the other hand, we are initiating a new multimedia research activity which emphasises the development of tools for creating constructive multimedia features for course-on-demand in distance learning. In this research, we investigated the development of a course-on-demand system which allows students to access a multimedia database containing recorded video courses over various mechanisms, such as TCP/IP LANs, PSTN, and ISDN. The system is based on a client-server model. The recorded courses are stored in the server which allows multiple clients to simultaneously access the multimedia database over the networks. Students can access these pre-recorded sequences through Web browsers (e.g., Netscape, Mosaic, and Microsoft Internet Explorer) by clicking on the specific course numbers (e.g., EE440, EE505, etc) or on the specific contents (e.g., Z-transform, FIR Filters, etc). To help course instructors to create this multimedia features we have developed a hyper video editor tool. This tool allows instructor to mark various portions of the class video and create the corresponding hyper links and multimedia features by editing on line forms.

We have designed and implemented a lipreading system that recognizes isolated words using only color video of human lips (without acoustic data). The system performs video recognition using "snakes" to extract visual features of geometric space, Karhunen-Loeve transform (KLT) to extract principal components in the color eigenspace, and hidden Markov models (HMM's) to recognize the combined visual features sequences. With the visual information alone, we were able to achieve 94% accuracy for ten isolated words.

The hidden Markov model (HMM) inversion algorithm, based on either the gradient search or the Baum-Welch reestimation of input speech features, is proposed and applied to the robust speech recognition tasks under general types of mismatch conditions. This algorithm stems from the gradient-based inversion algorithm of an artificial neural network (ANN) by viewing an HMM as a special type of ANN. Given input speech features s, the forward training of an HMM finds the model parameters lambda subject to an optimization criterion. On the other hand, the inversion of an HMM finds speech features, s, subject to an optimization criterion with given model parameters lambda. The gradient-based HMM inversion and the Baum-Welch HMM inversion algorithms can be successfully integrated with the model space optimization techniques, such as the robust MINIMAX technique, to compensate the mismatch in the joint model and feature space. The joint space mismatch compensation technique achieves better performance than the single space, i.e. either the model space or the feature space alone, mismatch compensation techniques. It is also demonstrated that approximately 10-dB signal-to-noise ratio (SNR) gain is obtained in the low SNR environments when the joint model and feature space mismatch compensation technique is used.

3D object recognition under partial object viewing is a difficult pattern recognition task. In this paper, we introduce a neural-network solution that is robust to partial viewing of objects and noise corruption. This method directly utilizes the acquired 3D data and requires no feature extraction. The object is first parametrically represented by a continuous distance transform neural network (CDTNN) trained by the surface points of the exemplar object. The CDTNN maps any 3D coordinate into a value that corresponds to the distance between the point and the nearest surface point of the object. Therefore, a mismatch between the exemplar object and an unknown object can be easily computed. When encountered with deformed objects, this mismatch information can be backpropagated through the CDTNN to iteratively determine the deformation in terms of affine transform. Application to 3D heart contour delineation and invariant recognition of 3D rigid-body objects is presented.

We have designed and implemented a lipreading system which recognises isolated words using only color motion video of human lips (without acoustic data). The lipreading system performs color motion video recognition using "snakes" (active contour models), principal component analysis (PCA), and hidden Markov models (HMM). The snake algorithm and PCA are used to extract two sets of visual features from every frame (image) in the video sequence. The snake algorithm looks for contour features in the geometric space, while PCA seeks principal components in the eigenspace. An HMM recognizer is used to train and recognise a sequence of the combined visual features. With the visual information alone, we were able to achieve 94% recognition accuracy for 10 isolated words of a single speaker without using any special marker or lipstick.

We propose a new method to estimate the multivariate conditional density, f(m|x), a density over the output space m conditioned on any given input x. In particular, we are interested in cases where the number of available training data points is relatively sparse within x space. We start from a priori considerations and establish certain desirable characteristics in kernel functions for conditional density estimation. We find that Gaussian kernels with expanding covariances, expanding as we move away from the data point of the kernel, satisfy these a priori considerations. We combine these expanding Gaussian kernels (EGK) according to Bayesian techniques. We compare the EGK with standard Gaussian kernel (SDK) methods, and find that EGK avoids multimodality, has diminishing confidence levels farther from training points, performs better asymptotically, and performs better with respect to the Kullback-Leibler criteria.

Cascade-correlation (Cascor) is a popular supervised learning architecture that dynamically grows layers of hidden neurons of fixed nonlinear activations (e.g., sigmoids), so that the network topology (size, depth) can be efficiently determined. Similar to a cascade-correlation learning network (CCLN), a projection pursuit learning network (PPLN) also dynamically grows the hidden neurons. Unlike a CCLN where cascaded connections from the existing hidden units to the new candidate hidden unit are required to establish high-order nonlinearity in approximating the residual error, a PPLN approximates the high-order nonlinearity by using trainable parametric or semi-parametric nonlinear smooth activations based on minimum mean squared error criterion. An analysis is provided to show that the maximum correlation training criterion used in a CCLN tends to produce hidden units that saturate and thus makes it more suitable for classification tasks instead of regression tasks as evidenced in the simulation results. It is also observed that this critical weakness in CCLN can also potentially carry over to classification tasks, such as the two-spiral benchmark used in the original CCLN paper.

The hidden Markov model (HMM) inversion algorithm is proposed and applied to robust speech recognition for general types of mismatched conditions. The Baum-Welch HMM inversion algorithm is a dual procedure to the Baum-Welch HMM reestimation algorithm, which is the most widely used speech recognition technique. The forward training of an HMM, based on the Baum-Welch reestimation, finds the model parameters /spl lambda/ that optimize some criterion, usually maximum likelihood (ML), with given speech inputs s. On the other hand, the inversion of a HMM finds speech inputs s that optimize some criterion with given model parameters /spl lambda/. The performance of the proposed HMM inversion, in conjunction with HMM reestimation, for robust speech recognition under additive noise corruption and microphone mismatch conditions is favorably compared with other noisy speech recognition techniques, such as the projection-based first-order cepstrum normalization (FOCN) and the robust minimax (MINIMAX) classification techniques.

Inverse problems have been often considered ill-posed, i.e., the statement of the problem does not thoroughly constrain the solution space. In this paper the authors take advantage of this lack of information by adding additional informative constraints to the problem solution using Bayesian methodology. Bayesian modeling gains much of its power from its ability to isolate and incorporate causal models as conditional probabilities. As causal models are accurately represented by forward models, the authors convert implicit functional models into data driven forward models represented by neural networks, to be used as engines in a Bayesian modeling setting. Remote sensing problems afford opportunities for inclusion of ground truth information, prior probabilities, noise distributions, and other informative constraints within a Bayesian probabilistic framework. They first apply these Bayesian methods to a synthetic remote sensing problem, showing that the performance is superior to a previously published method of iterative inversion of neural networks. Next, microwave brightness temperatures obtained from the Scanning Multichannel Microwave Radiometer (SMMR) over the African continent are inverted. The values of soil moisture, surface air temperature and vegetation moisture retrieved from the inversion produced contours that agree with the expected trends for that region.>

Contour finding of distinct features in 2-D/3-D images is essential for image analysis and computer vision. To overcome the potential problems associated with existing contour finding algorithms, we propose a framework, called the neural network-based stochastic active contour model (NNS-SNAKE), which integrates a neural network classifier for systematic knowledge building, an active contour model (also known as the "Snake") for automated contour finding using energy functions, and the Gibbs sampler to help the snake to find the most probable contour using a stochastic decision mechanism. Successful application of the NNS-SNAKE to extraction of several types of contours on magnetic resonance (MR) images is presented.

Cascaded correlation is a popular supervised learning architecture that dynamically grows layers of hidden neurons of fixed nonlinear activations (e.g., sigmoids), so that the network topology (size, depth) can be efficiently determined. Similar to a cascaded correlation learning network (CCLN), a projection pursuit learning network (PPLN) also dynamically grows the hidden neurons. Unlike a CCLN where cascaded connections from the existing hidden units to the new candidate hidden unit are required to establish high-order nonlinearity in approximating the residual error, a PPLN approximates the high-order nonlinearity by using trainable nonlinear nodal activation functions (e.g., hermite polynomials). To relax the necessity of predefined smoothness of nonlinearity (e.g., the order of hermite polynomials) in a PPLN, the authors propose a new learning network, called a cascaded projection pursuit network (CPPN), which combines both advantages of a PPLN and a CCLN. The training strategy of a CPPN is similar to that of PPLN and the added cascaded connections allow the CPNN to better capture the high order features without the necessity of proper selection of orders of polynomials. Simulation results show that a CPPN is suitable for nonlinear regression and outperforms a PPLN when the polynomial order used in the activation functions is less than the order of target functions.>

Contour finding of distinct features in 2D/3D images is essential for image analysis and computer vision. To overcome the potential problems associated with existing contour finding algorithms, the authors propose a neural network based active contour model (NN-SNAKE), which integrates a neural network classifier for systematic knowledge building, and an active contour model (also known as "Snake") for automated contour finding using energy functions. The paper describes work on image sequence classification using the proposed NN-SNAKE and hidden Markov models. The "snake" model was applied to extract visual features from a sequence of mouth images and a hidden Markov model was applied to perform word recognition on the visual features. With the visual information alone, the authors were able to achieve 93% recognition rate for 11 isolated words. The models performed lip-reading in a hand-free car audio system.>

Proposes a novel continuous distance transform neural network (CDTNN) to implement a continuous parametric 3D distance transform representation, which can be efficiently used in estimating the motion of 3D objects from a single view perspective. Our proposed CDTNN motion estimation approach consists of two stages of efforts. The 3D object is first converted by a CDTNN to a continuous 3D distance transform representation, i.e. the magnitudes of the responding neural network output values are linearly proportional to the distance of the points to the nearest surface of the object. When later presented with surface points of the oriented snapshots of a moving 3D object, this parametric CDTNN representation allows easy accumulation of the orientation mismatch information. More specifically, the mismatch information can be back-propagated through the CDTNN to iteratively determine the best similarity transform (orientation) required to align the oriented object with the represented exemplar object at each time instance. This orientation provide enough information for the motion of the estimated 3D object.>

Motion of the subject during magnetic resonance scan produces artifacts in the reconstructed images, which appear as blurring and ghost repetitions of the moving structures for the 2-dimensional Fourier Transform imaging methods. Several mathematical techniques have been proposed to correct the motion artifacts. Those techniques usually assume some types of motion models such as translational motion, rotational motion or linear expansion. In reality, motion can be far more complex, thus those techniques can be applied only in very limited cases. We present a new iterative algorithm to correct the corrupted data. Our method can be generalized to any arbitrary motion. The computer simulations demonstrate that a significant amount of improvement in motion artifact correction is achieved using this algorithm.>

We study and compare two types of connectionist learning methods for model-free regression problems: 1) the backpropagation learning (BPL); and 2) the projection pursuit learning (PPL) emerged in recent years in the statistical estimation literature. Both the BPL and the PPL are based on projections of the data in directions determined from interconnection weights. However, unlike the use of fixed nonlinear activations (usually sigmoidal) for the hidden neurons in BPL, the PPL systematically approximates the unknown nonlinear activations. Moreover, the BPL estimates all the weights simultaneously at each iteration, while the PPL estimates the weights cyclically (neuron-by-neuron and layer-by-layer) at each iteration. Although the BPL and the PPL have comparable training speed when based on a Gauss-Newton optimization algorithm, the PPL proves more parsimonious in that the PPL requires a fewer hidden neurons to approximate the true function. To further improve the statistical performance of the PPL, an orthogonal polynomial approximation is used in place of the supersmoother method originally proposed for nonlinear activation approximation in the PPL.

Through parallel processing, low precision fixed point hardware can be used to build a very high speed neural network computing engine where the low precision results in a drastic reduction in system cost. The reduced silicon area required to implement a single processing unit is taken advantage of by implementing multiple processing units on a single piece of silicon and operating them in parallel. The important question which arises is how much precision is required to implement neural network algorithms on this low precision hardware. A theoretical analysis of error due to finite precision computation was undertaken to determine the necessary precision for successful forward retrieving and back-propagation learning in a multilayer perceptron. This analysis can easily be further extended to provide a general finite precision analysis technique by which most neural network algorithms under any set of hardware constraints may be evaluated.>

The inversion of snow parameters from passive microwave remote sensing measurements is performed, using an iterative inversion of a neural network (NN) trained with a dense-media multiple-scattering model. Inversion of four parameters is performed based on five brightness temperatures. The four parameters are mean grain size of ice particles in snow, snow density, snow temperature, and snow depth. Iterative inversion of a data-driven forward NN model is justified on a theoretical and methodological basis. An error analysis is performed, comparing iterative inversion of a forward model with the use of an explicit inverse for the retrieval of independent snow parameters from their corresponding measurements. The NN iterative inversion algorithm is further illustrated by reconstructing a synthetic terrain of snow parameters from their corresponding measurements, inverting all four parameters simultaneously. The reconstructed parameter contours are in good agreement with the original synthetic parameter contours.>

Classifying objects that are distorted by similarity transforms and detection/occlusion noise is a difficult pattern recognition task. A novel and robust neural network solution based on detected surface boundary points is presented. The method operates in two stages. The object is first parametrically represented by a surface reconstruction neural network (SRNN) trained by the boundary points sampled from the exemplar object. When later presented with a distorted object, this parametric representation reduces the effects caused by detection/occlusion and also allows the mismatch information backpropagated through the SRNN to iteratively determine the best similarity transform of the distorted object. The distance measure can then be computed in the reconstructed representation domain between the exemplar object and the aligned distorted object.>

The Proteus architecture is a highly parallel MIMD, multiple instruction, multiple-data machine, optimized for large granularity tasks such as machine vision and image processing. The system can achieve 20 Giga-flops (80 Giga-flops peak). It accepts data via multiple serial links at a rate of up to 640 megabytes/second. The system employs a hierarchical reconfigurable interconnection network with the highest level being a circuit switched Enhanced Hypercube serial interconnection network for internal data transfers. The system is designed to use 256 to 1024 RISC processors. The processors use one megabyte external Read/Write Allocating Caches for reduced multiprocessor contention. The system detects, locates, and replaces faulty subsystems using redundant hardware to facilitate fault tolerance.>

An iterative constrained inversion technique is used to find the control inputs to the plant. That is, rather than training a controller network and placing this network directly in the feedback or feedforward paths, the forward model of the plant is learned, and iterative inversion is performed on line to generate control commands. The control approach allows the controllers to respond online to changes in the plant dynamics. This approach also attempts to avoid the difficulty of analysis introduced by most current neural network controllers, which place the highly nonlinear neural network directly in the feedback path. A neural network-based model reference adaptive controller is also proposed for systems having significant dynamics between the control inputs and the observed (or desired) outputs and is demonstrated on a simple linear control system. These results are interpreted in terms of the need for a dither signal for on-line identification of dynamic systems.

An improved neural network inversion technique that scales the search vector in accordance with the geometry of the problem has been developed. It searches in the direction of the gradient with a vector whose size is inversely related to the size of the gradient. To avoid unlimited growth of the search vector where the gradient is small, an upper bound is set on the size of the search vector. The network was trained by backpropagation and the training was halted when the network produced no error on the training set, where the output was categorized by binary thresholding. The results show the superior performance of the improved method. The technique was applied to automated cytology screening. A set of 400 object feature vectors randomly selected from a large database of 1929 feature vectors served as the initial training data.>

An approach is presented for query-based neural network learning. A layered perceptron partially trained for binary classification is considered. The single-output neuron is trained to be either a zero or a one. A test decision is made by thresholding the output at, for example, one-half. The set of inputs that produce an output of one-half forms the classification boundary. The authors adopted an inversion algorithm for the neural network that allows generation of this boundary. For each boundary point, the classification gradient can be generated. The gradient provides a useful measure of the steepness of the multidimensional decision surfaces. Conjugate input pairs are generated using the boundary point and gradient information and presented to an oracle for proper classification. These data are used to refine further the classification boundary, thereby increasing the classification accuracy. The result can be a significant reduction in the training set cardinality in comparison with, for example, randomly generated data points. An application example to power system security assessment is given.

The proposed systolic design for 2-D rank order filtering has a wide variety of applications in image processing. It derives its architecture mainly from a systolic design for 1-D rank order filtering proposed previously. The adopted systolic design, called the sample oriented rank order filter design, takes advantage of the evaluated rank values in the current window for the evaluation of the rank values in the next window without explicitly sorting the data in the window. This makes it possible to convert a 2-D windowed data sequence into a 1-D windowed data sequence with multiple data sample exchange at each window movement, and the 1-D systolic design is thus applicable. By cascading many such 1-D systolic arrays into a 2-D array, and supplied with simple parallel-in-serial-out (PISO) logic, this architecture can achieve the maximally available parallelism with nearly 100% efficiency if either the pipeline interleaving or processor sharing technique is used. The design does not require the preloading of the whole 2-D data array, and line scanned images can be processed with negligible time delay.>

Radar constant false alarm rate (CFAR) detectors, which are adaptive threshold detectors used to compensate for unknown noise environments, are discussed. Although the signal processing theory for CFAR detection is advanced, applications lag because of the high throughput required in radar. This intensive computational requirement (a data rate of at least 20 MHz for most search radars) cannot be met only by further advances in VLSI technology; parallel processing techniques are also needed. The feasibility of array processors based on VLSI technology enables the processing speed to increase by several orders of magnitude. One special-purpose VLSI architecture is the systolic array, which boasts massive concurrency. The concurrency in systolic arrays is derived from pipelined or parallel processing, and possibly both. Systolic array architectures are proposed for several important CFAR detectors.>

A novel approach is presented to solve a constrained inverse problem encountered in the design of frequency selective surfaces (FSSs). Due to the many-to-one nonlinear functional relationship between an FSS and its frequency response, there is no closed-form solution directly from the given desired frequency response to the corresponding surface. Therefore, to design an FSS for a given response, one has to search in the knowledge base through a laborious and tedious trial-and-error procedure. The authors' approach adopts an iterative regularized inversion technique, which starts with an inversion algorithm for multilayer perceptrons to generate the corresponding 2-D surface for the given desired frequency response. A constraint-satisfaction mechanism is then used to reshape the 2-D surface to satisfy the constraints, and the resulting surface is used as the initial point for the next inversion algorithm. This procedure is mathematically similar to the projection-onto-convex-set algorithm for constrained optimization problems

A novel approach to query-based neural network learning is presented. A layered perceptron partially trained for binary classification is considered. The single-output neuron is trained to be either a 0 or a 1. A test decision is made by thresholding the output at, for example, 1/2. The set of inputs that produce an output of 1/2 forms the classification boundary. For each boundary point, the classification gradient can be generated. The gradient provides a useful measure of the sharpness of the multidimensional decision surfaces. Conjugate input pair locations are generated using the boundary point and gradient information and are presented to the oracle for proper classification. These new data are used to further refine the classification boundary, thereby increasing the classification accuracy. The result can be a significant reduction in the training set cardinality in comparison with, for example, randomly generated data points. An application example to power security assessment is given

A generic iterative model is presented for a wide variety of artificial neural networks (ANNs): single-layer feedback networks, multilayer feed-forward networks, hierarchical competitive networks, and hidden Markov models. Unifying mathematical formulations are provided for both the retrieving and learning phases of ANNs. Based on the unifying mathematical formulation, a programmable universal ring systolic array is derived for both phases. It maximizes the strength of VLSI in terms of intensive and pipelined computing and yet circumvents the limitation on communication. Hardware implementation for the processing units based on CORDIC techniques is discussed.>

The authors propose a ring VLSI systolic architecture for implementing artificial neural networks (ANNs) with applications to robotic processing. Key design issues concerning algorithms, applications, and architectures are examined. A variety of neural networks is considered, including single-layer feedback neural networks, competitive learning networks, and multilayer feed-forward networks. It is demonstrated that the ANNs are suitable to all three levels of robotic processing applications including task planning, path planning, and path control levels. For these applications, a programmable systolic array is developed than can exploit the strength of VLSI to provide intensive and pipelined computing. Both the retrieving and learning phases are integrated in the design. The proposed architecture, which is more versatile than other existing ANNs, can accommodate all the useful neural networks for robotic processing.>

Systolic Kalman (SK) filter designs are presented which are based on a triangular array (triarray) configuration. In order to facilitate the systolic design, the original algorithm for the Kalman filter estimation is reformulated in a new least-squares formulation. The design has advantages in both numerical accuracy and computational efficiency. For the case of white additive noise, the SK-W filter design uses approximately n/sup 2//2 processors and provides a speed-up of n/sup 2//2, with a nearly 100% utilization rate. For the case of colored additive noise, the SK-C filter design also offers comparable speed-up performance.>

The authors discuss two important aspects in multilayer feed-forward neural nets: the optimal number of hidden units per layer, and the optimal number of synaptic weights between two adjacent layers. On the basis of simulations, they conjecture that the optimal number of hidden units shall be equal to or a little bit more than M-1 for efficient learning, where M is the number of pairs of training patterns used. Locally interconnected nets may be useful for some real applications where geometrical properties are significant. By introducing highway links into the locally interconnected nets, the convergence speed can be improved significantly.>