Recent advancements in vision-centric multi-task learning have greatly impacted autonomous driving, with a focus on constructing efficient and rich Bird’s Eye View (BEV) representations. While these methods achieve impressive performance, they often suffer from structural complexity and high computational costs due to the need for dense BEV representations. To address these challenges, we propose UniSparseBEV, a simple and efficient vision-based multi-task learning framework based on sparse queries. We introduce a set of learnable shared queries to facilitate information exchange across tasks. Additionally, we propose theZ-axis Deformable Cross-Attention (Z-DCA) module, which enables BEV segmentation task queries to directly extract information from image features without requiring dense BEV representations. To further enhance training efficiency, we incorporate 2D supervision into the network. Extensive experiments on the NuScenes dataset demonstrate that UniSparseBEV outperforms existing single-task methods in 3D object detection and BEV segmentation. A detailed robustness analysis is also conducted on the UniSparseBEV framework. We hope UniSparseBEV can serve as a strong baseline for multi-tasking in autonomous driving.

Accurate prognostic stratification for patients with colorectal liver metastases (CRLM) remains a clinical challenge due to the limited discriminative power of conventional models. This study developed and evaluated CRLMPred, a machine learning framework that integrates preoperative metabolic host factors and radiomic tumor features derived from contrast-enhanced CT to predict postoperative recurrence at 3,6, and 12 months. To ensure clinical applicability and avoid methodological bias, all input variables were restricted to preoperative baseline parameters, thereby eliminating the risk of temporal data leakage that inflated performance in preliminary models (AUC$>0.98$). The 3 -month prediction model demonstrated optimal performance, with an area under the curve (AUC) of 0.837 (95 % CI:0.674-0.955), a high negative predictive value (NPV) of 97.4 %, and stable cross-validation metrics. Decision curve analysis confirmed consistent net clinical benefit across a range of decision thresholds. Beyond its predictive accuracy, this study emphasizes the importance of rigorous model design to avoid data leakage and introduces a reproducible pipeline for reliable, translational deployment of artificial intelligence in clinical oncology. CRLM-Pred offers a promising clinical tool to support personalized surveillance and treatment planning in patients with CRLM.

In light of the dynamic nature of autonomous driving environments and stringent safety requirements, general MLLMs combined with CLIP alone often struggle to accurately represent driving-specific scenarios, particularly in complex interactions and long-tail cases. To address this, we propose the Hints of Prompt (HoP) framework, which introduces three key enhancements: Affinity hint to emphasize instance-level structure by strengthening token-wise connections, Semantic hint to incorporate high-level information relevant to driving-specific cases, such as complex interactions among vehicles and traffic signs, and Question hint to align visual features with the query context, focusing on question-relevant regions. These hints are fused through a Hint Fusion module, enriching visual representations by capturing driving-related representations with limited domain data, ensuring faster adaptation to driving scenarios. Extensive experiments confirm the effectiveness of the HoP framework, showing that it significantly outperforms previous state-of-the-art methods in all key metrics.

In the field of HER2 expression level assessment for breast cancer, clinical evaluations often rely on the synergistic analysis of both H&E and IHC stained images. However, acquiring dual-modality images for the same patient is frequently hindered by complex clinical workflows and high costs, resulting in missing modalities. To address this challenge, we propose an adaptive bimodal input prediction framework that flexibly supports both single-modality and dual-modality inputs. This framework employs a dynamic branch selection mechanism to overcome the rigid dependency of existing models on complete inputs, enabling accurate predictions using either H&E or IHC images alone, while retaining the ability for joint inference when both modalities are available. The core technical innovations include: a missing modality branch selector that dynamically activates either a modality completion process or an end-to-end dual-modality inference pipeline based on the available input; and a cross-modal generative adversarial network (CM-GAN) that facilitates context-aware reconstruction of the missing modality in the feature space. This design improves the prediction accuracy from 71.44% to 94.25% when using single-modality H&E images, significantly mitigating performance degradation caused by incomplete information. Experimental results demonstrate that the proposed framework achieves a prediction accuracy of 95.09% with full dual-modality input and maintains a high reliability of 90.28% under single-modality conditions. By adopting this ''dual-modality preferred, single-modality compatible'' flexible architecture, healthcare institutions can achieve near dual-modality accuracy without mandating synchronized acquisition of both image types. This is particularly valuable for regions with limited IHC staining infrastructure, offering a cost-effective clinical solution and substantially enhancing the accessibility of HER2 expression level assessment.

While Vision-Language Models (VLMs) based on large-scale models have shown revolutionary advancements across various vision-language tasks, research on improving VLM robustness remains underexplored. Existing studies primarily focus on attacking VLM after the pretrained visual or textual encoders, typically requiring obvious noise or long inference time. In this study, we look into VLM structure and highlight alignment module’s role as a protective filter that enhances VLM robustness against various perturbations. Motivated by these insights, we investigate VLM from both user and model developer perspectives and introduce the alignment perturbation strategy, which consists of multimodal, visual, and textual perturbations. Multimodal perturbation aims to achieve targeted textual output generation and is further utilized to enhance VLM robustness. Minimal perturbations to visual or textual inputs can lead to significant changes in the overall output of VLMs, revealing their sensitivity to both visual and textual input variations. Building on the alignment perturbation strategy, we propose alignment robust training, which efficiently improves VLM robustness by finetuning the parameters of alignment module without excessive resource consumption. Experiment results across various tasks and models demonstrate the effectiveness of the proposed alignment perturbation and alignment robust training. These methods deepen the understanding of VLM robustness, allowing for secure and reliable deployment towards diverse real-world scenarios. Codes are available at https://github.com/zhangconghhh/RobustVLMs.

In recent years, with the advancement of artificial intelligence technology, autonomous driving technologies have gradually emerged. 3D object detection using point clouds has become a key in this field. Multi-frame fusion of point clouds is a promising technique to enhance 3D object detection for autonomous driving systems. However, most existing multi-frame detection methods focus primarily on utilizing vehicle-side lidar data. Infrastructure-side detection remains relatively unexplored, yet can enhance vital vehicle-road coordination capabilities. To help with this coordination, we propose an efficient multi-frame aggregation multi-scale fusion network specifically for infrastructure-side 3D object detection. First, our key innovation is a novel multi-frame feature aggregation module that effectively integrates information from multiple past point cloud frames to improve detection accuracy. This module comprises a feature pyramid network to fuse multi-scale features, as well as a cross-attention mechanism to learn semantic correlations between different frames over time. Next, we incorporate deformable attention, which reduces the computational overhead of aggregation by sampling locations. We designed Multi-frame and Multi-scale modules, thereby we named the model MAMF-Net. Finally, through extensive experiments on two infrastructure-side datasets including the V2X-Seq-SPD dataset which was released by Baidu corporation, we demonstrate that MAMF-Net delivers consistent accuracy improvements over single frame detectors such as PointPillars, PV-RCNN and TED-S, especially boosting pedestrian detection by 5%. Our approach also surpasses other multi-frame methods designed for vehicle-side point clouds such as MPPNet.

The diffusion model is a new type of generative model. In this work, We propose a generative video quality enhancement method based on the diffusion model. Compared with the existing methods, our method requires only one model to improve the quality of videos encoded with different quantization parameters, and the structure of this model is shown in Figure 1 .

Deep Neural Networks (DNNs) are vulnerable to adversarial perturbations, raising significant concerns about their security. Numerous methods have been proposed to enhance DNN robustness. However, many methods, including adversarial training and noise injection, improve robustness by incorporating external data into the network. Exploring the network’s inherent potential is crucial to improve adversarial robustness. Inspired by principles in physical chemistry, where increased disorder leads to greater energetic stability, we introduce the Weight Decorrelation Loss. This method is simple but effective, enhancing robustness by disrupting the feature space’s ordered structure. The proposed loss achieves substantial performance improvements and state-of-the-art performance after being combined with Gaussian noise. We conduct comprehensive experiments on five datasets, comparing our approach to state-of-the-art defense methods. The results demonstrate our method’s effectiveness against several powerful white-box and black-box attacks.

There are two mainstream approaches for object detection: top-down and bottom-up. The state-of-the-art approaches are mainly top-down methods. In this paper, we demonstrate that bottom-up approaches show competitive performance compared with top-down approaches and have higher recall rates. Our approach, named CenterNet, detects each object as a triplet of keypoints (top-left and bottom-right corners and the center keypoint). We first group the corners according to some designed cues and confirm the object locations based on the center keypoints. The corner keypoints allow the approach to detect objects of various scales and shapes and the center keypoint reduces the confusion introduced by a large number of false-positive proposals. Our approach is an anchor-free detector because it does not need to define explicit anchor boxes. We adapt our approach to backbones with different structures, including 'hourglass'-like networks and 'pyramid'-like networks, which detect objects in single-resolution and multi-resolution feature maps, respectively. On the MS-COCO dataset, CenterNet with Res2Net-101 and Swin-Transformer achieve average precisions (APs) of 53.7% and 57.1%, respectively, outperforming all existing bottom-up detectors and achieving state-of-the-art performance. We also design a real-time CenterNet model, which achieves a good trade-off between accuracy and speed, with an AP of 43.6% at 30.5 frames per second (FPS).

DeepFakes have raised serious societal concerns, leading to a great surge in detection-based forensics methods in recent years. Face forgery recognition is a standard detection method that usually follows a two-phase pipeline,i.e., it extracts the face first and then determines its authenticity by classification. While those methods perform well in ideal experimental environment, they face challenges when dealing with DeepFakes in the wild involving complex background and multiple faces of varying sizes. Moreover, most face forgery recognition methods can only process one face at a time. One straightforward way to address this issue is to simultaneous process multi-face by integrating face extraction and forgery detection in an end-to-end fashion by adapting advanced object detection architectures. However, as these object detection architectures are designed to capture the discriminative features of different object categories rather than the subtle forgery traces among the faces, the direct adaptation suffers from limited representation ability. In this paper, we propose Contrastive Multi-FaceForensics (COMICS), an end-to-end framework for multi-face forgery detection. COMICS integrates face extraction and forgery detection in a seamless manner and adapts to the advanced object detection architectures. The core of the proposed framework is a bi-grained contrastive learning approach that explores face forgery traces at both the coarse- and fine-grained levels. Specifically, coarse-grained level contrastive learning captures the discriminative features among positive and negative proposal pairs at multiple layers produced by the proposal generator, and the fine-grained level contrastive learning captures the pixel-wise discrepancy between the forged and original areas of the same face and the pixel-wise content inconsistency among different faces. Extensive experiments on the OpenForensics and FFIW datasets demonstrate that our method outperforms other counterparts and shows great potential for being integrated into various architectures. Codes are available at https://github.com/zhangconghhh/COMICS.

AVS3 is the latest video coding standard of the AVS series. With the addition of advanced coding tools, AVS3 achieves better encoding performance than AVS2 and HEVC. However, these new features also lead to a significant increase in the complexity of AVS3, especially for integer-pixel motion estimation, which is difficult to be implemented on hardware. This paper proposes a LCU-level weighted search algorithm based on rate distortion optimization in integer-pixel motion estimation. Mean-while, a parallel and pipelined hardware architecture for AVS3 integer-pixel motion estimation is proposed and implemented on Xilinx Virtex VU440 FPGA. Experimental results show that the proposed architecture can process 3840x2160@60fps at 400 MHz clock frequency. Compared with the AVS3 reference software HPM4.0, the proposed algorithm only increases BjØntcgaard delta bitrate by 0.41 % in LDP (Low-Delay P) configuration.

The latest Versatile Video Coding (VVC) standard has significantly coding efficiency improvement compared with its ancestor High Efficiency Video Coding (HEVC) standard, but at the expense of over-high complexity. As measured by the VVC test model (VTM), the intra-mode comparison and selection in the rate-distortion optimization (RDO) search consume most of the encoding time. In this paper, we propose a deep multi-task learning based fast intra-mode decision approach via adaptively pruning off most redundant modes. First, we create a large-scale intra-mode database for VVC, including both normal angular modes and the newly introduced tools, i.e., intra sub-partition (ISP) and matrix-based intra prediction (MIP). Next, we propose a multi-task intra-mode decision network (MID-Net) model to effectively predict the most probable angular modes and whether to skip ISP and MIP modes. Then, a fast intra-coding workflow is designed accordingly, involving rough mode decision (RMD) acceleration and candidate mode list (CML) pruning. For the workflow output, the learning-oriented probability and the statistics-oriented probability are synthesized together to further improve the prediction accuracy, ensuring that only unnecessary intra-modes are skipped. Finally, experimental results show that our approach can significantly reduce 40.48% of encoding time of VVC intra-coding with negligible rate-distortion degradation, outperforming other state-of-the-art approaches.

In this work, we propose a low-complexity convolution neural network for compressed video post-processing. The main process can be expressed as follows:

We describe a proactive defense method to expose Deep-Fakes with training data contamination. Note that the existing methods usually focus on defending from general DeepFakes, which are synthesized by GAN using random noise. In contrast, our method is dedicated to defending from native Deep-Fakes, which is synthesized by auto-encoder that involves face swapping and encoding-decoding process that general DeepFakes do not have. Specifically, we design two types of traces namely sustainable traces and erasable traces, which are added on the faces to manipulate the training of DeepFake models. Consequently, the trained DeepFake model can synthesize faces with sustainable traces but no erasable traces. With the help of these two traces, we can expose DeepFakes proactively. Our method is compared with recent passive and proactive defense methods, which corroborates the efficacy of our method.

Human detection and pose estimation are essential for understanding human activities in images and videos. Mainstream multi-human pose estimation methods take a top-down approach, where human detection is first performed, then each detected person bounding box is fed into a pose estimation network. This top-down approach suffers from the early commitment of initial detections in crowded scenes and other cases with ambiguities or occlusions, leading to pose estimation failures. We propose the DetPoseNet, an end-to-end multi-human detection and pose estimation framework in a unified three-stage network. Our method consists of a coarse-pose proposal extraction sub-net, a coarse-pose based proposal filtering module, and a multi-scale pose refinement sub-net. The coarse-pose proposal sub-net extracts whole-body bounding boxes and body keypoint proposals in a single shot. The coarse-pose filtering step based on the person and keypoint proposals can effectively rule out unlikely detections, thus improving subsequent processing. The pose refinement sub-net performs cascaded pose estimation on each refined proposal region. Multi-scale supervision and multi-scale regression are used in the pose refinement sub-net to simultaneously strengthen context feature learning. Structure-aware loss and keypoint masking are applied to further improve the pose refinement robustness. Our framework is flexible to accept most existing top-down pose estimators as the role of the pose refinement sub-net in our approach. Experiments on COCO and OCHuman datasets demonstrate the effectiveness of the proposed framework. The proposed method is computationally efficient (5-6x speedup) in estimating multi-person poses with refined bounding boxes in sub-seconds.

In this paper, we propose a novel and efficient false positive reduction framework for early pulmonary nodules detection with physical examination Computer Tomography (CT) scan. First, a 3DU-NET model is used for detecting small pulmonary nodules in physical examination. Second, the grayscale history image (GHI) of the candidate nodules in multi-view is used as the input of the false positive reduction network to obtain their comprehensive information. Finally, an effective multi-branch false positive reduction network is built to avoid reducing the sensitivity of nodule detection. Experiments are performed in our dataset, i.e. 1,000 samples of Lung Nodules from Physical Examination (LNPE1000), in which nodules are with the average diameter of 5. 3 mm. The sensitivities of the proposed method are 79.9% and 85.8% with each CT contained 0.25 and 0.45 false positives on average respectively, which are more than 8.9% and 3.9% higher than baseline. Experimental results show that the proposed method is robust and efficient, which can successfully detect smaller and larger nodules in physical examination and can effectively reduce false positives while maintain the sensitivity of nodule detection. It may be helpful to further improve the efficiency of doctors in clinical diagnosis.

In this paper, we propose an efficient convolution neural network to enhance the quality of video compressed by HEVC standard. The model is composed of a max-pooling module and a hierarchical feature fusion module. The max-pooling module extracts feature from different scales and enlarges the receptive field of the model without stacking too many convolution layers. And the hierarchical feature fusion module accurately aligns features from different scales and fuses them efficiently. Two modules are applied in the proposed network, our model reconstructs compressed video frames with higher visual quality. Besides, the model is constructed in the full convolution network, thus it can adapt to videos in variable resolutions. The experiment results show that the proposed model outperforms existing models in the terms of PSNR under the same dataset.

Multivariate time series (MTS) prediction aims at predicting future time series by extracting multiple forms of dependencies of past time series. Traditional prediction methods and deep learning-based prediction methods focus on extracting the dynamic relationships of certain aspects of MTS, especially the temporal characteristics, often neglecting the spatial and temporal dynamic correlations of MTS. Inspired by convolution neural network (CNN) and attention mechanism, this paper proposes a convolution LSTM network model based on MTS prediction with two-stage attention. Specifically, we first propose a new MTS preprocessing method to perform convolution operations better. Then convolution layer extracts spatial correlation of MTS and LSTM model extracts temporal correlation. It is worth mentioning that the combination of attention mechanism and LSTM can effectively solve the problem of insufficient time dependency in MTS prediction. In addition, dual-stage attention mechanism can effectively eliminate irrelevant information, select the relevant exogenous sequence, give it higher weight, and increase the past value of the target sequence to further eliminate irrelevant information. Finally, the MTS spatio-temporal correlation is extracted to improve the prediction accuracy, and the model is interpreted. Experimental results show that the model has broad application prospects. Experiments based on typical datasets of finance, environment, and energy determine the optimal window size and hidden size of the prediction, and demonstrate that the model achieves the state-of-the-art effect compared to the other four deep learning models. On top of that, the model is not only suitable for single-step prediction of MTS, but also suitable for multistep prediction of time step in a certain range.

We propose a fast online video pose estimation method to detect and track human upper-body poses based on a conditional dynamic Bayesian modeling of pose modes without referring to future frames. The estimation of human body poses from videos is an important task with many applications. Our method extends fast image-based pose estimation to live video streams by leveraging the temporal correlation of articulated poses between frames. Video pose estimation is inferred over a time window using a conditional dynamic Bayesian network (CDBN), which we term time-windowed CDBN. Specifically, latent pose modes and their transitions are modeled and co-determined from the combination of three modules: 1) inference based on current observations; 2) the modeling of mode-to-mode transitions as a probabilistic prior; and 3) the modeling of state-to-mode transitions using a multimode softmax regression. Given the predicted pose modes, the body poses in terms of arm joint locations can then be determined more accurately and robustly. Our method is suitable to investigate high frame rate (HFR) scenarios, where pose mode transitions can effectively capture action-related temporal information to boost performance. We evaluate our method on a newly collected HFR-Pose dataset and four major video pose datasets (VideoPose2, TUM Kitchen, FLIC, and Penn_Action). Our method achieves improvements in both accuracy and efficiency over existing online video pose estimation methods.

Style transfer on images has achieved significant advances in recent years, with the deep convolutional neural network (CNN). Directly applying image style transfer algorithms to each frame of a video independently often leads to flickering and unstable results. In this work, we present a self-supervised space-time convolutional neural network (CNN) based method for online video style transfer, named as VTNet, which is end-to-end trained from nearly unlimited unlabeled video data to produce temporally coherent stylized videos in real-time. Specifically, our VTNet transfer the style of a reference image to the source video frames, which is formed by the temporal prediction branch and the stylizing branch. The temporal prediction branch is used to capture discriminative spatiotemporal features for temporal consistency, pretrained in an adversarial manner from unlabeled video data. The stylizing branch is used to transfer the style image to a video frame with the guidance from the temporal prediction branch to ensure temporal consistency. To guide the training of VTNet, we introduce the style-coherence loss net (SCNet), which assembles the content loss, the style loss, and the new designed coherence loss. These losses are computed based on high-level features extracted from a pretrained VGG-16 network. The content loss is used to preserve high-level abstract contents of the input frames, and the style loss introduces new colors and patterns from the style image. Instead of using optical flow to explicitly redress the stylized video frames, we design the coherence loss to make the stylized video inherit the dynamics and motion patterns from the source video to remove temporal flickering. Extensive subjective and objective evaluations on various styles demonstrate that the proposed method achieves favorable results against the state-of-the-arts with high efficiency.

AI-synthesized face-swapping videos, commonly known as DeepFakes, is an emerging problem threatening the trustworthiness of online information. The need to develop and evaluate DeepFake detection algorithms calls for datasets of DeepFake videos. However, current DeepFake datasets suffer from low visual quality and do not resemble DeepFake videos circulated on the Internet. We present a new large-scale challenging DeepFake video dataset, Celeb-DF, which contains 5,639 high-quality DeepFake videos of celebrities generated using improved synthesis process. We conduct a comprehensive evaluation of DeepFake detection methods and datasets to demonstrate the escalated level of challenges posed by Celeb-DF.

Detecting small objects is a challenging task due to their low resolution and noisy representation even using deep learning methods. In this paper, we propose a novel object detection method based on the channel-aware deconvolutional network (CADNet) for accurate small object detection. Specifically, we develop the channel-aware deconvolution (ChaDeConv) layer to exploit the correlations of feature maps in different channels across deeper layers, improving the recall rate of small objects at low additional computational costs. Following the ChaDeConv layer, the multiple region proposal sub-network (Multi-RPN) is employed to supervise and optimize multiple detection layers simultaneously to achieve better accuracy. The Multi-RPN module is only used in the training phase and does not increase the computation cost of the inference. In addition, we design a new anchor matching strategy based on the center point translation (CPTMatching) of anchors to select more extending anchors as positive samples in the training phase. The extensive experiments on the PASCAL VOC 2007/2012, MS COCO, and UAVDT datasets show that the proposed CADNet achieves state-of-the-art performance compared to the existing methods.

Model compression technique is now widely investigated to fit the high-complexity deep neural network into resource-constrained mobile devices in recent years, in which one of effective methods is knowledge distillation. In this paper we make a discussion on the temperature term introduced in knowledge distillation method. The temperature term in distill training is aimed at making it easier for the student network to learn the generalization capablityof teacher network by softening the labels from the teacher network. We analyze the situation of using the temperature term in ordinary training to soften the output of neural network instead of soften the target. In experiments, we show that by applying a proper temperature term in training process, a better performance can be gained on NABirds dataset than using the model without temperature term.

In object detection, keypoint-based approaches often experience the drawback of a large number of incorrect object bounding boxes, arguably due to the lack of an additional assessment inside cropped regions. This paper presents an efficient solution that explores the visual patterns within individual cropped regions with minimal costs. We build our framework upon a representative one-stage keypoint-based detector named CornerNet. Our approach, named CenterNet, detects each object as a triplet, rather than a pair, of keypoints, which improves both precision and recall. Accordingly, we design two customized modules, cascade corner pooling, and center pooling, that enrich information collected by both the top-left and bottom-right corners and provide more recognizable information from the central regions. On the MS-COCO dataset, CenterNet achieves an AP of 47.0 %, outperforming all existing one-stage detectors by at least 4.9%. Furthermore, with a faster inference speed than the top-ranked two-stage detectors, CenterNet demonstrates a comparable performance to these detectors. Code is available at https://github.com/Duankaiwen/CenterNet.

Generative adversary networks (GANs) have recently led to highly realistic image synthesis results. In this work, we describe a new method to expose GAN-synthesized images using the locations of the facial landmark points. Our method is based on the observations that the facial parts configuration generated by GAN models are different from those of the real faces, due to the lack of global constraints. We perform experiments demonstrating this phenomenon, and show that an SVM classifier trained using the locations of facial landmark points is sufficient to achieve good classification performance for GAN-synthesized faces.

Cascaded Regression (CR) based methods have been proposed to solve facial landmarks detection problem, which learn a series of descent directions by multiple cascaded regressors separately trained in coarse and fine stages. They outperform the traditional gradient descent based methods in both accuracy and running speed. However, cascaded regression is not robust enough because each regressor's training data comes from the output of previous regressor. Moreover, training multiple regressors requires lots of computing resources, especially for deep learning based methods. In this paper, we develop a Self-Iterative Regression (SIR) framework to improve the model efficiency. Only one self-iterative regressor is trained to learn the descent directions for samples from coarse stages to fine stages, and parameters are iteratively updated by the same regressor. Specifically, we proposed Landmarks-Attention Network (LAN) as our regressor, which concurrently learns features around each landmark and obtains the holistic location increment. By doing so, not only the rest of regressors are removed to simplify the training process, but the number of model parameters is significantly decreased. The experiments demonstrate that with only 3.72M model parameters, our proposed method achieves the state-of-the-art performance.

Human pose estimation is an important topic in computer vision with many applications including gesture and activity recognition. However, pose estimation from image is challenging due to appearance variations, occlusions, clutter background, and complex activities. To alleviate these problems, we develop a robust pose estimation method based on the recent deep conv-deconv modules with two improvements: (1) multi -scale supervision of body keypoints, and (2) a global regression to improve structural consistency of keypoints. We refine keypoint detection heatmaps using layer-wise multi-scale supervision to better capture local contexts. Pose inference via keypoint association is optimized globally using a regression network at the end. Our method can effectively disambiguate keypoint matches in close proximity including the mismatch of left-right body parts, and better infer occluded parts. Experimental results show that our method achieves competitive performance among state-of-the-art methods on the MPII and FLIC datasets.

Identifying global contrast enhancement in an image is an important task in forensics estimation. Several previous methods analyze the “peak-gap” fingerprints in graylevel histograms. However, images in real scenarios are often stored in the JPEG format with middle/low compression quality, resulting in less obvious “peak-gap” effect and then unsatisfactory performance. In this paper, we propose a novel deep Multi-Path Network (MPNet) based approach to learn discriminative features from graylevel histograms. Specifically, given the histograms, their high-level peaks and gaps information can be exploited effectively after several shared convolutional layers in the network, even in middle/low quality compressed images. Moreover, the proposed multi-path module is able to focus on dealing with specific forensics operations for more robustness on image compression. The experiments on three challenging datasets (i.e., Dresden, RAISE and UCID) demonstrate the effectiveness of the proposed method compared to existing methods.

To effectively solve the challenges in object tracking, such as large deformation and severe occlusion, many existing methods use graph-based models to capture target part relations, and adopt a sequential scheme of target part selection, part matching, and state estimation. However, such methods have two major drawbacks: 1) inaccurate part selection leads to performance deterioration of part matching and state estimation and 2) there are insufficient effective global constraints for local part selection and matching. In this paper, we propose a new object tracking method based on iterative graph seeking, which integrate target part selection, part matching, and state estimation using a unified energy minimization framework. Our method also incorporates structural information in local parts variations using the global constraint. We devise an alternative iteration scheme to minimize the energy function for searching the most plausible target geometric graph. Experimental results on several challenging benchmarks (i.e., VOT2015, OTB2013, and OTB2015) demonstrate improved performance and robustness in comparison with existing algorithms.

Inconsistency in contrast enhancement can be used to expose image forgeries. In this work, we describe a new method to estimate contrast enhancement operations from a single image. Our method takes advantage of the nature of contrast enhancement as a mapping between pixel values and the distinct characteristics it introduces to the image pixel histogram. Our method recovers the original pixel histogram and the contrast enhancement simultaneously from a single image with an iterative algorithm. Unlike previous works, our method is robust in the presence of additive noise perturbations that are used to hide the traces of contrast enhancement. Furthermore, we also develop an effective method to detect image regions undergone contrast enhancement transformations that are different from the rest of the image, and we use this method to detect composite images. We perform extensive experimental evaluations to demonstrate the efficacy and efficiency of our method.

The rapid advances of transportation infrastructure have led to a dramatic increase in the demand for smart systems capable of monitoring traffic and street safety. Fundamental to these applications are a community-based evaluation platform and benchmark for object detection and multi-object tracking. To this end, we organize the AVSS2017 Challenge on Advanced Traffic Monitoring, in conjunction with the International Workshop on Traffic and Street Surveillance for Safety and Security (IWT4S), to evaluate the state-of-the-art object detection and multi-object tracking algorithms in the relevance of traffic surveillance. Submitted algorithms are evaluated using the large-scale UA-DETRAC benchmark and evaluation protocol. The benchmark, the evaluation toolkit and the algorithm performance are publicly available from the website http://detrac-db.rit.albany.edu.

Graph-based representation is widely used in visual tracking field by finding correct correspondences between target parts in different frames. However, most graph-based trackers consider pairwise geometric relations between local parts. They do not make full use of the target's intrinsic structure, thereby making the representation easily disturbed by errors in pairwise affinities when large deformation or occlusion occurs. In this paper, we propose a geometric hypergraph learning-based tracking method, which fully exploits high-order geometric relations among multiple correspondences of parts in different frames. Then visual tracking is formulated as the mode-seeking problem on the hypergraph in which vertices represent correspondence hypotheses and hyperedges describe high-order geometric relations among correspondences. Besides, a confidence-aware sampling method is developed to select representative vertices and hyperedges to construct the geometric hypergraph for more robustness and scalability. The experiments are carried out on three challenging datasets (VOT2014, OTB100, and Deform-SOT) to demonstrate that our method performs favorably against other existing trackers.

We propose an efficient framework for large-scale 3D reconstruction from a large set of photos following the Structure-from-Motion (SfM) paradigm with divide-conquer and fusion. Our main novelty is to ensure commonality from overlaps between image sets corresponding to their reconstructions, which facilitates effective stitching and fusion. Specifically, such commonality is ensured by selecting a set of duplicated images (which are termed anchor images) in adjacent image sets prior to the 3D reconstruction. The anchor images can assist accurate fusion of the 3D point clouds. We describe an efficient RANSAC scheme for pairwise stitching. Our method is intuitively scalable to large site reconstruction via subdivision and fusion following a graph construct. We further describe another RANSAC algorithm to improve loop closure in our anchor image approach. Experimental results on reconstructing a large portion of a university campus demonstrate the efficacy of our method.

Recent advances in online visual tracking focus on designing part-based model to handle the deformation and occlusion challenges. However, previous methods usually consider only the pairwise structural dependences of target parts in two consecutive frames rather than the higher order constraints in multiple frames, making them less effective in handling large deformation and occlusion challenges. This paper describes a new and efficient method for online deformable object tracking. Different from most existing methods, this paper exploits higher order structural dependences of different parts of the tracking target in multiple consecutive frames. We construct a structure-aware hyper-graph to capture such higher order dependences, and solve the tracking problem by searching dense subgraphs on it. Furthermore, we also describe a new evaluating data set for online deformable object tracking (the Deform-SOT data set), which includes 50 challenging sequences with full annotations that represent realistic tracking challenges, such as large deformations and severe occlusions. The experimental result of the proposed method shows considerable improvement in performance over the state-of-the-art tracking methods.

Estimation of human body poses from video is an important problem in computer vision with many applications. Most existing methods for video pose estimation are offline in nature, where all frames in the video are used in the process to estimate the body pose in each frame. In this work, we describe a fast online video upper body pose estimation method (CDBN-MODEC) that is based on a conditional dynamic Bayesian network model, which predicts upper body pose in a frame without using information from future frames. Our method combines fast single image based pose estimation methods with the temporal correlation of poses between frames. We collect a new high frame rate upper body pose dataset that better reflects practical scenarios calling for fast online video pose estimation. When evaluated on this dataset and the VideoPose2 benchmark dataset, CDBN-MODEC achieves improvements in both performance and running efficiency over several state-of-art online video pose estimation methods.

Several important classes of images such as text, barcode and pattern images have the property that pixels can only take a distinct subset of values. This knowledge can benefit the restoration of such images, but it has not been widely considered in current restoration methods. In this work, we describe an effective and efficient approach to incorporate the knowledge of distinct pixel values of the pristine images into the general regularized least squares restoration framework. We introduce a new regularizer that attains zero at the designated pixel values and becomes a quadratic penalty function in the intervals between them. When incorporated into the regularized least squares restoration framework, this regularizer leads to a simple and efficient step that resembles and extends the rounding operation, which we term as soft-rounding. We apply the soft-rounding enhanced solution to the restoration of binary text/barcode images and pattern images with multiple distinct pixel values. Experimental results show that soft-rounding enhanced restoration methods achieve significant improvement in both visual quality and quantitative measures (PSNR and SSIM). Furthermore, we show that this regularizer can also benefit the restoration of general natural images.

This paper provides a multiple-layer parallel motion estimation (ME) scheme implemented on GPU for High Efficiency Video Coding (HEVC). The scheme is hierarchically structured, including four layers: coding tree unit (CTU), prediction unit (PU), motion vector (MV) selection and instruction optimization. In PU-layer, costs of various PU sizes were obtained through a SAD (sum of absolute differences) look-up table instead of progressive cost merging. And during MV selection, GPU's comparison instruction was used to avoid branches. At the same time, concurrent CTUs processing and SIMD (Single Instruction, Multiple Data) optimization also improve the performance significantly. Experimental results show that the proposed scheme can take full advantage of GPU and achieves over 90 times speedup compared with the HM10.0 using fast ME.

In this paper, we propose an adaptive weighted distortion optimization algorithm for improving the coding performance of video represented in RGB color space. The YCbCr is the color space, which is usually adopted in video coding. And with 4:2:0 planar format, it generally uses the ratio 6:1:1 metrics objective peak signal-to-noise ratio (PSNR) for quality assessment. In this paper, we still use 6:1:1 combined PSNR as a universal criterion to optimize video coded in RGB domain and measure the final quality of decoded video as well. The new distortion metric in RGB coding is derived from the optimal target, and the modification is only applied on the encoder side. The proposed algorithm achieves about 26% bit-saving on average and maximum 44% bit-saving in HEVC range extensions reference software HM12.0 RExt4.1. Additionally, the visual quality of video coded by the proposed algorithm is also improved.

High Efficiency Video Coding (HEVC) is the most up to dated video coding standard which reduces the bit rate by almost 40% with the same objective quality compared to H.264/AVC but at about 40% extra encoding complexity overhead. The main reason for HEVC complexity increase is inter prediction that accounts for 60-70% of the whole encoding time. Especially for surveillance video with huge amount of data, the reducing of coding complexity is a very pressing work. So in this paper, a novel fast method is proposed based on background model to speed up the searching procedure of surveillance video coding. This paper analyzes the proportion of different block partitions and reference frames for multiple types blocks, and puts the analysis results into practice by developing optimized mode decision and reference frame selection schemes for the HEVC encoder according to spatial-temporal corresponding CUs (coding units) or PUs (prediction units). Simulation results show that the proposed algorithm can achieve more than 41 % reduction of coding time with comparable rate distortion (RD) performance.

This demonstration system shows a kind of video decoder's implementation in Reconfigurable Video Coding (RVC) framework on Open RVC-CAL Compiler (Orcc) platform. Differently from tradition video decoder, the reconfigurable video decoder is not a decoder conforming a special video coding standard, but dynamically built according to actual bitsteams, which may not conform any standard. The reconfigurable video decoder receives not only the compressed video bitstream but also the decoder description. As an example, in this demo, we reconfigure AVS and H.264/AVC decoders using Just-In-Time Adaptive Decoder Engine (Jade).

With the real-time requirement for video conversation, the coding system needs to adopt low delay and low complexity strategy to encode conversational videos, which may result in a significant decline of the video quality under the constrained bandwidth of network. In conversational videos, the face region attracts most human attentions. Therefore, recovering the distortion of face region will effectively improve the visual quality of conversational video. Actually, the participants in a conversation are usually unchanged in a relative long period, and similar facial expressions of the participants would be often repetitive. However, conventional video coding methods just consider the correlation of several neighboring frames while the long-range correlation of similar face regions in the whole conversational video has not been fully used. In this paper, we propose a face distortion recovery system to improve the visual quality of decoded conversational video by online learning an own face feature database for each user. First, at the sender side, the face feature database is established and online updated to include different facial expressions of the person. Then, at the receiver side, the low quality face regions in decoded video are recovered with the face patches in the database. Experimental results show that, under low bits rates the proposed method achieves average 5.22 dB gain with small burden to update the database.

In a video conversation, the participants usually remain the same. As the conversation continues, similar facial expressions of the same person would occur intermittently. However, the correlation of similar face features has not been fully used since the conventional methods only focus on independent frames. We set up a face feature database and updated it online to include new facial expressions during the whole conversation. At the receiver side, the database is used to recover the face distortion and thus improve the visual quality. Additionally, the proposed method brings small burden to update the database and is generic to various CODEC.

Detecting abnormal events in crowded scenes remains challenging due to the diversity of events defined by various applications. Among the many application situations, motion analysis for event representation is suited for crowded scenes. In this paper, we propose a novel abnormal event detection method via likelihood estimation of dynamic-texture motion representation, called Structural Multi-scale Motion Interrelated Patterns (SMMIP). SMMIP combines both original motion patterns and their structural spatio-temporal information, which effectively represents localized events by different resolutions of motion patterns. To model normal events, the Gaussian mixture model is trained with the observed normal events, then the likelihood estimation for testing events is computed to judge whether they are abnormal. Meanwhile, the proposed model can be learned online by updating the parameters incrementally. The proposed approach is evaluated on several publicly available datasets and outperforms several other methods proposed before, which is shown that the structural spatio-temporal information added in motion representation helps increasing the anomalies detection rate.

In this paper, a novel self-adaptive curve, based on human visual model (HVM), is proposed for recovering details from low dynamic range (LDR) digital photographs, which are under-exposed or over-exposed or both. In order to improve the perceptual visibility, we utilize HVM to construct our method, which is able to take advantage of entire dynamic range to enhance the contrast of images. Extensive experiments demonstrate that our method consistently achieves satisfying results for unwell-exposed LDR photographs.

Cloud computing is emerging as a very promising technology for computing and storage services. However, the multi-resources load balancing over heterogeneous cluster or cloud is a NP-hard problem. To obtain an optimized solution, in this paper, we propose a heuristic algorithm named Minimum Longest Queue Finish Time (MLFT). In the proposed scheme, we first divide the high computation task into multiple sub-tasks, and re-organize all the tasks into multiple task queues to shorten the entire finish time of all the tasks submitted to the cluster and launched in parallel according to load balancing. In the task division process, an adaptive segmentation algorithm is proposed according to the complexity and maximum segmentation granularity of the input task. Based on the proposed algorithm, an efficient parallel video transcoding framework with cloud computing is presented. Experimental results show that the proposed algorithm outperforms the existing algorithms significantly on the entire finish time of the tasks and approaches to the optimal solution closely.

In traditional four-stage pipeline structures for H.264 video encoder hardware implementation, rate distortion optimization (RDO) based mode decision was turned off, and dual-port or ping-pang on-chip search window SRAM was used to achieve data reuse between the integer and fractional pixel motion estimation. To support RDO based mode decision for efficient high definition AVS video coding implementation, we propose an improved four-stage MB pipeline structure. Also on-chip buffer structure is optimized to achieve the balance between circuit consumption and coding performance. The Jizhun profile AVS video encoder is successfully mapped into hardware implementation with the proposed pipeline structure with small performance degradation.

This paper proposes a hardware friendly multi-resolution motion estimation algorithm and VLSI architecture for high definition MPEG-like video encoder hardware implementation. By parallel searching and utilizing the high correlation in multi-resolution reference pixels, huge throughput and computation due to large search window are alleviated considerably. Sixteen way parallel processing element arrays with configurable multiplying technologies achieve fast search with regular data access and efficient data reuse. Also, the parallel arrays can be efficiently reused at three hierarchical levels for sequential motion vector refinement. The modified algorithm reaches a good balance between implementation complexity and search performance. Also, the logic circuit and on-chip SRAM consumption of the VLSI architecture are moderate.

High throughput, heavy bandwidth requirement, huge on-chip memory consumption, and complex data flow control are major challenges in high definition integer motion estimation hardware implementation. This paper proposes an efficient very large scale integration architecture for integer multi-resolution motion estimation based on optimized algorithm. There are three major contributions in this paper. First, this paper proposes a hardware friendly multi-resolution motion estimation algorithm well-suited for high definition video encoder. Second, parallel processing element (PE) array structure is proposed to implement three-level hierarchical motion estimation, only 256PEs are enough for one reference frame real-time high definition motion estimation by efficient PE reuse. Third, efficient on-chip reference pixel buffer sharing mechanism between integer and fractional motion estimation is proposed with almost 50% SRAM saving and memory bandwidth reduction. The proposed multi-resolution motion estimation algorithm reached a good balance between complexity and performance with rate distortion optimized variable block size motion estimation support. Also, we have achieved moderate logic circuit and on-chip SRAM consumption. The proposed architecture is well-suited for all MPEG-like video coding standards such as H.264, audio video coding standard, and VC-1.

High external memory bandwidth requirement is one major challenge for efficient hardware motion estimation (ME) implementation. Large double-buffered on-chip search window (SW) buffer is usually employed to increase the throughput. In this paper, we focus on SW buffer structure optimization in a systematic viewpoint. An efficient buffer share mechanism is proposed to minimize the memory consumption, simultaneously alleviate the external memory access bandwidth. Moreover, variable block size ME (VBSME) and large SW in high definition (HD) video encoder are both supported with good trade off among throughput, data regularity, and rate distortion performance. The simplified algorithm can achieve nearly 50% SW buffer saving with less than 0.15 dB PSNR degradation at the worst case compared with full search VBSME.

This demonstration shows an AVS intra decoder configuration in the RVC framework. It explains how to use the dataflow mechanism offered by the RVC framework to support AVS decoder configuration. It also shows the flexibility and convenience to reconfigure decoders in the RVC framework. In this work, the AVS VTL is established containing FUs from AVS. The proposed AVS decoder configuration is implemented by connecting some FUs from AVS VTL and reusing some FUs from MPEG VTL. The demonstration shows that the decoder can decode AVS conformance bitstreams correctly in RVC simulator.

In this paper, a high accuracy and low-complexity fixed-point inverse discrete cosine transform (IDCT) based on AAN's fast algorithm is proposed for the approximation of the real IDCT. The proposed transform can be implemented with 16-bit multiplication and 24-bit addition operations. A common factor extraction algorithm is used to obtain a new set of real factors of IDCT whose integer approximations can be implemented with few arithmetic operations. A two-stage scale approach is proposed for constraining the dynamic range to a narrow range in scale part. The experimental results show that the proposed transform has significantly higher accuracy than the thresholds of IEEE1180-1190. The results of the proposed IDCT implemented into the decoder of MPEG-2 also show that the fixed-point IDCT achieves higher performance.

In this paper, a low-complexity adaptive block-size transform (ABT) scheme is proposed for the H.264/AVC. The integer 8x8/4x4 discrete cosine transforms with an extended relation are applied to the proposed ABT scheme for reducing the implementation complexities. In the proposed scheme, the 8x8/4x4 transforms can be merged together and share the same scale matrix. As a result, the transform units and storage resources can be effectively saved in both encoder and decoder. The 8x8 transform extended from the 4x4 transform of H.264/AVC can be easily implemented with several simple arithmetic operators, and its intermediate results can be strictly limited within 16-bit. The experimental results show that the proposed ABT scheme based on extended transforms achieves a similar coding performance to the original ABT scheme for H.264/AVC.

In this paper, a low-complexity 8times8/4times4 adaptive block-size transform (ABT) scheme is tentatively proposed for the Chinese audio and video coding standard (AVS). In the proposed ABT scheme, an integer 8times8 transform is derived from the integer 4times4 transform used in AVS according to a transform extension principle. The 8times8 transform not only has high energy compacted property but also can be merely implemented within several additions and shifts, and all intermediate results are limited with 16-bit. The 8times8 transform and 4times4 transform can be merged together and share the same scale matrix so that the hardware units and storage resources are efficiently saved for both encoder and decoder. The experimental results on numerous sequences show that the proposed ABT scheme can achieve significant performance improvement for AVS