In face detection, it is a common strategy to treat samples differently according to their difficulty for balancing training data distribution. However, we observe that widely used sampling strategies, such as OHEM and Focal loss, can lead to the performance imbalance between different tasks (e.g., classification and localization). Through analysis, we point out that, due to the driving of classification information, these sample-based strategies are difficult to coordinate the attention of different tasks during the training, thus leading to the above imbalance. Accordingly, we first confirm this by shifting the attention from the sample level to the task level. Then, we propose a fine-grained task attention method, a.k.a FTA, including inter-task importance and intra-task importance, which adaptively adjusts the attention of each item in the task from both global and local perspectives, so as to achieve finer optimization. In addition, we introduce transformer as a feature enhancer to assist our convolution network, and propose a context enhancement transformer, a.k.a CET, to mine the spatial relationship in the features towards more robust feature representation. Extensive experiments on WiderFace and FDDB benchmarks demonstrate that our method significantly boosts the baseline performance by 2.7%, 2.3% and 4.9% on easy, medium and hard validation sets respectively. Furthermore, the proposed FTAFace-light achieves higher accuracy than the state-of-the-art and reduces the amount of computation by 28.9%.

In this paper, the framework for a real-time view recommendation system is proposed. The proposed system comprises two parts: offline aesthetic modeling stage and efficient online aesthetic view finding process. A preference-aware aesthetic model is proposed to suggest views according to varied user-favorite photographic styles, where a bottom-up approach is developed to construct an aesthetic feature library with bag-of-aesthetics-preserving features instead of top-down methods that implement the heuristic guidelines (rule-specific features) listed in photography literatures, which is employed in previous works. A collection of scenic photos is used as the test set; however, the proposed method can be employed to other types of photo collection according to different application scenarios. The proposed model can cover both implicit and explicit aesthetic features and can adapt to users' preferences with a learning process. In the second part, the learned model is employed in a view finder to help the user to locate the most aesthetic view while taking a photograph. The experimental results show that the proposed features in the library (92.06% in accuracy) outperform the state-of-the-art rule-specific features (83.63% in accuracy) significantly in the photo aesthetic quality classification task, and the rule-specific features are also proved to be encompassed by the proposed features. Meanwhile, it is observed from experiments that the features extracted for contrast information are more effective than those for absolute information, which is consistent with the properties of human visual systems. Furthermore, the user studies for the view recommendation task confirm that the suggested views are consistent with users' preferences (81.25% agreements).

Feature matching is an indispensable process for object recognition, which is an important issue for wearable devices with video analysis functionalities. To implement a low-power SoC for object recognition, the proposed visual vocabulary processor (VVP) is employed to accelerate the speed of feature matching. The VVP can transform hundreds of 128-D SIFT vectors into a 64-D histogram for object matching by using the binary-tree-based architecture, and 16 calculators for the computations of the Euclidean distances are designed for each of the two processors in each level. A total of 126 visual words can be saved in the six-level hierarchical memory, which instantly offers the data required for the matching process, and more than 5 times of bandwidth can be saved compared with the non-binary-tree-based architecture. As a part of the recognition SoC, the VVP is implemented with the 65-nm CMOS technology, and the experimental results show that the gate count and the average power consumption are 280 K and 5.6 mW, respectively.

In this paper, an aesthetic modeling method for scenic photographs is proposed. A bottom-up approach is developed to construct an aesthetic library with bag-of-aesthetics preserving features instead of top-down methods that implement the heuristic guidelines (rule-specific features) listed in the photography literature, which is employed in previous works. The proposed method can cover both implicit and explicit aesthetic features with a learning process. The experimental results show that the proposed features in the library (92.06% in accuracy) outperform the state-of-the-art rule-specific features (83.63% in accuracy) significantly in the aesthetic quality assessment for scenic photos, and the rule-specific features are also proved to be encompassed by the proposed features. Meanwhile, it is observed from experiments that the features extracted for contrast information are more effective than those for absolute information, which is consistent with the properties of human visual systems.

A new photo retrieval system for mobile devices is proposed. The system can be used to search for photos with similar spatial layouts efficiently, and it adopts an image segmentation algorithm that extracts features of image regions based on K-Means clustering. Since K-Means is computationally intensive for real-time applications and prone to generate clustering results with local optima, parallel hardware architectures are designed to meet the real-time requirement of the retrieval process. Experiments show that the proposed algorithm in the photo retrieval system obtains better mean average precision than other methods, and it is tested with image recognition problems. The robustness of the algorithm is also evaluated with noise and image blurring. Besides, the proposed K-Means hardware can provide a trade-off between the execution time and the retrieval performance on the software and hardware cosimulation platform. The contribution of this work is twofold. The first is the development of a photo retrieval framework for mobile devices, where a new texture feature is employed in the algorithm to enhance the retrieval performance. The other is the integration of the K-Means hardware accelerator and the photo retrieval system. The hardware architecture is analyzed, and the specifications are compared with previous works.

K-Means is an important clustering algorithm that is widely applied to different applications, including color clustering and image segmentation. To handle large cluster numbers in embedded systems, a hardware architecture of hierarchical K-Means (HK-Means) is proposed to support a maximum cluster number of 1024. It adopts 10 processing elements for the Euclidean distance computations and the level-order binary-tree traversal. Besides, a hierarchical memory structure is integrated to offer a maximum bandwidth of 1280 bit/cycle to processing elements. The experiments show that applications such as video segmentation and color quantization can be implemented based on the proposed HK-Means hardware. Moreover, the gate count of the hardware is 414 K, and the maximum frequency achieves 333 MHz. It supports the highest cluster number and has the most flexible specifications among our works and related works.

A new image segmentation method is proposed to combine the edge information with the feature-space method, K-Means clustering. A procedure called seam processing, which is computationally efficient, is employed to search for horizontal and vertical seams that contain edge information. By transforming the spatial coordinates based on the seam detection results, the edge information can be added to the feature vectors, which are the inputs of K-Means algorithm. The experiments show that the proposed method can achieve edge-adaptive segmentation results, which can not be obtained using traditional methods based on K-Means clustering.

K-Means is a clustering algorithm that is widely applied in many fields, including pattern classification and multimedia analysis. Due to real-time requirements and computational-cost constraints in embedded systems, it is necessary to accelerate K-Means algorithm by hardware implementations in SoC environments, where the bandwidth of the system bus is strictly limited. In this paper, a bandwidth adaptive hardware architecture of K-Means clustering is proposed. Experiments show that the proposed hardware can be used in applications such as image segmentation, and it has the maximum clock speed 400-MHz and 440-K gate count with TSMC 90-nm technology. Moreover, the throughput of the proposed hardware reaches 16 dimension/cycle, and it can deal with feature vectors with different dimensions using five parallel modes to utilize the input bandwidth efficiently.

K-means is a clustering algorithm that is widely applied in many fields, including pattern classification and multimedia analysis. Due to real-time requirements and computational-cost constraints in embedded systems, it is necessary to accelerate k-means algorithm by hardware implementations in SoC environments, where the bandwidth of the system bus is strictly limited. In this paper, a bandwidth adaptive hardware architecture of k-means clustering is proposed. Experiments show that the proposed hardware has the maximum clock speed 400 MHz with TSMC 90 nm technology, and it can deal with feature vectors with different dimensions using five parallel modes to utilize the input bandwidth efficiently.

In the next-generation visual surveillance systems, content analysis tools will be integrated. In this paper, to accelerate these tools, it is proposed to integrate a hardware content analysis engine into a smart camera system-on-a-chip (SoC). A smart camera SoC hardware architecture with the proposed visual content analysis engine is first presented. This engine consists of dedicated accelerators and a programmable morphology coprocessor. Stream processing design concept, frame-level pipelining, and subword level parallelism are employed together to efficiently utilize the bandwidth of the system bus and achieve high throughput. The implementation results show that, with 168 K logic gates and 40.63 Kb on-chip memory, a processing speed of 30 640 x 480 frames/s can be achieved, while the operations of video object segmentation, object description and tracking, and face detection and scoring are supported.

A new and efficient algorithm to find the fingertips for human computer interface is proposed in this paper.The Fingertip Positioning algorithm of this paper combines particle filtering with Particle Random Diffusion to find the different fingertips quickly and robustly. There are two special methods used in this algorithm, which are Particle Random Diffusion and Fingertip Particle Selection. Without checking every pixel in the image of video sequence, this algorithm works well with low computational cost. This algorithm comes out with good performance in cluttered backgrounds and processes in real time. Based on the accurate positions of fingertips, hand gesture recognition can be done efficiently and robustly.

K-Means is a clustering algorithm that is widely applied in many fields, including pattern classification, multimedia analysis, and image retrieval. Due to real-time requirements of image segmentation in embedded systems, it is necessary to accelerate K-Means algorithm by hardware implementations. The contribution of this paper includes a series of K-means hardware analyses and a newly proposed SIP for image segmentation in SoC environments. Experiments show that the proposed SIP has the maximum clock speed 200MHz with TSMC 0.18μm technology, and that it can be successfully used for image segmentation on an FPGA board with AMBA AHB.

A fast and efficient approach for color image segmentation is proposed. In this work, a new quantization technique for HSV color space is implemented to generate a color histogram and a gray histogram for K-Means clustering, which operates across different dimensions in HSV color space. Compared with the traditional K-Means clustering, the initialization of centroids and the number of cluster are automatically estimated in the proposed method. In addition, a filter for post-processing is introduced to effectively eliminate small spatial regions. Experiments show that the proposed segmentation algorithm achieves high computational speed, and salient regions of images can be effectively extracted. Moreover, the segmentation results are close to human perceptions.

A robust video object segmentation algorithm for complex conditions in surveillance systems is proposed in this paper. This algorithm contains an unsupervised K-Means background clustering technique to model the temporal distribution in RGB domain for each spatial position. Based on the proposed background model, the object mask generation process integrates noise reduction, cast shadow cancellation, and improved watershed transform to obtain satisfying object masks. Experiments show that it can be applied on low-fame-rate and noisy video sequences in surveillance systems in which temporal tracking becomes impractical, and achieve better segmentation results than the previous works for complex lighting conditions and outdoor scenes.

The high performance of the conventional spatio-temporal image noise reduction filters comes at the cost of high computationally intensity. In this paper, a motion adaptive spatio-temporal filter is proposed for double-shot images to achieve high performance with low computing power requirement. Two images are used simultaneously to separate the static regions and dynamic regions, where different noise reduction approaches are employed. Temporal average and 2-D adaptive filter is applied for static regions, where spatio-temporal filter with motion compensation is only applied for dynamic regions. Experiments show that the proposed noise reduction filter can achieve better noise reduction performance with less computationally intensity than other previous spatio-temporal filters.

Facial images with low resolution in surveillance sequences are hard to detect with traditional approaches, and the quality of these faces is a significant factor for human face recognition. A new technique called face scoring, which determines the face scores based on face quality, is proposed. It combines spirits of image-based face detection and essences of video object segmentation to filter out face candidates. Besides, the face scoring technique includes eight scoring functions based on feature extraction technique, integrated by a single layer neural network training system to obtain an optimal linear combination to select high-quality faces. In the proposed algorithm, the way to choose input vector is quite different from traditional approaches and has good properties. Experiments show that the proposed algorithm effectively extracts low-resolution human faces, which traditional algorithm cannot handle well. It can also rank face candidates according to face scores, which is useful for surveillance video summary and indexing.

Facial images with low resolution in surveillance sequences are hard to detect with traditional approaches. An efficient face detection and face scoring technique in surveillance systems is proposed. It combines spirits of image-based face detection and essences of video object segmentation to filter out high-quality faces. The proposed face scoring technique, which is useful for surveillance video summary and indexing, includes four scoring functions based on feature extraction and is integrated by a neural network training system to select high-quality face. Experiments show that the proposed algorithm effectively extracts low-resolution human faces, which traditional face detection algorithm cannot handle well. It can also rank face candidates according to face scores, which determine face quality.