What is the current state-of-the-art for image restoration and enhancement applied to degraded images acquired under less than ideal circumstances? Can the application of such algorithms as a pre-processing step improve image interpretability for manual analysis or automatic visual recognition to classify scene content? While there have been important advances in the area of computational photography to restore or enhance the visual quality of an image, the capabilities of such techniques have not always translated in a useful way to visual recognition tasks. Consequently, there is a pressing need for the development of algorithms that are designed for the joint problem of improving visual appearance and recognition, which will be an enabling factor for the deployment of visual recognition tools in many real-world scenarios. To address this, we introduce the UG$^2$dataset as a large-scale benchmark composed of video imagery captured under challenging conditions, and two enhancement tasks designed to test algorithmic impact on visual quality and automatic object recognition. Furthermore, we propose a set of metrics to evaluate the joint improvement of such tasks as well as individual algorithmic advances, including a novel psychophysics-based evaluation regime for human assessment and a realistic set of quantitative measures for object recognition performance. We introduce six new algorithms for image restoration or enhancement, which were created as part of the IARPA sponsored UG$^2$Challenge workshop held at CVPR 2018. Under the proposed evaluation regime, we present an in-depth analysis of these algorithms and a host of deep learning-based and classic baseline approaches. From the observed results, it is evident that we are in the early days of building a bridge between computational photography and visual recognition, leaving many opportunities for innovation in this area.

This paper presents a learning-based method for recognizing chess pieces from depth information. The proposed method is integrated in a recreational robotic system that is designed to play games of chess against humans. The robot has two arms and an Ensenso N35 Stereo 3D camera. Our goal is to provide the robot visual intelligence so that it can identify the chess pieces on the chessboard using the depth information captured by the 3D camera. We build a convolutional neural network to solve this 3D object recognition problem. While training neural networks for 3D object recognition becomes popular these days, collecting enough training data is still a time-consuming task. We demonstrate that it is much more convenient and effective to generate the required training data from 3D CAD models. The neural network trained using the rendered data performs well on real inputs during testing. More specifically, the experimental results show that using the training data rendered from the CAD models under various conditions enhances the recognition accuracy significantly. When further evaluations are done on real data captured by the 3D camera, our method achieves 90.3% accuracy.

Automatic photo cropping is an important tool for improving visual quality of digital photos without resorting to tedious manual selection. Traditionally, photo cropping is accomplished by determining the best proposal window through visual quality assessment or saliency detection. In essence, the performance of an image cropper highly depends on the ability to correctly rank a number of visually similar proposal windows. Despite the ranking nature of automatic photo cropping, little attention has been paid to learning-to-rank algorithms in tackling such a problem. In this work, we conduct an extensive study on traditional approaches as well as ranking-based croppers trained on various image features. In addition, a new dataset consisting of high quality cropping and pairwise ranking annotations is presented to evaluate the performance of various baselines. The experimental results on the new dataset provide useful insights into the design of better photo cropping algorithms.

Since a proficient golf swing is a key element in success in golf, golfers always make much effort in improving their swing skills. However, if not under the supervision of a coach, a golfer cannot effectively perceive his/her faults in swings and comprehend how to change the swing mechanics. In this paper, we propose a vision-based golf training system, named "Improve My Golf Swing," which aims at improving the golfer's swing skills using intelligent glasses serving as a wearable displayer. The golfer's posture is analyzed through computation-efficient image processing techniques. Involving professional golf training knowledge, the proposed system automatically generates feature lines to evaluate whether the golfer's body is properly aligned. During a swing, the golfer can watch his/her own posture as well as the augmented feature lines displayed on the intelligent glasses. Instead of quantitative swing analysis, we focus on providing instructions of posture adjustment to the golfer in a more intuitive and unambiguous manner. Different audio alerts will be given to the golfer according to which part of a posture fault is detected. Experiments on different golf swing videos show encouraging results.

The purpose of this research is to recognize The Human faces by analyzing the brain waves, detected by EEG, of subject. Initially, a subject looked at an image of human face until the subject remembered the facial features of the face. Then we carried out the EEG experiment. Each face used in the experiment was divided into three regions: eyes, nose and mouth, and two other faces were randomly selected. Totally, nine regions, three for each face, were randomly selected and shown on the monitor for the subject. Based on the collected brain waves, the amplitude of P300 signals achieved a significant difference between the regions of the remembered face and other two faces. In this way, a face by combining eyes, nose and mouth could be generated which is the same as the remembered face. We had succeeded recognize a human face by retrieving the brain waves after the subject watching at the facial regions. This is very useful to find out the criminal by the witness who was only required to watch the facial features.

This paper presented a brain-computer Chinese Spelling System, which allow the users to spell out Chinese words via his/her brain signals. This system analyzed brain signals by ERPs to find out the subject's demand. To detect subject's demand, the flashing phonetic symbols, or Chinese words, were showed in a matrix for subject. The system would totally collected brain signals while matrix flashing, and then analyzed the collected signals by ERPs. Based on analyzed ERPs, the P300 achieved a significant difference between two events, target and non-target, in statistics. Based on the significant difference, the system could detect the subject's demands and showed the detected phonetic symbols in the monitor. When all the symbols had been selected, the system would automatically spell Chinese words. According to experiment, the accuracy of this system was up to 90 percent. The development of this system provides severe motor disabilities a tool to communicate with external environment.

We present a novel image representation that characterizes a color image by an intensity image and a small number of color pixels. Our idea is based on solving an inverse problem of colorization: Given a color image, we seek to obtain an intensity image and a small subset of color pixels, which are called landmark pixels, so that the input color image can be recovered faithfully using the intensity image and the color cues provided by the selected landmark pixels. We develop an algorithm to derive the landmark-based sparse color representations from color images, and use the representations in the applications of color transfer and color correction. The computational cost for these applications is low owing to the sparsity of the proposed representation. The landmark-based representation is also preferable to statistics-based representations, e.g. color histograms and Gaussian mixture models, when we need to reconstruct the color image from a given representation.