As an alternative to bar-code scanning, we are developing a real-time retail product detector for point-of-sale automation. The major challenge associated with image based object detection arise from occlusion and the presence of other objects in close proximity. For robust product detection under such conditions, it is crucial to train the detector on a rich set of images with varying degrees of occlusion and proximity between the products, which fairly represents a wide range of customer tendencies of placing products together. However, generating a fairly large database of such images traditionally requires a large amount of human effort. On the other hand, acquiring individual object images with their corresponding masks is a relatively easy task. We propose an realistic image synthesis approach which uses individual object images and their corresponding masks to create training images with desired properties (occlusion and congestion among the products). We train our product detector over images thus generated and achieve a consistent performance improvement across different types of test data. With the proposed approach, detector achieves an improvement of 46.2% (from 0.67 to 0.98) and 40% (from 0.60 to 0.84) over precision and recall respectively, compared to using a basic training dataset containing one product per image.

We propose a robust carried object recognition method from surveillance video. Though tremendous strides have been made in the object recognition by using the appearance-based method such as Faster R-CNN, YOLO, etc., recognizing carried object classes is still a difficult task. It is because the carried objects in surveillance video are of small size, low-resolution, occluded by people. To address the problem, we introduce prior knowledge of the location relation between carried objects and human body parts to improve the performance of the appearance-based method. Specifically, for the extracted human region, we detect human body parts and use prior knowledge to estimate the probability distribution of the existence of each class of carried objects. We update the score results of the appearance-based method by applying the probability distribution. In this way, we can reduce false positives and enhance confidence score of true positives. We evaluate the proposed method on four datasets and show that the proposed method outperforms Faster R-CNN by 6% in terms of average F-measure.

Recent advancements in image recognition technologies has enabled image recognition-based systems to be widely used in real world applications. In this talk, I will introduce NEC's image-based object recognition technologies targeted for recognizing various manufactured goods and retail products from a camera, and talk about their industrial applications which we have developed and commercialized. These image-based object recognition technologies enable highly efficient and cost-effective management of goods and products throughout their life-cycle (manufacturing, distribution, retail, and consumption), which otherwise cannot be achieved by human labor or by use of ID tags. Firstly, I will talk about a technology to recognize multiple objects from a single image using feature matching of compact local descriptors, combined with a more recent Deep Learning-based recognition. It enables large number of objects to be recognized at once, which greatly reduces human labor and time for various product inspection and checking works. Using this technology, we have developed and commercialized the product inspection system in warehouses, the planogram recognition system for retail shop shelves, and the self-service POS system for easy-to-use and fast checkout in retail stores. Secondly, I will talk about the "Fingerprint of Things'' technology. It enables individual identification of tiny manufactured parts (e.g. bolts and nuts) by identifying images of their unique surface patterns, just like human fingerprints. We have built a prototype of mass-produced parts traceability system, which enables users to easily track down the individual parts using a mobile device. In the talk, I will explain the key issues in realizing these industrial applications of image-based object recognition technologies.

In this paper, we propose a fast non-iterative camera pose voting method for 3D object identification. The proposed method improves the accuracy and speed upon the conventional local feature based 2D-to-3D matching between a 2D image and a 3D model reconstructed by the structure-from-motion (SfM) pipeline. Instead of performing iterative RANSAC based method for geometric verification, the proposed method computes a hypothesis of camera pose from each feature correspondence between the query image and the 3D model. The camera pose is computed using scale, orientation and coordinate of the local features, calibrated by the camera matrix of the database image used to construct the 3D model. Then the most likely hypothesis is found by carrying out a two-stage clustering on the estimated camera poses in the parameter space. Experiment results on the 3D machine datasets show that our method improves the identification accuracy from 82.8% to 84.9% when FPR is 1%, compared with conventional RANSAC based method. In addition, the processing speed for the geometric verification is improved up to 25 times compared to the conventional method.

This paper proposes a method to detect and identify multiple objects in an image using grid voting of object center positions estimated from local descriptor keypoint matches. For each keypoint match, the proposed method estimates the object center position using scale and orientation associated with the keypoints. Then, it casts a vote for an image grid where the estimated object center is located. For the grids with high number of votes, geometric verification of the keypoint matches is carried out to accurately localize multiple objects in the image. Since the computational complexity of the grid voting is O(n), where n is the number of estimated object centers, the proposed method runs faster than a conventional method using mean shift clustering with O(n2) complexity. Experimental results using images with 52 objects show that the proposed method reduces the computational time by approximately 60% compared to the conventional method, while identification accuracy is comparable. With the reduced computational complexity, industrial applications such as an efficient inventory management in retail using images are enabled in practical computational time.

This paper proposes a new scalable and compact binary local descriptor, named the BRIGHT (Binary ResIzable Gradient HisTogram) descriptor, for low-latency and high accuracy identification of real-world objects in images. The BRIGHT descriptor is extracted by first creating a hierarchical HOG (Histogram of Oriented Gradients) of a local patch centered around keypoints detected from an image. The elements of the histogram are then binarized, and the subset of bits is progressively selected forming a progressively scalable descriptor with a size ranging from only 32 bits to 150 bits. Experiment using images with objects taken under various camera viewpoints, lighting conditions, and occlusions, shows that the BRIGHT descriptor can robustly match objects with an identification accuracy comparable with that of SIFT descriptor, but at a descriptor size smaller than 1/10 of SIFT. With the reduced descriptor size, transmission of descriptors from a mobile device to a database server can be dramatically speeded up, enabling low-latency response in mobile search services.

This paper presents the core technologies of the video signature tools recently standardized by ISO/IEC Moving Picture Experts Group (MPEG) as an amendment to the MPEG-7 Standard (ISO/IEC 15938). The video signature is a high-performance content fingerprint that is suitable for desktop scale to web-scale deployment and provides high levels of robustness to common video editing operations and high temporal localization accuracy at extremely low false alarm rates, achieving a detection rate in the order of 96% at a false alarm rate in the order of five false matches per million comparisons. The applications of the video signature are numerous and include rights management and monetization, distribution management, usage monitoring, metadata association, and corporate or personal database management. In this paper, we review the prior work in the field, explain the standardization process and status, and provide details and evaluation results for the video signature tools.

This paper proposes detection of wipes and digital video effects (DVEs) in a video sequence based on a new pattern-independent model. This model is based on the characteristics of image boundary lines dividing the two image regions in the transitional frames. Wipes and DVEs are modeled as frame sequences where either (A) a single boundary line moves continuously in a time sequence, or (B) multiple boundary lines form a quadrilateral within a frame. The model is applied to the image boundary lines extracted from a video sequence to detect wipes and DVEs. An evaluation using news programs containing various patterns of wipes and DVEs shows that the proposed method achieves recall of 91.5 % and precision of 60.7 %, improving the conventional twin-comparison method by 29.6 % in recall and 46.5 % in precision.

This paper proposes an image signature robust to caption superimposition for video sequence identification. A new image signature which is a set of local features is developed for a high-speed frame-by-frame matching of video sequences. The signature of a frame is obtained by partitioning the image into blocks and extracting the local feature representing the dominant type of edge direction from each block. The similarity between the signatures is calculated by comparing the edge types of the corresponding blocks, and counting the number of the blocks having the same edge type. A weighting scheme based on the probability of caption superimposition for each block can be applied to the similarity calculation to improve the matching performance. The experimental results of the video sequence identification show that the proposed signature achieves precision of 99.65% and recall of 99.45%, improving both the precision and the recall by more than 30% compared with the conventional signature.