Facial Expression Recognition is an active area of research in computer vision with a wide range of applications. Several approaches have been developed to solve this problem for different benchmark datasets. However, Facial Expression Recognition in the wild remains an area where much work is still needed to serve real-world applications. To this end, in this paper we present a novel approach towards facial expression recognition. We fuse rich deep features with domain knowledge through encoding discriminant facial patches. We conduct experiments on two of the most popular benchmark datasets; CK and TFE. Moreover, we present a novel dataset that, unlike its precedents, consists of natural — not acted — expression images. Experimental results show that our approach achieves state-of-the-art results over standard benchmarks and our own dataset.

We introduce a novel approach towards scene recognition using semantic segmentation maps as image representation. Given a set of images and a list of possible categories for each image, our goal is to assign a category from that list to each image. Our approach is based on representing an image by its semantic segmentation map, which is a mapping from each pixel to a predefined set of labels. Among similar high-level approaches, ours has the capability of not only representing what semantic labels the scene contains, but also their shapes, sizes and locations. We also investigate the effect of varying experiment parameters, including varying labels used, semantic segmentation technique, and semantic training source. We obtain state-of-the-art results over Siftflow and MSRC-21 datasets.

AutoCaption is a system that helps a smartphone user generate a caption for their photos. It operates by uploading the photo to a cloud service where a number of parallel modules are applied to recognize a variety of entities and relations. The outputs of the modules are combined to generate a large set of candidate captions, which are returned to the phone. The phone client includes a convenient user interface that allows users to select their favorite caption, reorder, add, or delete words to obtain the grammatical style they prefer. The user can also select from multiple candidates returned by the recognition modules.

We introduce new approaches for augmenting annotated training datasets used for object detection tasks that serve achieving two goals: reduce the effort needed for collecting and manually annotating huge datasets and introduce novel variations to the initial dataset that help the learning algorithms. The methods presented in this work aim at relocating objects using their segmentation masks to new backgrounds. These variations comprise changes in properties of objects such as spatial location in the image, surrounding context and scale. We propose a model selection approach to arbitrate between the constructed model on a per class basis. Experimental results show gains that can be harvested using the proposed approach.

We investigate the problem of stitching timely synchronized video streams captured by freely moving devices. Recently, it was shown that using frame-to-frame correlation can greatly enhance the efficiency and effectiveness of video stitching algorithms [19]. In this paper, we address some of the shortcomings in [19], namely the simple blending approach, causing almost a third of stitching errors and the fact that the stitching algorithm is only tested on a frame-by-frame basis which does not realistically mimic the user perception of the output system quality as a complete video. We propose the use of a modified blending technique based on optimal seam selection and experimentally validate its superiority using precision, recall and F1measures on a frame-by-frame basis, while maintaining low computational complexity. Furthermore, we validate that the performance gains measured on a frame-by-frame basis are also evident when the stitched video output is evaluated as a single unit.

Among the approaches for solving the semantic image segmentation problem that has proven successful is in formulating an energy minimization expressed on top of a conditional random field (CRF) over image pixels. Recently, high order potentials (cliques of size greater than 2) over superpixels have been incorporated in the CRF energy function yielding promising results. These potentials encourage pixels within the same superpixel to take the same label by penalizing inconsistent labeling within the superpixel. While some of the earlier attempts modeled higher order potentials without considering the conditional dependencies between superpixels, others modeled these dependencies at the cost of oversimplified models at higher levels. In this paper, we propose incorporating superpixel neighborhood information within the high order potential, hence modeling dependencies between superpixels without the need of oversimplifying or constraining the model. Results show that the proposed method achieves state-of-the-art results on the challenging PASCAL VOC 2007 dataset.

Mobile phones are becoming more and more ubiquitous with a large number of these devices having image/video capturing capabilities, connection capabilities and built-in rich sensory. This has encouraged the common user to capture more image/video content than ever before. However, this has created two interrelated problems: 1) while capturing some scene the user may want to get more information about it to make a decision (e.g. a buying one) without painful textual input on these mobile devices and while accounting for multiple meanings associated with a single image, as an image is worth a thousand words and 2) videos captured cannot be easily searched afterwards and hence forgotten due to the lack of proper indexing techniques. In this paper, we are presenting a system addressing the above two problems through a single solution by providing users with real-time automatically generated tags of their currently captured videos. The user can select/deselect from the automatic tags, thus tags can serve as visual query suggestions helping bridging the user's query intent. This same set of tags will be stored with the video for enabling easy content access afterwards.

Constructing a panoramic video out of multiple incoming live mobile video streams is a challenging problem. This problem involves multiple users live streaming the same scene from different angles, using their mobile phones, with the objective of constructing a panoramic video of the scene. The main challenge in this problem is the lack of coordination between the streaming users, resulting in too much, too little, or no overlap between incoming streams. To add to the challenge, the streaming users are generally free to move, which means that the amounts of overlap between the different streams are dynamically changing. In this paper, we propose a method for automatically coordinating between the streaming users, such that the quality of the resulting panoramic video is enhanced. The method works by analyzing the incoming video streams, and automatically providing active feedback to the streaming users. We investigate different methods for generating the active feedback and presenting it to the streaming users resulting in an improved panoramic video output compared to the case where no feedback is utilized.

Object instance matching is a cornerstone component in many computer vision applications such as image search, augmented reality and unsupervised tagging. The common flow in these applications is to take an input image and match it against a database of previously enrolled images of objects of interest. This is usually difficult as one needs to capture an image corresponding to an object view already present in the database, especially in the case of 3D objects with high curvature where light reflection, viewpoint change and partial occlusion can significantly alter the appearance of the captured image. Rather than relying on having numerous views of each object in the database, we propose an alternative method of capturing a short video sequence scanning a certain object and utilize information from multiple frames to improve the chance of a successful match in the database. The matching step combines local features from a number of frames and incrementally forms a point cloud describing the object. We conduct experiments on a database of different object types showing promising matching results on both a privately collected set of videos and those freely available on the Web such that on YouTube. Increase in accuracy of up to 20% over the baseline of using a single frame matching is shown to be possible.

We investigate the problem of efficient panoramic video construction based on time-synchronized input video streams. No additional constraints are imposed regarding the motion of the capturing video cameras. The presented work is, to the best of our knowledge, the first attempt to construct in real-time a panoramic video stream from input video streams captured by freely moving cameras. The main contribution is in proposing an efficient panoramic video construction algorithm that exploits temporal information to avoid solving the stitching problem fully on a frame by frame basis. We provide detailed experimental evaluation of different methodologies that employ previous frames stitching results such as tracking interest points using optical flow and using areas of overlap to limit the search space for interest points. Our results clearly indicate that making use of temporal information in video stitching can achieve a significant reduction in execution time while providing a comparable effectiveness.

User generated videos with mobile phone cameras are becoming more and more ubiquitous allowing people to share live content with remote parties, possibly in real-time. However with limited mobile phone capabilities, these videos are usually of small resolution, resulting in a small field of view for acceptable quality. Fortunately with the proliferation of video capture-enabled mobile phones, there is high chance that one or more persons will be shooting the same scene from different views. In this demonstration, we are showing an end-to-end system which receives video streams coming from different mobile phones, time synchronizes the streams and produces a single composite mosaic video, and all of this is done in real-time. The proposed system operates without coordination between users. The system has been tested under various capturing conditions such as indoor, outdoor, day and night conditions.

Lately, the usage of mobile live video streaming for video sharing has been growing steadily. However, because of the limitations presented by the capabilities of mobile phone video capturing devices, these video streams end up having either a low resolution or a small field of view. On the other hand, the ubiquity of video capture-capable mobile phones make the probability that more than one user will be recording the same scene from different views relatively high.