Answering questions about complex situations in videos requires not only capturing the presence of actors, objects, and their relations but also the evolution of these relationships over time. A situation hyper-graph is a representation that describes situations as scene sub-graphs for video frames and hyper-edges for connected sub-graphs and has been proposed to capture all such information in a compact structured form. In this work, we propose an architecture for Video Question Answering (VQA) that enables answering questions related to video content by predicting situation hyper-graphs, coined Situation Hyper-Graph based Video Question Answering (SHG- VQA). To this end, we train a situation hyper-graph decoder to implicitly identify graph representations with actions and object/human-object relationships from the input video clip. and to use cross-attention between the predicted situation hyper-graphs and the question embedding to predict the correct answer. The proposed method is trained in an end-to-end manner and optimized by a VQA loss with the cross-entropy function and a Hungarian matching loss for the situation graph prediction. The effectiveness of the proposed architecture is extensively evaluated on two challenging benchmarks: AGQA and STAR. Our results show that learning the underlying situation hyper-graphs helps the system to significantly improve its performance for novel challenges of video question-answering tasks11Code will be available at https://github.com/aurooj/SHG-VQA.

The problem of grounding VQA tasks has seen an increased attention in the research community recently, with most attempts usually focusing on solving this task by using pretrained object detectors. However, pre-trained object detectors require bounding box annotations for detecting relevant objects in the vocabulary, which may not always be feasible for real-life large-scale applications. In this paper, we focus on a more relaxed setting: the grounding of relevant visual entities in a weakly supervised manner by training on the VQA task alone. To address this problem, we propose a visual capsule module with a query-based selection mechanism of capsule features, that allows the model to focus on relevant regions based on the textual cues about visual information in the question. We show that integrating the proposed capsule module in existing VQA systems significantly improves their performance on the weakly supervised grounding task. Overall, we demonstrate the effectiveness of our approach on two state-of-the-art VQA systems, stacked NMN and MAC, on the CLEVR-Answers benchmark, our new evaluation set based on CLEVR scenes with groundtruth bounding boxes for objects that are relevant for the correct answer, as well as on GQA, a real world VQA dataset with compositional questions. We show that the systems with the proposed capsule module consistently outperform the respective baseline systems in terms of answer grounding, while achieving comparable performance on VQA task.1

Due to the high (human) cost of image annotation, lack of large-scale annotation data prevents computer vision models from fully solving image classification tasks. On the other hand, the Bert model [1] has shown that training large scale models with non-annotation text data is possible. Inspired by this, we propose a Self-supervised Visual Feature Learning and Classification framework (SVFLC) that can be applied to large scale training data without annotation. This approach is based on the contrastive predictive learning (CPL) method. By refining CPL using special data augmentation and new contrastive learning mechanisms, learning shape-biased features can be emphasized. In the next step, these features are used to produce pseudo labels via a clustering algorithm. Inspired by the recent research on noisy labels, we proceed to employ distance of the sample from cluster centers to eliminate low-confidence labels, and use soft triplet loss and classification loss jointly for achieving robust performance in the final classification. In this unsupervised learning paradigm, on the Imagenet dataset, our framework outperforms commonly used approaches. The unsupervised performance is lower than supervised and semi-supervised learning approaches, but our proposed framework is more suitable for general cases, and serves as a baseline algorithm for future improvement to the unsupervised learning paradigm.

This paper introduces a new type of image enhancement problem. Compared to traditional image enhancement methods, which mostly deal with pixel-wise modifications of a given photo, our proposed task is to crop an image which is embedded within a photo and enhance the quality of the cropped image. We split our proposed approach into two deep networks: deep photo cropper and deep image enhancer. In the photo cropper network, we employ a spatial transformer to extract the embedded image. In the photo enhancer, we employ super-resolution to increase the number of pixels in the embedded image and reduce the effect of stretching and distortion of pixels. We use cosine distance loss between image features and ground truth for the cropper and the mean square loss for the enhancer. Furthermore, we propose a new dataset to train and test the proposed method. Finally, we analyze the proposed method with respect to qualitative and quantitative evaluations.

Mass utilization of body-worn cameras has led to a huge corpus of available egocentric video. Existing video summarization algorithms can accelerate browsing such videos by selecting (visually) interesting shots from them. Nonetheless, since the system user still has to watch the summary videos, browsing large video databases remain a challenge. Hence, in this work, we propose to generate a textual synopsis, consisting of a few sentences describing the most important events in a long egocentric videos. Users can read the short text to gain insight about the video, and more importantly, efficiently search through the content of a large video database using text queries. Since egocentric videos are long and contain many activities and events, using video-to-text algorithms results in thousands of descriptions, many of which are incorrect. Therefore, we propose a multi-task learning scheme to simultaneously generate descriptions for video segments and summarize the resulting descriptions in an end-to-end fashion. We Input a set of video shots and the network generates a text description for each shot. Next, visual-language content matching unit that is trained with a weakly supervised objective, identifies the correct descriptions. Finally, the last component of our network, called purport network, evaluates the descriptions all together to select the ones containing crucial information. Out of thousands of descriptions generated for the video, a few informative sentences are returned to the user. We validate our framework on the challenging UT Egocentric video dataset, where each video is between 3 to 5 hours long, associated with over 3000 textual descriptions on average. The generated textual summaries, including only 5 percent (or less) of the generated descriptions, are compared to groundtruth summaries in text domain using well-established metrics in natural language processing.1.

When the UAV goes to high altitudes such that the observed surface of the earth becomes planar, the structure and motion recovery of the earth's moving plane becomes ambiguous. This planar degeneracy has been pointed out very often in the literature; therefore, current navigation methods either completely fail or give many confusing solutions in such scenario. Interestingly, the horizon line in planar scenes is straight and distinctive; hence, easily detected. Therefore, we show in this paper that the horizon line provides two degrees of freedom that control the relative orientation between the camera coordinate system and the local surface of earth. The recovered degrees of freedom help linearize and disambiguate the planar flow, and therefore we obtain a unique solution for the UAV motion estimation. Unlike previous work which used the horizon to provide the roll angle and the pitch percentage and only employed them for flight stability, we extract the exact angles and directly use them to estimate the ego motion. Additionally, we propose a novel horizon detector based on the maximum a posteriori estimation of both motion and appearance features which outperforms the other detectors in planar scenarios. We thoroughly experimented on the proposed method against information from GPS and gyroscopes, and obtained promising results.

We present a novel, real-time, markerless vision-based tracking system, employing a rigid orthogonal configuration of two pairs of opposing cameras. Our system uses optical flow over sparse features to overcome the limitation of vision-based systems that require markers or a pre-loaded model of the physical environment. We show how opposing cameras enable cancellation of common components of optical flow leading to an efficient tracking algorithm. Experiments comparing our device with an electromagnetic tracker show that its average tracking accuracy is 80% over 185 frames, and it is able to track large range motions even in outdoor settings.

One emerging application of computing technology is that of interactive rooms and furniture. For instance, Interactive Tables allow for a workspace that is intuitive, natural, and conducive to creating multi-user collaborative work environments. Although several interactive table prototypes have been developed, we have engineered a method using a computer vision system instead of touch screen technology, which allows increased flexibility to the end user because of its ability to ignore or even make use of objects placed upon the table and its decreased likelihood of accidental input. In this work we present a method for implementing such a camera-driven interactive table with a ceiling-mounted camera and demonstrate some of its potential uses. The vision system makes use of a novel hand detection and segmentation technique designed to be tolerant of any level of background complexity on the display and any reasonable range of indoor lighting conditions, thus allowing the highest level of freedom to the end user. It searches the results of multi-scale line and curve finding systems to locate thimbleshaped finger models, marking them as candidate fingers and performing a set of geometric and texture-based tests on each to remove false positives. Finally, it groups finger detections that are similar to each other in location and appearance, while allowing the reintroduction of weak candidates that are supported by strong neighbors, into hand detections with finger and palm locations. Results demonstrate the system's ability extract enough information from images of hands in very complex backgrounds to allow for finger and palm placement recognition.

This paper contributes a new boosting paradigm to achieve detection of events in video. Previous boosting paradigms in vision focus on single frame detection and do not scale to video events. Thus new concepts need to be introduced to address questions such as determining if an event has occurred, localizing the event, handling same action performed at different speeds, incorporating previous classifier responses into current decision, using temporal consistency of data to aid detection and recognition. The proposed method has the capability to improve weak classifiers by allowing them to use previous history in evaluating the current frame. A learning mechanism built into the boosting paradigm is also given which allows event level decisions to be made. This is contrasted with previous work in boosting which uses limited higher level temporal reasoning and essentially makes object detection decisions at the frame level. Our approach makes extensive use of temporal continuity of video at the classifier and detector levels. We also introduce a relevant set of activity features. Features are evaluated at multiple zoom levels to improve detection. We show results for a system that is able to recognize 11 actions.

As the dissemination of images grows across the Internet, authentication of images is becoming an important research issue. Image watermarking techniques are the most used forms to authenticate an image, however they introduce distortions in the original image and are vulnerable to a multitude of attacks. In this paper we present a novel method to authenticate images. We propose that instead of watermarking images, we can authenticate an image by calculating an invariant feature vector which models the image based on its shape and illumination characteristics. We propose an algorithm to calculate such a robust feature vector which withstands a variety of attacks and hence can be a good alternative to watermarking techniques.

We discuss and present a computer vision system that monitors a sitting baby of ages of 6 months old to one-year-old. The system tracks a baby's mouth and hands and alarms the guardians if it finds some objects in the baby's hand going towards his or her mouth. We use skin detection, grayscale based sum of square differences (SSD) and template matching for tracking the hands and the head of a baby.

This paper presents a system for analyzing human driver visual attention. The system relies on estimation of global motion and color statistics to robustly track a person's head and facial features. The system is fully automatic, it can initialize automatically, and reinitialize when necessary. The system classifies rotation in all viewing directions, detects eye/mouth occlusion, detects eye blinking and eye closure, and recovers the three dimensional gaze of the eyes. In addition, the system is able to track both through occlusion due to eye blinking, and eye closure, large mouth movement, and also through occlusion due to rotation. Even when the face is fully occluded due to rotation, the system does not break down. Further the system is able to track through yawning, which is a large local mouth motion. Finally, results are presented, and future work on how this system can be used for more advanced driver visual attention monitoring is discussed.

We present a method for tracking the 3D position of a finger, using a single camera placed several meters away from the user. After skin detection, we use motion to identify the gesticulating arm. The finger point is found by analyzing the arm's outline. To derive a 3D trajectory, we first track 2D positions of the user's elbow and shoulder. Given that a human's upper arm and lower arm have consistent length, we observe that the possible locations of a finger and elbow form two spheres with constant radii. From the previously tracked body points, we can reconstruct these spheres, computing the 3D position of the elbow and finger. These steps are fully automated and do not require human intervention. The system presented can be used as a visualization tool, or as a user input interface, in cases when the user would rather not be constrained by the camera system.

We describe a system for analyzing human driver alertness. It relies on optical flow and color predicates to robustly track a person's head and facial features. Our system classifies rotation in all viewing directions, detects eye/mouth occlusion, detects eye blinking, and recovers the 3D gaze of the eyes. We show results and discuss how this system can be used for monitoring driver alertness.

This paper presents an automatic system for fire detection in video sequences. There are many previous methods to detect fire, however, all except two use spectroscopy or particle sensors. The two that use visual information suffer from the inability to cope with a moving camera or a moving scene. One of these is not able to work on general data, such as movie sequences. The other is too simplistic and unrestrictive in determining what is considered fire, so that it can be used reliably only in aircraft dry bays. Our system uses color and motion information computed from video sequences to locate fire. This is done by first using an approach that is based upon creating a Gaussian-smoothed color histogram to determine the fire-colored pixels, and then using the temporal variation of pixels to determine which of these pixels are actually fire. Unlike the two previous vision-based methods for pre detection, our method is applicable to more areas because of its insensitivity to camera motion. Two specific applications not possible with previous algorithms are the recognition of fire in the presence of global camera motion or scene motion and the recognition of fire in movies for possible use in an automatic rating system. We show that our method works in a variety of conditions, and that it can automatically determine when it has insufficient information.

Current indexing-based approaches build the hash table using either a large number of reference views or 3D models. In this paper, we propose building the hash table using algebraic functions of views. During preprocessing, we consider groups of model points and we represent all the views (i.e., images) that they can produce in a hash table. These views are computed using algebraic functions of a small number of reference views which contain the group. Fundamental to this procedure is a methodology based on singular value decomposition and interval arithmetic for estimating the ranges of values that the parameters of algebraic functions can assume. During recognition, scene groups are used to retrieve from the hash table the model groups that might have produced them. Using algebraic functions of views for indexing-based recognition offers a number of advantages. First of all, the hash table can be built easier without requiring 3D models or a large number of reference views. Second, recognition does not rely on the similarities between new and reference views. Third, verification becomes simpler. Finally, the approach is more general and extendible.

Motion in depth and/or zooming cause defocus blur. We show how the defocus blur in an image can be recovered simultaneously with affine motion. We introduce the theory, develop a solution method and demonstrate the validity of the theory and the solution by conducting experiments with real scenery.

A major problem associated with geometric hashing and methods which have emerged from it is the nonuniform distribution of invariants over the hash space. This has two serious effects on the performance of the method. First, it can result in an inefficient storage of data which can increase recognition time. Second, given that geometric hashing is highly amenable to parallel implementation, a nonuniform distribution of data poses difficulties in tackling the load-balancing problem. Finding a "good" geometric hash function which redistributes the invariants uniformly over the hash space is not easy. Current approaches make assumptions about the statistical characteristics of the data and then use techniques from probability theory to calculate a transformation that maps the nonuniform distribution of invariants to a uniform one. In this paper, a new approach is proposed based on an elastic hash table. In contrast to existing approaches which try to redistribute the invariants over the hash bins, we proceed oppositely by distributing the hash bins over the invariants. The key idea is to associate the hash bins with the output nodes of a self-organizing feature map (SOFM) neural network which is trained using the invariants as training examples. In this way, the location of a hash bin in the space of invariants is determined by the weight vector of the node associated with the hash bin. During training, the SOFM spreads the hash bins proportionally to the distribution of invariants (i.e., more hash bins are assigned to higher density areas while less hash bins are assigned to lower density areas) and adjusts their size so that they eventually hold almost the same number of invariants. The advantage of the proposed approach is that it is a process that adapts to the invariants through learning. Hence, it makes absolutely no assumptions about the statistical characteristics of the invariants and the geometric hash function is actually computed through learning. Furthermore, SOFM's "topology preserving" property ensures that the computed geometric hash function should be well behaved. The proposed approach, was shown to perform well on both artificial and real data.

There are at least two situations in practical computer vision where displacement of a point in an image is accompanied by a defocus blur. The first is when a camera of limited autofocal capability moves in depth, and the second is when a limited autofocal camera zooms. Motion and zooming are two popular strategies for acquiring more detail or for acquiring depth. The defocus blur has been considered noise or at best been ignored. However, the defocus blur is in itself a cue to depth, and hence we proceed to show how it can be calculated simultaneously with affine motion. We first introduce the theory, then develop a solution method and finally demonstrate the validity of the theory and the solution by conducting experiments with real scenery.

We present a theory and practical computations for automatically matching a police artist sketch to a set of true photographs. We locate facial features in both the sketch as well as the set of photograph images. Then, the sketch is photometrically standardized to facilitate comparison with a photo and then both the sketch and the photos are geometrically standardized. Finally, for matching, eigenanalysis is employed. Results using real police sketches and arrest photos are presented.

We consider the problem of learning the mapping between the image coordinates of unknown affine views of an object and the parameters of the affine transformation that can align a known view of the same object with them. A single layer neural network (SL-NN) is used to learn the mapping. Although the proposed approach is conceptually similar to other approaches in the literature, its practical advantages are more profound. The views used to train the SL-NN are not obtained by taking different pictures of the object but by sampling the space of its affine transformed views. This space is constructed by estimating the range of values that the parameters of affine transformation can assume using a single view and a methodology based on singular value decomposition and interval arithmetic. The proposed scheme is as accurate as traditional least-squares approaches but faster. A front-end stage to the SL-NN, based on principal components analysis increases its noise tolerance dramatically and guides us in deciding how many training views are necessary in order for it to learn a good mapping.

A major problem associated with geometric hashing and methods which have emerged from it is the non-uniform distribution of invariants over the hash space. This problem can affect the performance of the method significantly. Finding a "good" geometric hash function which redistributes the invariants uniformly over the hash space is not easy. In this paper, a new approach is proposed for alleviating the above problem. It is based on the use of an "elastic hash table" which is implemented as a self-organizing feature map neural network (SOFM-NN). In contrast to existing approaches which try to redistribute the invariants over the hash bins, we proceed oppositely, spreading the hash bins over the invariants. During training, the SOFM-NN resembles an elastic net which deforms over the hash space. The objective of the deformation process is to spread more hash bins in hash space areas which are heavily occupied and less hash bins in lower density areas. The advantage of the proposed approach is that it is a process that adapts to the invariants through learning. Hence, it makes absolutely no assumptions about the statistical characteristics of the invariants and the geometric hash function is actually computed through learning. Furthermore, the well known "topology preserving" property of the SOFM-NN guarantees that the computed geometric hash function should be well behaved. Finally, the proposed approach is inherently parallelizable.>

Wet surfaces are ubiquitous in our visual experience. Autonomous machines with vision systems will need to identify wet surfaces from dry. Wet surfaces (especially rough, absorbent ones) appear darker when wet. This paper presents the Lekner and Dorf (1988) model for describing the darkening caused by wetting. We explain how to use this optics model to transform intensity values of a region of an image to make that region appear wet. We also show how the model can be reversed in order to make a wet part of an image appear dry. It is also shown that this technique can be used to identify wet regions. This identification is contrasted with darkening caused by shadows. Comparisons of the gray-level histograms of these real images show the validity of this approach for distinguishing wet surfaces from dry.>

The ability to classify age from a facial image has not been pursued in computer vision. This research addresses the limited task of age classification of a facial image into a baby, young adult, and senior adult. This is the first reported work to classify age, and to successfully extract and use natural wrinkles. We present a theory and practical computations for visual age classification from facial images, based on cranio-facial changes in feature-position ratios, and on skin wrinkle analysis. Three age groups are classified.>

In the problem area of human facial image processing, the first computational task that needs to be solved is that of detecting a face under arbitrary scene conditions. Although some progress towards this has been reported in the literature, face detection remains a difficult problem. In this paper the authors report on a novel face-finding method that appears quite robust. First, "snakelets" are used to find candidate edges. Candidate ovals (face-locations) are then found from these snakelets using a voting method. For each of these candidate face-locations, the authors use a method introduced previously to find detailed facial features. If a substantial number of the facial features are found successfully, and their positions satisfy ratio-tests for being standard, the procedure positively reports the existence of a face at this location in the image.

Interreflections are an important visual effect that needs to be accounted for in any endeavour that exploits photometry and radiometry, such as shape from shading and color constancy. Previous work on interreflections has considered pure nonrough Lambertian reflection only, and this is a crude approximation to the physics of light reflection. In this paper we derive an interreflection model that accounts for the roughness of surfaces, and show how this roughness parameter affects the loci of contributing points from each of the surfaces of two interreflecting planes.

This paper presents a theory and practical computations for the recognition of a facial image given a police-artist sketch. First, the facial features of the police sketch are located automatically. Then, regions centered at these positions that contain high intensity variations are temporarily removed. Then a standardization step transforms the sketch image into a "photograph-like" image. After this transformation, the high intensity variation regions that were removed are placed back into the standardized image. The resulting image is then passed to the recognition part of our algorithm. In this stage, principle component analysis is performed on both the standardized image to be recognized and the facial images of the training set. Results using actual police sketches and arrest photographs are presented.>

A novel approach to computing egomotion and detecting points not moving rigidly with the scene when an observer moves with unrestricted motion is presented. The approach, using collinear image points, is based on an exact method for cancelling effects of the observer's rotation from optic flow. For each point only the component of velocity normal to the direction of the line joining the collinear points is needed. The algorithm is simple, appears robust, and is ideal for parallel implementation.>