We propose a method to generate optimal natural language for block placement directives generated by a machine’s planner during human-agent interactions in the blocks world. A non user-friendly machine directive, e.g., move(ObjId, toPos), is transformed into visually and contextually grounded referring expressions that are much easier for the user to comprehend. We describe an algorithm that progressively and generatively transforms the machine’s directive in ECI (Elementary Composable Ideas)-space, generating many alternative versions of the directive. We then define a cost function to evaluate the ease of comprehension of these alternatives and select the best option. The parameters for this cost function were derived empirically from a user study that measured utterance-to-action timings.

We conducted a small scale user study to understand user experiences with two different forms of dialogue interfaces - text-based and voice-based - to interact with a virtual agent in a Blocks World environment while perform tower building tasks. The participants also had the option of using deictic gestures in addition to the speech modality. We identify common types of errors/issues that led to communication failures and share our observations about how users reacted to these issues. We also present survey data that reflects users' evaluations of the dialogue interfaces and their interactions with the virtual agent.

The vast majority of edge detection literature has aimed at improving edge recall and precision, with relatively few addressing the accuracy of edge orientation estimates which are often based on gradient. We show that first-order estimates of orientation can have significant error and this can be remedied by employing Third-Order estimates. This paper aims at estimating differential geometry attributes of an edge, namely, localization, orientation, and curvature, as well as edge topology, and develop robust numerical techniques in gray-scale and color images, applicable to a variety of popular edge detectors, such as gradient-based, gPb and SE. Second, a combinatorial model of edge grouping in a small neighborhood is developed to capture all geometrically consistent grouping called curvels, which establish: (i) edge topology in the form of potential links between an edge and other edges; (ii) an accurate curvature estimate for each possible grouping, whose performance is comparable to methods which use global and multi-scale methods; (iii) a more accurate localization of an edge. These have been evaluated using four distinct methodologies (i) traditional human annotated datasets; (ii) using coherence measure; (iii) stability analysis under visual perturbation, and (iv) utilitarian evaluation, and show meaningful improvements.

We present a new collaborative visual storytelling platform, Aesop, for direction and animation. Aesop consists of a language parser, human gesture monitoring, composition graphs, dialogue state manager, and an interactive 3D animation software. Aesop thus enables 3D spatial and temporal reasoning which are both essential for storytelling. Our key innovation is to enable conversational AI using both verbal and non-verbal communication, which enables research in language, vision, and planning.

Collaborative problem solving (CPS) is seen as a key skill in K-12 education---in computer science as well as other subjects. Efforts to introduce children to computing rely on pair programming as a way of having young learners engage in CPS. Characteristics of quality collaboration are joint exploring or understanding, joint representation, and joint execution. We present a data driven approach to assessing and elucidating collaboration through modeling of multimodal student behavior and performance data.

We introduce the Tower Game Dataset for computational modeling of social interaction predicates. Existing research in affective computing has focused primarily on recognizing the emotional and mental state of a human based on external behaviors. Recent research in the social science community argues that engaged and sustained social interactions require the participants to jointly coordinate their verbal and non-verbal behaviors. With this as our guiding principle, we collected the Tower Game Dataset consisting of multimodal recordings of two players participating in a tower building game, in the process communicating and collaborating with each other. The format of the game was specifically chosen as it elicits spontaneous communication from the participants through social interaction predicates such as joint attention and entrainment. The dataset will be made public and we believe that it will foster new research in the area of computational social interaction modeling.

We propose to use action, scene and object concepts as semantic attributes for classification of video events in InTheWild content, such as YouTube videos. We model events using a variety of complementary semantic attribute features developed in a semantic concept space. Our contribution is to systematically demonstrate the advantages of this concept-based event representation (CBER) in applications of video event classification and understanding. Specifically, CBER has better generalization capability, which enables to recognize events with a few training examples. In addition, CBER makes it possible to recognize a novel event without training examples (i.e., zero-shot learning). We further show our proposed enhanced event model can further improve the zero-shot learning. Furthermore, CBER provides a straightforward way for event recounting/understanding. We use the TRECVID Multimedia Event Detection (MED11) open source event definitions and datasets as our test bed and show results on over 1400 hours of videos.

Low-level appearance as well as spatio-temporal features, appropriately quantized and aggregated into Bag-of-Words (BoW) descriptors, have been shown to be effective in many detection and recognition tasks. However, their effcacy for complex event recognition in unconstrained videos have not been systematically evaluated. In this paper, we use the NIST TRECVID Multimedia Event Detection (MED11 [1]) open source dataset, containing annotated data for 15 high-level events, as the standardized test bed for evaluating the low-level features. This dataset contains a large number of user-generated video clips. We consider 7 different low-level features, both static and dynamic, using BoW descriptors within an SVM approach for event detection. We present performance results on the 15 MED11 events for each of the features as well as their combinations using a number of early and late fusion strategies and discuss their strengths and limitations.

We present a framework for extracting image contours based on geometric and structural consistency among edge element locations and orientations. The paper presents two contributions. First, we observe that while the traditional edge orientation operators are based on first-order derivatives, orientation as tangent of a localized curve requires third-order derivatives. We derive a numerically stable third-order edge operator and show that it outperforms current techniques. Second, we consider all discrete n-tuples of edges in a local neighborhood (7times7) and retain those that are geometrically consistent with a third-order local curve model. This results in a number of ordered discrete combinations of edges, each represented by a bundle of curves. The resulting curve bundle map is a representation of all possible local groupings from which longer contour fragments are constructed. We validate our results and show that our framework outperforms traditional approaches to contour extraction.

Shape is an important cue for generic object recognition but can be insufficient without other cues such as object appearance. We explore a number of ways in which the geometric aspects of an object can be augmented with its appearance. The main idea is to construct a dense correspondence between the interior regions of two shapes based on a shape-based correspondence so that the intensity and gradient distributions can be compared, e.g., using a mutual information paradigm. Three methods for regional alignment are suggested and compared here, based on: (i) propagation of correspondences from the silhouette to parallel curves in the interior, (ii) intersection of line segments anchored on corresponding points on the contour, and (iii) correspondence of shape skeletons. These methods have been implemented and applied to vehicle category recognition from aerial videos under known viewing and illumination conditions. We have constructed a photo-realistic synthetic video database to explore the performance of these methods under controlled conditions. We have also tested these algorithms on real video collected for this purpose from a balloon. Our findings indicate that (i) augmenting shape with appearance significantly increases recognition rate, and (ii) the region correspondence induced by the shape skeleton yields the highest performance.