Active speaker detection is an important component in video analysis algorithms for applications such as speaker diarization, video re-targeting for meetings, speech enhancement, and human-robot interaction. The absence of a large, carefully labeled audio-visual active speaker dataset has limited evaluation in terms of data diversity, environments, and accuracy. In this paper, we present the AVA Active Speaker detection dataset (AVA-ActiveSpeaker) which has been publicly released to facilitate algorithm development and comparison. It contains temporally labeled face tracks in videos, where each face instance is labeled as speaking or not, and whether the speech is audible. The dataset contains about 3.65 million human labeled frames spanning 38.5 hours. We also introduce a state-of-the-art, jointly trained audio-visual model for real-time active speaker detection and compare several variants. The evaluation clearly demonstrates a significant gain due to audio-visual modeling and temporal integration over multiple frames.

Speech activity detection (or endpointing) is an important processing step for applications such as speech recognition, language identification and speaker diarization.Both audio-and vision-based approaches have been used for this task in various settings, often tailored toward end applications.However, much of the prior work reports results in synthetic settings, on taskspecific datasets, or on datasets that are not openly available.This makes it difficult to compare approaches and understand their strengths and weaknesses.In this paper, we describe a new dataset which we will release publicly containing densely labeled speech activity in YouTube videos 1 , with the goal of creating a shared, available dataset for this task.The labels in the dataset annotate three different speech activity conditions: clean speech, speech co-occurring with music, and speech cooccurring with noise, which enable analysis of model performance in more challenging conditions based on the presence of overlapping noise.We report benchmark performance numbers on AVA-Speech using off-the-shelf, state-of-the-art audio and vision models that serve as a baseline to facilitate future research.

Self-folding robot is usually modeled as rigid origami, a class of origami whose entire surface remains rigid during folding except at crease lines. In this work, we focus on finding valid folding motion that brings the origami from the unfolded state continuously to the folded state. Although recent computational methods allow rapid simulation of folding process of certain rigid origami, these methods can fail even when the input crease pattern is extremely simple but requires implicit folding orders. Moreover, due to the rigidity requirement, the probability of generating a valid configuration via uniform sampling is zero, which greatly hinders the applicability of traditional sampling-based motion planners. We propose a novel sampling strategy that samples in the discrete domain. Our experimental results show that the proposed method could efficiently generate valid configurations. Using those configurations, the planner successfully folds several types of rigid origami that the existing methods fail to fold and could discover multiple folding paths in different homotopies.

Recent advances in robotics engineering have enabled the realization of self-folding machines. Rigid origami is usually used as the underlying model for the self-folding machines whose surface remains rigid during folding except at joints. A key issue in designing rigid origami is foldability that concerns about finding folding steps from a flat sheet of crease pattern to a desired folded state. Although recent computational methods allow rapid simulation of folding process of certain rigid origamis, these methods can fail even when the input crease pattern is extremely simple. In this paper, we take on the challenge of planning folding and unfolding motion of origami tessellations, which are composed of repetitive crease patterns. The number of crease lines of a tessellation is usually large, thus searching in such a high dimensional configuration space with the requirement of maintaining rigidity is nontrivial. We propose a motion planner that takes symmetry into consideration and reuses folding path found on the essential crease pattern. Both of these strategies enable us to fold large origami tessellation much more efficiently than existing methods. Our experimental results show that the proposed method successfully folds several types of rigid origami tessellations that existing methods fail to fold.

Cut along the surface of a polyhedron and unfold it to a planar structure without overlapping is known as Unfolding Polyhedra problem which has been extensively studied in the mathematics literature for centuries. However, whether there exists a continuous unfolding motion such that the polyhedron can be continuously transformed to its unfolding has not been well studied. Recently, researchers started to recognize continuous unfolding as a key step in designing and implementation of self-folding robots. In this paper, we model the unfolding of a polyhedron as multi-link tree-structure articulated robot, and address this problem using motion planning techniques. Instead of sampling in continuous domain which traditional motion planners do, we propose to sample only in the discrete domain. Our experimental results show that sampling in discrete domain is efficient and effective for finding feasible unfolding paths.

While techniques that segment shapes into visually meaningful parts have generated impressive results, these techniques also have only focused on relatively simple shapes, such as those composed of a single object either without holes or with few simple holes. In many applications, shapes created from images can contain many overlapping objects and holes. These holes may come from sensor noise, may have important parts of the shape or may be arbitrarily complex. These complexities that appear in real-world 2D shapes can pose grand challenges to the existing part segmentation methods. In this paper, we propose a new decomposition method, called Dual-space Decomposition that handles complex 2D shapes by recognizing the importance of holes and classifying holes as either topological noise or structurally important features. Our method creates a nearly convex decomposition of a given shape by segmenting both the polygon itself and its complementary. We compare our results to segmentation produced by nonexpert human subjects. Based on two evaluation methods, we show that this new decomposition method creates statistically similar to those produced by human subjects.

Collision prediction is a fundamental operation for planning motion in dynamic environment. Existing methods usually exploit complex behavior models or use dynamic constraints in collision prediction. However, these methods all assume simple geometries, such as disc, which significantly limit their applicability. This paper proposes a new approach that advances collision prediction beyond disc robots and handles arbitrary polygons. Our new tool predicts collision by assuming that obstacles are adversarial. Comparing to an online motion planner that replans periodically at fixed time interval and planner that approximates obstacle with discs, our experimental results provide strong evidences that the new method significantly reduces the number of replans while maintaining higher success rate of finding a valid path. Our geometric-based collision prediction method provides a tool to handle highly complex shapes and provides a complimentary approach to those methods that consider behavior and dynamic constraints of objects with simple shapes.