Neural networks trained on datasets such as ImageNet have led to major advances in visual object classification. One obstacle that prevents networks from reasoning more deeply about complex scenes and situations, and from integrating visual knowledge with natural language, like humans do, is their lack of common sense knowledge about the physical world. Videos, unlike still images, contain a wealth of detailed information about the physical world. However, most labelled video datasets represent high-level concepts rather than detailed physical aspects about actions and scenes. In this work, we describe our ongoing collection of the “something-something” database of video prediction tasks whose solutions require a common sense understanding of the depicted situation. The database currently contains more than 100,000 videos across 174 classes, which are defined as caption-templates. We also describe the challenges in crowd-sourcing this data at scale.

Congruence is the geometric concept of being the same up to rotations and\ntranslations in Euclidean space. As congruence is a fundamental concept in\ngeometry, it has drawn broad attentions from the computational geometry\ncommunity for a long time whether the curse of dimensionality applies to\ncongruence testing. We developed a deterministic optimal-runningtime algorithm\nfor congruence testing in 4-space. To understand the importance of the main\nalgorithm in the historical context, we provide a survey about the\ncomputational model and the previous work on congruence testing algorithms.\nThe crucial ingredients of the algorithm are explained component by component.\nThese include general 4-dimensional rotations, angles between linear\nsubspaces, and the Plücker embedding. In the sequence of steps in the\nalgorithm, high regularities are forced in the structure of point sets. This\nlets us encounter beautiful mathematical structures on a 3-sphere and the\nsymmetry group of finite points: the Hopf fibration of a 3-sphere and the\nCoxeter group of four-dimensional point groups. We also give an elementary and\nself-contained overview about these two mathematical topics. The main\nalgorithm consists of five modules that are interesting in their own right.\nThe algorithm is complicated and we provide rather pessimistic estimates. This\nalgorithm, however, can be regarded as a big step forward to constructing a\nmore efficient algorithm in higher dimensions. In the same vein, the last part\nis devoted to the extendability of the algorithm to higher dimensions. This\npart concludes with discussing implementability and geometric properties that\nthe algorithm may imply.