The majority of current object detectors lack context: class predictions are made independently from other detections. We propose to incorporate context in object detection by post-processing the output of an arbitrary detector to rescore the confidences of its detections. Rescoring is done by conditioning on contextual information from the entire set of detections: their confidences, predicted classes, and positions. We show that AP can be improved by simply reassigning the detection confidence values such that true positives that survive longer (i.e., those with the correct class and large IoU) are scored higher than false positives or detections with small IoU. In this setting, we use a bidirectional RNN with attention for contextual rescoring and introduce a training target that uses the IoU with ground truth to maximize AP for the given set of detections. The fact that our approach does not require access to visual features makes it computationally inexpensive and agnostic to the detection architecture. In spite of this simplicity, our model consistently improves AP over strong pre-trained baselines (Cascade R-CNN and Faster R-CNN with several backbones), particularly by reducing the confidence of duplicate detections (a learned form of non-maximum suppression) and removing out-of-context objects by conditioning on the confidences, classes, positions, and sizes of the co-occurrent detections. Code is available at https://github.com/LourencoVazPato/seeing-without-looking/.

We discuss the usage of symmetric polynomials for representing 2D shapes in their most general form, i.e., arbitrary sets of unlabeled points in the plane. Although particular families of these polynomials have been used in the past, we present general results that pave the way for the development of new representations that exhibit key properties for shape recognition. We show that all monomial symmetric polynomials satisfy homogeneity, which enables leveraging on previous work on spectral invariants to obtain invariance/completeness with respect to shape orientation. Representations based on symmetric polynomials are invariant to shape point relabeling. We single out elementary symmetric polynomials and power sums as particular families of polynomials that further enable obtaining completeness with respect to point labeling. We discuss the efficient computation of these polynomials and study how perturbations in the shape point coordinates affect their values.

We address the representation of two-dimensional shape in its most general form, i.e., arbitrary sets of points, that may arise in multiple situations, e.g., sparse sets of specific landmarks, or dense sets of image edge points. Our goal are recognition tasks, where the key is balancing two contradicting demands: shapes that differ by rigid transformations or point re-labeling should have the same representation (invariance) but geometrically distinct shapes should have different representations (completeness). In the paper, we introduce a new shape representation that marries properties of the elementary symmetric polynomials and the bispectrum. Like the power spectrum, the bispectrum is insensitive to signal shifts; however, unlike the power spectrum, the bispectrum is complete. The elementary symmetric polynomials are complete and invariant to variable relabeling. We show that the elementary symmetric polynomials of the shape points depend on the shape orientation in a way that enables interpreting them in the frequency domain and building from them a bispectrum. The result is a shape representation that is complete and invariant to rigid transformations and point-relabeling. The paper also reports experiments that illustrate the proved properties.