Human actions in egocentric videos often feature hand-object interactions composed of a verb (performed by the hand) applied to an object. Despite their extensive scaling up, egocentric datasets still face two limitations — sparsity of action compositions and a closed set of interacting objects. This paper proposes a novel open vocabulary action recognition task. Given a set of verbs and objects observed during training, the goal is to generalize the verbs to an open vocabulary of actions with seen and novel objects. To this end, we decouple the verb and object predictions via an object-agnostic _verb encoder_ and a prompt-based _object encoder_. The prompting leverages CLIP representations to predict an open vocabulary of interacting objects. We create open vocabulary benchmarks on the EPIC-KITCHENS-100 and Assembly101 datasets; whereas closed-action methods fail to generalize, our proposed method is effective. In addition, our object encoder significantly outperforms existing open-vocabulary visual recognition methods in recognizing novel interacting objects.

Assembly101 is a new procedural activity dataset fea-turing 4321 videos of people assembling and disassembling 101 “take-apart” toy vehicles. Participants work without fixed instructions, and the sequences feature rich and natu-ral variations in action ordering, mistakes, and corrections. Assembly101 is the first multi-view action dataset, with si-multaneous static (8) and egocentric (4) recordings. Se-quences are annotated with more than 100K coarse and 1M fine-grained action segments, and I8M 3D hand poses. We benchmark on three action understanding tasks: recognition, anticipation and temporal segmentation. Ad-ditionally, we propose a novel task of detecting mistakes. The unique recording format and rich set of annotations al-low us to investigate generalization to new toys, cross-view transfer, long-tailed distributions, and pose vs. appearance. We envision that Assemblyl0l will serve as a new challenge to investigate various activity understanding problems.

Person re-identification in the wild needs to simultaneously (frame-wise) detect and re-identify persons and has wide utility in practical scenarios. However, such tasks come with an additional open-set re-ID challenge as all probe persons may not necessarily be present in the (frame-wise) dynamic gallery. Traditional or close-set re-ID systems are not equipped to handle such cases and raise several false alarms as a result. To cope with such challenges open-set metric learning (OSML), based on the concept of Large margin nearest neighbor (LMNN) approach, is proposed. We term our method Open-Set LMNN (OS-LMNN). The goal of separating impostor samples from the genuine samples is achieved through a joint optimization of the Weibull distribution and the Mahalanobis metric learned through this OS-LMNN approach. The rejection is performed based on low probability over distance of imposter pairs. Exhaustive experiments with other metric learning techniques over the publicly available PRW dataset clearly demonstrate the robustness of our approach.