Video procedure planning, i.e., planning a sequence of action steps given the video frames of start and goal states, is an essential ability for embodied AI. Recent works utilize Large Language Models (LLMs) to generate enriched action step description texts to guide action step decoding. Although LLMs are introduced these methods decode the action steps into a closed-set of one-hot vectors, limiting the model's capability of generalizing to new steps or tasks. Additionally, fixed action step descriptions based on world-level commonsense may contain noise in specific instances of visual states. In this paper, we propose PlanLLM, a cross-modal joint learning framework with LLMs for video procedure planning. We propose an LLM-Enhanced Planning module which fully uses the generalization ability of LLMs to produce free-form planning output and to enhance action step decoding. We also propose Mutual Information Maximization module to connect world-level commonsense of step descriptions and sample-specific information of visual states, enabling LLMs to employ the reasoning ability to generate step sequences. With the assistance of LLMs, our method can both closed-set and open vocabulary procedure planning tasks. Our PlanLLM achieves superior performance on three benchmarks, demonstrating the effectiveness of our designs.

Periocular recognition has been emerging as an effective biometric identification approach, especially under less constrained environments where face and/or iris recognition is not applicable. This paper proposes a new deep learning-based architecture for robust and more accurate periocular recognition which incorporates attention model to emphasize important regions in the periocular images. The new architecture adopts multi-glance mechanism, in which part of the intermediate components are configured to incorporate emphasis on important semantical regions, i.e., eyebrow and eye, within a periocular image. By focusing on these regions, the deep convolutional neural network is able to learn additional discriminative features, which in turn improves the recognition capability of the whole model. The superior performance of our method strongly suggests that eyebrow and eye regions are important for periocular recognition, and deserve special attention during the deep feature learning process. This paper also presents a customized verification-oriented loss function, which is shown to provide higher discriminating power than conventional contrastive/triplet loss functions. Extensive experiments on six publicly available databases are performed to evaluate the proposed approach. The reproducible experimental results indicate that our approach significantly outperforms several state-of-the-art methods for the periocular recognition.

This paper proposes an accurate and generalizable deep learning framework for iris recognition. The proposed framework is based on a fully convolutional network (FCN), which generates spatially corresponding iris feature descriptors. A specially designed Extended Triplet Loss (ETL) function is introduced to incorporate the bit-shifting and non-iris masking, which are found necessary for learning discriminative spatial iris features. We also developed a sub-network to provide appropriate information for identifying meaningful iris regions, which serves as essential input for the newly developed ETL. Thorough experiments on four publicly available databases suggest that the proposed framework consistently outperforms several classic and state-of-the-art iris recognition approaches. More importantly, our model exhibits superior generalization capability as, unlike popular methods in the literature, it does not essentially require database-specific parameter tuning, which is another key advantage over other approaches.

Accurate biometric identification under real environments is one of the most critical and challenging tasks to meet growing demand for higher security. This paper proposes a new framework to efficiently and accurately match periocular images that are automatically acquired under less-constrained environments. Our framework, referred to as semantics-assisted convolutional neural networks (SCNNs) in this paper, incorporates explicit semantic information to automatically recover comprehensive periocular features. This strategy enables superior matching accuracy with the usage of relatively smaller number of training samples, which is often an issue with several biometrics. Our reproducible experimental results on four different publicly available databases suggest that the SCNN-based periocular recognition approach can achieve outperforming results, both in achievable accuracy and matching time, for less-constrained periocular matching. Additional experimental results presented in this paper also indicate that the effectiveness of proposed SCNN architecture is not only limited to periocular recognition but it can also be useful for generalized image classification. Without increasing the volume of training data, the SCNN is able to automatically extract more discriminative features from the input data than a single CNN, therefore can consistently improve the recognition performance. The experimental results presented in this paper validate such an approach to enable faster and more accurate periocular recognition under less constrained environments.

This paper proposes a novel and more accurate iris segmentation framework to automatically segment iris region from the face images acquired with relaxed imaging under visible or near-infrared illumination, which provides strong feasibility for applications in surveillance, forensics and the search for missing children, etc. The proposed framework is built on a novel total-variation based formulation which uses l1 norm regularization to robustly suppress noisy texture pixels for the accurate iris localization. A series of novel and robust post processing operations are introduced to more accurately localize the limbic boundaries. Our experimental results on three publicly available databases, i.e., FRGC, UBIRIS.v2 and CASIA.v4-distance, achieve significant performance improvement in terms of iris segmentation accuracy over the state-of-the-art approaches in the literature. Besides, we have shown that using iris masks generated from the proposed approach helps to improve iris recognition performance as well. Unlike prior work, all the implementations in this paper are made publicly available to further advance research and applications in biometrics at-d-distance.