Video question answering is a non-trivial task that demands joint understanding of visual contents and linguistic questions as well as temporal reasoning across video frames. Recent agent-based approaches address this by conducting multi-step reasoning with large language models (LLMs) across frame-level captions generated by vision-language models, but encounter limited temporal coherence across frames. A possible direction based on video language models (VideoLMs) directly captures temporal dynamics via video-level descriptions, but often lacks fine-grained visual cues due to a restricted number of input frames and a large dependency on input prompts. To tackle these challenges, we propose RefineVQA, a training-free framework that can easily be plugged into existing VideoLMs with iterative, LLM-guided description refinements. Specifically, the VideoLM produces an initial description, followed by LLM feedback determining whether the description suffices for the question and guiding further visual extraction, which in turn enhances the description quality while preserving temporal context. Plugged into state-of-the-art VideoLMs, ReFineVQA yields consistent gains across diverse benchmarks–NExT-QA, EgoSchema, VideoMME, ActivityNet, and StreamingBench–even with a small external LLM of 3.8B parameters.

In recent text-video retrieval, the use of additional captions from vision-language models has shown promising effects on the performance. However, existing models using additional captions often have struggled to capture the rich semantics, including temporal changes, inherent in the video. In addition, incorrect information caused by generative models can lead to inaccurate retrieval. To address these issues, we propose a new framework, Narrating the Video (NarVid), which strategically leverages the comprehensive information available from frame-level captions, the narration. The proposed NarVid exploits narration in multiple ways: 1) feature enhancement through cross-modal interactions between narration and video, 2) query-aware adaptive filtering to suppress irrelevant or incorrect information, 3) dual-modal matching score by adding query-video similarity and query-narration similarity, and 4) hard-negative loss to learn discriminative features from multiple perspectives using the two similarities from different views. Experimental results demonstrate that NarVid achieves state-of-the-art performance on various benchmark datasets.

Estimating heart activities and physiological signals from facial video without any contact, known as remote photoplethysmography and remote heart rate estimation, holds significant potential for numerous applications. In this letter, we present a novel approach for remote heart rate measurement leveraging a Spatial-Temporal SwiftFormer architecture (STSPhys). Our model addresses the limitations of existing methods that rely heavily on 3D CNNs or 3D visual transformers, which often suffer from increased parameters and potential instability during training. By integrating both spatial and temporal information from facial video data, STSPhys achieves robust and accurate heart rate estimation. Additionally, we introduce a hybrid loss function that integrates constraints from both the time and frequency domains, further enhancing the model's accuracy. Experimental results demonstrate that STSPhys significantly outperforms existing state-of-the-art methods on intra-dataset and cross-dataset tests, achieving superior performance with fewer parameters and lower computational complexity.

Hand pose recognition has received increasing attention in an area of human–computer interaction. With the recent spread of many low‐cost three‐dimensional (3D) cameras, research into understanding more natural gestures has increased. In this study, the authors present a method for hand part classification and joint estimation from a single depth image. They apply random decision forests (RDFs) for hand part classification. Foreground pixels in the hand image are estimated by RDF. Then hand joints are estimated based on the classified hand parts. They suggest a robust feature extraction method for per‐pixel classification, which enhances the accuracy of hand part classification. They also propose a tree selection algorithm using legacy trained RDF to classify unseen test data. Selecting trees using the proposed method show better performance than using all the trees as in conventional method. Depth images and label images synthesised by 3D hand mesh model were used for training forests and algorithm verification. The authors’ experiments show that the enhanced algorithm outperforms the state‐of‐the‐art method in accuracy.

In order to make machines able to recognize various patterns, it is important to define an appropriate function for measuring similarities between different objects. Conventional similarity measures are devised mainly for 1D vector data, which may lead to loss of information of 2D matrix data. We cast the calculation of similarity between two matrices as a neural network problem, and design the architecture for learning a similarity measure. We provide experiments on real 2D matrix data in the face recognition and gesture recognition, where we show that the learning of a similarity measure leads to improvements in the performance of the recognition problem. Also we compare the performance of the proposed measure with conventional distance measures for 2D matrix data.

Facial expression recognition is an important topic in the field of human-agent interaction, because facial expression is simple and impressive signal which human can send to others. Though there have been numerous studies on facial image analysis, the performance of expression recognition is still not acceptable due to the diversity of human expression and enormous variations in facial images. In this paper, we try to improve the performance of facial expression recognition by using multi-task learning techniques of neural networks. Through computational experiments on a benchmark database, we show positive possibility of performance improvement using multi-task learning.

Recently, gesture recognition using visual sensors has been paid increased attention as an intelligent technology for human-computer interaction. To achieve good recognition performance in vision-based gesture recognition, it is important to find an efficient representation of complex visual signals and a robust classification method that can deal with diverse variations of gesture data. To treat these challenging topics, we propose a hierarchical structure for feature extraction and measuring similarity between two gestures obtained through RGB-D video sensors such as Kinect. The efficiency of the proposed method is confirmed through computational experiments on a public benchmark database.

Image processing and machine learning communities have long addressed the problems involved in the analysis of large high-dimensional data sets. To deal with high-dimensional data efficiently, learning core properties of given data set is important. The manifold learning methods such as ISOMap try to identify a low-dimensional manifold from a set of unorganized samples. ISOMap method is an extension of the classical multidimensional scaling method for dimension reduction, which find a linear subspace in which dissimilarity between data points is preserved. In order to measure dissimilarity, ISOMap uses the geodesic distances on the manifold instead of Euclidean distance. In this paper, we propose a modification of ISOMap using class information, which is often given in company with input data in many applications such as pattern classification. Since the conventional ISOMap does not use class information in approximating true geodesic distance between each pair of data points, it is difficult to construct a data structure related to class-membership that may give important information for given task such as data visualization and classification. The proposed method utilizes class-membership for measuring distance of data pair so as to find a low-dimensional manifold preserving the distance between classes as well as the distance between data points. Through computational experiments on artificial data sets and real facial data sets, we confirm that the proposed method gives better performance than the conventional ISOMap.

The parameter spaces of hierarchical systems such as multilayer perceptrons include singularities due to the symmetry and degeneration of hidden units. A parameter space forms a geometrical manifold, called the neuromanifold in the case of neural networks. Such a model is identified with a statistical model, and a Riemannian metric is given by the Fisher information matrix. However, the matrix degenerates at singularities. Such a singular structure is ubiquitous not only in multilayer perceptrons but also in the gaussian mixture probability densities, ARMA time-series model, and many other cases. The standard statistical paradigm of the Cramér-Rao theorem does not hold, and the singularity gives rise to strange behaviors in parameter estimation, hypothesis testing, Bayesian inference, model selection, and in particular, the dynamics of learning from examples. Prevailing theories so far have not paid much attention to the problem caused by singularity, relying only on ordinary statistical theories developed for regular (nonsingular) models. Only recently have researchers remarked on the effects of singularity, and theories are now being developed. This article gives an overview of the phenomena caused by the singularities of statistical manifolds related to multilayer perceptrons and gaussian mixtures. We demonstrate our recent results on these problems. Simple toy models are also used to show explicit solutions. We explain that the maximum likelihood estimator is no longer subject to the gaussian distribution even asymptotically, because the Fisher information matrix degenerates, that the model selection criteria such as AIC, BIC, and MDL fail to hold in these models, that a smooth Bayesian prior becomes singular in such models, and that the trajectories of dynamics of learning are strongly affected by the singularity, causing plateaus or slow manifolds in the parameter space. The natural gradient method is shown to perform well because it takes the singular geometrical structure into account. The generalization error and the training error are studied in some examples.

Natural gradient learning is known to be efficient in escaping plateau, which is a main cause of the slow learning speed of neural networks. The adaptive natural gradient learning method for practical implementation also has been developed, and its advantage in real-world problems has been confirmed. In this letter, we deal with the generalization performances of the natural gradient method. Since natural gradient learning makes parameters fit to training data quickly,the overfitting phenomenon may easily occur, which results in poor generalization performance. To solve the problem, we introduce the regularization term in natural gradient learning and propose an efficient optimizing method for the scale of regularization by using a generalized Akaike information criterion (network information criterion). We discuss the properties of the optimized regularization strength by NIC through theoretical analysis as well as computer simulations. We confirm the computational efficiency and generalization performance of the proposed method in real-world applications through computational experiments on benchmark problems.

Singularities are ubiquitous in the parameter space of hierarchical models such as multilayer perceptrons. At singularities, the Fisher information matrix degenerates, and the Cramer-Rao paradigm does no more hold, implying that the classical model selection the(cid:173) ory such as AIC and MDL cannot be applied. It is important to study the relation between the generalization error and the training error at singularities. The present paper demonstrates a method of analyzing these errors both for the maximum likelihood estima(cid:173) tor and the Bayesian predictive distribution in terms of Gaussian random fields, by using simple models.

The natural gradient learning method is known to have ideal performances for on-line training of multilayer perceptrons. It avoids plateaus, which give rise to slow convergence of the backpropagation method. It is Fisher efficient, whereas the conventional method is not. However, for implementing the method, it is necessary to calculate the Fisher information matrix and its inverse, which is practically very difficult. This article proposes an adaptive method of directly obtaining the inverse of the Fisher information matrix. It generalizes the adaptive Gauss-Newton algorithms and provides a solid theoretical justification of them. Simulations show that the proposed adaptive method works very well for realizing natural gradient learning.