Barlow Twins and VICReg are self-supervised representation learning models that use regularizers to decorrelate features. Although these models are as effective as conventional representation learning models, their training can be computationally demanding if the dimension$d$of the projected embeddings is high. As the regularizers are defined in terms of individual elements of a cross-correlation or covariance matrix, computing the loss for$n$samples takes$O(nd^{2})$time. In this paper, we propose a relaxed decorre-lating regularizer that can be computed in$O(nd\log d)$time by Fast Fourier Transform. We also propose an inexpensive technique to mitigate undesirable local minima that develop with the relaxation. The proposed regularizer exhibits accuracy comparable to that of existing regularizers in down-stream tasks, whereas their training requires less memory and is faster for large$d$. The source code is available.11https://github.com/yutaro-s/scalable-decorrelation-ssl.git

Numerous practical datasets have been developed to recognize human actions from videos. However, many of them were constructed by collecting videos within a limited domain; thus, a model trained using one of the existing datasets often fails to classify videos in a different domain accurately. A possible solution for this drawback is to enhance the domain of each action label, i.e., to import videos associated with a given action label from the other datasets, and then, to train a model using the enhanced dataset. To realize this solution, we constructed a meta video dataset from the existing datasets for human action recognition, referred to as MetaVD. MetaVD comprises six popular human action recognition datasets, which we integrated by annotating 568,015 relation labels in total. These relation labels reflect equality, similarity, and hierarchy between action labels of the original datasets. We further present simple yet effective dataset enhancement methods using MetaVD, which are useful for training models with higher generalization performance, as established by experiments on human action classification. As a further contribution of MetaVD, we show that its analysis can provide useful insight into the datasets.

In recent years, automatic video caption generation has attracted considerable attention. This paper focuses on the generation of Japanese captions for describing human actions. While most currently available video caption datasets have been constructed for English, there is no equivalent Japanese dataset. To address this, we constructed a large-scale Japanese video caption dataset consisting of 79,822 videos and 399,233 captions. Each caption in our dataset describes a video in the form of “who does what and where.” To describe human actions, it is important to identify the details of a person, place, and action. Indeed, when we describe human actions, we usually mention the scene, person, and action. In our experiments, we evaluated two caption generation methods to obtain benchmark results. Further, we investigated whether those generation methods could specify “who does what and where.”

Most interactive programs have been assuming interaction with a single user.We propose the notion of "Social Interaction" as a new interaction paradigm between multiple humans and computers.Social interaction requires that first a computer has the multiple participants model, second its behaviors are not only determined by internal logic but also affected by perceived external situations, and finally it actively joins the interaction.An experimental system with these features was developed.It consists of three subsystems, a vision subsystem that processes motion video input to examine an external situation, an action/reaction subsystem that generates an action based on internal logic of a task and a situated reaction triggered by perceived external situation, and a facial animation subsystem that generates a threedimensional face capable of various facial displays.From the experiment using the system with a number of subjects, we found that subjects generally tended to try to interpret facial displays of the computer.Such involvement prevented them from concentrating on a task.We also found that subjects never recognized situated reactions of the computer that were unrelated to the task although they unconsciously responded to them.These findings seem to imply subliminal involvement of the subjects caused by facial displays and situated reactions.

Human face-to-face conversation is an ideal model for human-computer dialogue.One of the major features of face-to-face communication is its multiplicity of communication channels that act on multiple modalities.To realize a natural multimodal dialogue, it is necessary to study how humans perceive information and determine the information to which humans are sensitive.A face is an independent communication channel that conveys emotional and conversational signals, encoded as facial expressions.We have developed an experimental system that integrates speech dialogue and facial animation, to investigate the effect of introducing communicative facial expressions as a new modality in human-computer conversation.Our experiments have showen that facial expressions are helpful, especially upon first contact with the system.We have also discovered that featuring facial expressions at an early stage improves subsequent interaction.