It was our pleasure and privilege to welcome you to Istanbul for the 18th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2024). We hope your experience at FG was rewarding both professionally and personally!

Sign language recognition (SLR) has recently achieved a breakthrough in performance thanks to deep neural networks trained on large annotated sign datasets. Of the many different sign languages, these annotated datasets are only available for a select few. Since acquiring gloss-level labels on sign language videos is difficult, learning by transferring knowledge from existing annotated sources is useful for recognition in under-resourced sign languages. This study provides a publicly available cross-dataset transfer learning benchmark from two existing public Turkish SLR datasets. We use a temporal graph convolution-based sign language recognition approach to evaluate five supervised transfer learning approaches and experiment with closed-set and partial-set cross-dataset transfer learning. Experiments demonstrate that improvement over finetuning based transfer learning is possible with specialised supervised transfer learning methods.

State-of-the-art Sign Language Recognition (SLR) frameworks based on Graph Convolutional Networks (GCNs) require a skeleton-based graph topology. Although upper body skeleton configuration is often considered, Sign Languages (SLs) mainly comprises hand shapes, upper body movements, and facial gestures. Notably, the hand plays a major role in performing the sign. This paper investigates optimal choices for the hand graph topology essential for improving the recognition performance. Our experiments on two benchmark Turkish SL datasets, BosphorusSign22k and AUTSL, demonstrate that hand-based topology substantially contributes to performance that is competitive with full body-based topologies.

Automatically detecting conversational groups from video footage is an important research area with applications in social robotics, human-computer interaction, and video analysis. In this paper, we use a novel approach based on Graph Convolutional Networks (GCN) for the social group detection problem. We base our approach on a method from the community detection domain called Deep Modularity Networks (DMoN). As an input to the DMoN method, we construct a graph representation using body view frustums, which indicates affinities among the individuals. In addition to the frame-level group detector, the proposed system utilizes temporal information to improve the group detection accuracy. Experiments conducted on the standard benchmark dataset SALSA confirm that our GCN-based approach improves the group detection quality over state-of-the-art group detection methods.

Sign Language Translation has attained considerable success recently, raising hopes for improved communication with the Deaf. A pre-processing step called tokenization improves the success of translations. Tokens can be learned from sign videos if supervised data is available. However, data annotation at the gloss level is costly, and annotated data is scarce. The paper utilizes Adversarial, Multitask, Transfer Learning to search for semi-supervised tokenization approaches without burden of additional labeling. It provides extensive experiments to compare all the methods in different settings to conduct a deeper analysis. In the case of no additional target annotation besides sentences, the proposed methodology attains 13.25 BLUE-4 and 36.28 ROUGE scores which improves the current state-of-the-art by 4 points in BLUE-4 and 5 points in ROUGE.

Recognition of sign language is a difficult task which often requires tedious annotations by sign language experts. End-to-end learning attempts that bypass frame level annotations have achieved some success in limited datasets, but it has been shown that high quality annotations improve performance drastically. Recent unsupervised learning methods using deep neural networks have achieved successes in learning feature extraction. Yet a technique for high quality frame level classification using unsupervised techniques does not exist. In this paper, we assign labels of an isolated Sign Language(SL) dataset using end-to-end neural network architectures that have proven success in unsupervised discovery of sub-word acoustic units in speech processing. We observe that key-hand-shape s(KHS), which are meaningful visual basic parts of signs in a SL dataset can be detected using unsupervised clustering techniques. Sparse autoencoders can successfully retrieve and cluster KHS s used in isolated signs. In addition, using correspondent frames in an autoencoder scheme has the power to continue the learning process.

Deep neural network models owe their representational power to the high number of learnable parameters. It is often infeasible to run these largely parametrized deep models in limited resource environments, like mobile phones. Network models employing conditional computing are able to reduce computational requirements while achieving high representational power, with their ability to model hierarchies. We propose Conditional Information Gain Networks, which allow the feed forward deep neural networks to execute conditionally, skipping parts of the model based on the sample and the decision mechanisms inserted in the architecture. These decision mechanisms are trained using cost functions based on differentiable Information Gain, inspired by the training procedures of decision trees. These information gain based decision mechanisms are differentiable and can be trained end-to-end using a unified framework with a general cost function, covering both classification and decision losses. We test the effectiveness of the proposed method on MNIST and recently introduced Fashion MNIST datasets and show that our information gain based conditional execution approach can achieve better or comparable classification results using significantly fewer parameters, compared to standard convolutional neural network baselines.

The rapid growth of storage capacity and processing power has caused machine learning applications to increasingly rely on using immense amounts of labeled data. It has become more important than ever to have fast and inexpensive ways to annotate vast amounts of data. With the emergence of crowdsourcing services, the research direction has gravitated toward putting the wisdom of crowds to better use. Unfortunately, spammers and inattentive annotators pose a threat to the quality and trustworthiness of the consensus. Thus, high quality consensus estimation from crowd annotated data requires a meticulous choice of the candidate annotator and the sample in need of a new annotation. Due to time and budget limitations, it is of utmost importance that this choice is carried out while the annotation collection is in progress. We call this process active crowd-labeling. To this end, we propose an active crowd-labeling approach for actively estimating consensus from continuous-valued crowd annotations. Our method is based on annotator models with unknown parameters, and Bayesian inference is employed to reach a consensus in the form of ordinal, binary, or continuous values. We introduce ranking functions for choosing the candidate annotator and sample pair for requesting an annotation. In addition, we propose a penalizing method for preventing annotator domination, investigate the explore-exploit trade-off for incorporating new annotators into the system, and study the effects of inducing a stopping criterion based on consensus quality. We also introduce the crowd-labeled Head Pose Annotations datasets. Experimental results on the benchmark datasets used in the literature and the Head Pose Annotations datasets suggest that our method provides high-quality consensus by using as few as one fifth of the annotations (~80% cost reduction), thereby providing a budget and time-sensitive solution to the crowd-labeling problem.

In this paper, we focus on describing the method we designed for automatic perceived personality prediction. We present a simple model that uses three different sets of features: nonverbal audio cues, visual cues from video, and facial landmark points. The model uses a random decision forest to do regression from the extracted features. As we discuss in Section 4, this multimodal model performs relatively well in the task of personality analysis and recognition of certain personality traits from short video clips. Using this model, we came in 4th place in the first track of the second round of Joint Contest on Multimedia Challenges Beyond Visual Analysis at ICPR 2016 [1].

Face is a highly utilized biometric, and 3D modality is preferred due to better handling of variations such as pose and illumination. However, occlusions covering the face alter the 3D surface and degrade the recognition performance. To improve recognition rates, the occluded parts should be detected prior to any surface comparison. In this paper, we consider two different occlusion detection approaches: The first one is based on statistical facial surface modeling, where pixel-wise Gaussian Mixture Models are trained. The second algorithm considers occlusion detection as a binary image segmentation problem: The regional cues of depth values are incorporated with neighborhood cues, and the acquired surface is modeled as a graph. The surface pixels are labeled as either face or occlusion via the graph cut technique. Experiments on the Bosphorus and the UMB-DB databases, including realistic occlusion variations, show that both methods improve occlusion detection and face recognition rates as compared to the baseline technique.

https://doi.org/10.21437/interspeech.2014-267

With advances in sensor technology, the three-dimensional (3-D) face has become an emerging biometric modality, preferred especially in high security applications. However, dealing with occlusions covering the facial surface is a great challenge, which should be handled to enable applicability to fully automatic security systems. In this paper, we propose a fully automatic 3-D face recognition system which is robust to occlusions. We basically consider two problems: 1) occlusion handling for surface registration, and 2) missing data handling for classification based on subspace analysis techniques. For the alignment problem, we employ an adaptively-selected-model-based registration scheme, where a face model is selected for an occluded face such that only the valid nonoccluded patches are utilized. After registering to the model, occlusions are detected and removed. In the classification stage, a masking strategy, which we call masked projection, is proposed to enable the use of subspace analysis techniques with incomplete data. Furthermore, a regional scheme suitable for occlusion handling is incorporated in classification to improve the overall results. Experimental results on two databases with realistic facial occlusions, namely, the Bosphorus and the UMB-DB, are reported. Experimental results confirm that registration based on the adaptively selected model together with the masked subspace analysis classification offer an occlusion robust face recognition system.

In this paper, we present an intelligent information kiosk called MIDAS (Multimodal Interactive-Dialogue Automaton for Self-service), including its hardware and software architecture, stages of deployment of speech recognition and synthesis technologies. MIDAS uses the methodology Wizard of Oz (WOZ) that allows an expert to correct speech recognition results and control the dialogue flow. User statistics of the multimodal human computer interaction (HCI) have been analyzed for the operation of the kiosk in the automatic and automated modes. The infokiosk offers information about the structure and staff of laboratories, the location and phones of departments and employees of the institution. The multimodal user interface is provided with a touch screen, natural speech input and head and manual gestures, both for ordinary and physically handicapped users.

Falls are identified as a major health risk for the elderly and a major obstacle to independent living. Considering the remarkable increase in the elderly population of developed countries, methods for fall detection have been a recent active area of research. However, existing methods often use only wearable sensors, such as acceloremeters, or cameras to detect falls. In this demonstration, in contrast to the state of the art solutions, we focus on the use of multi-modal wireless sensor networks within the WeCare framework. WeCare system is developed as a solution for independent living applications by remotely monitoring the health and well-being of its users. We describe the general structure of WeCare and demonstrate its fall detection method. Our set-up not only includes scalar sensors to detect falls and motion but also consists of embedded cameras and RFID tags and uses sensor fusion techniques to improve the success of fall detection and minimize the false positives.

A new multimodal biometric database designed and acquired within the framework of the European BioSecure Network of Excellence is presented. It is comprised of more than 600 individuals acquired simultaneously in three scenarios: 1) over the Internet, 2) in an office environment with desktop PC, and 3) in indoor/outdoor environments with mobile portable hardware. The three scenarios include a common part of audio/video data. Also, signature and fingerprint data have been acquired both with desktop PC and mobile portable hardware. Additionally, hand and iris data were acquired in the second scenario using desktop PC. Acquisition has been conducted by 11 European institutions. Additional features of the BioSecure Multimodal Database (BMDB) are: two acquisition sessions, several sensors in certain modalities, balanced gender and age distributions, multimodal realistic scenarios with simple and quick tasks per modality, cross-European diversity, availability of demographic data, and compatibility with other multimodal databases. The novel acquisition conditions of the BMDB allow us to perform new challenging research and evaluation of either monomodal or multimodal biometric systems, as in the recent BioSecure Multimodal Evaluation campaign. A description of this campaign including baseline results of individual modalities from the new database is also given. The database is expected to be available for research purposes through the BioSecure Association during 2008.

Biometric identification from three-dimensional (3-D) facial surface characteristics has become popular, especially in high security applications. In this paper, we propose a fully automatic expression insensitive 3-D face recognition system. Surface deformations due to facial expressions are a major problem in 3-D face recognition. The proposed approach deals with such challenging conditions in several aspects. First, we employ a fast and accurate region-based registration scheme that uses common region models. These common models make it possible to establish correspondence to all the gallery samples in a single registration pass. Second, we utilize curvature-based 3-D shape descriptors. Last, we apply statistical feature extraction methods. Since all the 3-D facial features are regionally registered to the same generic facial component, subspace construction techniques may be employed. We show that linear discriminant analysis significantly boosts the identification accuracy. We demonstrate the recognition ability of our system using the multiexpression Bosphorus and the most commonly used 3-D face database, Face Recognition Grand Challenge (FRGCv2). Our experimental results show that in both databases we obtain comparable performance to the best rank-1 correct classification rates reported in the literature so far: 98.19% for the Bosphorus and 97.51% for the FRGCv2 database. We have also carried out the standard receiver operating characteristics (ROC III) experiment for the FRGCv2 database. At an FAR of 0.1%, the verification performance was 86.09%. This shows that model-based registration is beneficial in identification scenarios where speed-up is important, whereas for verification one-to-one registration can be more beneficial.

Deformations caused by facial expression variations complicate the task of 3D face registration which is vital for successful 3D face recognition systems. In this work, we propose to use a hierarchical component-based face registration technique capable of handling the difficulties caused by non-rigid nature of faces. Local components independently registered by the iterative closest point (ICP) algorithm provides a fast registration with the use of a generic face model and does not suffer from non-rigidity of human facial surface. Invariance of the proposed approach is further increased by utilizing curvature-based 3D surface descriptors. Identification experiments conducted on the multi-expression Bosphorus database reveal that the accuracy of the classical ICP-based approach can be significantly increased under extreme expression variations.

In this work, we propose a multi-class classification strategy based on Fisher kernels. Fisher kernels combine the powers of discriminative and generative classifiers by mapping variable-length sequences to a new fixed length feature space. The mapping is based on a single generative model and the classifier is intrinsically binary. We apply a multi-class classification, instead of a binary classification, on each Fisher score space and combine the decisions of multi-class classifiers. We show, through experiments on gesture and sign sequences, that the Fisher scores extracted from the HMM of one class provide discriminative information for other classes as well. Comparisons with other strategies show that the proposed method enhances the performance of the base classifier the most.

In this paper, we present an extensive study of 3-D face recognition algorithms and examine the benefits of various score-, rank-, and decision-level fusion rules. We investigate face recognizers from two perspectives: the data representation techniques used and the feature extraction algorithms that match best each representation type. We also consider novel applications of various feature extraction techniques such as discrete Fourier transform, discrete cosine transform, nonnegative matrix factorization, and principal curvature directions to the shape modality. We discuss and compare various classifier combination methods such as fixed rules voting- and rank-based fusion schemes. We also present a dynamic confidence estimation algorithm to boost fusion performance. In identification experiments performed on FRGC v1.0 and FRGC v2.0 face databases, we tried to find the answers to the following questions: 1) the relative importance of the face representation technique vis-à-vis the types of features extracted; 2) the impact of the gallery size; 3) the conditions, under which subspace methods are preferable, and the compression factor; 4) the most advantageous fusion level and fusion methods; 5) the role confidence votes in improving fusion and the style of selecting experts in the fusion; and 6) the consistency of the conclusions across different databases.

In this paper, we propose a method to determine the quality of deployment in a surveillance wireless sensor network in an environment that contains obstacles. The maximum detection probability on the potential breach paths is used as the deployment quality measure. The watershed segmentation algorithm is applied on the graph formed by the iso-sensing curves to identify the possible breach paths. An algorithm is proposed to convert the watershed segmentation to an auxiliary graph which is then employed to determine the deployment quality measure.

In local feature-based face recognition systems, the topographical locations of feature extractors directly affect the discriminative power of a recognizer. Better recognition accuracy can be achieved by the determination of the positions of salient image locations. Most of the facial feature selection algorithms in the literature work with two assumptions: one, that the importance of each feature is independent of the other features, and two, that the kernels should be located at fiducial points. Under these assumption, one can only get a sub-optimal solution. In this paper, we present a methodology that tries to overcome this problem by relaxing the two assumptions using a formalism of subset selection problem. We use a number of feature selection algorithms and a genetic algorithm. Comparative results on the FERET dataset confirm the viability of our approach.

Parallel pattern recognition requires great computational resources; it is NP-complete. From an engineering point of view it is desirable to achieve good performance with limited resources. For this purpose, we develop a serial model for visual pattern recognition based on the primate selective attention mechanism. The idea in selective attention is that not all parts of an image give us information. If we can attend only to the relevant parts, we can recognize the image more quickly and using less resources. We simulate the primitive, bottom-up attentive level of the human visual system with a saliency scheme and the more complex, top-down, temporally sequential associative level with observable Markov models. In between, there is a neural network that analyses image parts and generates posterior probabilities as observations to the Markov model. We test our model first on a handwritten numeral recognition problem and then apply it to a more complex face recognition problem. Our results indicate the promise of this approach in complicated vision applications.

Color quantization reduces the number of the colors in a color image, while the subsequent dithering operation attempts to create the illusion of more colors with this reduced palette. In quantization, the palette is designed to minimize the mean squared error (MSE). However, the dithering that follows enhances the color appearance at the expense of increasing the MSE. We introduce three joint quantization and dithering algorithms to overcome this contradiction. The basic idea is the same in two of the approaches: introducing the dithering error to the quantizer in the training phase. The fuzzy C-means (FCM) and the fuzzy learning vector quantization (FLVQ) algorithms are used to develop two combined mechanisms. In the third algorithm, we minimize an objective function including an inter-cluster separation (ICS) term to obtain a color palette which is more suitable for dithering. The goal is to enlarge the convex hull of the quantization colors to obtain the illusion of more colors after error diffusion. The color contrasts of images are also enhanced with the proposed algorithm. We test the results of these three new algorithms using quality metrics which model the perception of the human visual system and illustrate that substantial improvements are achieved after dithering.

In this work pose-invariant face recognition is attempted using a two-stage approach. In the first stage, the orientation of the face is recognized, and in the second stage, the face is recognized among a subset of faces with the same orientation in the training set. We have generated our own database and tried several different techniques for pose invariant face recognition. We have used both linear techniques such as principal component analysis (PCA) and linear discriminant analysis (LDA) and unsupervised clustering techniques such as C-means and fuzzy C-means. The classification algorithms used with all of these techniques are nearest mean and k-nearest neighbor (k-NN). This work presents a comparison of these techniques on our database. With all techniques, it is observed that the recognition performance is enhanced when view information is incorporated as a preliminary step.

Quantization errors are generally hidden by performing a dithering operation on the image. A common method is to utilize error diffusion. However, this method is prone to error accumulation, resulting in color impulses and streaks. This paper presents a new approach to error diffusion dithering through a fuzzy error diffusion algorithm. In this method, the amount of error to be diffused is determined by considering the relative location of the pixel not only to the closest codebook vector, but to all other palette entries. The goal is to hide the quantization errors by error diffusion, while preventing the excess accumulation of errors. This is achieved through an attraction-repulsion schema according to a fuzzy membership function. We also explored methods to speed up the fuzzy error diffusion process through a L-filter approach by determining a fixed set of membership values. We have implemented the fuzzy error diffusion algorithm for color images and achieved drastic improvements, resulting in superior quality dithered images and significantly lower mean squared error values. A different error measure modeling the characteristic of the human visual system also indicates the superiority of our method.

Quantization errors are generally hidden by performing a dithering operation on the image. A common method is to utilize error diffusion. However, this method is prone to error accumulation, resulting in color impulses and streaks. This paper presents an improvement on error diffusion dithering through a fuzzy error diffusion algorithm. In this method, the amount of error to be diffused is determined by considering the relative location of the pixel not only to the closest codebook vector, but also to the color cluster it belongs to. The goal is to hide the quantization errors by error diffusion, while preventing the excess accumulation of errors. This is achieved through an attraction-repulsion schema according to a fuzzy membership function. We have implemented the fuzzy error diffusion algorithm for color images and achieved drastic improvements, resulting in superior quality dithered images and significantly lower mean squared error values.

Most color image printing and display devices do not have the capability of reproducing true color images. A common remedy is the use of dithering techniques that take advantage of the lower sensitivity of the eye to spatial resolution and exchange higher color resolution with lower spatial resolution. An adaptive error diffusion method for color images is presented. The error diffusion filter coefficients are updated by a normalized least mean square-type (LMS-type) algorithm to prevent textural contours, color impulses, and color shifts, which are among the most common side effects of the standard dithering algorithms. Another novelty of the new method is its vector character: previous applications of error diffusion have treated the individual color components of an image separately. We develop a general vector approach and demonstrate through simulation studies that superior results are achieved.

Quantization and dithering operations are generally performed sequentially: first quantization and then, dithering. The latter is aware of the former. Dithering manipulates the quantization error to hide it. The same can not be said about the quantizer. The quantizer often works to minimize the quantization error without regard to the fact that the results will be changed by the ditherer. We aim to increase the overall image quality by designing the quantizer in such a way that better results will be obtained after dithering.

Most color image printing and display devices do not have the capability of reproducing true color images. A common remedy is the use of dithering techniques that exploit the lower sensitivity of the eye to spatial resolution and exchange higher color resolution with lower spatial resolution. In this paper an adaptive error diffusion method is presented. The error diffusion filter coefficients are updated by a normalized LMS type algorithm to prevent textural contours, color impulses and color shifts which are among the the most common side effects of the standard dithering algorithms.

We introduce a new median filtering technique. The proposed decimated median filter (DMF) achieves the same impulse removing performance as standard median filters of larger window length. It requires fewer operations and presents less blurring due to the shorter window length.

When evaluating the quality of rank order filtered images, quantitative measures like the mean absolute error (MAE) criterion have been used only with caution because they do not always agree with the results of subjective tests. In general, a large part of the error in a median filtered image falls into one of two categories: remaining impulses or edge smearing. It is known that the human eye responds differently to these two components, whereas measures like the MAE do not distinguish between them. The authors have decomposed the error components, and by computing the MAE separately for each, have found a pair of perceptually significant error measures.>