Test-time adaptation aims to adapt to realistic environments in an online manner by learning during test time. Entropy minimization has emerged as a principal strategy for test-time adaptation due to its efficiency and adaptability. Nevertheless, it remains underexplored in continual test-time adaptation, where stability is more important. We observe that the entropy minimization method often suffers from model collapse, where the model converges to predicting a single class for all images due to a trivial solution. We propose ranked entropy minimization to mitigate the stability problem of the entropy minimization method and extend its applicability to continuous scenarios. Our approach explicitly structures the prediction difficulty through a progressive masking strategy. Specifically, it gradually aligns the model’s probability distributions across different levels of prediction difficulty while preserving the rank order of entropy. The proposed method is extensively evaluated across various benchmarks, demonstrating its effectiveness through empirical results.

Continual learning aims to refine model parameters for new tasks while retaining knowledge from previous tasks. Recently, prompt-based learning has emerged to leverage pre-trained models to be prompted to learn subsequent tasks without the reliance on the rehearsal buffer. Although this approach has demonstrated outstanding results, existing methods depend on preceding task-selection process to choose appropriate prompts. However, imperfectness in task-selection may lead to negative impacts on the performance particularly in the scenarios where the number of tasks is large or task distributions are imbalanced. To address this issue, we introduce a novel task-agnostic approach that focuses on the visual semantic information of image tokens eliminating the preceding task prediction. By leveraging the ability of the pre-trained model to discriminate between similar tokens, our method not only subdivides the prompt but also eliminates the need for additional forward pass. Consequently, we achieve competitive performance on four benchmarks while significantly reducing training time compared to state-of-the-art methods. The code is available at https://github.com/pilsHan/I-Prompt

Domain adaptation for object detection typically entails transferring knowledge from one visible domain to another visible domain. However, there are limited studies on adapting from the visible to the thermal domain, because the domain gap between the visible and thermal domains is much larger than expected, and traditional domain adaptation can not successfully facilitate learning in this situation. To overcome this challenge, we propose a Distinctive Dual-Domain Teacher (D3T) framework that employs distinct training paradigms for each domain. Specifically, we segregate the source and target training sets for building dual-teachers and successively deploy exponential moving average to the student model to individual teachers of each domain. The framework further incorporates a zigzag learning method between dual teachers, facilitating a gradual transition from the visible to thermal domains during training. We validate the superiority of our method through newly designed experimental protocols with wellknown thermal datasets, i.e., FLIR and KAIST. Source code is available at https://github.com/EdwardDo69/D3T.

Unsupervised domain adaptation for object detection (UDAOD) aims to reduce the gap between the labeled source domain and the unlabeled target domain. In the driving scenes, there are distinct unique characteristics that differentiate between the objects, both spatially and categorically. These properties largely maintain their invariance across domains, enabling the effective training of object detectors in the target domain. To consider this, we introduce the domain-invariant object concentration framework, which combines instance-level and image-level approaches to efficiently utilizing domain-invariant object knowledge. At the instance-level, we propose a target surrogate selection module. This module leverages the unique information of object categories to match regions of interest (ROIs) between both domains, thereby enabling effective training on the unlabeled target domain. At the image level, we introduce PutMix, which utilizes domain-invariant knowledge of common object positions in driving scenes based on our statistically defined object crowded area. To validate our method, we experiment across four driving scenarios using four different datasets.

BACKGROUND: Locating small molecule binding sites in target proteins, in the resolution of either pocket or residue, is critical in many drug-discovery scenarios. Since it is not always easy to find such binding sites using conventional methods, different deep learning methods to predict binding sites out of protein structures have been developed in recent years. The existing deep learning based methods have several limitations, including (1) the inefficiency of the CNN-only architecture, (2) loss of information due to excessive post-processing, and (3) the under-utilization of available data sources. METHODS: We present a new model architecture and training method that resolves the aforementioned problems. First, by layering geometric self-attention units on top of residue-level 3D CNN outputs, our model overcomes the problems of CNN-only architectures. Second, by configuring the fundamental units of computation as residues and pockets instead of voxels, our method reduced the information loss from post-processing. Lastly, by employing inter-resolution transfer learning and homology-based augmentation, our method maximizes the utilization of available data sources to a significant extent. RESULTS: The proposed method significantly outperformed all state-of-the-art baselines regarding both resolutions-pocket and residue. An ablation study demonstrated the indispensability of our proposed architecture, as well as transfer learning and homology-based augmentation, for achieving optimal performance. We further scrutinized our model's performance through a case study involving human serum albumin, which demonstrated our model's superior capability in identifying multiple binding sites of the protein, outperforming the existing methods. CONCLUSIONS: We believe that our contribution to the literature is twofold. Firstly, we introduce a novel computational method for binding site prediction with practical applications, substantiated by its strong performance across diverse benchmarks and case studies. Secondly, the innovative aspects in our method- specifically, the design of the model architecture, inter-resolution transfer learning, and homology-based augmentation-would serve as useful components for future work.

Point-cloud based 3D object detectors recently have achieved remarkable progress. However, most studies are limited to the development of network architectures for improving only their accuracy without consideration of the computational efficiency. In this paper, we first propose an autoencoder-style framework comprising channel-wise compression and decompression via interchange transfer-based knowledge distillation. To learn the map-view feature of a teacher network, the features from teacher and student networks are independently passed through the shared autoencoder; here, we use a compressed representation loss that binds the channel-wised compression knowledge from both student and teacher networks as a kind of regularization. The decompressed features are transferred in opposite directions to reduce the gap in the interchange reconstructions. Lastly, we present an head attention loss to match the 3D object detection information drawn by the multi-head self-attention mechanism. Through extensive experiments, we verify that our method can train the lightweight model that is well-aligned with the 3D point cloud detection task and we demonstrate its superiority using the well-known public datasets; e.g., Waymo and nuScenes.11Our code is available at https://github.com/hyeon-jo/interchange-transfer-KD.

In knowledge distillation, since a single, omnipotent teacher network cannot solve all problems, multiple teacher-based knowledge distillations have been studied recently. However, sometimes their improvements are not as good as expected because some immature teachers may transfer the false knowledge to the student. In this paper, to overcome this limitation and take the efficacy of the multiple networks, we divide the multiple networks into teacher and student groups, respectively. That is, the student group is a set of immature networks that require learning the teacher’s knowledge, while the teacher group consists of the selected networks that are capable of teaching successfully. We propose our online role change strategy where the top-ranked networks in the student group are able to promote to the teacher group at every iteration. After training the teacher group using the error samples of the student group to refine the teacher group’s knowledge, we transfer the collaborative knowledge from the teacher group to the student group successfully. We verify the superiority of the proposed method on CIFAR-10, CIFAR-100, and ImageNet which achieves high performance. We further show the generality of our method with various backbone architectures such as ResNet, WRN, VGG, Mobilenet, and Shufflenet.1

The teacher-student framework, prevalent in semi-supervised semantic segmentation, mainly employs the exponential moving average (EMA) to update a single teacher's weights based on the student's. However, EMA updates raise a problem in that the weights of the teacher and student are getting coupled, causing a potential performance bottleneck. Furthermore, this problem may become more severe when training with more complicated labels such as segmentation masks but with few annotated data. This paper introduces Dual Teacher, a simple yet effective approach that employs dual temporary teachers aiming to alleviate the coupling problem for the student. The temporary teachers work in shifts and are progressively improved, so consistently prevent the teacher and student from becoming excessively close. Specifically, the temporary teachers periodically take turns generating pseudo-labels to train a student model and maintain the distinct characteristics of the student model for each epoch. Consequently, Dual Teacher achieves competitive performance on the PASCAL VOC, Cityscapes, and ADE20K benchmarks with remarkably shorter training times than state-of-the-art methods. Moreover, we demonstrate that our approach is model-agnostic and compatible with both CNN- and Transformer-based models. Code is available at https://github.com/naver-ai/dual-teacher.

The objective of human parsing is to partition a human in an image into constituent parts. This task involves labeling each pixel of the human image according to the classes. Since the human body comprises hierarchically structured parts, each body part of an image can have its sole position distribution characteristic. Probably, a human head is less likely to be under the feet, and arms are more likely to be near the torso. Inspired by this observation, we make instance class distributions by accumulating the original human parsing label in the horizontal and vertical directions, which can be utilized as supervision signals. Using these horizontal and vertical class distribution labels, the network is guided to exploit the intrinsic position distribution of each class. We combine two guided features to form a spatial guidance map, which is then superimposed onto the baseline network by multiplication and concatenation to distinguish the human parts precisely. We conducted extensive experiments to demonstrate the effectiveness and superiority of our method on three well-known benchmarks: LIP, ATR, and CIHP databases.††Our code is available at https://github.com/tjpulkl/CDGNet.

Unsupervised domain adaptation (UDA) methods for learning domain invariant representations have achieved remarkable progress. However, most of the studies were based on direct adaptation from the source domain to the target domain and have suffered from large domain discrepancies. In this paper, we propose a UDA method that effectively handles such large domain discrepancies. We introduce a fixed ratio-based mixup to augment multiple intermediate domains between the source and target domain. From the augmented-domains, we train the source-dominant model and the target-dominant model that have complementary characteristics. Using our confidence-based learning methodologies, e.g., bidirectional matching with high-confidence predictions and self-penalization using low-confidence predictions, the models can learn from each other or from its own results. Through our proposed methods, the models gradually transfer domain knowledge from the source to the target domain. Extensive experiments demonstrate the superiority of our proposed method on three public benchmarks: Office-31, Office-Home, and VisDA-2017.1

With the success of deep neural networks, knowledge distillation which guides the learning of a small student network from a large teacher network is being actively studied for model compression and transfer learning. However, few studies have been performed to resolve the poor learning issue of the student network when the student and teacher model sizes significantly differ. In this paper, we propose a densely guided knowledge distillation using multiple teacher assistants that gradually decreases the model size to efficiently bridge the large gap between the teacher and student networks. To stimulate more efficient learning of the student network, we guide each teacher assistant to every other smaller teacher assistants iteratively. Specifically, when teaching a smaller teacher assistant at the next step, the existing larger teacher assistants from the previous step are used as well as the teacher network. Moreover, we design stochastic teaching where, for each mini-batch, a teacher or teacher assistants are randomly dropped. This acts as a regularizer to improve the efficiency of teaching of the student network. Thus, the student can always learn salient distilled knowledge from the multiple sources. We verified the effectiveness of the proposed method for a classification task using CIFAR-10, CIFAR-100, and ImageNet. We also achieved significant performance improvements with various backbone architectures such as ResNet, WideResNet, and VGG.1

Electrocardiogram (ECG) signals offer rich information for analyzing and understanding the cardiac activity of a person. The continuous monitoring of ECG can help diagnose cardiac disorders, such as arrhythmia, effectively. While many wearable healthcare platforms offer continuous ECG monitoring, these devices are cumbersome in the fact that they need to be continuously attached to the human body, which causes uncomfortableness, and limits their usage when monitoring a person's ECG throughout the night as they sleep. In this work, we propose a fully non-intrusive sensing system for monitoring the ECG of a person while in bed. Specifically, we present Heartquake, a geophone-based sensing system for extracting ECG patterns using heartbeat vibrations that penetrate through the mattress. The cardiac activity-originated vibration patterns are captured on the geophone and sent to a server, where the data is filtered to remove external noise and passed on to a bidirectional long short term memory (Bi-LSTM) deep learning model for ECG waveform extraction. Our experimental results with 21study participants suggest that Heartquake can detect all five ECG peaks (e.g., P, Q, R, S, T) with an average error of as low as 16 msec when participants are stationary on the bed. With additional noise factors, this error shows an increase, but can be mitigated from model personalization to still be sufficient enough as a screening tool to detect urgent situations.

Metric learning has attracted significant attention in the past decades, because of its appealing advances in various real-world tasks, e.g., person re-identification and face recognition. Traditional supervised metric learning attempts to seek a discriminative metric, which could minimize the pairwise distance of within-class data samples, while maximizing the pairwise distance of data samples from various classes. However, it is still a challenge to build a robust and discriminative metric, especially for corrupted data in the real-world application. In this paper, we propose a Robust Discriminative Metric Learning algorithm through fast low-rank representation and denoising strategy. To be specific, the metric learning problem is guided by a discriminative regularization by incorporating the pair-wise or class-wise information. Moreover, the low-rank basis learning is jointly optimized with the metric to better uncover the global data structure and remove noise. Furthermore, the fast low-rank representation is implemented to mitigate the computational burden and ensure the scalability on large-scale datasets. Finally, we evaluate our learned metric on several challenging tasks, e.g., face recognition/verification, object recognition, image clustering, and person re-identification. The experimental results verify the effectiveness of our proposed algorithm in comparison to many metric learning algorithms, even deep learning ones.

A new face detector with very high accuracy and real-time speed for mobile phone is introduced. The method achieves the fastest speed and the highest accuracy compared with other similar methods. A series of ideas are proposed in order to accelerate detection speed of the traditional Adaboost detector. First of all, the threshold for weak classifier is learned based on a new multiple instance pruning method regarding not only positive samples but also negative samples, by which, weak classifier is able to reject background more efficiently. Then, a coarse-to-fine scan is applied. Coarse scan is used to find possible face location, and the fine scan refines the face location and rejects false alarms. We further improve the speed of multi-scale face detection by introducing two different template sizes for detector training. By which, the smaller faces can be rapidly detected and the high performance is kept for larger faces. The proposed method is evaluated on public dataset FDDB, the result shows competitive performance against all Adaboost based methods. The method has been implemented on mobile phone and the speed is superior to all competitors.

Unconstrained face recognition under varying views is one of the most challenging tasks, since the difference in appearances caused by poses may be even larger than that due to identity. In this paper, we exploit and analyze a novel pose normalization scheme for facial images under varying views via robust 3D shape reconstruction from single, unconstrained photos in the wild. Specifically, to address the problem of ambiguous 2D-to-3D landmark correspondence and imperfect landmark detector, for each input 2D face, the 3D shape is suggested to be learned by iteratively refining the 3D landmarks and the weighting coefficients of each landmark. Experimental results on both LFW and a large-scale self-collected face databases demonstrate that the proposed approach performs better than the existing representative technologies.

In this paper, we propose a novel unifying framework using a Markov network to learn the relationships among multiple classifiers. In face recognition, we assume that we have several complementary classifiers available, and assign observation nodes to the features of a query image and hidden nodes to those of gallery images. Under the Markov assumption, we connect each hidden node to its corresponding observation node and the hidden nodes of neighboring classifiers. For each observation-hidden node pair, we collect the set of gallery candidates most similar to the observation instance, and capture the relationship between the hidden nodes in terms of a similarity matrix among the retrieved gallery images. Posterior probabilities in the hidden nodes are computed using the belief propagation algorithm, and we use marginal probability as the new similarity value of the classifier. The novelty of our proposed framework lies in the method that considers classifier dependence using the results of each neighboring classifier. We present the extensive evaluation results for two different protocols, known and unknown image variation tests, using four publicly available databases: 1) the Face Recognition Grand Challenge ver. 2.0; 2) XM2VTS; 3) BANCA; and 4) Multi-PIE. The result shows that our framework consistently yields improved recognition rates in various situations.

In this letter, we propose a novel method to extract illumination-invariant features for face recognition and verification under varying illuminations. Inspired by the fact that normalized coefficients of the singular value decomposition (SVD) are insensitive to different illumination conditions, we exploit a simple, yet powerful scheme for describing underlying structures of faces, so-called SVD face. In contrast to previous approaches still suffering from the loss of details, our SVD face greatly preserves textures of the original image based on the relaxation of SVD coefficients. Theoretical analysis shows that our SVD face is an illumination-invariant measure and has an ability to discover meaningful components (e.g., eyes, mouth, etc.) of face images while suppressing the effect of various illuminations. Experimental results on both Yale B and our illuminated face (IF) datasets demonstrate that the SVD face is effective for face recognition and verification compared to previous approaches proposed in literature.

Device-to-device (D2D) communication has received considerable attention in recent years as one of the key technologies for future communication system. Among the typical D2D communication systems, FlashLinQ (FLQ) adopted single-tone orthogonal frequency division multiplexing (OFDM) transmission which enables wide-sense discovery by time domain spreading and very low peak to average power ratio (PAPR). Although synchronization based on a common external source (CES) is basically assumed in FLQ, a means to support devices when they are unable to use a CES is still necessary. In OFDM system, generally, carrier frequency offset (CFO) induces inter-carrier interference (ICI) which degrades overall system performance drastically. Especially in D2D system, ICI can be amplified due to not only different frequency selectivity but also different path losses between links. Accordingly, a precise estimation and compensation of CFO are very important. Therefore, in this paper, we propose a new CFO estimation method for OFDM based D2D system. In the proposed method, the estimation of CFO is obtained by using two correlation results in a symbol. The estimation procedure of the proposed method enables the independent estimation of CFO between links and the estimation range can be adaptively defined depending on the distance between the two windows. By numerical analysis and performance evaluation, we verified that the proposed method achieves precise estimation performance compared to the conventional method.

We propose a novel unifying framework using a Markov network to learn the relationship between multiple classifiers in face recognition. We assume that we have several complementary classifiers and assign observation nodes to the features of a query image and hidden nodes to the features of gallery images. We connect each hidden node to its corresponding observation node and to the hidden nodes of other neighboring classifiers. For each observation-hidden node pair, we collect a set of gallery candidates that are most similar to the observation instance, and the relationship between the hidden nodes is captured in terms of the similarity matrix between the collected gallery images. Posterior probabilities in the hidden nodes are computed by the belief-propagation algorithm. The novelty of the proposed framework is the method that takes into account the classifier dependency using the results of each neighboring classifier. We present extensive results on two different evaluation protocols, known and unknown image variation tests, using three different databases, which shows that the proposed framework always leads to good accuracy in face recognition.

We propose a new face-recognition framework to learn the relationship between multiple classifiers using a Markov network. For each image, we make three face models based on different distances between two eye locations. The novelty of the proposed method lies in that the method not only compares the query and target images at the three different levels, but also takes into account the statistical dependency between the three different models. This dependency is captured by a Markov network, which we describe by a graphical model, where query models are observation nodes, target models are hidden nodes, and the network line represents their relationships. For each observation-hidden node pair, we collect a set of target candidates that are most similar to the observation, and the relationship between the hidden nodes is captured in terms of the similarity between target images. Posterior probabilities at the three hidden nodes of the Markov network are computed by a belief-propagation algorithm. We evaluate the proposed method using FRGC ver 2.0, XM2VTS, BANCA, and PIE databases, which demonstrates its superiority under the untrained variations.

The authors present a robust face recognition system for large-scale data sets taken under uncontrolled illumination variations. The proposed face recognition system consists of a novel illumination-insensitive preprocessing method, a hybrid Fourier-based facial feature extraction, and a score fusion scheme. First, in the preprocessing stage, a face image is transformed into an illumination-insensitive image, called an "integral normalized gradient image," by normalizing and integrating the smoothed gradients of a facial image. Then, for feature extraction of complementary classifiers, multiple face models based upon hybrid Fourier features are applied. The hybrid Fourier features are extracted from different Fourier domains in different frequency bandwidths, and then each feature is individually classified by linear discriminant analysis. In addition, multiple face models are generated by plural normalized face images that have different eye distances. Finally, to combine scores from multiple complementary classifiers, a log likelihood ratio-based score fusion scheme is applied. The proposed system using the face recognition grand challenge (FRGC) experimental protocols is evaluated; FRGC is a large available data set. Experimental results on the FRGC version 2.0 data sets have shown that the proposed method shows an average of 81.49% verification rate on 2-D face images under various environmental variations such as illumination changes, expression changes, and time elapses.

In this paper, we propose a novel method using gender information for achieving better performances of face recognition systems. Gender is one of the important factors for recognizing appearance of human faces and there are many studies on gender classifications such. However, the gender information is not actively applied in vision-based face recognition tasks, because we cannot find out human identity using only gender information. Therefore, we design the face recognition system based on the gender-based facial features with global facial features, and moreover, gender-based score normalization method for verification task. For fair evaluations, we use FRGC database known as a large size face image database.

The face recognition system based on the only single classifier considering the restricted information can not guarantee the generality and superiority of performances in a real situation. To challenge such problems, we propose the hybrid Fourier features extracted from different frequency bands and multiple face models. The hybrid Fourier feature comprises three different Fourier domains; merged real and imaginary components, Fourier spectrum and phase angle. When deriving Fourier features from three Fourier domains, we define three different frequency bandwidths, so that additional complementary features can be obtained. After this, they are individually classified by Linear Discriminant Analysis. This approach makes possible analyzing a face image from the various viewpoints to recognize identities. Moreover, we propose multiple face models based on different eye positions with a same image size, and it contributes to increasing the performance of the proposed system. We evaluated this proposed system using the Face Recognition Grand Challenge (FRGC) experimental protocols known as the largest data sets available. Experimental results on FRGC version 2.0 data sets has proven that the proposed method shows better verification rates than the baseline of FRGC on 2D frontal face images under various situations such as illumination changes, expression changes, and time elapses.

In this paper, we propose an ICA (Independent Component Analysis) based face recognition algorithm, which is robust to illumination and pose variation. Generally, it is well known that the first few eigenfaces represent illumination variation rather than identity. Most PCA (Principal Component Analysis)-based methods have overcome illumination variation by discarding the projection to a few leading eigenfaces. The space spanned after removing a few leading eigenfaces is called the "residual face space". We found that ICA in the residual face space provides more efficient encoding in terms of redundancy reduction and robustness to pose variation as well as illumination variation, owing to its ability to represent non-Gaussian statistics. Moreover, a face image is separated into several facial components, local spaces, and each local space is represented by the ICA bases (independent components) of its corresponding residual space. The statistical models of face images in local spaces are relatively simple and facilitate classification by a linear encoding. Various experimental results show that the accuracy of face recognition is significantly improved by the proposed method under large illumination and pose variations.

We propose a method of efficient face description for facial image retrieval from a large data set. The novel descriptor is obtained by decomposing the face image into several components and then combining the component features. The decomposition combined with LDA (linear discriminant analysis) provides discriminative facial features that are less sensitive to light and pose changes. Each component is represented in its Fisher space and another LDA is then applied to compactly combine the features of the components. To enhance retrieval accuracy further, a simple pose classification and transformation technique is performed, followed by recursive matching. The experimental results obtained on the MPEG-7 data set show an impressive accuracy of our algorithm as compared with the conventional PCA/ICA/LDA methods.

We present a component-based face descriptor with LDA (Linear Discriminant Analysis) and a simple pose classification. Our algorithm has been developed to deal with face image retrieval in huge database such as those in internet environments. Such retrieval requires a compact face descriptor and an efficient recognition algorithm that is robust to variations in lighting and facial poses. Partitioning of a face image into components facilitates the development of an efficient and robust algorithm as follows. First, compensation for light and pose variations is much more easily done on individual components than on the whole image. Second, pose variation is compensated by classifying facial pose and aligning facial components. Finally, LDA is more effective at the component level which has simplified statistics than the whole image. Experimental results on MPEG-7 database show an impressive accuracy of our algorithm compared with conventional LDA methods. 1.