Recent advancements in text-guided audio generation have yielded promising results in diverse domains, including sound effects, speech, and music.However, jointly generating speech with environmental audio remains challenging due to the inherent disparities in their acoustic patterns and temporal dynamics.We propose ImmersiveTTS, an environment-aware text-to-speech (TTS) model that generates natural speech seamlessly integrated within environmental contexts by explicitly modeling cross-modal interactions.Our model builds on a multimodal diffusion transformer and fuses transcript-aligned speech latent with textconditioned environmental context via joint attention.To enhance semantic consistency, we introduce a domain-specific representation alignment objective tailored to environmentaware TTS, leveraging complementary selfsupervised representations from speech and audio encoders.Experimental results show that ImmersiveTTS achieves higher naturalness, intelligibility, and audio fidelity than existing approaches across objective metrics and human listening tests.

• We propose Consciousness-ECG Transformer capturing HRV for conscious state estimation. • We develop a real-time conscious state estimation system robust to noise. • Our method surpasses baselines in sleep staging and anesthesia level monitoring. • System validated on ECG data from real surgeries, proving clinical applicability. Conscious state estimation is important in various medical settings, including sleep staging and anesthesia management, to ensure patient safety and optimize health outcomes. Traditional methods predominantly utilize electroencephalography (EEG), which faces challenges such as high sensitivity to noise and the requirement for controlled environments. In this study, we propose the consciousness-ECG transformer that leverages electrocardiography (ECG) signals for non-invasive and reliable conscious state estimation. Our approach employs a transformer with decoupled query attention to effectively capture heart rate variability features that distinguish between conscious and unconscious states. We implemented the conscious state estimation system with real-time monitoring and validated our system on datasets involving sleep staging and anesthesia level monitoring during surgeries. Experimental results demonstrate that our model outperforms baseline models, achieving accuracies of 0.877 on sleep staging and 0.880 on anesthesia level monitoring. Moreover, our model achieves the highest area under curve values of 0.786 and 0.895 on sleep staging and anesthesia level monitoring, respectively. The proposed system offers a practical and robust alternative to EEG-based methods, particularly suited for dynamic clinical environments. Our results highlight the potential of ECG-based consciousness monitoring to enhance patient safety and advance our understanding of conscious states.

Sleep is important to human health. To improve sleep quality, recorded EEG signals have been utilized for automated sleep staging either in real time during sleep or after sleep. However, because previous approaches classify events that have already occurred rather than forecasting the future, their effectiveness is limited for personalized sleep management. This study proposes the sleep stage forecaster with sleep EEG tokenizer (SSF-SET) framework, which predicts the future sleep-stage sequence using only earlier EEG of the current sleep. SET combines a multi-branch transformer for epoch-level representations, an LSTM-based sequence encoder-decoder, and quantization to convert continuous EEG features into sleep-informative tokens. This quantization makes an information bottleneck that reveals the latent transition structure and suppresses artifacts, enabling reliable next-stage prediction. The decoder-only transformer SSF is first pretrained for next-token prediction with causal attention, then fine-tuned via reinforcement learning that uses sequence-level macro-F1 and token-consistency rewards; throughout it does not access future EEG at inference. With subject-wise cross-validation on the SleepEDF20 and SleepEDF78 datasets, SSF-SET consistently outperformed direct forecasting and forecasting with predicted sleep stages. On SleepEDF20, accuracy was 0.596 and macro-F1 was 0.516. In addition, we achieved an accuracy of 0.611 and a macro-F1 of 0.537 on SleepEDF78. These results show that quantized sleep EEG tokens are effective for autoregressive prediction and demonstrate that future sleep stages can be predicted without future EEG. We believe that SSF-SET is an important component for developing closed-loop and personalized sleep interventions that can act before disruptive transitions occur, and we expect it to improve sleep quality.

Deep learning-based molecular generation models have shown great potential in efficiently exploring vast chemical spaces by generating potential drug candidates with desired properties. However, these models often produce chemically invalid molecules, which limits the usable scope of the learned chemical space and poses significant challenges for practical applications. To address this issue, we propose ChemFixer, a framework designed to correct invalid molecules into valid ones. ChemFixer is built on a transformer architecture, pre-trained using masking techniques, and fine-tuned on a large-scale dataset of valid/invalid molecular pairs that we constructed. Through comprehensive evaluations across diverse generative models, ChemFixer improved molecular validity while effectively preserving the chemical and biological distributional properties of the original outputs. This indicates that ChemFixer can recover molecules that could not be previously generated, thereby expanding the diversity of potential drug candidates. Furthermore, ChemFixer was effectively applied to a drug-target interaction (DTI) prediction task using limited data, improving the validity of generated ligands and discovering promising ligand-protein pairs. These results suggest that ChemFixer is not only effective in data-limited scenarios, but also extensible to a wide range of downstream tasks. Taken together, ChemFixer shows promise as a practical tool for various stages of deep learning-based drug discovery, enhancing molecular validity and expanding accessible chemical space.

Current noninvasive electroencephalography (EEG)-based brain–computer interface (BCI) systems face a fundamental scalability barrier: they either suffer catastrophic forgetting (CF) when learning from new users or require centralized management and use of sensitive neural data from previous users-making real-world deployment impractical. To address this, we introduce subject-incremental continual adaptation (SI-CA), a novel paradigm that models cross-subject continual learning (CL), where knowledge transfer and limited replay sustain stable performance as new subjects are introduced, enabling continual decoding without forgetting. Building on this paradigm, we propose a novel CL framework that achieves memory-efficient adaptation by integrating an extendable architecture with prototype-based consistency regularization and limited replay to mitigate CF. The effectiveness of our proposed method has been validated on three benchmark EEG-BCI datasets. Experimental results demonstrate that the proposed method can effectively reduce reliance on historical samples during CL, while maintaining stable decoding performance for previously learned individuals and ensuring reliable motor decoding for newly encountered ones. This holds significant importance for the development of scalable, privacy-preserving, and stable neural interface systems.

Sleep is associated with mental and physical health; therefore, it is important to pre-assess sleep quality for daily life. However, previous sleep studies, such as self-reporting and sleep stage (SS) classification, have limitations in their ability to assess sleep quality at an early stage. Therefore, a new approach based on the pre-estimation system for sleep quality using a sensor-based model is required. In this study, we propose a distribution-based deep learning model to pre-assess sleep quality using the estimation of the falling asleep process (FAP) at the early SS. Sleep onset latency (SOL) is considered to pre-estimate the FAP because SOL is associated with insomnia and could be observable at the beginning of sleep. Moreover, multimodal polysomnography (PSG) signals, including two-channel electroencephalogram (EEG), electrocardiogram (ECG), and electromyogram (EMG), are used to consider the complexity of sleep dynamics. The advantage of the proposed method is combining each distribution from PSG with the importance score to compare influence and show the probability of wake over time. To evaluate the model performance, the feature extraction and temporal encoder with the previous SS module are compared, respectively, and the proposed model achieved the best mean absolute error (MAE) (8.65) and concordance index (C-index) (0.708) among baseline models, including AttnSleep, TinySleepNet, and MEDI-SOL, while maintaining a competitive Brier score (BS) (0.041) and negative binomial log-likelihood (NBLL) (0.145). As a result, the main contribution of this study is that it can pre-assess sleep quality by estimating a probability over time of when a subject will fall asleep.

3D Human Pose Estimation (HPE) is a one-to-many problem by nature, making it challenging to estimate an accurate 3D pose from a single 2D pose. Some prior works have attempted to tackle this problem by using a conditional generative network. They generate 3D poses from a given 2D pose with noises from a standard Gaussian distribution, while the depth distribution is dependent on each posture and more complex than the standard Gaussian distribution. This may lead to inaccurate distribution learning. In this paper, we propose a probabilistic framework called ProPose to address this issue. ProPose employs Pose Instance-Level Gaussian Distribution (PILGD) derived from 3D pose-based self-representation learning to obtain reliable distribution which is able to address pose-dependent depth distribution. To access this PILGD, we utilize normalizing flow, which learns a mapping function between the PILGD and a 2D Pose-Adaptive Gaussian Distribution (PAGD). This converts the problem of directly estimating 3D poses from 2D poses to a mapping problem between PILGD and PAGD using a normalizing flow. Extensive experiments show the advantages of utilizing the PILGD and PAGD. ProPose achieves comparable performances to previous state-of-the-art probabilistic methods in a multi-hypothesis setting. Notably, ProPose in a single-hypothesis setting demonstrates comparable generalization ability to existing state-of-the-art deterministic methods.

The feature attribution method reveals the contribution of input variables to the decision-making process to provide an attribution map for explanation. Existing methods grounded on the information bottleneck principle compute information in a specific layer to obtain attributions, compressing the features by injecting noise via a parametric damping ratio. However, the attribution obtained in a specific layer neglects evidence of the decision-making process distributed across layers. In this paper, we introduce a comprehensive information bottleneck (CoIBA), which discovers the relevant information in each targeted layer to explain the decision-making process. Our core idea is applying information bottleneck in multiple targeted layers to estimate the comprehensive information by sharing a parametric damping ratio across the layers. Leveraging this shared ratio complements the over-compressed information to discover the omitted clues of the decision by sharing the relevant information across the targeted layers. We suggest the variational approach to fairly reflect the relevant information of each layer by upper bounding layer-wise information. Therefore, CoIBA guarantees that the discarded activation is unnecessary in every targeted layer to make a decision. The extensive experimental results demonstrate the enhancement in faithfulness of the feature attributions provided by CoIBA.

In this paper, we examine a key limitation in query-based detectors for temporal action detection (TAD), which arises from their direct adaptation of originally designed architectures for object detection. Despite the effectiveness of the existing models, they struggle to fully address the unique challenges of TAD, such as the redundancy in multi-scale features and the limited ability to capture sufficient temporal context. To address these issues, we propose a multi-dilated gated encoder and central-adjacent region integrated decoder for temporal action detection transformer (DiGIT). Our approach replaces the existing encoder that consists of multi-scale deformable attention and feedforward network with our multi-dilated gated encoder. Our proposed encoder reduces the redundant information caused by multi-level features while maintaining the ability to capture fine-grained and long-range temporal information. Furthermore, we introduce a central-adjacent region integrated decoder that leverages a more comprehensive sampling strategy for deformable cross-attention to capture the essential information. Extensive experiments demonstrate that DiGIT achieves state-of-the-art performance on THUMOS14, ActivityNet v1.3, and HACS-Segment. Code is available at: https://github.com/Dotori-HJ/DiGIT

Attribution-based explanation techniques capture key patterns to enhance visual interpretability; however, these patterns often lack the granularity needed for insight in fine-grained tasks, particularly in cases of model misclassification, where explanations may be insufficiently detailed. To address this limitation, we propose a fine-grained counterfactual explanation framework that generates both object-level and part-level interpretability, addressing two fundamental questions: (1) which fine-grained features contribute to model misclassification, and (2) where dominant local features influence counterfactual adjustments. Our approach yields explainable counterfactuals in a non-generative manner by quantifying similarity and weighting component contributions within regions of interest between correctly classified and misclassified samples. Furthermore, we introduce a saliency partition module grounded in Shapley value contributions, isolating features with region-specific relevance. Extensive experiments demonstrate the superiority of our approach in capturing more granular, intuitively meaningful regions, surpassing fine-grained methods.

Recent advancements in speech synthesis have significantly improved the audio quality and pronunciation of synthesized speech.To further advance toward human-like conversational speech synthesis, this paper presents Filler-Speech, a novel speech synthesis framework that enables natural filler insertion and control over filler style.To address this, we construct a filler-inclusive speech data, derived from the open-source large-scale speech corpus.This data includes fillers with pitch and duration information.For the generation and style control of natural fillers, we propose a method that tokenizes the filler style and utilizes crossattention with the input text.Furthermore, we introduce a large language model-based filler prediction method that enables natural insertion of fillers even when only text input is provided.The experimental results demonstrate that the constructed dataset is valid and that our proposed methods for filler style control and filler prediction are effective.

Large Language Models (LLMs) have shown significant progress in Open-Domain Question Answering (ODQA), yet most evaluations focus on English and assume locale-invariant answers across languages.This assumption neglects the cultural and regional variations that affect question understanding and answer, leading to biased evaluation in multilingual benchmarks.To address these limitations, we introduce XLQA, a novel benchmark explicitly designed for locale-sensitive multilingual ODQA.XLQA contains 3,000 English seed questions expanded to eight languages, with careful filtering for semantic consistency and human-verified annotations distinguishing locale-invariant and locale-sensitive cases.Our evaluation of five state-of-the-art multilingual LLMs reveals notable failures on localesensitive questions, exposing gaps between English and other languages due to a lack of locale-grounding knowledge.We provide a systematic framework and scalable methodology for assessing multilingual QA under diverse cultural contexts, offering a critical resource to advance the real-world applicability of multilingual ODQA systems.Our findings suggest that disparities in training data distribution contribute to differences in both linguistic competence and locale-awareness across models.

Recently, there has been a growing demand for conversational speech synthesis (CSS) that generates more natural speech by considering the conversational context. To address this, we introduce JELLY, a novel CSS framework that integrates emotion recognition and context reasoning for generating appropriate speech in conversation by fine-tuning a large language model (LLM) with multiple partial LoRA modules. We propose an Emotion-aware Q-former encoder, which enables the LLM to perceive emotions in speech. The encoder is trained to align speech emotions with text, utilizing datasets of emotional speech. The entire model is then fine-tuned with conversational speech data to infer emotional context for generating emotionally appropriate speech in conversation. Our experimental results demonstrate that JELLY excels in emotional context modeling, synthesizing speech that naturally aligns with conversation, while mitigating the scarcity of emotional conversational speech datasets.

Referring video object segmentation aims to segment objects within a video corresponding to a given text description. Existing transformer-based temporal modeling approaches face challenges related to query inconsistency and the limited consideration of context. Query in-consistency produces unstable masks of different objects in the middle of the video. The limited consideration of context leads to the segmentation of incorrect objects by failing to adequately account for the relationship between the given text and instances. To address these issues, we propose the Multi-context Temporal Consistency Module (MTCM), which consists of an Aligner and a Multi-Context Enhancer (MCE). The Aligner removes noise from queries and aligns them to achieve query consistency. The MCE predicts text-relevant queries by considering multi-context. We applied MTCM to four different models, increasing performance across all of them, particularly achieving 47.6 $\mathcal{J}\& \mathcal{F}$ on the MeViS. Code is available at https://github.com/Choi58/MTCM.

Decoding text, speech, or images from human neural signals holds promising potential both as neuroprosthesis for patients and as innovative communication tools for general users. Although neural signals contain various information on speech intentions, movements, and phonetic details, generating informative outputs from them remains challenging, with mostly focusing on decoding short intentions or producing fragmented outputs. In this study, we developed a diffusion model-based framework to decode visual speech intentions from speech-related non-invasive brain signals, to facilitate face-to-face neural communication. We designed an experiment to consolidate various phonemes to train visemes of each phoneme, aiming to learn the representation of corresponding lip formations from neural signals. By decoding visemes from both isolated trials and continuous sentences, we successfully reconstructed coherent lip movements, effectively bridging the gap between brain signals and dynamic visual interfaces. The results highlight the potential of viseme decoding and talking face reconstruction from human neural signals, marking a significant step toward dynamic neural communication systems and speech neuroprosthesis for patients.

Audio super-resolution is challenging owing to its ill-posed nature. Recently, the application of diffusion models in audio super-resolution has shown promising results in alleviating this challenge. However, diffusion-based models have limitations, primarily the necessity for numerous sampling steps, which causes significantly increased latency when synthesizing high-quality audio samples. In this paper, we propose FLowHigh, a novel approach that integrates flow matching, a highly efficient generative model, into audio super-resolution. We also explore probability paths specially tailored for audio super-resolution, which effectively capture high-resolution audio distributions, thereby enhancing reconstruction quality. The proposed method generates high-fidelity, high-resolution audio through a single-step sampling process across various input sampling rates. The experimental results on the VCTK benchmark dataset demonstrate that FLowHigh achieves state-of-the-art performance in audio super-resolution, as evaluated by log-spectral distance and ViSQOL while maintaining computational efficiency with only a single-step sampling process.

Bearing in mind the limited parametric knowledge of Large Language Models (LLMs), retrieval-augmented generation (RAG) which supplies them with the relevant external knowledge has served as an approach to mitigate the issue of hallucinations to a certain extent. However, uniformly retrieving supporting context makes response generation source-inefficient, as triggering the retriever is not always necessary, or even inaccurate, when a model gets distracted by noisy retrieved content and produces an unhelpful answer. Motivated by these issues, we introduce Semantic Uncertainty Guided Adaptive Retrieval (SUGAR), where we leverage context-based entropy to actively decide whether to retrieve and to further determine between single-step and multi-step retrieval. Our empirical results show that selective retrieval guided by semantic uncertainty estimation improves the performance across diverse question answering tasks, as well as achieves a more efficient inference.

Recently, universal waveform generation tasks have been investigated conditioned on various out-of-distribution scenarios. Although one-step GAN-based methods have shown their strength in fast waveform generation, they are vulnerable to train-inference mismatch scenarios such as two-stage text-to-speech. Meanwhile, diffusion-based models have shown their powerful generative performance in other domains; however, they stay out of the limelight due to slow inference speed in waveform generation tasks. Above all, there is no generator architecture that can explicitly disentangle the natural periodic features of high-resolution waveform signals. In this paper, we propose PeriodWave, a novel universal waveform generation model from Mel-spectrogram and neural audio codec. First, we introduce a period-aware flow matching estimator that effectively captures the periodic features of the waveform signal when estimating the vector fields. Additionally, we utilize a multi-period estimator that avoids overlaps to capture different periodic features of waveform signals. Although increasing the number of periods can improve the performance significantly, this requires more computational costs. To reduce this issue, we also propose a single period-conditional universal estimator that can feed-forward parallel by period-wise batch inference. Additionally, we first introduce FreeU to reduce the high-frequency noise for waveform generation. Furthermore, we demonstrate the effectiveness of the proposed method in neural audio codec decoding task, and present the streaming generation framework of non-autoregressive model for speech language models. The experimental results demonstrated that our model outperforms the previous models in reconstruction tasks from Mel-spectrogram and discrete token, and text-to-speech tasks. Source code is available at https://github.com/sh-lee-prml/PeriodWave

Abstractive compression utilizes smaller langauge models to condense query-relevant context, reducing computational costs in retrieval-augmented generation (RAG). However, retrieved documents often include information that is either irrelevant to answering the query or misleading due to factual incorrect content, despite having high relevance scores. This behavior indicates that abstractive compressors are more likely to omit important information essential for the correct answer, especially in long contexts where attention dispersion occurs. To address this issue, we categorize retrieved documents in a more fine-grained manner and propose Abstractive Compression Robust against Noise (ACoRN), which introduces two novel training steps. First, we use offline data augmentation on the training dataset to enhance compressor robustness against two distinct types of retrieval noise. Second, since the language model-based compressor cannot fully utilize information from multiple retrieved documents and exhibits positional bias, we perform fine-tuning to generate summaries centered around key information that directly supports the correct answer. Our experiments demonstrate that T5-large, trained with ACoRN as a compressor, improves EM and F1 scores while preserving the answer string, which could serve as direct evidence. ACoRN excels on datasets with many accuracy-reducing documents, making it highly useful in real-world scenarios.

Large Language Models (LLMs) trained on extensive datasets often learn sensitive information, which raises significant social and legal concerns under principles such as the "Right to be forgotten." Retraining entire models from scratch to remove undesired information is both costly and impractical. Furthermore, existing single-domain unlearning methods fail to address multi-domain scenarios, where knowledge is interwoven across domains such as privacy and copyright, creating overlapping representations that lead to excessive knowledge removal or degraded performance. To tackle these issues, we propose GRAIL (GRadient-based AdaptIve unLearning), a novel multi-domain unlearning framework. GRAIL leverages gradient information from multiple domains to precisely distinguish the unlearning scope from the retention scope, and applies an adaptive parameter-wise localization strategy to selectively remove targeted knowledge while preserving critical parameters for each domain. Experimental results on unlearning benchmarks show that GRAIL achieves unlearning success on par with the existing approaches, while also demonstrating up to 17% stronger knowledge retention success compared to the previous state-of-art method. Our findings establish a new paradigm for effectively managing and regulating sensitive information in large-scale pretrained language models.

Neural operators have emerged as promising frameworks for learning mappings governed by partial differential equations (PDEs), serving as data-driven alternatives to traditional numerical methods. While methods such as the Fourier neural operator (FNO) have demonstrated notable performance, their reliance on uniform grids restricts their applicability to complex geometries and irregular meshes. Recently, Transformer-based neural operators with linear attention mechanisms have shown potential in overcoming these limitations for large-scale PDE simulations. However, these approaches predominantly emphasize global feature aggregation, often overlooking fine-scale dynamics and localized PDE behaviors essential for accurate solutions. To address these challenges, we propose the Locality-Aware Attention Transformer (LA2Former), which leverages K-nearest neighbors for dynamic patchifying and integrates global-local attention for enhanced PDE modeling. By combining linear attention for efficient global context encoding with pairwise attention for capturing intricate local interactions, LA2Former achieves an optimal balance between computational efficiency and predictive accuracy. Extensive evaluations across six benchmark datasets demonstrate that LA2Former improves predictive accuracy by over 50% relative to existing linear attention methods, while also outperforming full pairwise attention under optimal conditions. This work underscores the critical importance of localized feature learning in advancing Transformer-based neural operators for solving PDEs on complex and irregular domains. Code is available at https://github.com/komingsu/LA2Former.

Semi-Supervised semantic segmentation (SSSS) aims to improve segmentation performance by utilising large amounts of unlabeled data with limited labeled samples. Existing methods often suffer from coupling, where over-reliance on initial labeled data leads to suboptimal learning; confirmation bias, where incorrect predictions reinforce themselves repeatedly; and boundary blur caused by limited boundary-awareness and ambiguous edge cues. To address these issues, we propose CW-BASS, a novel framework for SSSS. In order to mitigate the impact of incorrect predictions, we assign confidence weights to pseudo-labels. Additionally, we leverage boundary-delineation techniques, which, despite being extensively explored in weakly-supervised semantic segmentation (WSSS), remain underutilized in SSSS. Specifically, our method: (1) reduces coupling via a confidence-weighted loss that adjusts pseudo-label influence based on their predicted confidence scores, (2) mitigates confirmation bias with a dynamic thresholding mechanism that learns to filter out pseudo-labels based on model performance, (3) tackles boundary blur using a boundary-aware module to refine segmentation near object edges, and (4) reduces label noise through a confidence decay strategy that progressively refines pseudo-labels during training. Extensive experiments on Pascal VOC 2012 and Cityscapes demonstrate that CW-BASS achieves state-of-the-art performance. Notably, CW-BASS achieves a 65.9% mIoU on Cityscapes under a challenging and underexplored 1/30 (3.3%) split (100 images), highlighting its effectiveness in limited-label settings. Our code is available at https://github.com/psychofict/CW-BASS.

Controlling singing style is crucial for achieving an expressive and natural singing voice. Among the various style factors, vibrato plays a key role in conveying emotions and enhancing musical depth. However, modeling vibrato remains challenging due to its dynamic nature, making it difficult to control in singing voice conversion. To address this, we propose VibESVC, a controllable singing voice conversion model that explicitly extracts and manipulates vibrato using discrete wavelet transform. Unlike previous methods that model vibrato implicitly, our approach decomposes the F0 contour into frequency components, enabling precise transfer. This allows vibrato control for enhanced flexibility. Experimental results show that VibE-SVC effectively transforms singing styles while preserving speaker similarity. Both subjective and objective evaluations confirm high-quality conversion.

3D Human Pose Estimation (HPE) is a fundamental task in the computer vision. Generalization in 3D HPE task is crucial due to the need for robustness across diverse environments and datasets. Existing methods often focus on learning relationships between joints to enhance the generalization capability, but the role of the loss landscape, which is closely tied to generalization, remains underexplored. In this paper, we empirically visualize the loss landscape of the 3D HPE task, revealing its complexity and the challenges it poses for optimization. To address this, we first introduce a simple adaptive scaling mechanism that smooths the loss landscape. We further observe that different solutions on this smoothed loss landscape exhibit varying generalization behaviors. Based on this insight, we propose an efficient ensemble approach that combines diverse solutions on the smooth loss landscape induced by our adaptive scaling mechanism. Extensive experimental results demonstrate that our approach improves the generalization capability of 3D HPE models, and can be easily applied, regardless of model architecture, with consistent performance gains.

Electroencephalography (EEG) is a fundamental modality for cognitive state monitoring in brain-computer interfaces (BCIs). However, it is highly susceptible to intrinsic signal errors and human-induced labeling errors, which lead to label noise and ultimately degrade model performance. To enhance EEG learning, multimodal knowledge distillation (KD) has been explored to transfer knowledge from visual models with rich representations to EEG-based models. Nevertheless, KD faces two key challenges: modality gap and soft label misalignment. The former arises from the heterogeneous nature of EEG and visual feature spaces, while the latter stems from label inconsistencies that create discrepancies between ground truth labels and distillation targets. This paper addresses semantic uncertainty caused by ambiguous features and weakly defined labels. We propose a novel cross-modal knowledge distillation framework that mitigates both modality and label inconsistencies. It aligns feature semantics through a prototype-based similarity module and introduces a task-specific distillation head to resolve label-induced inconsistency in supervision. Experimental results demonstrate that our approach improves EEG-based emotion regression and classification performance, outperforming both unimodal and multimodal baselines on a public multimodal dataset. These findings highlight the potential of our framework for BCI applications.

Recently, personalized portrait generation with a text-to-image diffusion model has significantly advanced with Textual Inversion, emerging as a promising approach for creating high-fidelity personalized images. Despite its potential, cur-rent Textual Inversion methods struggle to maintain consistent facial identity due to semantic misalignments between textual and visual embedding spaces regarding identity. We introduce ID-EA, a novel framework that guides text embeddings to align with visual identity embeddings, thereby improving identity preservation in a personalized generation. ID-EA comprises two key components: the ID-driven Enhancer (ID-Enhancer) and the ID-conditioned Adapter (ID-Adapter). First, the ID-Enhancer integrates identity embeddings with a textual ID anchor, refining visual identity embeddings derived from a face recognition model using representative text embeddings. Then, the ID-Adapter leverages the identity-enhanced embedding to adapt the text condition, ensuring identity preservation by adjusting the cross-attention module in the pre-trained UNet model. This process encourages the text features to find the most related visual clues across the foreground snippets. Extensive quantitative and qualitative evaluations demonstrate that ID-EA substantially outperforms state-of-the-art methods in identity preservation metrics while achieving remarkable computational efficiency, generating personalized portraits approximately 15 times faster than existing approaches.

Large language models (LLMs) have demonstrated state-of-the-art performance across a wide range of natural language processing tasks. However, high-performing models are typically accessible only via APIs, incurring substantial inference costs. Cascade methods address this by initially employing a cheaper model and escalating to a stronger one only when necessary. Nevertheless, existing cascade approaches struggle to select a reliable representative response and assess the overall reliability of free-form outputs, as they rely on exact text matching. To overcome these limitations, we propose Keyword-inspired Cascade (KiC), a novel framework for cost-efficient free-form text generation. KiC identifies the most representative answer among multiple outputs from a weaker model and evaluates the semantic alignment of other responses with it. Based on the degree of alignment, KiC determines whether to accept the weaker model’s output or escalate to a stronger model. Experiments on three free-form text generation benchmarks show that KiC achieves 97.53% of GPT-4’s accuracy while reducing API costs by 28.81% on average, and even outperforms GPT-4 in a specific benchmark.

Conventional reinforcement learning (RL) approaches often struggle to learn effective policies under sparse reward conditions, necessitating the manual design of complex, task-specific reward functions. To address this limitation, reinforcement learning from human feedback (RLHF) has emerged as a promising strategy that complements hand-crafted rewards with human-derived evaluation signals. However, most existing RLHF methods depend on explicit feedback mechanisms such as button presses or preference labels, which disrupt the natural interaction process and impose a substantial cognitive load on the user. We propose a novel reinforcement learning from implicit human feedback (RLIHF) framework that utilizes non-invasive electroencephalography (EEG) signals, specifically error-related potentials (ErrPs), to provide continuous, implicit feedback without requiring explicit user intervention. The proposed method adopts a pre-trained decoder to transform raw EEG signals into probabilistic reward components, enabling effective policy learning even in the presence of sparse external rewards. We evaluate our approach in a simulation environment built on the MuJoCo physics engine, using a Kinova Gen2 robotic arm to perform a complex pick-and-place task that requires avoiding obstacles while manipulating target objects. The results show that agents trained with decoded EEG feedback achieve performance comparable to those trained with dense, manually designed rewards. These findings validate the potential of using implicit neural feedback for scalable and human-aligned reinforcement learning in interactive robotics.

Stable diffusion is an outstanding image generation model for text-to-image, but its time-consuming generation process remains a challenge due to the quadratic complexity of attention operations. Recent token merging methods improve efficiency by reducing the number of tokens during attention operations, but often overlook the characteristics of attention-based image generation models, limiting their effectiveness. In this paper, we propose local representative token guided merging (ReToM), a novel token merging strategy applicable to any attention mechanism in image generation. To merge tokens based on various contextual information, ReToM defines local boundaries as windows within attention inputs and adjusts window sizes. Furthermore, we introduce a representative token, which represents the most representative token per window by computing similarity at a specific timestep and selecting the token with the highest average similarity. This approach preserves the most salient local features while minimizing computational overhead. Experimental results show that ReToM achieves a 6.2% improvement in FID and higher CLIP scores compared to the baseline, while maintaining comparable inference time. We empirically demonstrate that ReToM is effective in balancing visual quality and computational efficiency.

Video question answering (VQA) is a multimodal task that requires the interpretation of a video to answer a given question. Existing VQA methods primarily utilize question and answer (Q&A) pairs to learn the spatio-temporal characteristics of video content. However, these annotations are typically event-centric, which is not enough to capture the broader context of each video. The absence of essential details such as object types, spatial layouts, and descriptive attributes restricts the model to learning only a fragmented scene representation. This issue limits the model’s capacity for generalization and higher-level reasoning. In this paper, we propose a fundamental question generation with the integration of question embeddings for video question answering (FIQ), a novel approach designed to strengthen the reasoning ability of the model by enhancing the fundamental understanding of videos. FIQ generates Q&A pairs based on descriptions extracted from videos, enriching the training data with fundamental scene information. Generated Q&A pairs enable the model to understand the primary context, leading to enhanced generalizability and reasoning ability. Furthermore, we incorporate a VQ-CAlign module that assists task-specific question embeddings with visual features, ensuring that essential domain-specific details are preserved to increase the adaptability of downstream tasks. Experiments on SUTD-TrafficQA demonstrate that our FIQ achieves state-of-the-art performance compared to existing baseline methods. Code is available at https://github.com/juyoungohjulie/FIQ

With recent advancements in text-to-image (T2I) models, effectively generating multiple instances within a single image prompt has become a crucial challenge. Existing methods, while successful in generating positions of individual instances, often struggle to account for relationship discrepancy and multiple attributes leakage. To address these limitations, this paper proposes the relation-aware disentangled learning (RaDL) framework. RaDL enhances instance-specific attributes through learnable parameters and generates relation-aware image features via Relation Attention, utilizing action verbs extracted from the global prompt. Through extensive evaluations on benchmarks such as COCO-Position, COCO-MIG, and DrawBench, we demonstrate that RaDL outperforms existing methods, showing significant improvements in positional accuracy, multiple attributes consideration, and the relationships between instances. Our results present RaDL as the solution for generating images that consider both the relationships and multiple attributes of each instance within the multi-instance image.

Composed image retrieval (CIR) is a vision-language task that retrieves a target image using a reference image and modification text, enabling intuitive specification of desired changes. While effectively fusing visual and textual modalities is crucial, existing methods typically adopt either early or late fusion. Early fusion tends to excessively focus on explicitly mentioned textual details and neglect visual context, whereas late fusion struggles to capture fine-grained semantic alignments between image regions and textual tokens. To address these issues, we propose FAR-Net, a multi-stage fusion framework designed with enhanced semantic alignment and adaptive reconciliation, integrating two complementary modules. The enhanced semantic alignment module (ESAM) employs late fusion with cross-attention to capture fine-grained semantic relationships, while the adaptive reconciliation module (ARM) applies early fusion with uncertainty embeddings to enhance robustness and adaptability. Experiments on CIRR and FashionIQ show consistent performance gains, improving Recall@1 by up to 2.4% and Recall@50 by 1.04% over existing state-of-the-art methods, empirically demonstrating that FAR-Net provides a robust and scalable solution to CIR tasks.

Composed Image Retrieval (CIR) is the task of retrieving a target image from a gallery using a composed query consisting of a reference image and a modification text. Among various CIR approaches, training-free zero-shot methods based on pre-trained models are cost-effective but still face notable limitations. For example, sequential VLM-LLM pipelines process each modality independently, which often results in information loss and limits cross-modal interaction. In contrast, methods based on multimodal large language models (MLLMs) often focus exclusively on applying changes indicated by the text, without fully utilizing the contextual visual information from the reference image. To address these issues, we propose multi-faceted Chain-of-Thought with re-ranking (MCoT-RE), a training-free zero-shot CIR framework. MCoT-RE utilizes multi-faceted Chain-of-Thought to guide the MLLM to balance explicit modifications and contextual visual cues, generating two distinct captions: one focused on modification and the other integrating comprehensive visual-textual context. The first caption is used to filter candidate images. Subsequently, we combine these two captions and the reference image to perform multi-grained re-ranking. This two-stage approach facilitates precise retrieval by aligning with the textual modification instructions while preserving the visual context of the reference image. Through extensive experiments, MCoT-RE achieves state-of-the-art results among training-free methods, yielding improvements of up to 6.24% in Recall@10 on FashionIQ and 8.58% in Recall@1 on CIRR.

Low-light image enhancement (LLIE) is a crucial task in computer vision aimed at enhancing the visual fidelity of images captured under low-illumination conditions. Conventional methods frequently struggle with noise, overexposure, and color distortion, leading to significant image quality degradation. To address these challenges, we propose LUMINA-Net, an unsupervised deep learning framework that learns adaptive priors from low-light image pairs by integrating multi-stage illumination and reflectance modules. To assist the Retinex decomposition, inappropriate features in the raw image can be removed using a simple self-supervised mechanism. First, the illumination module intelligently adjusts brightness and contrast while preserving intricate textural details. Second, the reflectance module incorporates a noise reduction mechanism that leverages spatial attention and channel-wise feature refinement to mitigate noise contamination. Through extensive experiments on LOL and SICE datasets, evaluated using PSNR, SSIM, and LPIPS metrics, LUMINA-Net surpasses state-of-the-art methods, demonstrating its efficacy in low-light image enhancement.

Integrating multiple personalized concepts into a single image has recently gained attention in text-to-image (T2I) generation. However, existing methods often suffer from performance degradation in complex scenes due to distortions in non-personalized regions and the need for additional fine-tuning, limiting their practicality. To address this issue, we propose FlipConcept, a novel approach that seamlessly integrates multiple personalized concepts into a single image without requiring additional tuning. We introduce guided appearance attention to enhance the visual fidelity of personalized concepts. Additionally, we introduce mask-guided noise mixing to protect non-personalized regions during concept integration. Lastly, we apply background dilution to minimize concept leakage, i.e., the undesired blending of personalized concepts with other objects in the image. In our experiments, we demonstrate that the proposed method, despite not requiring tuning, outperforms existing models in both single and multiple personalized concept inference. These results demonstrate the effectiveness and practicality of our approach for scalable, high-quality multi-concept personalization.

Malignant arrhythmias (MA), stemming from abnormalities in the neuronal signaling of the cardiac muscle , necessitate sophisticated predictive models for effective clinical management. Traditional machine learning models primarily rely on single-step forecasting and fail to capture the complex temporal dynamics of underlying arrhythmogenic processes. This paper propose the first multi-step forecasting framework for MA, leveraging a brain-inspired approach that emulates and captures the neuronal signal transmission patterns embedded in electrocardiogram (ECG) data. Our framework comprises three key modules: (i) input module, (ii) multi-path propagation module, and (iii) multi-step forecasting module. The multi-path propagation module incorporates short-term and long-term paths that reflect the different time scales of neural information processing . We further introduce novel brain-inspired information processing units within this module. First, the local and global synaptic plasticity units extract the local and global temporal patterns in the ECG using temporal convolution blocks and cosine-similarity based pattern matching. The processed information is transmitted to the subsequent unit, as well as the Hebb-based learning unit, designed to model the neuromodulation of spike- and feature-level activations and connection strength of the pre- and post-synaptic neurons. Evaluated on two benchmark datasets, our model outperforms existing state-of-the-art models and baseline multi-step models in both short-term and long-term forecasting tasks. The results not only demonstrate the potential of our model in providing a robust clinical tool for fine-grained arrhythmia intervention but also offer valuable insights for advancing multi-step forecasting in other applications.

• Proposing an Intuitive BCI Paradigm through high-level visual imagery. • Novel EEG interpretation using phase-locking with deep learning. • Enhanced task classification across subjects for practical BCI application. • Conducted pseudo-online tests to assess the real-world applicability of the BCIs. This study introduces a pioneering approach in brain-computer interface (BCI) technology, featuring our novel concept of high-level visual imagery for non-invasive electroencephalography (EEG)-based communication. High-level visual imagery, as proposed in our work, involves the user engaging in the mental visualization of complex upper limb movements. This innovative approach significantly enhances the BCI system, facilitating the extension of its applications to more sophisticated tasks such as EEG-based robotic arm control. By leveraging this advanced form of visual imagery, our study opens new horizons for intricate and intuitive mind-controlled interfaces. We developed an advanced deep learning architecture that integrates functional connectivity metrics with a convolutional neural network-image transformer. This framework is adept at decoding subtle user intentions, addressing the spatial variability in high-level visual tasks, and effectively translating these into precise commands for robotic arm control. Our comprehensive offline and pseudo-online evaluations demonstrate the framework’s efficacy in real-time applications, including the nuanced control of robotic arms. The robustness of our approach is further validated through leave-one-subject-out cross-validation, marking a significant step towards versatile, subject-independent BCI applications. This research highlights the transformative impact of advanced visual imagery and deep learning in enhancing the usability and adaptability of BCI systems, particularly in robotic arm manipulation.

Recently, denoising diffusion models have demonstrated remarkable performance among generative models in various domains. However, in the speech domain, there are limitations in complexity and controllability to apply diffusion models for time-varying audio synthesis. Particularly, a singing voice synthesis (SVS) task, which has begun to emerge as a practical application in the game and entertainment industries, requires high-dimensional samples with long-term acoustic features. To alleviate the challenges posed by model complexity in the SVS task, we propose HiddenSinger, a high-quality SVS system using a neural audio codec and latent diffusion models. To ensure high-fidelity audio, we introduce an audio autoencoder that can encode audio into an audio codec as a compressed representation and reconstruct the high-fidelity audio from the low-dimensional compressed latent vector. Subsequently, we use the latent diffusion models to sample a latent representation from a musical score. In addition, our proposed model is extended to an unsupervised singing voice learning framework, HiddenSinger-U, to train the model using an unlabeled singing voice dataset. Experimental results demonstrate that our model outperforms previous models regarding audio quality. Furthermore, the HiddenSinger-U can synthesize high-quality singing voices of speakers trained solely on unlabeled data.

Recent Mixup-based data augmentation methods have integrated saliency information for richer supervisory signals. However, they often face significant computational burdens, require additional modules, or are constrained by specific architectures. To overcome these limitations, we present GuidedMixup, a model-agnostic, saliency-aware mixup strategy. Unlike previous methods that struggle with random pairings of discordant source and target images, we focus on matching harmonious pairs among mini-batch images and develop an efficient algorithm to identify image pairs with minimal conflict in salient regions. Thanks to these effective pairs, GuidedMixup employs simplified but fine-grained mask generation and adjusts the pixel-wise mixing ratio based solely on the relative saliency strength of paired images, avoiding complex optimization. Additionally, we introduce GuidedMixup++, which incorporates an optimal location search for efficiently relocating target images. GuidedMixup++ resizes target images and calculates minimal conflict for each pair candidate by considering all possible positions of target images, which is remarkably efficient powered by convolution operations. This information is then used to select pairs for mixing. Experimental results demonstrate that the proposed methods surpass other saliency-based techniques in terms of efficiency, generalization performance, and robustness against corrupted or reduced datasets, as well as in downstream tasks like object detection and instance segmentation.

With the remarkable success of deep neural networks, there is a growing interest in research aimed at providing clear interpretations of their decision-making processes. In this paper, we introduce Attribution Equilibrium, a novel method to decompose output predictions into fine-grained attributions, balancing positive and negative relevance for clearer visualization of the evidence behind a network decision. We carefully analyze conventional approaches to decision explanation and present a different perspective on the conservation of evidence. We define the evidence as a gap between positive and negative influences among gradient-derived initial contribution maps. Then, we incorporate antagonistic elements and a user-defined criterion for the degree of positive attribution during propagation. Additionally, we consider the role of inactivated neurons in the propagation rule, thereby enhancing the discernment of less relevant elements such as the background. We conduct various assessments in a verified experimental environment with PASCAL VOC 2007, MS COCO 2014, and ImageNet datasets. The results demonstrate that our method outperforms existing attribution methods both qualitatively and quantitatively in identifying the key input features that influence model decisions.

Emotional text-to-speech (TTS) has advanced significantly, but challenges persist due to the complexity of emotions and limitations in emotional speech datasets and models. A key issue with previous studies is the reliance on limited emotional speech datasets or extensive manual annotations, which restrict generalization across different speakers and emotional styles. To address this, we propose EmoSphere++, an emotioncontrollable zero-shot TTS model capable of generating expressive speech with fine-grained control over emotional style and intensity—without requiring manual annotations. We introduce a novel emotion-adaptive spherical vector that effectively captures emotional style and intensity, along with a joint attribute style encoder that enhances generalization to both seen and unseen speakers. To further improve emotion transfer in zero-shot scenarios, we introduce an additional disentanglement method to enhance the style transfer performance for zero-shot scenarios. Through both objective and subjective evaluations, we demonstrate the benefits of the proposed model in emotion style and intensity modeling, as well as its effectiveness in enhancing emotional expressiveness across both seen and unseen speakers

Emotional voice conversion (EVC) involves modifying various acoustic characteristics, such as pitch and spectral envelope, to match a desired emotional state while preserving the speaker’s identity. Existing EVC methods often rely on text transcriptions or time-alignment information and struggle to handle varying speech durations effectively. In this paper, we propose DurFlex-EVC, a duration-flexible EVC framework that operates without the need for text or alignment information. We introduce a unit aligner that models contextual information by aligning speech with discrete units representing content, eliminating the need for text or speech-text alignment. Additionally, we design a style autoencoder that effectively disentangles content and emotional style, allowing precise manipulation of the emotional characteristics of the speech. We further enhance emotional expressiveness through a hierarchical stylize encoder that applies the target emotional style at multiple hierarchical levels, refining the stylization process to improve the naturalness and expressiveness of the converted speech. Experimental results from subjective and objective evaluations demonstrate that our approach outperforms baseline models, effectively handling duration variability and enhancing emotional expressiveness in the converted speech.

Developing natural, intuitive, and human-centric input systems for mobile human-machine interaction (HMI) poses significant challenges. Existing gaze or gesture-based interaction systems are often constrained by their dependence on continuous visual engagement, limited interaction surfaces, or cumbersome hardware. To address these challenges, we propose MetaSkin, a novel neurohaptic interface that uniquely integrates neural signals with on-skin interaction for bare-handed, eyes-free interaction by exploiting human's natural proprioceptive capabilities. To support the interface, we developed a deep learning framework that employs multiscale temporal-spectral feature representation and selective feature attention to effectively decode neural signals generated by on-skin touch and motion gestures. In experiments with 12 participants, our method achieved offline accuracies of 81.95% for touch location discrimination, 71.00% for motion type identification, and 46.08% for 10-class touch-motion classification. In pseudo-online settings, accuracies reached 99.43% for touch onset detection, and 80.34% and 67.02% for classification of touch location and motion type, respectively. Neurophysiological analyses revealed distinct neural activation patterns in the sensorimotor cortex, underscoring the efficacy of our multiscale approach in capturing rich temporal and spectral dynamics. Future work will focus on optimizing the system for diverse user populations and dynamic environments, with a long-term goal of advancing human-centered, neuroadaptive interfaces for next-generation HMI systems. This work represents a significant step toward a paradigm shift in design of brain-computer interfaces, bridging sensory and motor paradigms for building more sophisticated systems.

As transportation systems grow in complexity and autonomous vehicle technologies advance, detecting the mental states of drivers and passengers is crucial for enhancing traffic safety and user experience. Recent studies have explored multi-biosignal fusion to monitor these states. Despite these efforts, existing methods fail to reflect the complexity and usefulness of various modalities, and do not consider the informativeness of biosignals based on signal quality or the contribution of each modality as biomarkers. To address this issue, we introduce dynamic multi-biosignal fusion (DMBF) to detect the mental states of drivers and passengers. DMBF employs a dynamic gate mechanism that estimates reliability based on data quality through confidence-aware learning, integrating this information into the learning process. Moreover, a spatial-temporal attention module is utilized to capture and combine key biosignal patterns across channels and time. Extensive evaluations were conducted on five datasets related to motion sickness, drowsiness, distraction, emotions, and sustained attention. Each dataset yielded F1 scores of 0.5569, 0.7187, 0.6647, 0.9378, and 0.8092, respectively, outperforming existing baseline models. The experimental results and ablation studies have demonstrated that DMBF is a more robust and versatile method for detecting the mental states of drivers and passengers. DMBF provides a foundation for future enhancements in traffic safety and passenger experience and has the potential for broader applications in multi-biosignal monitoring systems.

Drowsy driving is one of the greatest threats to road safety, which increases the importance of intelligent systems that can monitor driver drowsiness. Electroencephalogram (EEG)–based monitoring systems have gained attention because EEG is known to directly measure brain activities that reflect the mental state of the driver. However, calibration is necessary before using the system because EEG signals vary between and within subjects. Therefore, generalized EEG-based drowsiness estimation has become challenging. In this paper, we propose an EEG-based driver drowsiness classification framework without the need for calibration, which can be generalized to unseen subjects. We augment the features of unseen domains (i.e., subjects) with a Dirichlet mixup between prototypes of source domains to complement other domain knowledge. The parameter$\boldsymbol{\alpha}$vector of the Dirichlet distribution adjusts the intensity of the mixup, allowing for diverse enhancement. Furthermore, we utilize an auxiliary batch normalization module for augmented samples to avoid inaccurate estimation by the difference in distribution. The experiments were carried out using two EEG datasets, each measured using different drowsiness indicators, the Karolinska sleepiness scale, and reaction time. In leave-one-subject-out cross-validation, the proposed framework achieved outstanding performance in both datasets, anF1-score of 62.69% and 70.33% and an area under the receiver operating characteristic curve (AUROC) of 71.73% and 73.80%, respectively. The experimental results demonstrate the potential for practical applications of brain-computer interfaces without calibration.

Recent advances in deep learning have demonstrated excellent results for face attribute recognition (FAR), which is generally trained with large-scale labeled data. Despite the significant progress in this field, most existing works mainly rely on large-scale labeled data, which is not practical in many real-world FAR applications. Numerous studies have been conducted to address this problem, but they require either large external face datasets or complex auxiliary tasks for pretraining the backbone network. In this article, we propose a new method named language-driven spatial-semantic cross-attention (LSA) that does not require any pretraining steps with additional datasets or auxiliary tasks. Driven by the impressive outcomes of recent computer vision studies using language models, we harness language-based relational information to enhance attribute recognition. The core of LSA is to combine and balance the learned scaled-dot product attention with the attention constructed based on language-driven knowledge. To this end, we propose a correlation dictionary, obtained with the similarity between text embeddings of facial attributes and facial regions to represent relationships. The correlation dictionary then creates a cross-attention form and is combined into the cross-attention with balancing parameters. Thus, we can compensate for the lack of data information by providing prior knowledge directly to the network. Extensive experiments demonstrate that our method surpasses state-of-the-art techniques, achieving an average improvement of 0.29% on the CelebA dataset and 0.39% on the LFWA dataset with limited labeling data, even without additional dataset training.

Conventional semi-supervised learning (SSL) ideally assumes that labeled and unlabeled data share an identical class distribution; however, in practice, this assumption is easily violated, as unlabeled data often includes unknown class data, i.e., outliers. The outliers are treated as noise, considerably degrading the performance of SSL models. To address this drawback, we propose a novel framework, diversify and conquer (DAC), to enhance SSL robustness in the context of open-set SSL (OSSL). In particular, we note that existing OSSL methods rely on prediction discrepancies between inliers and outliers from a single model trained on labeled data. This approach can be easily failed when the labeled data are insufficient, leading to performance degradation that is worse than naive SSL that do not account for outliers. In contrast, our approach exploits prediction disagreements among multiple models that are differently biased toward the unlabeled distribution. By leveraging the discrepancies arising from training on unlabeled data, our method enables robust outlier detection, even when the labeled data are underspecified. Our key contribution is constructing a collection of differently biased models through a single training process. By encouraging divergent heads to be differently biased toward outliers while making consistent predictions for inliers, we exploit the disagreement among these heads as a measure to identify unknown concepts. Extensive experiments demonstrate that our method significantly surpasses state-of-the-art OSSL methods across various protocols.

Large language model (LLM)-based speech synthesis has been widely adopted in zero-shot speech synthesis. However, they require a large-scale data and possess the same limitations as previous autoregressive speech models, including slow inference speed and lack of robustness. This article proposes HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech (TTS) and voice conversion (VC). We verified that hierarchical speech synthesis frameworks could significantly improve the robustness and expressiveness of the synthetic speech. Furthermore, we significantly improve the naturalness and speaker similarity of synthetic speech even in zero-shot speech synthesis scenarios. For TTS, we adopt the text-to-vec (TTV) framework, which generates a self-supervised speech representation and an F0 representation based on text representations and prosody prompts. Then, HierSpeech++ generates speech from the generated vector, F0, and voice prompt. We further introduce a high-efficient speech super-resolution (SpeechSR) framework from 16 to 48 kHz. The experimental results demonstrated that the hierarchical variational autoencoder could be a strong zero-shot speech synthesizer given that it outperforms LLM-based and diffusion-based models. Moreover, we achieved the first human-level quality zero-shot speech synthesis. Audio samples and source code are available at https://github.com/hierspeechpp/code.

With the rise of explainable artificial intelligence (XAI), counterfactual (CF) explanations have gained significant attention. Effective CFs must be valid (classified as the CF class), practical (minimally deviated from the input), and plausible (close to the CF data manifold). However, practicality and plausibility often conflict, making valid CF generation challenging. To address this, we propose a novel framework that generates CFs by adjusting only semantic information in the disentangled latent space of a diffusion model. This shifts the sample closer to the CF manifold and across the decision boundary. In our framework, the latent vector mapping step occasionally produces invalid CFs or CFs insufficiently close to the decision boundary, resulting in dissimilarity to the input. Our method overcomes this with a two-stage latent vector adjustment: 1) linear interpolation and 2) time-step-wise optimization during reverse diffusion within the space accommodating linear changes in class information from the input. Experiments demonstrate that our approach generates more valid, plausible, and practical CFs by effectively leveraging the properties of the disentangled latent space.

Human–machine interfaces (HMIs) face several challenges that hinder their long-term performance and adaptability, such as severe overfitting of machine learning models due to limited calibration data on individual users, data distribution shifts owing to changes in user-state over time, and disruptions from outlier samples caused by user distractions. To address this, we propose a novel framework called brain-guided self-paced curriculum learning (BG-SPCL) that leverages user-state information to effectively constrain the learning space for the user intention decoder (curriculum learning) and dynamically adapts the learning curriculum based on the decoder state information self-paced learning (SPL). In the curriculum learning stage, we extract the level of user distraction from brain signals and determine the feasible curriculum region. In the SPL stage, sample difficulty is inferred from the decoder loss, and the sample weights are dynamically adjusted such that the decoder progressively learns more difficult samples. We evaluated the effectiveness of our approach by conducting extensive experiments on three public brain–machine interface (BMI) benchmarks, which constitute an HMI scenario where the user’s brain signals are naturally available. Our results showed superior performance of the proposed method compared to baseline in both offline and online learning settings with no labeled user data, demonstrating the potential for practical application of our framework in both BMI and HMI systems. Our code is available at:https://github.com/yeonoi3488/bg-spcl.

Clinical decision support systems (CDSSs) have been widely utilized to support the decisions made by cardiologists when detecting and classifying arrhythmia from electrocardiograms. However, forming a CDSS for the arrhythmia classification task is challenging due to the varying lengths of arrhythmias. Although the onset time of arrhythmia varies, previously developed methods have not considered such conditions. Thus, we propose a framework that consists of 1) local and global extraction and 2) local-global information fusion with attention to enable arrhythmia detection and classification within a constrained input length. The framework’s performance was evaluated in terms of 10-class and 4-class arrhythmia detection, focusing on identifying the onset and ending point of arrhythmia episodes and their duration using the MIT-BIH arrhythmia database (MITDB) and the MIT-BIH atrial fibrillation database (AFDB). Duration, episode, and Dice score performances resulted in overall F1-scores of 96.45%, 82.05%, and 96.31% on the MITDB and 97.57%, 98.31%, and 97.45% on the AFDB, respectively. The results demonstrated statistically superior performance compared to those of the benchmark models. To assess the generalization capability of the proposed method, an MITDB-trained model and MIT-BIH malignant ventricular arrhythmia database-trained model were tested AFDB and MITDB, respectively. Superior performance was attained compared with that of a state-of-the-art model. The proposed method effectively captures both local and global information and dynamics without significant information loss. Consequently, arrhythmias can be detected with greater accuracy, and their occurrence times can be precisely determined, enabling the clinical field to develop more accurate treatment plans based on the proposed method.

Diffusion-based generative models have recently exhibited powerful generative performance. However, as many attributes exist in the data distribution and owing to several limitations of sharing the model parameters across all levels of the generation process, it remains challenging to control specific styles for each attribute. To address the above problem, we introduce decoupled denoising diffusion models (DDDMs) with disentangled representations, which can enable effective style transfers for each attribute in generative models. In particular, we apply DDDMs for voice conversion (VC) tasks, tackling the intricate challenge of disentangling and individually transferring each speech attributes such as linguistic information, intonation, and timbre. First, we use a self-supervised representation to disentangle the speech representation. Subsequently, the DDDMs are applied to resynthesize the speech from the disentangled representations for style transfer with respect to each attribute. Moreover, we also propose the prior mixup for robust voice style transfer, which uses the converted representation of the mixed style as a prior distribution for the diffusion models. The experimental results reveal that our method outperforms publicly available VC models. Furthermore, we show that our method provides robust generative performance even when using a smaller model size. Audio samples are available at https://hayeong0.github.io/DDDM-VC-demo/.

Recent advances in semi-supervised learning (SSL) have relied on the optimistic assumption that labeled and unlabeled data share the same class distribution. However, this assumption is often violated in real-world scenarios, where unlabeled data may contain out-of-class samples. SSL with such uncurated unlabeled data leads training models to be corrupted. In this paper, we propose a robust SSL method for learning from uncurated real-world data within the context of open-set semi-supervised learning (OSSL). Unlike previous works that rely on feature similarity distance, our method exploits uncertainty in logits. By leveraging task-dependent predictions of logits, our method is capable of robust learning even in the presence of highly correlated outliers. Our key contribution is to present an unknown-aware graph regularization (UAG), a novel technique that enhances the performance of uncertainty-based OSSL frameworks. The technique addresses not only the conflict between training objectives for inliers and outliers but also the limitation of applying the same training rule for all outlier classes, which are existed on previous uncertainty-based approaches. Extensive experiments demonstrate that UAG surpasses state-of-the-art OSSL methods by a large margin across various protocols. Codes are available at https://github.com/heejokong/UAGreg.

In this paper, we investigate that the normalized co-ordinate expression is a key factor as reliance on hand-crafted components in query-based detectors for tempo-ral action detection (TAD). Despite significant advancements towards an end-to-end framework in object detection, query-based detectors have been limited in achieving full end-to-end modeling in TAD. To address this is-sue, we propose TE-TAD, a full end-to-end temporal action detection transformer that integrates time-aligned co-ordinate expression. We reformulate coordinate expression utilizing actual time line values, ensuring length-invariant representations from the extremely diverse video duration environment. Furthermore, our proposed adaptive query selection dynamically adjusts the number of queries based on video length, providing a suitable solution for varying video durations compared to a fixed query set. Our approach not only simplifies the TAD process by eliminating the needfor hand-crafted components but also significantly improves the performance of query-based detectors. Our TE-TAD outperforms the previous query-based detectors and achieves competitive performance compared to state-of-the-art methods on popular benchmark datasets. Code is available at: https://github.com/Dotori-HJ/TE-TAD

Recently, singing voice synthesis (SVS) models have shown significant progress with generative models. However, previous SVS models inaccurately predict prior and fundamental frequency (F0) for unseen speakers, resulting in a low-quality generated singing voice. To address these issues, in this paper, we propose MIDI-Voice for expressive singing voice synthesis and robust zero-shot singing voice style transfer. We employ a MIDI-based prior to a score-based diffusion model for better singing voice style adaptation. We first generate a MIDI-driven prior from the musical score, and this only includes the note information, not speaker information resulting in high-quality singing voice style adaptation. We also propose a DDSP-based MIDI-style prior for synthesizing a more expressive singing voice and for singing style adaptation, although it requires additional information from the audio. The experimental results show that MIDI-Voice outperforms the previous models in synthesizing an expressive singing voice, and also the superiority in zero-shot singing voice style transfer performance.

Although there has been significant advancement in the field of speech-to-speech translation, conventional models still require language-parallel speech data between the source and target languages for training. In this paper, we introduce TranSentence, a novel speech-to-speech translation without language-parallel speech data. To achieve this, we first adopt a language-agnostic sentence-level speech encoding that captures the semantic information of speech, irrespective of language. We then train our model to generate speech based on the encoded embedding obtained from a language-agnostic sentence-level speech encoder that is pre-trained with various languages. With this method, despite training exclusively on the target language’s monolingual data, we can generate target language speech in the inference stage using language-agnostic speech embedding from the source language speech. Furthermore, we extend TranSentence to multilingual speech-to-speech translation. The experimental results demonstrate that TranSentence is superior to other models.

The extraction of meaningful representations from sleep electroencephalogram (EEG) with limited labels poses a significant challenge. To deal with the lack of labels, existing sleep EEG approaches employ self-supervised learning (SSL) with negative pairs. However, they limit available augmentations because similar features are repelled in the training process. To address this issue, we propose a positive-only approach with a novel augmentation technique tailored for EEG signals. This technique involves clustering subsequences from the signal and preserving the closest subsequences to the cluster centers while transforming the remaining sequence to enhance overall representation. Our SSL framework with a clustering-based augmentation preserves key signal features, involving a convolutional encoder to capture complex EEG patterns. Through experiments, we demonstrate that our proposed method outperforms state-of-the-art models in sleep stage classification, achieving an accuracy of 87.1% on the Sleep-EDF-20, 84.8% on the Sleep-EDF-78, and 85.1% on the SHHS datasets. Especially, our method proves to be effective in scenarios with limited labeled data. These results highlight its potential to advance the field of biomedical signal processing.

In the field of 3D Human Pose Estimation (HPE), scalability and generalization across diverse real-world scenarios remain significant challenges. This paper addresses two key bottlenecks to scalability: limited data diversity caused by 'popularity bias' and increased 'one-to-many' depth ambiguity arising from greater pose diversity. We introduce the Biomechanical Pose Generator (BPG), which leverages biomechanical principles, specifically the normal range of motion, to autonomously generate a wide array of plausible 3D poses without relying on a source dataset, thus overcoming the restrictions of popularity bias. To address depth ambiguity, we propose the Binary Depth Coordinates (BDC), which simplifies depth estimation into a binary classification of joint positions (front or back). This method decomposes a 3D pose into three core elements—2D pose, bone length, and binary depth decision—substantially reducing depth ambiguity and enhancing model robustness and accuracy, particularly in complex poses. Our results demonstrate that these approaches increase the diversity and volume of pose data while consistently achieving performance gains, even amid the complexities introduced by increased pose diversity.

Zero-Shot Temporal Action Detection (ZSTAD) aims to classify and localize action segments in untrimmed videos for unseen action categories. Most existing ZSTAD methods utilize a foreground-based approach, limiting the integration of text and visual features due to their reliance on pre-extracted proposals. In this paper, we introduce a cross-modal ZSTAD baseline with mutual cross-attention, integrating both text and visual information throughout the detection process. Our simple approach results in superior performance compared to previous methods. Despite this improvement, we further identify a common-action bias issue that the cross-modal baseline over-focus on common sub-actions due to a lack of ability to discriminate text-related visual parts. To address this issue, we propose Text-infused attention and Foreground-aware Action Detection (Ti-FAD), which enhances the ability to focus on text-related sub-actions and distinguish relevant action segments from the background. Our extensive experiments demonstrate that Ti-FAD outperforms the state-of-the-art methods on ZSTAD benchmarks by a large margin: 41.2\% (+ 11.0\%) on THUMOS14 and 32.0\% (+ 5.4\%) on ActivityNet v1.3. Code is available at: https://github.com/YearangLee/Ti-FAD.

Adversarial attacks mislead deep neural network classifiers with slight perturbations, underscoring the necessity for the development of robust defenses to ensure the secure and responsible use of artificial intelligence. Recent research has shown that diffusion-based adversarial purification methods have emerged as a promising defense technique, but often suffer from computational inefficiencies and suboptimal results. To address these issues, we propose RoPAR, an innovative approach that enhances robustness against adversarial attacks by aggregating purified images at intermediate steps of the diffusion process. Our method improves model robustness while reducing the required diffusion steps. We also introduce a technique for reordering Gaussian noise to minimize semantic information loss while removing adversarial perturbations. These enhancements significantly reduce the number of function evaluations from 200 to 6, achieving a robust accuracy of 92.39 % against preprocessor-blind PGD attacks on CIFAR-10, a 2.29 percentage point improvement over state-of-the-art. Moreover, our method demonstrates its effectiveness in real-world scenarios, achieving 87.46 % accuracy on CIFAR-10C.

Adversarial attacks pose a significant threat to security-critical applications by deliberately deceiving model predictions. Numerous works attempt to create robust models by encoding useful information to intermediate representations. However, they still contain too much information about the training data which hinders improving the robustness of the model. To mitigate this issue, we propose a novel approach, CHBaR, that incorporates class-conditioned information into intermediate representations. The class-conditioned information plays the role of weight components which are multiplied with the intermediate representations to produce class-conditioned representations. We utilize an attribution-based explanation method to obtain this class-conditioned information. As a result, the weight components emphasize class-relevant features by highlighting relevant information from the target class. This weighting process easily integrates the target class without complex computations and conceals useless representations, thus enhancing model predictions by masking features unrelated to the class. Extensive experiments demonstrate the effectiveness of our proposed method in enhancing adversarial robustness. Especially, on the SVHN dataset, our proposed method shows an increment of 6.98 % points compared to the baseline model in PGD40adversarial attack with the TRADES training setting.

Brain-computer interface (BCI) technology enables communication between humans and devices by reflecting users' status and intentions. Electroencephalography (EEG) signals are utilized to capture brain electrical activity with no surgical operation. When conducting motor imagery (MI), one of the endogenous BCI paradigms, the users imagine the movement of muscles used when performing a certain movement without actual physical movement. However, not all subjects show outstanding classification performance in decoding MI-based EEG signals. We propose the novel method that utilizes the weights of the pre-trained model to generate the personalized weights, effectively combining the general MI features with the personalized features. We used the 5-fold cross-validation for evaluating the performances, and conducted the experiments in 3 different pre-trained models (Top-3, Top-5, and Top-7). We compared the performances of our proposed method using the baseline and the full fine-tuning. In comparison to our proposed method with the baseline, our proposed method achieved the improvement of the average accuracies in all pre-trained models, and those values were 0.123, 0.138, and 0.143, respectively. When comparing our proposed method with the full fine-tuning, the average accuracies of our proposed method were the highest in all pre-trained models, and the differences in the average accuracies were 0.012, 0.019, and 0.009, respectively. Hence, we demonstrated the possibility of improving the precision and effectiveness of the EEG-based systems by reflecting the individual differences in EEG signals among the subjects with low classification accuracy.

Text-to-speech (TTS) technologies have recently expanded to incorporate natural language prompts for user-friendly control of speech styles, driven by significant advancements in language models. Traditional prompt-based TTS research, however, typically requires large-scale prompt generation that often necessitates costly human annotations. To address this challenge, we propose PromotiCon, a model that leverages prompts generated without human annotations to control emotions in speech. Our model utilizes abundant prompts generated using a large language model. Additionally, we propose an emotion distance-based prompt-speech matching method to appropriately pair the generated prompts with the most resembling speech data. To enhance speaker adaptation, we utilize a semi-supervised approach that allows the joint utilization of multi-speaker data without emotion labels. As a result, our model facilitates zero-shot emotional speech synthesis. Our experimental results confirm the effectiveness of our approach. Audio samples are available at https://promoticon.github.io/.

Recent advancements in large language models (LLMs) highlight their fluency in generating responses to diverse prompts. However, these models sometimes generate plausible yet incorrect “hallucinated” facts, undermining trust. A frequent but often overlooked cause of such errors is the use of poorly structured or vague prompts by users, leading LLMs to base responses on assumed rather than actual intentions. To mitigate hallucinations induced by these ill-formed prompts, we introduce Curative Prompt Refinement (CPR), a plug-and-play framework for curative prompt refinement that 1) cleans ill-formed prompts, and 2) generates additional informative task descriptions to align the intention of the user and the prompt using a fine-tuned small language model. When applied to language models, we discover that CPR significantly increases the quality of generation while also mitigating hallucination. Empirical studies show that prompts with CPR applied achieves over a 90 % win rate over the original prompts without any external knowledge.

The inherent complexity and extensive architecture of deep neural networks often lead to overfitting, compromising their ability to generalize to new, unseen data. One of the regularization techniques, data augmentation, is now considered vital to alleviate this, and mixup, which blends pairs of images and labels, has proven effective in enhancing model generalization. Recently, incorporating saliency in mixups has shown performance gains by retaining salient regions in mixed results. While these methods have become mainstream at the input level, their applications at the feature level remain under-explored. Our observations indicate that outcomes from naive applications of input saliency-based methods did not consistently lead to enhancements in performance. In this paper, we attribute these observations primarily to two challenges: ‘Hard Boundary Issue’ and ‘Saliency Mismatch.’ The Hard Boundary Issue describes a situation where masks with distinct, sharp edges work well at the input level, but lead to unintended distortions in the deeper layers. The Saliency Mismatch points to the disparity between saliency masks generated from input images and the saliency of feature maps. To tackle these challenges, we present a novel method called ‘attention-based mixup mask adaptation’ (MMA). This approach employs an attention mechanism to effectively adapt mixup masks, which are designed to maximize saliency at the input level, for feature augmentation purposes. We reduce the Saliency Mismatch problem by incorporating the spatial significance of the feature map into the mixup mask. Additionally, we address the Hard Boundary Issue by applying softmax to smoothen the adjusted mixup mask. Through comprehensive experiments, we validate our observations and confirm the effectiveness of applying MMA to saliency-aware mixup approaches at the feature level, as evidenced by the performance improvements on multiple benchmarks and the robustness improvements against corruption and deformation.

This paper proposes Fre-Painter, a high-fidelity audio super-resolution system that utilizes robust speech representation learning with various masking strategies. Recently, masked autoencoders have been found to be beneficial in learning robust representations of audio for speech classification tasks. Following these studies, we leverage these representations and investigate several masking strategies for neural audio super-resolution. In this paper, we propose an upper-band masking strategy with the initialization of the mask token, which is simple but efficient for audio super-resolution. Furthermore, we propose a mix-ratio masking strategy that makes the model robust for input speech with various sampling rates. For practical applicability, we extend Fre-Painter to a text-to-speech system, which synthesizes high-resolution speech using low-resolution speech data. The experimental results demonstrate that Fre-Painter outperforms other neural audio super-resolution models.

Expressive text-to-speech systems have undergone significant advancements owing to prosody modeling, but conventional methods can still be improved. Traditional approaches have relied on the autoregressive method to predict the quantized prosody vector; however, it suffers from the issues of long-term dependency and slow inference. This study proposes a novel approach called DiffProsody in which expressive speech is synthesized using a diffusion-based latent prosody generator and prosody conditional adversarial training. Our findings confirm the effectiveness of our prosody generator in generating a prosody vector. Furthermore, our prosody conditional discriminator significantly improves the quality of the generated speech by accurately emulating prosody. We use denoising diffusion generative adversarial networks to improve the prosody generation speed. Consequently, DiffProsody is capable of generating prosody 16 times faster than the conventional diffusion model. The superior performance of our proposed method has been demonstrated via experiments.

The global prevalence of childhood and adolescent obesity is a major concern due to its association with chronic diseases and long-term health risks. Artificial intelligence technology has been identified as a potential solution to accurately predict obesity rates and provide personalized feedback to adolescents. This study highlights the importance of early identification and prevention of obesity-related health issues. To develop effective algorithms for the prediction of obesity rates and provide personalized feedback, factors such as height, weight, waist circumference, calorie intake, physical activity levels, and other relevant health information must be taken into account. Therefore, by collecting health datasets from 321 adolescents who participated in Would You Do It! application, we proposed an adolescent obesity prediction system that provides personalized predictions and assists individuals in making informed health decisions. Our proposed deep learning framework, DeepHealthNet, effectively trains the model using data augmentation techniques, even when daily health data are limited, resulting in improved prediction accuracy (acc: 0.8842). Additionally, the study revealed variations in the prediction of the obesity rate between boys (acc: 0.9320) and girls (acc: 0.9163), allowing the identification of disparities and the determination of the optimal time to provide feedback. Statistical analysis revealed that the performance of the proposed deep learning framework was more statistically significant (p 0.001) compared to the other general models. The proposed system has the potential to effectively address childhood and adolescent obesity.

Sleep onset latency (SOL) is an important factor relating to the sleep quality of a subject. Therefore, accurate prediction of SOL is useful to identify individuals at risk of sleep disorders and to improve sleep quality. In this study, we estimate SOL distribution and falling asleep function using an electroencephalogram (EEG), which can measure the electric field of brain activity. We proposed a Multi Ensemble Distribution model for estimating Sleep Onset Latency (MEDi-SOL), consisting of a temporal encoder and a time distribution decoder. We evaluated the performance of the proposed model using a public dataset from the Sleep Heart Health Study. We considered four distributions, Normal, log-Normal, Weibull, and log-Logistic, and compared them with a survival model and a regression model. The temporal encoder with the ensemble log-Logistic and log-Normal distribution showed the best and second-best scores in the concordance index (C-index) and mean absolute error (MAE). Our MEDi-SOL, multi ensemble distribution with combining log-Logistic and log-Normal distribution, shows the best score in C-index and MAE, with a fast training time. Furthermore, our model can visualize the process of falling asleep for individual subjects. As a result, a distribution-based ensemble approach with appropriate distribution is more useful than point estimation.

As deep neural networks (DNNs) have gained considerable attention in recent years, there have been several cases applying DNNs to portfolio management (PM). Although some researchers have experimentally demonstrated its ability to make a profit, it is still insufficient to use in real situations because existing studies have failed to answer how risky investment decisions are. Furthermore, even though the objective of PM is to maximize returns within a risk tolerance, they overlook the predictive uncertainty of DNNs in the process of risk management. To overcome these limitations, we propose a novel framework called risk-sensitive multiagent network (RSMAN), which includes risk-sensitive agents (RSAs) and a risk adaptive portfolio generator (RAPG). Standard DNNs do not understand the risks of their decision, whereas RSA can take risk-sensitive decisions by estimating market uncertainty and parameter uncertainty. Acting as a trader, this agent is trained via reinforcement learning from dynamic trading simulations to estimate the distribution of reward and via unsupervised learning to assess parameter uncertainty without labeled data. We also present an RAPG that can generate a portfolio fitting the user's risk appetite without retraining by exploiting the estimated information from the RSAs. We tested our framework on the U.S. and Korean real financial markets to demonstrate the practicality of the RSMAN.

Revealing the transparency of Deep Neural Networks (DNNs) has been widely studied to describe the decision mechanisms of network inner structures. In this paper, we propose a novel post-hoc framework, Unfold and Conquer Attribution Guidance (UCAG), which enhances the explainability of the network decision by spatially scrutinizing the input features with respect to the model confidence. Addressing the phenomenon of missing detailed descriptions, UCAG sequentially complies with the confidence of slices of the image, leading to providing an abundant and clear interpretation. Therefore, it is possible to enhance the representation ability of explanation by preserving the detailed descriptions of assistant input features, which are commonly overwhelmed by the main meaningful regions. We conduct numerous evaluations to validate the performance in several metrics: i) deletion and insertion, ii) (energy-based) pointing games, and iii) positive and negative density maps. Experimental results, including qualitative comparisons, demonstrate that our method outperforms the existing methods with the nature of clear and detailed explanations and applicability.

Translating imagined speech from human brain activity into voice is a challenging and absorbing research issue that can provide new means of human communication via brain signals. Efforts to reconstruct speech from brain activity have shown their potential using invasive measures of spoken speech data, but have faced challenges in reconstructing imagined speech. In this paper, we propose NeuroTalk, which converts non-invasive brain signals of imagined speech into the user's own voice. Our model was trained with spoken speech EEG which was generalized to adapt to the domain of imagined speech, thus allowing natural correspondence between the imagined speech and the voice as a ground truth. In our framework, an automatic speech recognition decoder contributed to decomposing the phonemes of the generated speech, demonstrating the potential of voice reconstruction from unseen words. Our results imply the potential of speech synthesis from human EEG signals, not only from spoken speech but also from the brain signals of imagined speech.

Pre-trained language models (PLMs) learn gender bias from imbalances in human-written corpora. This bias leads to critical social issues when deploying PLMs in real-world scenarios. However, minimizing bias is limited by the trade-off due to the degradation of language modeling performance. It is particularly challenging to detach and remove biased representations in the embedding space because the learned linguistic knowledge entails bias. To address this problem, we propose a compensatory debiasing strategy to reduce gender bias while preserving linguistic knowledge. This strategy utilizes two types of sentences to distinguish biased knowledge: stereotype and non-stereotype sentences. We assign small angles and distances to pairs of representations of the two gender groups to mitigate bias for the stereotype sentences. At the same time, we maximize the agreement for the representations of the debiasing model and the original model to maintain linguistic knowledge for the non-stereotype sentences. To validate our approach, we measure the performance of the debiased model using the following evaluation metrics: SEAT, StereoSet, CrowS-Pairs, and GLUE. Our experimental results demonstrate that the model fine-tuned by our strategy has the lowest level of bias while retaining knowledge of PLMs.

In deep learning, unsupervised domain adaptation (UDA) is commonly utilized when the availability of abundant labeled data is often limited. Several methods have been proposed for UDA to overcome the difficulty of distinguishing between semantically similar classes, such as person vs. rider and road vs. sidewalk. The confusion of the classes results from the collapse of the distance, caused by the domain shift, between classes in the feature space. In this work, we present a versatile approach based on text-image correlation-guided domain adaptation (TigDA), which maintains a distance to properly adjust the decision boundaries between classes in the feature space. In our approach, the feature information is extracted through text embedding of classes and the aligning capability of the text features with the image features is achieved using the cross-modality. The resultant cross-modal features play an essential role in generating pseudo-labels and calculating an auxiliary pixel-wise cross-entropy loss to assist the image encoder in learning the distribution of cross-modal features. Such a guiding process allows the extension of the distance between similar classes in feature space so that a proper distance for adjusting the decision boundary is maintained. Our TigDA achieved the highest performance among other UDA methods in both single-resolution and multi-resolution cases with the help of GTA5 and SYNTHIA for the source domain and Cityscapes for the target domain. The simplicity and versatility of TigDA will be widely applicable for enhancing the self-training capabilities of most UDA methods.

Brain-computer interface (BCI) enables the communication between humans and devices by reflecting humans' intentions and status. Endogenous BCI is the imagined-based BCI and it has the advantage that the fatigue level of the body, especially the eyes, is relatively low and no additional equipment for offering stimulation is required. When conducting imagined speech, one of the endogenous BCI paradigms, the users imagine the pronunciation as if actually speaking. In contrast, overt speech is that the users directly pronounce the words. We proposed the transfer learning-based method from overt speech- to imagined speech-based electroencephalogram (EEG) signals (TOINet). The proposed method utilizes an encoder to extract the feature vector of imagined speech from EEG signals, which is subsequently reconstructed into overt speech signals using the decoder. Through this process, the model can identify the significant and common features present in EEG signals for both overt and imagined speech, facilitating the classification of EEG signals associated with imagined speech. Eight subjects participated in the experiment. The average accuracy of the TOINet was 0.4841 for classifying four words and the EEG features of overt speech improved the performance by 0.0742. Hence, we demonstrated that EEG features of overt speech could improve the decoding performance of imagined speech.

Brain-to-speech systems, which enable communication through neural activity, have gathered significant attention as potential neuroprosthesis for patients and as novel communication tools for broader individuals. To date, most non-invasive brain-to-speech research has focused on word-level decoding, while sentence-level reconstruction remains challenging. In this study, we introduce a sentence reconstruction method using a restricted range of 16 unique words and compare two different approaches: word-in-sentence reconstruction and natural sentence generation. The focus is on efficiently generating sentences by utilizing the temporal convolutional network model to extract features from EEG signals and create word embeddings that considers the contextual relevance between words. The language model and keyword density measuring are applied to evaluate the sentence reconstruction performance for each approach. The results show that the word-in-sentence approach with language model leads to a significant reduction in the word error rate of$31.58\pm 18.58\ \%$for spoken speech and$56.01\pm 7.57\ \%$for imagined speech. The natural sentence generation approach significantly improved the words per minute performance, enabling more natural mode of brain-to-speech. We conducted an online demo to verify the potential of the proposed approaches, generating audible speech from brain signals in real-time. These findings demonstrate the feasibility of natural brain-to-speech systems by considering the contextual relevance, allowing users to freely communicate natural sentences in real life.

In computer vision, long-tailed multi-label visual recognition is a challenging problem due to the imbalance between classes and the recognition of rare classes. Previous methods for resolving data imbalance mostly originate from single-label classification, which can be obstructed by the label co-occurrence issue, and they attempt to minimize or compensate for it. In this paper, we propose a novel tail class priority sampling method for long-tailed multi-label classification that untangles both issues. Our method samples tail class more often and earlier in order to make the model learn tail classes before it has bias toward common classes. Due to label co-occurrence, other classes will be spontaneously learned in same iteration, ensuring a balanced representation of the head and medium classes. To further enhance the recognition performance, we modify to a bilateral structure that samples both original and proposed sampling distribution to better represent the tail classes. We evaluate our proposed method on two widely used datasets in long-tailed version, COCO-LT and VOC-LT, and compare it with previous methods. The experimental results show that our method achieves a new state-of-the-art performance for tail classes on both datasets. Our method is applicable in various real-world scenarios, making rare class recognition achievable, and can be easily incorporated into conventional recognition frameworks.

Previous video-based human pose estimation methods have shown promising results by leveraging aggregated features of consecutive frames. However, most approaches compromise accuracy to mitigate jitter or do not sufficiently comprehend the temporal aspects of human motion. Furthermore, occlusion increases uncertainty between consecutive frames, which results in unsmooth results. To address these issues, we design an architecture that exploits the keypoint kinematic features with the following components. First, we effectively capture the temporal features by leveraging individual keypoint’s velocity and acceleration. Second, the proposed hierarchical transformer encoder aggregates spatio-temporal dependencies and refines the 2D or 3D input pose estimated from existing estimators. Finally, we provide an online cross-supervision between the refined input pose generated from the encoder and the final pose from our decoder to enable joint optimization. We demonstrate comprehensive results and validate the effectiveness of our model in various tasks: 2D pose estimation, 3D pose estimation, body mesh recovery, and sparsely annotated multi-human pose estimation. Our code is available at https://github.com/KyungMinJin/HANet.

Temporal Action Localization (TAL) is a significant and challenging task that searches for subtle human activities in an untrimmed video. To extract snippet-level video features, existing TAL methods commonly use video encoders pre-trained on short-video classification datasets. However, the snippet-level features can incur ambiguity between consecutive frames due to short and poor temporal information, disrupting the precise prediction of action instances. Several methods incorporating temporal relations have been proposed to mitigate this problem; however, they still suffer from poor video features. To address this issue, we propose a novel temporal action localization framework called an Action-aware Masking Network (AMNet). Our method simultaneously refines video features using action-aware attention and considers inherent temporal relations using self-attention and cross-attention mechanisms. First, we present an Action Masking Encoder (AME) that generates an action-aware mask to represent positive characteristics, which is then used to refine snippet-level features to be more salient around actions. Second, we design a Group Attention Module (GAM), which models relations of temporal information and exchanges mutual information by dividing the features into two groups, i.e., long and short-groups. Extensive experiments and ablation studies on two primary benchmark datasets demonstrate the effectiveness of AM-Net, and our method achieves state-of-the-art performances on THUMOS-14 and ActivityNet1.3.

This study focuses on improving the quality of pseudolabeling in the context of semi-supervised semantic segmentation. Previous studies have adopted confidence thresholding to reduce erroneous predictions in pseudo-labeled data and to enhance their qualities. However, numerous pseudolabels with high confidence scores exist in the early training stages even though their predictions are incorrect, and this ambiguity limits confidence thresholding substantially. In this paper, we present a novel method to resolve the ambiguity of confidence scores with the guidance of network pruning. A recent finding showed that network pruning severely impairs the network generalization ability on samples that are not yet well learned or represented. Inspired by this finding, we refine the confidence scores by reflecting the extent to which the predictions are affected by pruning. Furthermore, we adopted a curriculum learning strategy for the confidence score, which enables the network to learn gradually from easy to hard samples. This approach resolves the ambiguity by suppressing the learning of noisy pseudolabels, the confidence scores of which are difficult to trust owing to insufficient training in the early stages. Extensive experiments on various benchmarks demonstrate the superiority of our framework over state-of-the-art alternatives.

Electroencephalogram (EEG)-based brain-machine interface (BMI) has been utilized to help patients regain motor function and has recently been validated for its use in healthy people because of its ability to directly decipher human intentions. In particular, neurolinguistic research using EEGs has been investigated as an intuitive and naturalistic communication tool between humans and machines. In this study, the human mind directly decoded the neural languages based on speech imagery using the proposed deep neurolinguistic learning. Through real-time experiments, we evaluated whether BMI-based cooperative tasks between multiple users could be accomplished using a variety of neural languages. We successfully demonstrated a BMI system that allows a variety of scenarios, such as essential activity, collaborative play, and emotional interaction. This outcome presents a novel BMI frontier that can interact at the level of human-like intelligence in real time and extends the boundaries of the communication paradigm.

Currently, the expectations for autonomous vehicles (AVs) are increasing. However, it is expected to take at least a decade to develop a fully AV, where human intervention is completely unrequired. By then, human driving is required if necessary. Currently, when the AV hands over control to the driver, a safe driving environment can be created only if it is possible to determine whether the driver is in an abnormal state. Unfortunately, according to the sensory conflict theory, the risk of motion sickness (MS) is higher in AV than in ordinary vehicles. This is because neither passengers nor drivers can predict the movement path of the vehicle under AV, so there is more dissonance between vision and perception. Because the technology to remove MS when it occurs has not yet been developed, the best way to maintain the driver’s good condition is to quickly predict MS through the driver’s bio-signals and establish a system to prevent MS through advanced driver assistance systems. It is necessary to quickly predict early MS and provide feedback before it becomes severe. In this study, we collected dry electroencephalogram (EEG) data to predict MS in a real-world driving environment. For MS-based feature extraction, a normalized sample covariance matrix-based feature representation method was used, and they were classified using convolutional neural networks. As a result, we achieved 89.05% (±5.76) accuracy when averaging all four experimental sessions we conducted. We expect our proposed model to be a useful indicator for resolving MS issues in AV environments.

Detection of the pilots’ mental states is particularly critical because their abnormal mental states (AbSs) could cause catastrophic accidents. In this study, we presented the feasibility of classifying the various specific AbSs (namely, low fatigue, high fatigue, low workload, high workload, low distraction, and high distraction) by applying the deep learning method. To the best of our knowledge, this study is the first attempt to classify multiple AbSs of pilots. We proposed the hybrid deep neural networks with five convolutional blocks and two long short-term memory layers for decoding multiple AbSs. We designed the model to extract the informative features from electroencephalography signals. A total of ten pilots conducted the experiment in a simulated flight environment. Compared with five conventional models, our proposed model achieved the highest grand-average accuracy of 68.04$(\pm$5.26)% which is at least 6.55% higher than other conventional models for classifying seven mental states across all subjects. Our proposed model could distinguish and classify low and high levels for each status category and give appropriate feedback to the subjects. In addition, we found nine indicators that showed the statistically significant differences between two mental states (p$<$0.05). Hence, we believe that it will contribute significantly to autonomous driving or autopilot advances based on artificial intelligence technology in the future.

Synthetic Aperture Radar (SAR) images inevitably contain speckle noise. Despeckling SAR images are typically represented as linear forms due to the consistency of denoising network training/inference settings on a linear scale. However, this leads to controversial problems when the linear SAR images are seen with the naked eye: i) restriction of representation for dark areas and ii) excessive expression for bright areas due to the high-intensity range. To overcome these problems, we propose a denoising framework that simultaneously eliminates speckle and thermal noise in the decibel (dB) domain through novel noise modeling. Our noise modeling allows the network to learn in the dB domain of the desired dynamic range, enabling stable end-to-end learning without separate spatial transformations. Our modeling is the first attempt to consider thermal noise. Experimental results show that our method is superior in quantitative and visual performance compared to the existing methods.

Recent studies for similarity-based self-supervised representation learning tend to consider only fixed temporal coverage from a given video. However, this approach limits that a model learns temporally persistent representations since it cannot reflect spatial and temporal information gaps from resolution variations. To overcome the limitation, this paper proposes a Temporal Adaptive Teacher-Student (TATS) framework that encourages the trained model to be robust on spatio-temporal variations. Our key approach is optimizing similarity-based learning that utilizes several views with dynamic temporal resolutions. From a given video, TATS captures spatio-temporal invariant clues for temporally persistent representation with cross-resolution correspondence between local and global views. Extensive experiments show that our TATS achieves competitive downstream (action recognition and video retrieval) performances on benchmarks (UCF101 and HMDB51).

Although recent advances in text-to-speech (TTS) have shown significant improvement, it is still limited to emotional speech synthesis. To produce emotional speech, most works utilize emotion information extracted from emotion labels or reference audio. However, they result in monotonous emotional expression due to the utterance-level emotion conditions. In this paper, we propose EmoQ-TTS, which synthesizes expressive emotional speech by conditioning phoneme-wise emotion information with fine-grained emotion intensity. Here, the intensity of emotion information is rendered by distance-based intensity quantization without human labeling. We can also control the emotional expression of synthesized speech by conditioning intensity labels manually. The experimental results demonstrate the superiority of EmoQ-TTS in emotional expressiveness and controllability.

Although recent advances in neural vocoder have shown significant improvement, most of these models have a trade-off between audio quality and computational complexity. Since the large model has a limitation on the low-resource devices, a more efficient neural vocoder should synthesize high-quality audio for practical applicability. In this paper, we present Fre-GAN 2, a fast and efficient high-quality audio synthesis model. For fast synthesis, Fre-GAN 2 only synthesizes low and high-frequency parts of the audio, and we leverage the inverse discrete wavelet transform to reproduce the target-resolution audio in the generator. Additionally, we also introduce adversarial periodic feature distillation, which makes the model synthesize high-quality audio with only a small parameter. The experimental results show the superiority of Fre-GAN 2 in audio quality. Furthermore, Fre-GAN 2 has a 10.91× generation acceleration, and the parameters are compressed by 21.23× than Fre-GAN.

Adaptive text-to-speech (TTS) has attracted increasing interests for the purpose of training TTS systems without tons of high quality data. Nevertheless, existing adaptive TTS systems still show low adaptation quality for novel speakers, since it is hard to learn an extensive speaking style with limited data. To address this issue, we propose progressive variational autoencoder (PVAE) which generates data with adapting to style gradually. PVAE learns a progressively style-normalized representation, which is a key component of progressive style adaptation. We extend PVAE to PVAE-TTS, a multi-speaker adaptive TTS model which generates natural speech with high adaptation quality for novel speakers. To further improve the adaptation quality, we also propose dynamic style layer normalization (DSLN) which utilizes a convolution operation. The experimental results demonstrate the superiority of PVAE-TTS in terms of both subjective and objective evaluations.

Voice conversion converts the voice while maintaining the language information. It uses two samples to synthesize speech: the source sample is used for content, the target sample is used for style representation. Therefore, VC has been progressed to design information flow to disentangle content and style in a speech. However, separated representations are damaged while passing sparse subspace. Besides, VC models suffer from the training-inference mismatch problem: they only use one sample in training. Accordingly, the model extracts inappropriate content and style representation and generates low-quality speech during inference. To address the mismatch scenario problem, we propose a StyleVC, which utilizes adversarial style generalization. First, we propose style generalization, which captures global style representation and restricts the model from copying information. Second, we use a pitch predictor to estimate pitch information according to content and style representation. Third, we further use adversarial training to make the model generate more realistic speech. Finally, we demonstrate our proposed model can generate high-quality speech. The experimental results also show that the proposed StyleVC significantly outperforms to extract the desired features and improve audio quality during inference.

Most object detection frameworks use backbone architectures originally designed for image classification, conventionally with pre-trained parameters on ImageNet. However, image classification and object detection are essentially different tasks and there is no guarantee that the optimal backbone for classification is also optimal for object detection. Recent neural architecture search (NAS) research has demonstrated that automatically designing a backbone specifically for object detection helps improve the overall accuracy. In this paper, we introduce a neural architecture adaptation method that can optimize the given backbone for detection purposes, while still allowing the use of pre-trained parameters. We propose to adapt both the micro- and macro-architecture by searching for specific operations and the number of layers, in addition to the output channel dimensions of each block. It is important to find the optimal channel depth, as it greatly affects the feature representation capability and computation cost. We conduct experiments with our searched backbone for object detection and demonstrate that our backbone outperforms both manually designed and searched state-of-the-art backbones on the COCO dataset.

Human pose estimation is an interesting and underlying topic in various fields such as action recognition and human-computer interaction. Although many methods have been developed recently, they are still far from perfect in accuracy and speed at a time. In this paper, we propose a Classification-based Pose Estimation Network with Multi-task Learning (CPENML) based on the low-resolution feature map to improve accuracy and inference time simultaneously. The proposed CPENML consists of two ideas. Firstly, novel proposed keypoint and offset estimation tasks based on classification achieve better performance than regression. Secondly, the proposed Multi-Scale Network (MSN) makes robust feature maps and balances the keypoint and offset tasks to maximize performance. To prove the effectiveness of the proposed method, we conduct ablation studies on the COCO dataset for proposed ideas. Compared to benchmarks, we demonstrate the superiority of our proposed method on COCO dataset in terms of inference time and accuracy.

Few-shot object detection has gained significant attention in recent years as it has the potential to greatly reduce the reliance on large amounts of manually annotated bounding boxes. While most existing few-shot object detection literature primarily focuses on bounding box classification by obtaining as discriminative feature embeddings as possible, we emphasize the necessity of handling the lack of intersection-over-union (IoU) variations induced by a biased distribution of novel samples. In this paper, we analyze the IoU imbalance that is caused by the relatively high number of low-quality region proposals, and reveal that it plays a critical role in improving few-shot learning capabilities. The well-known two stage fine-tuning technique causes insufficient quality and quantity of the novel positive samples, which hinders the effective object detection of unseen novel classes. To alleviate this issue, we present a few-shot object detection model with proposal balance refinement, a simple yet effective approach in learning object proposals using an auxiliary sequential bounding box refinement process. This process enables the detector to be optimized on the various IoU scores through additional novel class samples. To fully exploit our sequential stage architecture, we revise the fine-tuning strategy and expose the Region Proposal Network to the novel classes in order to provide increased learning opportunities for the region-of-interest (RoI) classifiers and regressors. Our extensive assessments on PASCAL VOC and COCO demonstrate that our framework substantially outperforms other existing few-shot object detection approaches.

Recently, advanced technologies have unlimited potential in solving various problems with a large amount of data. However, these technologies have yet to show competitive performance in brain-computer interfaces (BCIs) which deal with brain signals. Basically, brain signals are difficult to collect in large quantities, in particular, the amount of information would be sparse in spontaneous BCIs. In addition, we conjecture that high spatial and temporal similarities between tasks increase the prediction difficulty. We define this problem as sparse condition. To solve this, a factorization approach is introduced to allow the model to obtain distinct representations from latent space. To this end, we propose two feature extractors: A class-common module is trained through adversarial learning acting as a generator; Class-specific module utilizes loss function generated from classification so that features are extracted with traditional methods. To minimize the latent space shared by the class-common and class-specific features, the model is trained under orthogonal constraint. As a result, EEG signals are factorized into two separate latent spaces. Evaluations were conducted on a single-arm motor imagery dataset. From the results, we demonstrated that factorizing the EEG signal allows the model to extract rich and decisive features under sparse condition.

Video Anomaly Detection(VAD) has been traditionally tackled in two main methodologies: the reconstruction-based approach and the prediction-based one. As the reconstruction-based methods learn to generalize the input image, the model merely learns an identity function and strongly causes the problem called generalizing issue. On the other hand, since the prediction-based ones learn to predict a future frame given several previous frames, they are less sensitive to the generalizing issue. However, it is still uncertain if the model can learn the spatio-temporal context of a video. Our intuition is that the understanding of the spatio-temporal context of a video plays a vital role in VAD as it provides precise information on how the appearance of an event in a video clip changes. Hence, to fully exploit the context information for anomaly detection in video circumstances, we designed the transformer model with three different contextual prediction streams: masked, whole and partial. By learning to predict the missing frames of consecutive normal frames, our model can effectively learn various normality patterns in the video, which leads to a high reconstruction error at the abnormal cases that are unsuitable to the learned context. To verify the effectiveness of our approach, we assess our model on the public benchmark datasets: USCD Pedestrian 2, CUHK Avenue and ShanghaiTech and evaluate the performance with the anomaly score metric of reconstruction error. The results demonstrate that our proposed approach achieves a competitive performance compared to the existing video anomaly detection methods.

Traditionally convolutional neural network architectures have been designed by stacking layers on top of each other to form deeper hierarchical networks. The cortex in the brain however does not just stack layers as done in standard convolution neural networks, instead different regions are organized next to each other in a large single sheet of neurons. Biological neurons self organize to form topographic maps, where neurons encoding similar stimuli group together to form logical clusters. Here we propose new self-organization principles that allow for the formation of hierarchical cortical regions (i.e. layers) in a completely unsupervised manner without requiring any predefined architecture. Synaptic connections are dynamically grown and pruned, which allows us to actively constrain the number of incoming and outgoing connections. This way we can minimize the wiring cost by taking into account both the synaptic strength and the connection length. The proposed method uses purely local learning rules in the form of spike-timing-dependent plasticity (STDP) with lateral excitation and inhibition. We show experimentally that these self-organization rules are sufficient for topographic maps and hierarchical layers to emerge. Our proposed Self-Organizing Neural Sheet (SONS) model can thus form traditional neural network layers in a completely unsupervised manner from just a single large pool of unstructured spiking neurons.

This paper presents HierSpeech, a high-quality end-to-end text-to-speech (TTS) system based on a hierarchical conditional variational autoencoder (VAE) utilizing self-supervised speech representations. Recently, single-stage TTS systems, which directly generate raw speech waveform from text, have been getting interest thanks to their ability in generating high-quality audio within a fully end-to-end training pipeline. However, there is still a room for improvement in the conventional TTS systems. Since it is challenging to infer both the linguistic and acoustic attributes from the text directly, missing the details of attributes, specifically linguistic information, is inevitable, which results in mispronunciation and over-smoothing problem in their synthetic speech. To address the aforementioned problem, we leverage self-supervised speech representations as additional linguistic representations to bridge an information gap between text and speech. Then, the hierarchical conditional VAE is adopted to connect these representations and to learn each attribute hierarchically by improving the linguistic capability in latent representations. Compared with the state-of-the-art TTS system, HierSpeech achieves +0.303 comparative mean opinion score, and reduces the phoneme error rate of synthesized speech from 9.16% to 5.78% on the VCTK dataset. Furthermore, we extend our model to HierSpeech-U, an untranscribed text-to-speech system. Specifically, HierSpeech-U can adapt to a novel speaker by utilizing self-supervised speech representations without text transcripts. The experimental results reveal that our method outperforms publicly available TTS models, and show the effectiveness of speaker adaptation with untranscribed speech.

Many people experience motion sickness. In order to analyze a driver’s motion sickness state and prevent accidents, a method of estimating the degree of motion sickness based on bio-signals is emerging. The brain-computer interface (BCI) systems using electroencephalogram (EEG) are used as the most direct method of estimating motion sickness conditions. However, EEG-based systems suffer from variability between subjects and over time, so a calibration process is required for every use. To address this problem, we mitigate the need for calibration through cross-subject transfer learning between the target data and the multi-subjects source data. All experiments were conducted in a domain adaptation setting. Meanwhile, we assume that there is an ordinal relationship between motion sickness scores. Thus, we performed an ordinal classification task so that the feature vectors were mapped by reflecting the ordinal characteristics according to the motion sickness state. In this paper, we propose a motion sickness classification BCI framework in combination with ordinal classification, resting-state prototype-based ordinal distance learning, and a subject-specific embedding module. Taking into account constraints of ordinal rank, the feature extractor is trained with prototype-based ordinal distance learning to measure the relative distance between the resting-state and motion sickness state. We further utilize an embedding module that encodes subject-specific information combined with task discriminative features to be effective for domain adaptation tasks. The proposed framework achieved the highest performance (accuracy 60.21 %) through comparative experiments with other models.

Motion sickness is an unpleasant physiological response to situations involving the perception of motion. Research on motion sickness focuses on its manifestation by analyzing biosignals to observe physiological changes coinciding with the perception of motion sickness. Meanwhile, multimodal data fusion has gained attention for its ability to reflect the multimodality of real-life tasks and enhance the robustness of machine learning models. In this study, we aimed to find a deep learning-based multimodal framework for integrative analysis of multiple biosignals with the highest performance in classifying the level of carsickness. To do so, we first generated a dataset consisting of five different types of biosignals collected under real driving conditions: electroencephalogram (EEG), electrocardiogram (ECG), respiration (RESP), photoplethysmogram (PPG), and galvanic skin response (GSR). Then, we compared six deep learning-based unimodal classification models which have shown competency in signal classification. Lastly, we compared four different fusion methods for multimodal classification frameworks using either all five biosignals or three signals, which include RESP, ECG, and PPG. As a result, we found out that the fusion method combining self-attention and the tensor fusion network outperformed other unimodal and multimodal models with categorical accuracy of 76.26 % regardless of the number of biosignals used.

Incomplete reconstruction of abnormal samples using convolutional autoencoders trained only on normal samples has been the key principle of anomaly detection. Such detection mechanisms utilize reconstruction error differences between normal and abnormal frames. This is not consistent, however, causing the normal and abnormal samples undistin-guishable. To handle this problem, we propose a shuffle-and-sort strategy for learning the temporal context of normality. The purpose of the strategy is to reconstruct shuffled input frames into an output with the correct order using a self-attention mechanism. Consequently, the proposed method can model the temporal context of normal events, which prevents the successful completion of reconstructing anomalies by the convolutional layers. We demonstrated the detection efficiency of the proposed method using public benchmark datasets: UCSD Pedestrian 2, CUHK Avenue, and ShanghaiTech Campus Datasets.

Although many approaches for multi-human pose estimation in videos have shown profound results, they require densely annotated data which entails excessive man labor. Furthermore, there exists occlusion and motion blur that inevitably lead to poor estimation performance. To address these problems, we propose a method that leverages an attention mask for occluded joints and encodes temporal dependency between frames using transformers. First, our framework composes different combinations of sparsely annotated frames that denote the track of the overall joint movement. We propose an occlusion attention mask from these combinations that enable encoding occlusion-aware heatmaps as a semi-supervised task. Second, the proposed temporal encoder employs transformer architecture to effectively aggregate the temporal relationship and keypoint-wise attention from each time step and accurately refines the target frame’s final pose estimation. We achieve state-of-the-art pose estimation results for PoseTrack2017 and PoseTrack2018 datasets and demonstrate the robustness of our approach to occlusion and motion blur in sparsely annotated video data.

Temporal action localization (TAL) is a task of identifying a set of actions in a video, which involves localizing the start and end frames and classifying each action instance. Existing methods have addressed this task by using predefined anchor windows or heuristic bottom-up boundary-matching strategies, which are major bottlenecks in inference time. Additionally, the main challenge is the inability to capture long-range actions due to a lack of global contextual information. In this paper, we present a novel anchor-free framework, referred to as HTNet, which predicts a set of $\langle$start time, end time, class$\rangle$ triplets from a video based on a Transformer architecture. After the prediction of coarse boundaries, we refine it through a background feature sampling (BFS) module and hierarchical Transformers, which enables our model to aggregate global contextual information and effectively exploit the inherent semantic relationships in a video. We demonstrate how our method localizes accurate action instances and achieves state-of-the-art performance on two TAL benchmark datasets: THUMOS14 and ActivityNet 1.3.

Drowsy driving causes severe road traffic accidents and significantly threatens road driving. Recently, electroencephalogram (EEG)-based drowsiness state classification has gained attention in the field of brain-computer interface (BCI). Because of the inter-and intra-subject variability of EEG signals, EEG-based drowsiness state classification is still challenging in developing an estimator applicable to unseen subjects. Generally, calibration sessions are required to tune the model with subject-specific data. In this paper, we propose an EEG-based driver drowsiness state (i.e., alert and drowsy) classification framework that improves the generalization performance to unseen subjects. Style features of multi-domain instances are mixed to generate unseen domains, and the distance of labels within classes is minimized to learn robust representations. Experiments were conducted on EEG data acquired from a drowsy driving experiment in a simulated-driving environment. Our proposed framework achieved an accuracy of 77.26%, an F1-score of 0.6266, and a recall of 0.6813 across eleven subjects in leave-one-subject-out cross-validation. The experimental results showed an improvement in the generalization performance for novel target subjects in driver drowsiness state classification and demonstrated the potential for calibration-free BCI.

Brain-computer interface (BCI) is a technology that controls computers by reflecting users’ intentions. Especially the electroencephalogram (EEG)-based BCI systems have been developed because of their potential utility. In BCI studies, controlling the drone swarm is one of the important issues since it improves work efficiency and safety. Also, current research has investigated how the drone swarms are controlled by imagining their formations using visual imagery (VI)-based EEG signals. The raw EEG signals and the spectrogram are widely used as input representations for decoding EEG signals. However, the decoding performance of the VI-based EEG signals is low to control the drone swarm due to noise in the raw EEG signals and information loss problems that may arise in the spectrogram. In this paper, we develop the EEGgram generator that extracts spectrogram-like features from the raw EEG signals minimizing information loss problems. Also, we propose the EEGgramNet, which could extract the significant information from VI-based EEG signals using both the spectrogram and the EEGgram as inputs. The proposed method outperforms an accuracy of 0.643, which is 8.4 % higher than that of the best conventional method. Hence, we demonstrate the possibility of constructing a VI-based BCI system to control the drone swarm by imagining its formations.

Although various face-related tasks have significantly advanced in recent years, occlusion and extreme pose still impede the achievement of higher performance. Existing face rotation or de-occlusion methods only have emphasized the aspect of each problem. In addition, the lack of high-quality paired data remains an obstacle for both methods. In this work, we present a self-supervision strategy called Swap-R&R to overcome the lack of ground-truth in a fully unsupervised manner for joint face rotation and de-occlusion. To generate an input pair for self-supervision, we transfer the occlusion from a face in an image to an estimated 3D face and create a damaged face image, as if rotated from a different pose by rotating twice with the roughly de-occluded face. Furthermore, we propose Complete Face Recovery GAN (CFR-GAN) to restore the collapsed textures and disappeared occlusion areas by leveraging the structural and textural differences between two rendered images. Unlike previous works, which have selected occlusion-free images to obtain ground-truths, our approach does not require human intervention and paired data. We show that our proposed method can generate a de-occluded frontal face image from an occluded profile face image. Moreover, extensive experiments demonstrate that our approach can boost the performance of facial recognition and facial expression recognition. The code is publicly available1

Several voice conversion (VC) methods using a simple autoencoder with a carefully designed information bottleneck have recently been studied. In general, they extract content information from a given speech through the information bottleneck between the encoder and the decoder, providing it to the decoder along with the target speaker information to generate the converted speech. However, their performance is highly dependent on the downsampling factor of an information bottleneck. In addition, such frame-by-frame conversion methods cannot convert speaking styles associated with the length of utterance, such as the duration. In this paper, we propose a novel duration controllable voice conversion (DCVC) model, which can transfer the speaking style and control the speed of the converted speech through a phoneme-based information bottleneck. The proposed information bottleneck does not need to find an appropriate downsampling factor, achieving a better audio quality and VC performance. In our experiments, DCVC outperformed the baseline models with a 3.78 MOS and a 3.83 similarity score. It can also smoothly control the speech duration while achieving a 39.35x speedup compared with a Seq2seq-based VC in terms of the inference speed.

Brain-computer interfaces (BCIs) have been widely employed to identify and estimate a user's intention to trigger a robotic device by decoding motor imagery (MI) from an electroencephalogram (EEG). However, developing a BCI system driven by MI related to natural hand-grasp tasks is challenging due to its high complexity. Although numerous BCI studies have successfully decoded large body parts, such as the movement intention of both hands, arms, or legs, research on MI decoding of high-level behaviors such as hand grasping is essential to further expand the versatility of MI-based BCIs. In this study, we propose NeuroGrasp, a dual-stage deep learning framework that decodes multiple hand grasping from EEG signals under the MI paradigm. The proposed method effectively uses an EEG and electromyography (EMG)-based learning, such that EEG-based inference at test phase becomes possible. The EMG guidance during model training allows BCIs to predict hand grasp types from EEG signals accurately. Consequently, NeuroGrasp improved classification performance offline, and demonstrated a stable classification performance online. Across 12 subjects, we obtained an average offline classification accuracy of 0.68 (±0.09) in four-grasp-type classifications and 0.86 (±0.04) in two-grasp category classifications. In addition, we obtained an average online classification accuracy of 0.65 (±0.09) and 0.79 (±0.09) across six high-performance subjects. Because the proposed method has demonstrated a stable classification performance when evaluated either online or offline, in the future, we expect that the proposed method could contribute to different BCI applications, including robotic hands or neuroprosthetics for handling everyday objects.

With the development of sensing technologies and machine learning, techniques that can identify emotions and inner states of a human through physiological signals, known as electroencephalography (EEG), have been actively developed and applied to various domains, such as automobiles, robotics, healthcare, and customer-support services. Thus, the demand for acquiring and analyzing EEG signals in real-time is increasing. In this paper, we aimed to acquire a new EEG dataset based on the discrete emotion theory, termed as WeDea (Wireless-based eeg Data for emotion analysis), and propose a new combination for WeDea analysis. For the collected WeDea dataset, we used video clips as emotional stimulants that were selected by 15 volunteers. Consequently, WeDea is a multi-way dataset measured while 30 subjects are watching the selected 79 video clips under five different emotional states using a convenient portable headset device. Furthermore, we designed a framework for recognizing human emotional state using this new database. The practical results for different types of emotions have proven that WeDea is a promising resource for emotion analysis and can be applied to the field of neuroscience.

Convolutional neural networks (CNNs) have recently been applied to electroencephalogram (EEG)-based brain-computer interfaces (BCIs). EEG is a noninvasive neuroimaging technique, which can be used to decode user intentions. Because the feature space of EEG data is highly dimensional and signal patterns are specific to the subject, appropriate methods for feature representation are required to enhance the decoding accuracy of the CNN model. Furthermore, neural changes exhibit high variability between sessions, subjects within a single session, and trials within a single subject, resulting in major issues during the modeling stage. In addition, there are many subject-dependent factors, such as frequency ranges, time intervals, and spatial locations at which the signal occurs, which prevent the derivation of a robust model that can achieve the parameterization of these factors for a wide range of subjects. However, previous studies did not attempt to preserve the multivariate structure and dependencies of the feature space. In this study, we propose a method to generate a spatiospectral feature representation that can preserve the multivariate information of EEG data. Specifically, 3-D feature maps were constructed by combining subject-optimized and subject-independent spectral filters and by stacking the filtered data into tensors. In addition, a layer-wise decomposition model was implemented using our 3-D-CNN framework to secure reliable classification results on a single-trial basis. The average accuracies of the proposed model were 87.15% (±7.31), 75.85% (±12.80), and 70.37% (±17.09) for the BCI competition data sets IV_2a, IV_2b, and OpenBMI data, respectively. These results are better than those obtained by state-of-the-art techniques, and the decomposition model obtained the relevance scores for neurophysiologically plausible electrode channels and frequency domains, confirming the validity of the proposed approach.

While generative adversarial networks (GANs) based neural text-to-speech (TTS) systems have shown significant improvement in neural speech synthesis, there is no TTS system to learn to synthesize speech from text sequences with only adversarial feedback. Because adversarial feedback alone is not sufficient to train the generator, current models still require the reconstruction loss compared with the ground-truth and the generated mel-spectrogram directly. In this paper, we present Multi-SpectroGAN (MSG), which can train the multi-speaker model with only the adversarial feedback by conditioning a self-supervised hidden representation of the generator to a conditional discriminator. This leads to better guidance for generator training. Moreover, we also propose adversarial style combination (ASC) for better generalization in the unseen speaking style and transcript, which can learn latent representations of the combined style embedding from multiple mel-spectrograms. Trained with ASC and feature matching, the MSG synthesizes a high-diversity mel-spectrogram by controlling and mixing the individual speaking styles (e.g., duration, pitch, and energy). The result shows that the MSG synthesizes a high-fidelity mel-spectrogram, which has almost the same naturalness MOS score as the ground-truth mel-spectrogram.

The clear transparency of Deep Neural Networks (DNNs) is hampered by complex internal structures and nonlinear transformations along deep hierarchies. In this paper, we propose a new attribution method, Relative Sectional Propagation (RSP), for fully decomposing the output predictions with the characteristics of class-discriminative attributions and clear objectness. We carefully revisit some shortcomings of backpropagation-based attribution methods, which are trade-off relations in decomposing DNNs. We define hostile factor as an element that interferes with finding the attributions of the target and propagate it in a distinguishable way to overcome the non-suppressed nature of activated neurons. As a result, it is possible to assign the bi-polar relevance scores of the target (positive) and hostile (negative) attributions while maintaining each attribution aligned with the importance. We also present the purging techniques to prevent the decrement of the gap between the relevance scores of the target and hostile attributions during backward propagation by eliminating the conflicting units to channel attribution map. Therefore, our method makes it possible to decompose the predictions of DNNs with clearer class-discriminativeness and detailed elucidations of activation neurons compared to the conventional attribution methods. In a verified experimental environment, we report the results of the assessments: (i) Pointing Game, (ii) mIoU, and (iii) Model Sensitivity with PASCAL VOC 2007, MS COCO 2014, and ImageNet datasets. The results demonstrate that our method outperforms existing backward decomposition methods, including distinctive and intuitive visualizations.

Despite the recent success of 3D human reconstruction methods, recovering the accurate and smooth 3D human motion from video is still challenging. Designing a temporal model in the encoding stage is not sufficient enough to settle the trade-off problem between the per-frame accuracy and the motion smoothness. To address this problem, we approach some of the fundamental problems of 3D reconstruction tasks, simultaneously predicting 3D pose and 3D motion dynamics. First, we utilize the power of uncertainty to address the problem of multiple 3D configurations resulting in the same 2D projections. Second, we confirmed that dividing the body into local regions shows outstanding results for estimating 3D motion dynamics. In this paper, we propose (i) an encoder that makes two different estimations: a static feature that presents 2D pose feature as distribution and a dynamic feature that includes optical flow information and (ii) a decoder that divides the body into five different local regions to estimate the 3D motion dynamics of each region. We demonstrate how our method recovers the accurate and smooth motion and achieves the state-of-the-art results for both constrained and in-the-wild videos.

Text-to-speech (TTS) synthesis is the process of producing synthesized speech from text or phoneme input. Traditional TTS models contain multiple processing steps and require external aligners, which provide attention alignments of phoneme-to-frame sequences. As the complexity increases and efficiency decreases with every additional step, there is expanding demand in modern synthesis pipelines for end-to-end TTS with efficient internal aligners. In this work, we propose an end-to-end text-to-waveform network with a novel reinforcement learning based duration search method. Our proposed generator is feed-forward and the aligner trains the agent to make optimal duration predictions by receiving active feedback from actions taken to maximize cumulative reward. We demonstrate accurate alignments of phoneme-to-frame sequence generated from trained agents enhance fidelity and naturalness of synthesized audio. Experimental results also show the superiority of our proposed model compared to other state-of-the-art TTS models with internal and external aligners.

Although recent works on neural vocoder have improved the quality of synthesized audio, there still exists a gap between generated and ground-truth audio in frequency space. This difference leads to spectral artifacts such as hissing noise or reverberation, and thus degrades the sample quality. In this paper, we propose Fre-GAN which achieves frequency-consistent audio synthesis with highly improved generation quality. Specifically, we first present resolution-connected generator and resolution-wise discriminators, which help learn various scales of spectral distributions over multiple frequency bands. Additionally, to reproduce high-frequency components accurately, we leverage discrete wavelet transform in the discriminators. From our experiments, Fre-GAN achieves high-fidelity waveform generation with a gap of only 0.03 MOS compared to ground-truth audio while outperforming standard models in quality.

Although recent advances in voice conversion have shown significant improvement, there still remains a gap between the converted voice and target voice. A key factor that maintains this gap is the insufficient decomposition of content and voice style from the source speech. This insufficiency leads to the converted speech containing source speech style or losing source speech content. In this paper, we present VoiceMixer which can effectively decompose and transfer voice style through a novel information bottleneck and adversarial feedback. With self-supervised representation learning, the proposed information bottleneck can decompose the content and style with only a small loss of content information. Also, for adversarial feedback of each information, the discriminator is decomposed into content and style discriminator with self-supervision, which enable our model to achieve better generalization to the voice style of the converted speech. The experimental results show the superiority of our model in disentanglement and transfer performance, and improve audio quality by preserving content information.

The 3D Morphable Model (3DMM), which is a Principal Component Analysis (PCA) based statistical model that represents a 3D face using linear basis functions, has shown promising results for reconstructing 3D faces from single-view in-the-wild images. However, 3DMM has restricted representation power due to the limited number of 3D scans and global linear basis. To address the limitations of 3DMM, we propose a straightforward learning-based method that reconstructs a 3D face mesh through Free-Form Deformation (FFD) for the first time. FFD is a geometric modeling method that embeds a reference mesh within a parallelepiped grid and deforms the mesh by moving the sparse control points of the grid. As FFD is based on mathematically defined basis functions, it has no limitation in representation power. Thus, we can recover accurate 3D face meshes by estimating the appropriate deviation of control points as deformation parameters. Although both 3DMM and FFD are parametric models, deformation parameters of FFD are easier to interpret in terms of their effect on the final shape. This practical advantage of FFD allows the resulting mesh and control points to serve as a good starting point for 3D face modeling, in that ordinary users can fine-tune the mesh by using widely available 3D software tools. Experiments on multiple datasets demonstrate how our method successfully estimates the 3D face geometry and facial expressions from 2D face images, achieving comparable performance to the state-of-the-art methods.

Few-shot speaker adaptation is a specific Text-to-Speech (TTS) system that aims to reproduce a novel speaker’s voice with a few training data. While numerous attempts have been made to the few-shot speaker adaptation system, there is still a gap in terms of speaker similarity to the target speaker depending on the amount of data. To bridge the gap, we propose GC-TTS which achieves high-quality speaker adaptation with significantly improved speaker similarity. Specifically, we leverage two geometric constraints to learn discriminative speaker representations. Here, a TTS model is pre-trained for base speakers with a sufficient amount of data, and then fine-tuned for novel speakers on a few minutes of data with two geometric constraints. Two geometric constraints enable the model to extract discriminative speaker embeddings from limited data, which leads to the synthesis of intelligible speech. We discuss and verify the effectiveness of GC-TTS by comparing it with popular and essential methods. The experimental results demonstrate that GC-TTS generates high-quality speech from only a few minutes of training data, outperforming standard techniques in terms of speaker similarity to the target speaker.

Brain-computer interface (BCI) is used for communication between humans and devices by recognizing humans’ status and intention. Communication between humans and a drone using electroencephalogram (EEG) signals is one of the most challenging issues in the BCI domain. In particular, the control of drone swarms (the direction and formation) has more advantages compared to the control of a drone. The visual imagery (VI) paradigm is that subjects visually imagine specific objects or scenes. Reduction of the variability among subjects’ EEG signals is essential for practical BCI-based systems. In this study, we proposed the subepoch-wise feature encoder (SEFE) to improve the performances in the subject-independent tasks by using the VI dataset. This study is the first attempt to demonstrate the possibility of generalization among subjects in the VI-based BCI. We used the leave-one-subject-out cross-validation for evaluating the performances. We obtained higher performances when including our proposed module than excluding our proposed module. The DeepConvNet with SEFE showed the highest performance of 0.72 among six different decoding models. Hence, we demonstrated the feasibility of decoding the VI dataset in the subject-independent task with robust performances by using our proposed module.

Computer-aided diagnosis has recently received attention for its advantage of low cost and time efficiency. Although deep learning played a major role in the recent success of acne detection, there are still several challenges such as color shift by inconsistent illumination, variation in scales, and high density distribution. To address these problems, we propose an acne detection network which consists of three components, specifically: Composite Feature Refinement, Dynamic Context Enhancement, and Mask-Aware Multi-Attention. First, Composite Feature Refinement integrates semantic information and fine details to enrich feature representation, which mitigates the adverse impact of imbalanced illumination. Then, Dynamic Context Enhancement controls different receptive fields of multi-scale features for context enhancement to handle scale variation. Finally, Mask-Aware Multi-Attention detects densely arranged and small acne by suppressing uninformative regions and highlighting probable acne regions. Experiments are performed on acne image dataset ACNE04 and natural image dataset PASCAL VOC 2007. We demonstrate how our method achieves the state-of-the-art result on ACNE04 and competitive performance with previous state-of-the-art methods on the PASCAL VOC 2007.

Automatic pancreas segmentation is crucial to the diagnostic assessment of diabetes or pancreatic cancer. However, the relatively small size of the pancreas in the upper body, as well as large variations of its location and shape in retroperitoneum, make the segmentation task challenging. To alleviate these challenges, in this article, we propose a cascaded multitask 3-D fully convolution network (FCN) to automatically segment the pancreas. Our cascaded network is composed of two parts. The first part focuses on fast locating the region of the pancreas, and the second part uses a multitask FCN with dense connections to refine the segmentation map for fine voxel-wise segmentation. In particular, our multitask FCN with dense connections is implemented to simultaneously complete tasks of the voxel-wise segmentation and skeleton extraction from the pancreas. These two tasks are complementary, that is, the extracted skeleton provides rich information about the shape and size of the pancreas in retroperitoneum, which can boost the segmentation of pancreas. The multitask FCN is also designed to share the low- and mid-level features across the tasks. A feature consistency module is further introduced to enhance the connection and fusion of different levels of feature maps. Evaluations on two pancreas datasets demonstrate the robustness of our proposed method in correctly segmenting the pancreas in various settings. Our experimental results outperform both baseline and state-of-the-art methods. Moreover, the ablation study shows that our proposed parts/modules are critical for effective multitask learning.

An electroencephalogram (EEG) is the most extensively used physiological signal in emotion recognition using biometric data. However, these EEG data are difficult to analyze, because of their anomalous characteristic where statistical elements vary according to time as well as spatial-temporal correlations. Therefore, new methods that can clearly distinguish emotional states in EEG data are required. In this paper, we propose a new emotion recognition method, named AsEmo. The proposed method extracts effective features boosting classification performance on various emotional states from multi-class EEG data. AsEmo Automatically determines the number of spatial filters needed to extract significant features using the explained variance ratio (EVR) and employs a Subject-independent method for real-time processing of Emotion EEG data. The advantages of this method are as follows: (a) it automatically determines the spatial filter coefficients distinguishing emotional states and extracts the best features; (b) it is very robust for real-time analysis of new data using a subject-independent technique that considers subject sets, and not a specific subject; (c) it can be easily applied to both binary-class and multi-class data. Experimental results on real-world EEG emotion recognition tasks demonstrate that AsEmo outperforms other state-of-the-art methods with a 2-8% improvement in terms of classification accuracy.

As Deep Neural Networks (DNNs) have demonstrated superhuman performance in a variety of fields, there is an increasing interest in understanding the complex internal mechanisms of DNNs. In this paper, we propose Relative Attributing Propagation (RAP), which decomposes the output predictions of DNNs with a new perspective of separating the relevant (positive) and irrelevant (negative) attributions according to the relative influence between the layers. The relevance of each neuron is identified with respect to its degree of contribution, separated into positive and negative, while preserving the conservation rule. Considering the relevance assigned to neurons in terms of relative priority, RAP allows each neuron to be assigned with a bi-polar importance score concerning the output: from highly relevant to highly irrelevant. Therefore, our method makes it possible to interpret DNNs with much clearer and attentive visualizations of the separated attributions than the conventional explaining methods. To verify that the attributions propagated by RAP correctly account for each meaning, we utilize the evaluation metrics: (i) Outside-inside relevance ratio, (ii) Segmentation mIOU and (iii) Region perturbation. In all experiments and metrics, we present a sizable gap in comparison to the existing literature.

3D Morphable Model (3DMM) is a statistical model of facial shape and texture using a set of linear basis functions. Most of the recent 3D face reconstruction methods aim to embed the 3D morphable basis functions into Deep Convolutional Neural Network (DCNN). However, balancing the requirements of strong regularization for global shape and weak regularization for high level details is still ill-posed. To address this problem, we properly control generality and specificity in terms of regularization by harnessing the power of uncertainty. Additionally, we focus on the concept of nonlinearity and find out that Graph Convolutional Neural Network (Graph CNN) and Generative Adversarial Network (GAN) are effective in reconstructing high quality 3D shapes and textures respectively. In this paper, we propose to employ (i) an uncertainty-aware encoder that presents face features as distributions and (ii) a fully nonlinear decoder model combining Graph CNN with GAN. We demonstrate how our method builds excellent high quality results and outperforms previous state-of-the-art methods on 3D face reconstruction tasks for both constrained and in-the-wild images.

A brain-computer interface (BCI) is used to control external devices for healthy people as well as to rehabilitate motor functions for motor-disabled patients. Decoding movement intention is one of the most significant aspects for performing arm movement tasks using brain signals. Decoding movement execution (ME) from electroencephalogram (EEG) signals have shown high performance in previous works, however movement imagination (MI) paradigm-based intention decoding has so far failed to achieve sufficient accuracy. In this study, we focused on a robust MI decoding method with transfer learning for the ME and MI paradigm. We acquired EEG data related to arm reaching for 3D directions. We proposed a BCI-transfer learning method based on a Relation network (BTRN) architecture. Decoding performances showed the highest performance compared to conventional works. We confirmed the possibility of the BTRN architecture to contribute to continuous decoding of MI using ME datasets.

A brain-computer interface (BCI) provides a direct communication pathway between user and external devices. EEG-based motor imagery paradigm is widely used in non-invasive BCI to obtain encoded signals contained user intention of movement execution. However, EEG has intricate and non-stationary properties resulting in insufficient decoding performance. By imagining numerous movements of a single-arm, decoding performance can be improved without artificial command matching. In this study, we collected intuitive EEG data contained the nine different types of movements of a single-arm from 9 subjects. We propose an end-to-end role assigned convolutional neural network (ERA-CNN) which considers discriminative features of each upper limb region by adopting the principle of a hierarchical CNN architecture. The proposed model outperforms previous methods on 3-class, 5-class and two different types of 7-class classification tasks. Hence, we demonstrate the possibility of decoding user intention by using only EEG signals with robust performance using the ERA-CNN.

Remote sensing image retrieval (RSIR) is the process of ranking database images depending on the degree of similarity compared to the query image. As the complexity of RSIR increases due to the diversity in shooting range, angle, and location of remote sensors, there is an increasing demand for methods to address these issues and improve retrieval performance. In this work, we introduce a novel method for retrieving aerial images by merging group convolution with attention mechanism and metric learning, resulting in robustness to rotational variations. For refinement and emphasis on important features, we applied channel attention in each group convolution stage. By utilizing the characteristics of group convolution and channel-wise attention, it is possible to acknowledge the equality among rotated but identically located images. The training procedure has two main steps: (i) training the network with Aerial Image Dataset (AID) for classification, (ii) fine-tuning the network with triplet-loss for retrieval with Google Earth South Korea and NWPU-RESISC45 datasets. Results show that the proposed method performance exceeds other state-of-the-art retrieval methods in both rotated and original environments. Furthermore, we utilize class activation maps (CAM) to visualize the distinct difference of main features between our method and baseline, resulting in better adaptability in rotated environments.

In recent works, a flow-based neural vocoder has shown significant improvement in real-time speech generation task. The sequence of invertible flow operations allows the model to convert samples from simple distribution to audio samples. However, training a continuous density model on discrete audio data can degrade model performance due to the topological difference between latent and actual distribution. To resolve this problem, we propose audio dequantization methods in flow-based neural vocoder for high fidelity audio generation. Data dequantization is a well-known method in image generation but has not yet been studied in the audio domain. For this reason, we implement various audio dequantization methods in flow-based neural vocoder and investigate the effect on the generated audio. We conduct various objective performance assessments and subjective evaluation to show that audio dequantization can improve audio generation quality. From our experiments, using audio dequantization produces waveform audio with better harmonic structure and fewer digital artifacts.

The ability to perceive and recognize objects is fundamental for the interaction with the external environment. Studies that investigate them and their relationship with brain activity changes have been increasing due to the possible application in an intuitive brain-machine interface (BMI). In addition, the distinctive patterns when presenting different visual stimuli that make data differentiable enough to be classified have been studied. However, reported classification accuracy still low or employed techniques for obtaining brain signals are impractical to use in real environments. In this study, we aim to decode electroencephalography (EEG) signals depending on the provided visual stimulus. Subjects were presented with 72 photographs belonging to 6 different semantic categories. We classified 6 categories and 72 exemplars according to visual stimuli using EEG signals. In order to achieve a high classification accuracy, we proposed an attention driven convolutional neural network and compared our results with conventional methods used for classifying EEG signals. We reported an accuracy of 50.37 ± 6.56% and 26.75 ± 10.38% for 6-class and 72-class, respectively. These results statistically outperformed other conventional methods. This was possible because of the application of the attention network using human visual pathways. Our findings showed that EEG signals are possible to differentiate when subjects are presented with visual stimulus of different semantic categories and at an exemplar-level with a high classification accuracy; this demonstrates its viability to be applied it in a real-world BMI.

Conventional methods for object detection usually require substantial amounts of training data and annotated bounding boxes. If there are only a few training data and annotations, the object detectors easily overfit and fail to generalize. It exposes the practical weakness of the object detectors. On the other hand, human can easily master new reasoning rules with only a few demonstrations using previously learned knowledge. In this paper, we introduce a few-shot object detection via knowledge transfer, which aims to detect objects from a few training examples. Central to our method is prototypical knowledge transfer with an attached meta-learner. The meta-learner takes support set images that include the few examples of the novel categories and base categories, and predicts prototypes that represent each category as a vector. Then, the prototypes reweight each RoI (Region-of-Interest) feature vector from a query image to remodels R-CNN predictor heads. To facilitate the remodeling process, we predict the prototypes under a graph structure, which propagates information of the correlated base categories to the novel categories with explicit guidance of prior knowledge that represents correlations among categories. Extensive experiments on the PASCAL VOC dataset verifies the effectiveness of the proposed method.

In this study, we adopted visual motion imagery, which is a more intuitive brain-computer interface (BCI) paradigm, for decoding the intuitive user intention. We developed a 3-dimensional BCI training platform and applied it to assist the user in performing more intuitive imagination in the visual motion imagery experiment. The experimental tasks were selected based on the movements that we commonly used in daily life, such as picking up a phone, opening a door, eating food, and pouring water. Nine subjects participated in our experiment. We presented statistical evidence that visual motion imagery has a high correlation from the prefrontal and occipital lobes. In addition, we selected the most appropriate electroencephalography channels using a functional connectivity approach for visual motion imagery decoding and proposed a convolutional neural network architecture for classification. As a result, the averaged classification performance of the proposed architecture for 4 classes from 16 channels was 67.50 (±1.52)% across all subjects. This result is encouraging, and it shows the possibility of developing a BCI-based device control system for practical applications such as neuroprosthesis and a robotic arm.

Motor imagery (MI) classification is important as the emerging research interest of brain computer interface (BCI) due to its potential about real-world application. Advancing manipulation and control technology of external devices such as robotics, the need of MI for complex and human-like movements is growing. The two most important procedures that influence the performance of MI-BCI are feature extraction and classification. Although there have been recent studies on feature extraction for complex, there is no consensus on the classifier suitable for complex MI. This study aimed to identify the best classifier for complex MI decoding.Electroencephalography (EEG) recordings measured during complex MI, which are hand grasping, spreading, pronation and supination, were used for binary (grasp vs. twist) and quaternary classification. Time domain parameter, which have shown suitability for complex movement decoding in previous works, was used as the EEG feature. Four types of ten machine learning classifiers, which have been applied to MI-BCI, were compared.Shrinkage regularized linear discriminant analysis (SRLDA) exhibited the best classification accuracy in both binary (92.8%) and quaternary (55.2%). In the case of training and testing time, a small amount of time for real-time analysis were needed, except random forest and logistic regression.This study showed that SRLDA is an appropriate classifier for complex MI classification, due to its ability to handle stationary and high dimensionality feature, TDP. The findings suggest that complex MI-BCI could gain more benefit from applying linear and shrinkage regularized model (i.e., SRLDA).

Imagined speech is spotlighted as a new trend in the brain-machine interface due to its application as an intuitive communication tool. However, previous studies have shown low classification performance, therefore its use in real-life is not feasible. In addition, no suitable method to analyze it has been found. Recently, deep learning algorithms have been applied to this paradigm. However, due to the small amount of data, the increase in classification performance is limited. To tackle these issues, in this study, we proposed an end-to-end framework using Siamese neural network encoder, which learns the discriminant features by considering the distance between classes. The imagined words (e.g., arriba (up), abajo (down), derecha (right), izquierda (left), adelante (forward), and atrás (backward)) were classified using the raw electroencephalography (EEG) signals. We obtained a 6-class classification accuracy of 31.40 ± 2.73% for imagined speech, which significantly outperformed other methods. This was possible because the Siamese neural network, which increases the distance between dissimilar samples while decreasing the distance between similar samples, was used. In this regard, our method can learn discriminant features from a small dataset. The proposed framework would help to increase the classification performance of imagined speech for a small amount of data and implement an intuitive communication system.

The assessment of consciousness and unconsciousness is a challenging issue in modern neuroscience. Consciousness is closely related to memory consolidation in that memory is a critical component of conscious experience. So far, many studies have been reported on memory consolidation during consciousness, but there is little research on memory consolidation during unconsciousness. Therefore, we aim to assess the unconsciousness in terms of memory consolidation using electroencephalogram signals. In particular, we used unconscious state during the nap; because sleep is the only state in which consciousness disappears under normal physiological conditions. Seven participants performed two memory tasks (word-pairs and visuo-spatial) before and after the nap to assess the memory consolidation during unconsciousness. As a result, spindle power in central, parietal, occipital regions during unconsciousness was positively correlated with the difference in location memory performance. With the difference in memory performance, there was also a negative correlation between delta connectivity and word-pairs memory, alpha connectivity and location memory, and spindle connectivity and word-pairs memory. Additionally, brain activity and connectivity for differences according to nap and unconsciousness during memory recall were explored. These findings could help present new insights into the assessment of unconsciousness by exploring the relationship with memory consolidation.

Ear-electroencephalography (EEG) is a promising tool for practical brain-computer interface (BCI) applications because it is more unobtrusive, comfortable, and mobile than a typical scalp-EEG system. However, an ear-EEG has a natural constraint of electrode location (e.g., limited in or around the ear) for acquiring informative brain signals sufficiently. Achieving reliable performance of ear-EEG in specific BCI paradigms that do not utilize brain signals on the temporal lobe around the ear is difficult. For example, steady-state visual evoked potentials (SSVEPs), which are mainly generated in the occipital area, have a significantly attenuated and distorted amplitude in ear-EEG. Therefore, preserving the high level of decoding accuracy is challenging and essential for SSVEP BCI based on ear-EEG. In this paper, we first investigate linear and nonlinear regression methods to increase the decoding accuracy of ear-EEG regarding SSVEP paradigm by utilizing the estimated target EEG signals on the occipital area. Then, we investigate an ensemble method to consider the prediction variability of the regression methods. Finally, we propose an error correction regression (ECR) framework to reduce the prediction errors by adding an additional nonlinear regression process (i.e., kernel ridge regression). We evaluate the ECR framework in terms of single session, session-to-session transfer, and subject-transfer decoding. We also validate the online decoding ability of the proposed framework with a short-time window size. The average accuracies are observed to be 91.11±9.14%, 90.52±8.67%, 86.96±12.13%, and 78.79±12.59%. This paper demonstrates that SSVEP BCI based on ear-EEG can achieve reliable performance with the proposed ECR framework.

The continuous orientation estimation of a moving pedestrian is a crucial issue in autonomous driving that requires the detection of a pedestrian intending to cross a road. It is still a challenging task owing to several reasons, including the diversity of pedestrian appearances, the subtle pose difference between adjacent orientations, and similar poses with different orientations such as axisymmetric orientations. These problems render the task highly difficult. Recent studies involving convolutional neural networks (CNNs) have attempted to solve these problems. However, their performance is still far from satisfactory for application in intelligent vehicles. In this paper, we propose a CNN-based two-stream network for continuous orientation estimation. The network can learn representations based on the spatial co-occurrence of visual patterns among pedestrians. To boost estimation performance, we applied a coarse-to-fine learning approach that consists of two learning stages. We investigated continuous orientation performance on the TUD Multiview Pedestrian dataset and the KITTI dataset and compared them with the state-of-the-art methods. The results show that our method outperforms other existing methods.

In this article, we consider the problem of few-shot learning for classification. We assume a network trained for base categories with a large number of training examples, and we aim to add novel categories to it that have only a few, e.g., one or five, training examples. This is a challenging scenario because: 1) high performance is required in both the base and novel categories; and 2) training the network for the new categories with a few training examples can contaminate the feature space trained well for the base categories. To address these challenges, we propose two geometric constraints to fine-tune the network with a few training examples. The first constraint enables features of the novel categories to cluster near the category weights, and the second maintains the weights of the novel categories far from the weights of the base categories. By applying the proposed constraints, we extract discriminative features for the novel categories while preserving the feature space learned for the base categories. Using public data sets for few-shot learning that are subsets of ImageNet, we demonstrate that the proposed method outperforms prevalent methods by a large margin.

For a brain-computer interface (BCI) system, a calibration procedure is required for each individual user before he/she can use the BCI. This procedure requires approximately 20-30 min to collect enough data to build a reliable decoder. It is, therefore, an interesting topic to build a calibration-free, or subject-independent, BCI. In this article, we construct a large motor imagery (MI)-based electroencephalography (EEG) database and propose a subject-independent framework based on deep convolutional neural networks (CNNs). The database is composed of 54 subjects performing the left- and right-hand MI on two different days, resulting in 21 600 trials for the MI task. In our framework, we formulated the discriminative feature representation as a combination of the spectral-spatial input embedding the diversity of the EEG signals, as well as a feature representation learned from the CNN through a fusion technique that integrates a variety of discriminative brain signal patterns. To generate spectral-spatial inputs, we first consider the discriminative frequency bands in an information-theoretic observation model that measures the power of the features in two classes. From discriminative frequency bands, spectral-spatial inputs that include the unique characteristics of brain signal patterns are generated and then transformed into a covariance matrix as the input to the CNN. In the process of feature representations, spectral-spatial inputs are individually trained through the CNN and then combined by a concatenation fusion technique. In this article, we demonstrate that the classification accuracy of our subject-independent (or calibration-free) model outperforms that of subject-dependent models using various methods [common spatial pattern (CSP), common spatiospectral pattern (CSSP), filter bank CSP (FBCSP), and Bayesian spatio-spectral filter optimization (BSSFO)].

As the population becomes older worldwide, accurate computer-aided diagnosis for Alzheimer's disease (AD) in the early stage has been regarded as a crucial step for neurodegeneration care in recent years. Since it extracts the low-level features from the neuroimaging data, previous methods regarded this computer-aided diagnosis as a classification problem that ignored latent featurewise relation. However, it is known that multiple brain regions in the human brain are anatomically and functionally interlinked according to the current neuroscience perspective. Thus, it is reasonable to assume that the extracted features from different brain regions are related to each other to some extent. Also, the complementary information between different neuroimaging modalities could benefit multimodal fusion. To this end, we consider leveraging the coupled interactions in the feature level and modality level for diagnosis in this paper. First, we propose capturing the feature-level coupled interaction using a coupled feature representation. Then, to model the modality-level coupled interaction, we present two novel methods: 1) the coupled boosting (CB) that models the correlation of pairwise coupled-diversity on both inconsistently and incorrectly classified samples between different modalities and 2) the coupled metric ensemble (CME) that learns an informative feature projection from different modalities by integrating the intrarelation and interrelation of training samples. We systematically evaluated our methods with the AD neuroimaging initiative data set. By comparison with the baseline learning-based methods and the state-of-the-art methods that are specially developed for AD/MCI (mild cognitive impairment) diagnosis, our methods achieved the best performance with accuracy of 95.0% and 80.7% (CB), 94.9% and 79.9% (CME) for AD/NC (normal control), and MCI/NC identification, respectively.

Electroencephalography (EEG) is the primary modality for estimating the user intention in brain-computer interface (BCI). However, the suppression of the inter-subject variability (ISV) remains as a major challenge in constructing a reliable EEG-based BCI model. Subject-specific classification models have been widely used to avoid ISV, however these inherently involve time-consuming individual calibration process. This study speculated that the calibration could be minimized via clustering BCI subjects into subgroups by their respective similarity in brain power distribution at the resting state and conducted a proof-of-concept investigation. EEG recordings of twenty-nine healthy subjects from open motor imagery (MI) dataset were used in this study. K-means clustering based on brain activation in α, low β and high β-band at resting state divided the subjects into three subgroups. The efficacy of band-clustering was evaluated by comparing its MI classification performance (leftor right-hand gripping) to subject-specific and general models. Among the subjects in a cluster, ISV was lower than that in twenty-nine subjects, especially in the α-band. The MI classification accuracy using the cluster-specific model on the α-band was marked high performance (median accuracy 68.8%). The cluster-specific model had significantly high accuracy compared to general model (median accuracy = 64.6%). Furthermore, the difference of MI classification accuracy between the cluster-specific model on the α band and subject-specific model is not significant (median accuracy = 69.3%). Consequently, establishing a model by grouping clusters using similar brain activation patterns was highly beneficial for the MI classification without individual calibration process.

In this paper, we propose a matching network for gradually estimating the geometric transformation parameters between two aerial images taken in the same area but in different environments. To precisely matching two aerial images, there are important factors to consider such as different time, a variation of viewpoint, size, and rotation. The conventional methods for matching aerial image pairs with the large variations are extremely time-consuming process and have the limitations finding correct correspondences, because the image gradient and grayscale intensity for generating the feature descriptors are not robust to the variations. We design the network architecture as an end-to-end trainable deep neural network to reflect the characteristics of aerial images. The hierarchical structures that orderly estimate the rotation and the affine transformations make it possible to reduce the range of predictions and minimize errors caused by misalignment, resulting in more precise matching performance. Furthermore, we apply transfer learning to make the feature extraction networks more robust and suitable for the aerial image domain with the large variations. For the experiment, we apply the remote sensing image datasets from Google Earth and International Society for Photogrammetry and Remote Sensing (ISPRS). To evaluate our method quantitatively, we measure the probability of correct keypoints (PCK) metrics for objectively comparing the degree of matching. In terms of qualitative and quantitative assessment, our method demonstrates the state-of-the-art performances compared to the existing methods.

The understanding of human consciousness based on brain connectivity is considered important for brain- machine interfacing. In this study, we investigated changes in causal connectivity in electroencephalography data related to conscious and unconscious experiences during non-rapid eye movement sleep after parietal transcranial magnetic stimulation (TMS). A serial awakening paradigm was used to determine whether subjects had had a conscious experience or not. We calculated direct transfer function (DTF) as a measure of effective connectivity in five frequency bands focusing on frontal and parietal-occipital regions. The DTF showed significant differences in frontal-to-parietal flow between reported unconsciousness and consciousness. During the first 100 ms after TMS, the outward links of the parietal region at low frequencies were higher in no conscious experience than in conscious experience. During the next 100 ms, however, the outward links of the frontal region were higher in the conscious experience than the no conscious experience at low frequencies. Changes with causal connectivity over time after TMS indicate that the spatial roles in brain regions associated with consciousness are different. These findings may help clarify the cortical mechanisms related to conscious experience.

Near-infrared spectroscopy (NIRS) is receiving much attention in the fields of brain-computer interface (BCI) due to its noninvasiveness, usability, and high performance. However, NIRS is susceptible to the motion artifacts, and its high morphological variety hinders the existing signals artifacts elimination techniques from being applied to NIRS. This study proposes a novel feature extraction method for classifying motion artifacts. NIRS data from an open access dataset containing five types of motion artifact (i.e., eye blinking, head movement, eyes movement, teeth clenching, and mouth opening) from 28 healthy subjects were analyzed. The efficacy of the proposed wavelet statistical feature extraction method in artifact classification was compared to various existing feature extraction methods for BCI designs. Each feature was learned by four conventional machine learning models for classifying NIRS motion artifacts. Shrinkage regularized linear discriminant analysis (SRLDA) with the proposed features derived from oxy and deoxy-hemoglobin NIRS signal achieved 90% accuracy and 0.85 Cohen's kappa coefficient for classifying types of motion artifacts. Mann-Whitney U test and paired T-test indicate that SRLDA with the proposed feature had significantly higher classification performance compared to the other models. The proposed method can reliably classify the motion artifacts to standardize the morphology types of NIRS artifact. It could be used to augment signal quality control technique with further extension in future NIRS-based BCI systems.

Communication using brain-computer interface (BCI) has developed in attempts toward an intuitive system by decoding the imagined speech or visual imagery. However, discrimination between the two paradigms may be ambiguous because the user intention contains their original meaning. A clear distinction between the two paradigms may facilitate the active use of them leading to an intuitive BCI conversation system. In this study, we compared imagined speech and visual imagery in the perspective of its presence, spatial features, and classification performance based on electroencephalography. Seven subjects performed both imagined speech and visual imagery of twelve words/phrases. We showed the presence of the two paradigms, having distinct brain region from each other. The maximum thirteen-class classification accuracy including rest class was 34.2 % for imagined speech and 26.7 % for visual imagery. Therefore, we investigated the possibility of multiclass classification of more than ten classes in both paradigms, showing the potential of them to be used in the real world communication system. These findings could further be utilized in the intuitive communication for locked-in patients sending commands to the external world simply by thinking of `the very thing' that the user wants to deliver.

With deep learning emerging as a powerful machine learning tool to build Brain Computer Interface (BCI) systems, researchers are investigating the use of different type of networks architectures and representations of brain activity to attain superior classification accuracy compared to state-of-the-art machine learning approaches, that rely on processed signal and optimally extracted features. This paper presents a deep learning driven electroencephalography (EEG) -BCI system to perform decoding of hand motor imagery using deep convolution neural network architecture, with spectrally localized time-domain representation of multi-channel EEG as input. A significant increase in decoding performance in terms of accuracy of +6.47% is obtained compared to a wideband EEG representation. We further illustrate the movement class specific feature patterns for both the architectures and demonstrate that higher difference between classes is observed using the proposed architecture. We conclude that the network trained by taking into account the dynamic spatial interactions in distinct frequency bands of EEG, can offer better decoding performance and aid in better interpretation of learned features.

Brain-machine interface (BMI) provides a new control strategy for both patients and healthy people. An endogenous paradigm such as motor imagery (MI) for BMI is commonly used for detecting user intention without external stimuli. However, manipulating the dexterous robotic arm by using limited MI commands is challenging issues. In this paper, we designed a shared robotic arm control system using the intuitive MI and vision guidance. To accomplish the user's intention on the robotic arm, we used arm reach MI (left, right, and forward), hand grasp MI, and wrist twist MI by using electroencephalogram (EEG) signals. The Kinect sensor is used to match the decoded user intention with the detected object based on the location of the workspace. In addition, to decode intuitive MI successfully, we propose a novel convolutional neural network (CNN) based user intention decoding model. Ten subjects participated in our experiments, and five of them were selected to perform online tasks. The proposed method could decode various user intention (five intuitive MI classes and resting state) with a grand-averaged classification accuracy of 55.91% in offline analysis. For sufficient control on the online shared robotic arm control, the proposed online system was only started, once the patient shows higher performance than 60% in the offline analysis. For the online drinking tasks, we confirmed the averaged 78% success rate. Hence, we confirmed the possibility of the shared robotic arm control based on intuitive BMI and vision guidance with high performance.

Many real-world applications of reinforcement learning require an agent to select optimal actions from continuous spaces. Recently, deep neural networks have successfully been applied to games with discrete actions spaces. However, deep neural networks for discrete actions are not suitable for devising strategies for games where a very small change in an action can dramatically affect the outcome. In this paper, we present a new self-play reinforcement learning framework which equips a continuous search algorithm which enables to search in continuous action spaces with a kernel regression method. Without any hand-crafted features, our network is trained by supervised learning followed by self-play reinforcement learning with a high-fidelity simulator for the Olympic sport of curling. The program trained under our framework outperforms existing programs equipped with several hand-crafted features and won an international digital curling competition.

Most artificial intelligence (AI) based learning systems act in virtual or laboratory environments. Here we demonstrate an AI-based curling robot system named `Curly' that competes on a real-world curling ice sheet. Curly encompasses (1) an AI-based curling strategy and simulation engine under consideration of the high `icy' uncertainty, (2) the thrower robot enabled by autonomous driving with traction control, and (3) the skip robot that allows to recognize the curling field and stone configuration based on vision technology. The Curly performed well both: in classical game situations and when interacting with human opponents, namely, the top-ranked Korean amateur high school curling team.

The development of brain-computer interface (BCI) systems that are based on electroencephalography (EEG), and driven by spontaneous movement intentions, is useful for rehabilitation and external device control. In this study, we analyzed the decoding of five different hand executions and imageries from EEG signals, for a robot hand control. Five healthy subjects participated in this experiment. They executed and imagined five sustained hand motions. In this motor execution (ME) and motor imagery (MI) experiment, we proposed a subject-specific time interval selection method, and we used common spatial patterns (CSP) and the regularized linear discriminant analysis (RLDA) for the data analysis. As a result, we classified the five different hand motions offline and obtained average classification accuracies of 56.83% for ME, and 51.01% for MI, respectively. Both results were higher than the obtained accuracies from a comparison method that used a standard fixed time interval method. This result is encouraging, and the proposed method could potentially be used in future applications, such as a BCI-driven robot hand control.

This paper presents the feasibility of an electroencephalography (EEG)-based robot arm control system using a decoding of multi-directional arm reaching movement imagery. To do that, we have designed and implemented an experimental environment that can acquire non-invasive brain signals about multi-directional arm reaching movement. Five subjects participated in our experiments and the subjects performed four directional reaching tasks (Left, right, forward, and backward) with actual movement and movement imagery. The filter-bank common spatial pattern (FBCSP) was applied to extract spatio-frequency features from the acquired EEG signals. The regularized linear discriminant analysis (RLDA) was also applied as a classifier. As a result, the averaged classification accuracies of the actual movement and movement imagery were represented 67.04% and 59.19%, respectively. These results showed a feasibility of the EEG-based robot arm control system based on multi-directional arm reaching movement imagery.

This paper proposes a method for line segment detection in digital images. We propose a novel linelet-based representation to model intrinsic properties of line segments in rasterized image space. Based on this, line segment detection, validation, and aggregation frameworks are constructed. For a numerical evaluation on real images, we propose a new benchmark dataset of real images with annotated lines called YorkUrban-LineSegment. The results show that the proposed method outperforms state-of-the-art methods numerically and visually. To our best knowledge, this is the first report of numerical evaluation of line segment detection on real images.

Motor imagery (Ml)-based brain-computer interface (BCI) allows users to control external devices using the brain signal patterns induced by the imagination of movements. Since these patterns have high variability between subjects and sessions, the BCI system necessarily requires 20-30 minutes for the calibration process each time the system is used. This time-consuming process requires a high level of the user's concentration; most users experience uncomfortable feelings such as tiredness, exhaustion, and loss of attention, which are symptoms of mental fatigue. In this paper, we introduce a self-paced training that terminates the calibration process within a few minutes. In this training paradigm, users perform MI tasks continuously without an inter-stimulus-interval (ISI). Also, we propose a data selection method to extract the most prominent features from the short calibration data by assuming the data distribution probabilistically and using the prior knowledge of event-related desynchronization (ERD) patterns. The results from 19 subjects indicate that the proposed method gained a comparable classification performance to the conventional method but with a much shorter calibration period (12 min/73.8%, 30 min/76.1%, respectively). In this regard, the proposed method could be of great benefit for real-world BCI applications by providing a quicker calibration process.

As the brain is a unique biological system that reflects the subtle distinctions in the mental attributes of individual humans, electroencephalographic (EEG) signals have been regarded as one of the most promising and potent biometric signals for discriminating between individuals. However, existing EEG-based user-recognition methods present only a limited range of individual distinctions. In this paper, we propose a novel system of decoding cognitive EEG signals for individual identification with high accuracy. Specifically, we investigate the feasibility of our system, which can recognize an individual based on the discriminative patterns of source-level causal connectivity among brain regions, estimated from scalp-level EEG signals. The EEG signals were produced by a steady-state visual evoked potential-inducing grid-shaped top-down paradigm. This system can, in principle, use top-down cognitive features analyzed by individuals' differently characterized neurodynamic causal connectivities. In this paper, we achieved a maximal accuracy of 98.60% on average in 20 subjects, for whom we estimated causal connectivity in 16 brain regions using 5-s intervals of EEG signals. Our system shows promising initial results toward building a practical identification technology able to recognize individuals by means of brain neurodynamics.

In this paper, a robust lane detection method based on vanishing point estimation is proposed. Estimating a vanishing point can be helpful in detecting lanes, because parallel lines converge on the vanishing point in a projected 2-D image. However, it is not easy to estimate the vanishing point correctly in an image with a complex background. Thus, a robust vanishing point estimation method is proposed that uses a probabilistic voting procedure based on intersection points of line segments extracted from an input image. The proposed voting function is defined with line segment strength that represents relevance of the extracted line segments. Next, candidate line segments for lanes are selected by considering geometric constraints. Finally, the host lane is detected by using the proposed score function, which is designed to remove outliers in the candidate line segments. Also, the detected host lane is refined by using inter-frame similarity that considers location consistency of the detected host lane and the estimated vanishing point in consecutive frames. Furthermore, in order to reduce computational costs in the vanishing point estimation process, a method using a lookup table is proposed. Experimental results show that the proposed method efficiently estimates the vanishing point and detects lanes in various environments.

In this paper, we address the problem of recognizing unfinished human activity from partially observed videos. Specifically, we propose a novel human activity descriptor, which can represent pairwise relationships among human activities in a compact manner using pre-trained Convolutional Neural Networks (CNNs) by capturing the discriminative sub-volume. The potentially important relationship among all pairwise sub-volumes, called key-volumes, is automatically captured using global and local motion activation and the ratio of the participant. The captured key-volumes without prior knowledge hold discriminative information related to the unfinished activity. The key-volume information is considered in the descriptor construction procedure. Training a CNN model for a particular purpose requires a lot of resources, such as large amount of labeled data and computing power, despite its representational power. Thus, we develop a method to utilize pre-trained CNN without any additional model training procedure. The low-level features can be extracted through existing CNN toolkits. For a real application, the proposed method may be more cost-effective while implementing a smart surveillance system to understand human activity. In our experiments, we compare the performances of the proposed method with other state-of-the-art human activity prediction methods for two public datasets; the results of the experiments show that the proposed method outperforms these competing methods.

Decoding of various motor intentions for generating command is one of the important factors in brain-based wheelchair system. The goal of this study focuses on classifying four types of trunk-related motor execution and imagery intentions. By brain components which are related to the trunk-related movements (waist, shoulder, and trunk) are generated in the very small and very close brain areas; therefore, decoding of trunk-related motor intentions are not easy for providing reliable system commands. To the best of our knowledge, the problems mentioned above have not been explored in the literature. In this study, we first validated the decoding accuracy of trunk-related motor intention based movement-related cortical potential. A set of binary classification performance which are shoulder extension (SE), waist rotation (WR), trunk flexion (TF), and rest (RE) have validated in respect to execution movement as well as imagery movement across six subjects. All binary classification results showed performance that is higher than the chance level. The best decoding accuracy shows 68.5% in the motor imagery task of shoulder extension vs. waist rotation.

Nowadays, the electroencephalography (EEG)-based wheelchair driving system, one of the major applications of brain-computer interface (BCI), that allows an individual with mobility impairments to perform daily living activities independently. In this context, user's intention identifying methods were developed by several research groups using various paradigms for the wheelchair driving. In this study, we use a steady-state somatosensory evoked potential (SSSEP) paradigm, which elicits brain responses to vibro-tactile stimulation of specific frequencies, for a user's intention identification to driving a wheelchair. The main focus of this study is to validate an effectiveness of our SSSEP-based wheelchair driving system via an online experiment with more challenging tasks than our recent study. In our system, a subject concentrated on one of vibro-tactile stimuli (attached on left-hand, right-hand, and foot) selectively for driving wheelchair (corresponding to turn-left, turn-right, and move-forward). Five healthy subjects participated in the online experiment, and the experimental results show that our SSSEP paradigm is suitable to EEG-based intelligent wheelchair driving system.

Recently, steady-state visual evoked potential (SSVEP)-based brain-machine interface (BMI) speller systems have shown a great performance increase with high information transfer rate (ITR) and short response time. In previous BMI speller systems, however, users should utilize the systems at fixed viewing distance environment for evoking SSVEP signals because a variation of the SSVEP signals according to changes of viewing distance was not considered during system design process. For a real-world application of BMI speller, reliable speller systems which are robust to various viewing distance environment are needed. In this study, hence, we investigate the effects of viewing distance on SSVEP by changing distance between a user and visual stimuli. Here, we used four visual stimuli which have different frequencies using LED monitor. In the subsequent analysis, we present classification results with several methods. Our analysis and results show a possibility that SSVEP under various viewing distance environment could be facilitated.

Recently, there has been an increased demand for Brain-Machine Interface (BMI) toolboxes for neuroscientifc research. In many BMI applications, speller systems can provide an efficient communication channel for users with disabilities. Here, we introduce an open-source BMI toolbox termed `OpenBMI', which supports the various signal processing chains for common BMI paradigms, such as event-related potentials (ERPs) and steady-state visual evoked potentials (SSVEP). The OpenBMI framework consists of ready-to-use experimental paradigms, offline data analysis techniques, online feedback as well as evaluation modules. The data analysis modules provide essential pre-processing steps (segmentation, baseline correction, etc.) as well as signal processing algorithms such as temporal and spatial filtering, artifact rejection, among others. The experimental paradigms of ERP and SSVEP are available with fully open-sourced demo scripts. Users can easily modify or extend the demo scripts for their needs. In this article, the OpenBMI framework, its features as well as its future development plan is introduced.

Lately, electroencephalography (EEG)-based auth- entication has received considerable attention from the scientific community. However, the limited usability of wet EEG electrodes as well as low accuracy for large numbers of users have so far prevented this new technology to become commonplace. In this study a novel EEG-based authentication system is presented, which is based on the rapid serial visual presentation paradigm and uses a knowledge-based approach for authentication. Twenty-nine subjects’ data were recorded and analyzed with wet EEG electrodes as well as dry ones. A true acceptance rate of 100% can be reached for all subjects with an average required login time of 13.5 s for wet and 27 s for dry electrodes. Average false acceptance rates for the dry electrode setup were estimated to be$3.33 \times 10^{-5}$.

Diversified approaches for development of braking assistant system have been employed to assure the safety of driver and pedestrian. Recently, neurophysiological studies related to driver's mental state during driving under specific conditions have been a growing interest for development of driving assistant system based on brain-computer interface (BCI). In this article, the feasibility of online braking assistant system which could detect driver's braking intention based on BCI is investigated. The results of the online experiment verified that driver's braking intention could be robustly detected based on neurophysiological characteristics proposed by previous study. The performance of the online experiment was evaluated based on reaction times for emergency situations and detection accuracy.

For the last decade, a brain-computer interface (BCI) has gained great interests in the fields and successfully applied to various applications. In this work, we focus on a steady-state somatosensory evoked potential (SSSEP) based brain-controlled wheelchair that allows people with mobility impairment to improve activities of daily living. In our system, a user concentrated on three vibration stimuli (attached on left hand, right-hand, and foot) selectively for control wheelchair. The three stimuli were associated with three commands of wheelchair: turn-left, turn-right, and move-forward. Four healthy subjects participated in wheelchair control experiments to validate performance. As a results, all subjects controlled wheelchair successfully from the start to goal line without any collision.

In this paper, we address the problem of recognizing group activities of human objects based on their motion trajectory analysis. In order to resolve the complexity and ambiguity problems caused by a large number of human objects, we propose a Group Interaction Zone (GIZ) to detect meaningful groups in a scene to effectively handle noisy information. Two novel features, Group Interaction Energy (GIE) feature and Attraction and Repulsion Features, are proposed to better describe group activities within a GIZ. We demonstrate the performance of our method in two ways by (i) comparing the performance of the proposed method with the previous methods and (ii) analyzing the influence of the proposed features and GIZ-based meaningful group detection on group activity recognition using public datasets.

In this paper, we propose a novel method for unusual human activity detection in crowded scenes. Specifically, rather than detecting or segmenting humans, we devised an efficient method, called a motion influence map, for representing human activities. The key feature of the proposed motion influence map is that it effectively reflects the motion characteristics of the movement speed, movement direction, and size of the objects or subjects and their interactions within a frame sequence. Using the proposed motion influence map, we further developed a general framework in which we can detect both global and local unusual activities. Furthermore, thanks to the representational power of the proposed motion influence map, we can localize unusual activities in a simple manner. In our experiments on three public datasets, we compared the performances of the proposed method with that of other state-of-the-art methods and showed that the proposed method outperforms these competing methods.

In this paper; we propose a novel crowd behavior representation method to detect abnormal behaviors in videos. An adaptive optical flow filtering method is proposed to utilize low-level optical flow informations. Furthermore, a simple framework is developed to detect and to localize abnormal crowd behavior using adaptive optical flow filtering result. The proposed method is more robust than other modeling methods in representing different behaviors. In this model, a normal behavior is presented by the general value. Some outliers in the temporal domain or spatial domain are presented by a higher value. Spatio-temporal cuboids are extracted from the filtering result to present the likelihood of anomaly in the frame. Experimental evaluations are performed on two public datasets with comparison to the provisos abnormal behavior detection methods in the literature. Experimental results show that the proposed methods outperform previous abnormal behavior detection techniques in the literature.

In this paper we study the role of context in existing state-of-the-art detection and segmentation approaches. Towards this goal, we label every pixel of PASCAL VOC 2010 detection challenge with a semantic category. We believe this data will provide plenty of challenges to the community, as it contains 520 additional classes for semantic segmentation and object detection. Our analysis shows that nearest neighbor based approaches perform poorly on semantic segmentation of contextual classes, showing the variability of PASCAL imagery. Furthermore, improvements of existing contextual models for detection is rather modest. In order to push forward the performance in this difficult scenario, we propose a novel deformable part-based model, which exploits both local context around each candidate detection as well as global context at the level of the scene. We show that this contextual reasoning significantly helps in detecting objects at all scales.

In this paper, we address the problem of recognizing group activities that include interactions between human objects based on their motion trajectory analysis. In order to resolve the complexity and ambiguity problems caused by a large number of human objects, we propose a Group Interaction Zone (GIZ) to detect meaningful groups in a scene so as to be robust against noisy information. Two novel features, Group Interaction Energy feature and Attraction and Repulsion Features, are proposed to better describe group activities within a GIZ. We demonstrate the effectiveness of our method with other methods on the public BEHAVE dataset.

The problem of viewpoint changes is an important issue in the study of human action recognition. In this paper, we propose the use of spatial features in a spatiotemporal self-similarity matrix (SSM) based on action recognition that is robust in viewpoint changes from depth sequences. The spatial features represent a discriminative density of 3D point clouds in a 3D grid. We construct the spatiotemporal SSM for the spatial features that change along with frames. To obtain the spatiotemporal SSM, we compute the Euclidean distance of each spatial feature between two frames. The spatiotemporal SSM represents similarity of human action robust in viewpoint changes. Our proposed method is robust in viewpoint changes and various length of action sequence. This method is evaluated on ACTA2 dataset containing the multi-view RGBD human action data, and MSRAction3D dataset. In the experimental validation, the spatiotemporal SSM is a good solution for the problem of viewpoint changes in a depth sequence.

Recently neurophysiological studies have been concerned with using brain signals for driving assistance technologies. These studies verified that neurophysiological characteristics could be used for detection of emergency situations during simulated driving. However, it is hard to develop the braking assistant system which could control the vehicle continuously using this approach. In this article, the method for decoding of driver's braking intention based on analysis of neural correlates is proposed to control the braking of vehicle continuously. The participants' braking intention is decoded by kernel ridge regression (KRR) model to overcome the limitation of classification approach. In addition, the combination of three different features is employed to enhance the decoding performance. The decoding performances are evaluated by the correlation coefficient (r-value) and the normalized root-mean square error (NRMSE).

We developed and studied a Steady-State Visual Evoked Potential (SSVEP) based BCI system using a high frequency visual stimuli (>25Hz) design for reducing visual fatigue. Existing SSVEP based BCI designs primarily use low frequency visual stimuli (<;20Hz) for eliciting relatively higher SSVEP signal, while the low frequency stimuli can provocate photosensitivity epileptic seizure. On the other hand, high frequency stimuli are visually more comfortable and cause less visual fatigue and seizure. To detect the weak high frequency SSVEP signal, we used multi-channel EEG and introduced canonical correlation analysis to identify the elicited SSVEP frequency. We designed and built a 30-character SSVEP BCI speller system without calibration and evaluated the performance metrics including classification accuracy and subjective fatigue ratings, in both high-frequency and low frequency SSVEP modes. The result indicates that the high frequency stimuli system archieved higher classification accuracy (averaged 80% in the 30-class classification) comparable to that by the low frequency system. Moreover, no subjects rated the visual feeling as unacceptable or uncomfortable with the high frequency system.

In this paper, we present a new joint factorization algorithm, called Nonnegative Tensor Co-Factorization (NTCoF). The key idea is to simultaneously factorize multiple visual features of the same data into nonnegative dimensionality-reduced representations, and meanwhile, to maximize the correlations of the low-dimensional representations. The data is generally encoded as tensors of arbitrary order, rather than vectors, to preserve the original data structures. NTCoF provides a simple and efficient way to fuse multiple complementary features for enhancing the discriminative power of the desired rank-reduced representations under the nonnegative constraints. We formulate the related objectives with a block-wise quadratic nonnegative function. To optimize, a unified convergence provable solution is developed. This solution is applicable for any nonnegative optimization problems with block-wise quadratic objective functions, and thus offer an unified platform based on which specific solution can be directly derived by skipping over tedious proof about algorithmic convergence. We apply the proposed algorithm and solution on three image tasks, face recognition, multi-class image categorization and multi-label image annotation. Results with comparisons on public challenging datasets show that the proposed algorithm can outperform both the traditional nonnegative methods and the popular feature combination methods.

Tracking human pose is attractive to many applications such as Human Robot Interface (HRI), motion capture system, video surveillance, action recognition, etc. Though various methods were introduced during last decades, including both color and depth camera based, it is still considered that feature sets for them are not discriminative enough. In this paper, we propose a human pose tracking method based on a graphical model which incorporates global and local feature sets including Histogram of Oriented Gradients (HOG) and color distribution. HumanEva-I dataset is used for testing effectiveness of the proposed method.

As there has been a paradigm shift in the learning load from a human subject to a computer, machine learning has been considered as a useful tool for Brain-Computer Interfaces (BCIs). In this paper, we propose a novel Bayesian framework for discriminative feature extraction for motor imagery classification in an EEG-based BCI in which the class-discriminative frequency bands and the corresponding spatial filters are optimized by means of the probabilistic and information-theoretic approaches. In our framework, the problem of simultaneous spatiospectral filter optimization is formulated as the estimation of an unknown posterior probability density function (pdf) that represents the probability that a single-trial EEG of predefined mental tasks can be discriminated in a state. In order to estimate the posterior pdf, we propose a particle-based approximation method by extending a factored-sampling technique with a diffusion process. An information-theoretic observation model is also devised to measure discriminative power of features between classes. From the viewpoint of classifier design, the proposed method naturally allows us to construct a spectrally weighted label decision rule by linearly combining the outputs from multiple classifiers. We demonstrate the feasibility and effectiveness of the proposed method by analyzing the results and its success on three public databases.

Face recognition has been regarded as an effective method for subject identification at a distance because of its covert and remote sensing capability. However, face images have a low resolution when they are captured at a distance (say, larger than 5 meters) thereby degrading the face matching performance. To address this problem, we propose an imaging system consisting of static and pan-tilt-zoom (PTZ) cameras to acquire high resolution face images up to a distance of 12 meters. We propose a novel coaxial-concentric camera configuration between the static and PTZ cameras to achieve the distance invariance property using a simple calibration scheme. We also use a linear prediction model and camera motion control to mitigate delays in image processing and mechanical camera motion. Our imaging system was used to track 50 different subjects and their faces at distances ranging from 6 to 12 meters. The matching scenario consisted of these 50 subjects as probe and additional 10 000 subjects as gallery. Rank-1 identification accuracy of 91.5% was achieved compared to 0% rank-1 accuracy of the conventional camera system using a state-of-the-art matcher. The proposed camera system can operate at a larger distance (up to 50 meters) by replacing the static camera with a PTZ camera to detect a subject at a larger distance and control the second PTZ camera to capture the high-resolution face image.

Pose tracking technique has great potential for many applications such as marker-free human motion capture system, Human Computer Interactions (HCI), and video surveillance. Though many methods are introduced during last decades, self-occlusion - one body part is occluded by another one - is still considered one of the most difficult problems for 3D human pose tracking. In this paper, we propose a self-occlusion state estimation method. A MRF (Markov Random Field) is used to model the occlusion state which represents the pairwise depth order between two human body parts. A novel estimation method is proposed to infer a body pose and an occlusion state separately. HumanEva dataset is used for testing the proposed method. In order to evaluate and quantify how often the occlusion state changes, we label the ground truth of occlusion state.

Performance of automatic face recognition algorithm has increased considerably over the past decades. However, face recognition under changes in lighting conditions remains a challenging issue for computers. In this paper, we propose a novel face recognition algorithm inspired by information taken from human fixation patterns. We augment a LGBP (Local Gabor Binary Pattern) algorithm - a well-known face recognition algorithm - to allocate different weights to each facial part during processing. For deriving the weights, we analyzed data from a human face recognition experiment using eye-tracking. Eye-tracking allows us to determine the facial parts during the recognition process which represent salient regions for human processing. Face images are pre-processed during the recognition step using a weight mask based on the salient regions from the eye-tracking data. A comparison with the standard non-weighted LGBP approach demonstrates the efficacy of our method with the weighted method performing better under lighting changes.

Smoking is a typical exemplar in drug addiction researches for the wide range of tobacco users. The presented study uses functional magnetic resonance imaging (fMRI) to explore the resting-state (RS) neural mechanisms associated with smoking deprivation. We propose a novel analysis approach which applies nonnegative matrix factorization (NMF) algorithm on spatially concatenated two group datasets to investigate smoking related RS features in spatio-spectral domain. The NMF algorithm decomposed the magnitude spectra of fMRI time series into distinct frequency-specific basis functions and corresponding nonnegative spatial maps. After a two sample T-test on the z-scored spatial maps between groups, representative feature regions such as insula, anterior cingulate cortex and precuneus were found to be associated with smoking. These regions were consistent with that revealed by previous literatures studying in spatio-temporal domain. It indicates that our proposed analysis method provides another option for exploring neural mechanism differences between two groups, which might be used in a range of applications.

In human body pose estimation, manifold learning is a useful method for reducing the dimension of 2D images and 3D body configuration data. Most commonly, body pose is estimated from silhouettes derived from images or image sequences. A major problem when applying manifold estimation, however, is its vulnerability to silhouette variation. In this paper, we propose a novel approach to solving viewpoint-induced silhouette variation by introducing biased label distances for learning manifolds that are able to represent variations in viewpoint, pose, and 3D body configuration. We demonstrate the effectiveness of the approach on a synthetic and a real-world dataset.

A 3D morphable face model represents the complex shape and appearance structure of faces as a compact representation in a vector space. Such a model has proven extremely useful in various fields such as face recognition, facial expression synthesis, face animation, as well as in the perceptual and cognitive sciences. One of the most important steps in creating a 3D morphable model is to establish correspondence between all faces - a highly non-trivial process because of the complex topology of faces. In this paper, we propose to use implicit surfaces for this task, which enables a robust method for establishing dense 3D correspondences. This algorithm is used to build a morphable face model from a large database of Korean faces that were collected. We show that our model is in correspondence with the well-known MPI model so that our face space can be expanded from Asian to Caucasian faces. In addition, with the resulting control parameters of the model, we demonstrate realistic changes of high-level facial attributes.

In this paper, we propose a novel method of online learning of sparse pseudo-data, representative of the whole training data, for Gaussian Process (GP) regressions. We call the proposed method Incremental Sparse Pseudo-input Gaussian Process (ISPGP) regression. The proposed ISPGP algorithm allows for training from either a huge amount of training data by scanning through it only once or an online incremental training dataset. Thanks to the nature of the incremental learning algorithm, the proposed ISPGP algorithm can theoretically work with infinite data to which the conventional GP or SPGP algorithm is not applicable. From our experimental results on the KIN40K dataset, we can see that the proposed ISPGP algorithm is comparable to the conventional GP algorithm using the same number of training data. Although the proposed ISPGP algorithm performs slightly worse than Snelson and Ghahramani's SPGP algorithm, the level of performance degradation is acceptable.

In this paper, we devise a novel method that incrementally learns pseudo-data, which represent the whole training data set for Gaussian Process (GP) regression. The method involves sparse approximation of the GP by extending the work of Snelson and Ghahramani. We call the proposed method Incremental Sparse Pseudo-input Gaussian Process (ISPGP) regression. Unlike the Snelson and Ghahramani's work, the proposed ISPGP algorithm allows for training from either a huge amount of training data by scanning through it only once or an online incremental training data set. We also design a likelihood weighting scheme to incrementally determine pseudo-data while maintaining the representational power. Due to the nature of the incremental learning algorithm, the proposed ISPGP algorithm can theoretically work with infinite data to which the conventional GP or Sparse Pseudo-input Gaussian Process (SPGP) algorithm is not applicable. From our experimental results on the KIN40K data set, we can see that the proposed ISPGP algorithm is comparable to the conventional GP algorithm using the same number of training data. It also significantly reduces the computational cost and memory requirement in regression and is scalable to a large training data set without significant performance degradation. Although the proposed ISPGP algorithm performs slightly worse than Snelson and Ghahramani's SPGP algorithm, the level of performance degradation is acceptable.

Face recognition at a distance is gaining wide attention in order to augment the surveillance systems with face recognition capability. However, face recognition at a distance in nighttime has not yet received adequate attention considering the increased security threats at nighttime. We introduce a new face image database, called Near-Infrared Face Recognition at a Distance Database (NFRAD-DB). Images in NFRAD-DB are collected at a distance of up to 60 meters with 50 different subjects using a near-infrared camera, a telescope, and near-infrared illuminator. We provide face recognition performance using FaceVACS, DoG-SIFT, and DoG-MLBP representations. The face recognition test consisted of NIR images of these 50 subjects at 60 meters as probe and visible images at 1 meter with additional mug shot images of 10,000 subjects as gallery. Rank-1 identification accuracy of 28 percent was achieved from the proposed method compared to 18 percent rank-1 accuracy of a state of the art face recognition system, FaceVACS. These recognition results are encouraging given this challenging matching problem due to the illumination pattern and insufficient brightness in NFRAD images.

Detecting text regions in natural scenes is an important part of computer vision. We propose a novel text detection algorithm that extracts six different classes features of text, and uses Modest AdaBoost with multi-scale sequential search. Experiments show that our algorithm can detect text regions with a f= 0.70, from the ICDAR 2003 datasets which include images with text of various fonts, sizes, colors, alphabets and scripts.

In this paper, we investigate the maneuverability performance of human teleoperators on multi-robots. First, we propose that maneuverability performance can be assessed by a frequency response function that jointly considers the input force of the operator and the position errors of the multi-robot system that is being maneuvered. Doing so allows us to evaluate maneuverability performance in terms of the human teleoperator's interaction with the controlled system. This allowed us to effectively determine the suitability of different haptic cue algorithms in improving teleoperation maneuverability. Performance metrics based on the human teleoperator's frequency response function indicate that maneuverability performance is best supported by a haptic feedback algorithm which is based on an obstacle avoidance force.

EEG-based motor imagery classification has been widely studied for Brain-Computer Interfaces (BCIs) due to its asynchronous and continuous elicitation and its great potential to many applications. Many research groups have devoted their efforts to either the frequency band selection or optimal spatial filters learning via the Common Spatial Pattern (CSP) algorithm. However, since the spectral filtering and the spatial filtering are generally operated in order in a motor imagery classification system the optimization of the spatial filters and the spectral filters should be considered simultaneously in a unified framework. In this paper, we propose a novel probabilistic approach for the spatio-spectral filters optimization in an EEG-based BCI with a particle-filter algorithm and mutual information between feature vectors and class labels. There are two main contributions of the proposed method. The one is that it finds the optimal frequency bands that maximally discriminate the feature vectors of two classes in terms of an information theoretic approach. The other is that we construct a spectrally-weighted label decision rule by linearly combining the outputs from multiple SVMs, one for each frequency band, with the weight of the corresponding frequency band. From our experiments with two publicly available dataset, we confirm that the proposed method outperforms the other competing methods.

We propose a novel method of analyzing human interactions based on the walking trajectories of human subjects, which provide elementary and necessary components for understanding and interpretation of complex human interactions in visual surveillance tasks. Our principal assumption is that an interaction episode is composed of meaningful small unit interactions, which we call “sub-interactions”. We model each sub-interaction by a dynamic probabilistic model and propose a modified factorial hidden Markov model (HMM) with factored observations. The complete interaction is represented with a network of dynamic probabilistic models (DPMs) by an ordered concatenation of sub-interaction models. The rationale for this approach is that it is more effective in utilizing common components, i.e., sub-interaction models, to describe complex interaction patterns. By assembling these sub-interaction models in a network, possibly with a mixture of different types of DPMs, such as standard HMMs, variants of HMMs, dynamic Bayesian networks, and so on, we can design a robust model for the analysis of human interactions. We show the feasibility and effectiveness of the proposed method by analyzing the structure of network of DPMs and its success on four different databases: a self-collected dataset, Tsinghua University's dataset, the public domain CAVIAR dataset, and the Edinburgh Informatics Forum Pedestrian dataset.

In human body pose estimation, manifold learning is a popular technique for reducing the dimension of 2D images and 3D body configuration data. This technique, however, is especially vulnerable to silhouette variation such as caused by viewpoint changes. In this paper, we propose a novel approach that combines three separate manifolds for representing variations in viewpoint, pose and 3D body configuration. We use biased manifold learning to learn these manifolds with appropriately weighted distances. A set of four mapping functions are then learned by a generalized regression neural network for added robustness. Despite using only three manifolds, we show that this method can reliably estimate 3D body poses from 2D images with all learned viewpoints.

Sign language spotting is the task of detection and recognition of signs (words in the predefined vocabulary) and fingerspellings (a combination of continuous alphabets that are not found in signs) in a signed utterance. The internal structures of signs and fingerspellings differ significantly. Therefore, it is difficult to spot signs and fingerspellings simultaneously. In this paper, a novel method for spotting signs and fingerspellings is proposed, which can distinguish signs, fingerspellings, and nonsign patterns. This is achieved through a hierarchical framework consisting of three steps; (1) Candidate segments of signs and fingerspellings are discriminated with a two-layer conditional random field (CRF). (2) Hand shapes of detected signs and fingerspellings are verified by BoostMap embeddings. (3) The motions of fingerspellings are verified in order to distinguish those which have similar hand shapes and differ only in hand trajectories. Experiments demonstrate that the proposed method can spot signs and fingerspellings from utterance data at rates of 83% and 78%, respectively.

Sign language spotting is the task of detecting the start and end points of signs from continuous data and recognizing the detected signs in the predefined vocabulary. The difficulty with sign language spotting is that instances of signs vary in terms of both motion and shape. Moreover, signs have variable motion in terms of both trajectory and length. Especially, variable sign lengths result in problems with spotting signs in a video sequence, because short signs involve less information and fewer changes than long signs. In this paper, we propose a method for spotting variable lengths signs based on semi-CRF (semi-Markov Conditional Random Field). We performed experiments with ASL (American Sign Language) and KSL (Korean Sign Language) datasets of continuous sign sentences to demonstrate the efficiency of the proposed method. Experimental results showed that the proposed method outperforms both HMM and CRF.

Recently, many studies on gestural control methods for substituting for keyboard and mouse devices have been conducted because of their conveniences and intuitiveness. This paper presents a Virtual Mouse interface which is a gesture-based mouse interface and Two-layered Bayesian Network (TBN) for robust hand gesture recognition in real-time. The TBN provides robust recognition of hand gestures, as it compensates for an incorrectly recognized hand posture and its location via the preceding and following information. Experiments demonstrate that the proposed model recognizes hand gestures with a recognition rate of 93.78% and 85.15% for a simple and cluttered background, respectively.

In this paper, we propose a novel method for analyzing human interactions based on the walking trajectories of human subjects. Our principal assumption is that an interaction episode is composed of meaningful smaller unit interactions, which we call `sub-interactions.' The whole interaction is represented by an ordered concatenation or a network of sub-interaction models. From the experiments, we could confirm the effectiveness and robustness of the proposed method by analyzing the internal work of an interaction network and comparing the performance with other previous approaches.

Sign language spotting is the task of detecting and recognizing signs in a signed utterance, in a set vocabulary. The difficulty of sign language spotting is that instances of signs vary in both motion and appearance. Moreover, signs appear within a continuous gesture stream, interspersed with transitional movements between signs in a vocabulary and nonsign patterns (which include out-of-vocabulary signs, epentheses, and other movements that do not correspond to signs). In this paper, a novel method for designing threshold models in a conditional random field (CRF) model is proposed which performs an adaptive threshold for distinguishing between signs in a vocabulary and nonsign patterns. A short-sign detector, a hand appearance-based sign verification method, and a subsign reasoning method are included to further improve sign language spotting accuracy. Experiments demonstrate that our system can spot signs from continuous data with an 87.0 percent spotting rate and can recognize signs from isolated data with a 93.5 percent recognition rate versus 73.5 percent and 85.4 percent, respectively, for CRFs without a threshold model, short-sign detection, subsign reasoning, and hand appearance-based sign verification. Our system can also achieve a 15.0 percent sign error rate (SER) from continuous data and a 6.4 percent SER from isolated data versus 76.2 percent and 14.5 percent, respectively, for conventional CRFs.

In this paper, we propose a spatio-temporal silhouette representation, called silhouette energy image (SEI), and variability models, to characterize motion and shape properties for automatic recognition of human actions in daily life. To address the variability in the recognition of human actions, several parameters, such as anthropometry of the person, speed of the action, phase (starting and ending state of an action), camera observations (distance from camera, slanting motion, and rotation of human body), and view variations are proposed. We construct the variability models based on SEI and the variability parameters. The global shape-based motions express the spatio-temporal properties of SEI and variability models. Our construction of the optimal model for each action and view is based on the support vectors of motion descriptions of combined action models. We recognize different daily human actions of different styles successfully in the indoor and outdoor environment. Our experimental results show that the proposed method of human action recognition is robust, flexible and efficient.

In this paper, we present a real-time 3D pointing gesture recognition algorithm for natural human-robot interaction (HRI). The recognition errors in previous pointing gesture recognition algorithms are mainly caused by the low performance of the hands tracking module and by the unreliability of the direction estimate itself, therefore our proposed algorithm uses 3D particle filter for achieving reliability in hand tracking and cascade hidden Markov model (HMM) for a robust estimate for the pointing direction. When someone enters the field of view of the camera, his or her face and two hands are located and tracked using the particle filters. The first stage HMM takes the hand position estimate and maps it to a more accurate position by modeling the kinematic characteristics of finger pointing. The resulting 3D coordinates are used as an input to the second stage HMM that discriminates pointing gestures from others. Finally the pointing direction is estimated in the case of pointing state. The proposed method can deal with both large and small pointing gestures. The experiment shows better than 89% gesture recognition results and 99% target selection results.

In this paper, we describe a dynamic Bayesian network or DBN based approach to both two-hand gestures and one-hand gestures. Unlike wired glove-based approaches, the success of camera-based methods depends greatly on image processing and feature extraction results. So the proposed method of DBN-based inference is preceded by fail-safe steps of motion tracking. Then a new gesture recognition model for a set of both one-hand and two-hand gestures is proposed based on the dynamic Bayesian network framework which makes it easy to represent the relationship among features and incorporate new information to the model. In an experiment with ten isolated gestures, we obtained a recognition rate upwards of 99.59% with cross validation. The proposed model is believed to have a strong potential for successful applications to other related problems such as sign languages.

In this paper, we propose a new gesture recognition model for a set of both one-hand and two-hand gestures based on the dynamic Bayesian network framework which makes it easy to represent the relationship among features and incorporate new information to the model. Unlike the coupled HMM, the proposed model has room for common hidden variables which are believed to be shared between two variables. In an experiment with ten isolated gestures, we obtained a recognition rate upwards of 99.59% with leave-one-out cross validation. The proposed model is believed to have a strong potential for successful applications to other related problems such as sign languages.

Many methods for human gesture recognition have been researched. Bayesian Network (BN) and Dynamic Bayesian Network (DBN) are representative powerful tools for the gesture recognition. However, conventional BN is not appropriate in sequential data, and conventional DBN does not always guarantee that a sequence has relatively higher probability in a true class than in other classes. Moreover, the complexity of the DBN is increased exponentially with increasing number of hidden nodes and large number of training data is needed to guarantee the performance. Therefore, we propose a Semi-DBN (Semi-Dynamic Bayesian Network) which outperforms the conventional BNs and DBNs while it requires much less computational cost.

In this paper, we present a walking guidance system for the visually impaired pedestrians. The system has been designed to help the visually impaired by responding intelligently to various situations that can occur in unrestricted natural outdoor environments when walking and finding the destinations. It involves the main functions of people detection, text recognition, face recognition. In addition, added sophisticated functions of walking path guidance using Differential Global Positioning System, obstacle detection using a stereo camera and voice user, interface are included. In order to operate all functions concurrently, we develop approaches in real situations and integrate them. Finally, we experiment on a prototype system under natural environments in order to verify our approaches. The results show that our approaches are applicable to real situations.

3D face reconstruction is a popular area within the computer vision domain. 3D face reconstruction should ideally be achieved easily and cost-effectively, without requiring specialized equipment to estimate 3D shapes. As a result of this, many techniques for retrieving 3D shapes from 2D images have been proposed. In this paper, a novel method for 3D face reconstruction based on photometric stereo, which estimates the surface normal from shading information in multiple images, hence recovering the 3D shape of a face, is proposed. In order to overcome the problems of previous approaches related to prior-knowledge regarding lighting conditions and iterative algorithms, the exemplar is synthesized with known lighting conditions from at least three images, under arbitrary lighting conditions and using an illumination reference. Experiments in 3D face reconstruction were made by verifying the proposed approach using the illumination subset of the Max-Planck Institute face database and Yale face database B. Experimental results demonstrate that the proposed method is effective for 3D shape reconstruction of faces from 2D images.

This paper proposes a face hallucination method for the reconstruction of high-resolution facial images from single-frame, low-resolution facial images. The proposed method has been derived from example-based hallucination methods and morphable face models. First, we propose a recursive error back-projection method to compensate for residual errors, and a region-based reconstruction method to preserve characteristics of local facial regions. Then, we define an extended morphable face model, in which an extended face is composed of the interpolated high-resolution face from a given low-resolution face, and its original high-resolution equivalent. Then, the extended face is separated into an extended shape and an extended texture. We performed various hallucination experiments using the MPI, XM2VTS, and KF databases, compared the reconstruction errors, structural similarity index, and recognition rates, and showed the effects of face detection errors and shape estimation errors. The encouraging results demonstrate that the proposed methods can improve the performance of face recognition systems. Especially the proposed method can enhance the resolution of single-frame, low-resolution facial images.

In this paper, we propose an embedded face verification system against image degradation using a TMS320C6711 DSP chip. Our proposed system has several advantages over general systems in terms of price, size and applicability. As well as physical merits, it has an efficient approach for degraded facial images based on noise parameter estimation under real-life environments. The proposed method linearly combines image vector and noise parameters into one vector for training. When estimating noise parameters, we calculate the optimal coefficients of linear decomposition of an input image vector only. The noise parameters can be obtained from the linear composition step using these optimal coefficients. In contrast to conventional methods, we add the estimated noises to original images instead of removing them. Finally we perform a verification step with the input and synthesized image. The experimental results of this system show that the proposed method can estimate noise parameters accurately while improving the performance of photo image verification.

This paper presents a full-body gesture database which contains 2D video data and 3D motion data of 14 normal gestures, 10 abnormal gestures and 30 command gestures for 20 subjects. We call this database the Korea University Gesture (KUG) database. Using 3D motion cameras and 3 sets of stereo cameras, we captured 3D motion data and 3 pairs of stereo-video data in 3 different directions for normal and abnormal gestures. In case of command gestures, 2 pairs of stereo-video data were obtained by 2 sets of stereo cameras with different focal lengths in order to capture views of whole body and upper body, simultaneously. The 2D silhouette data was synthesized by separating a subject and background in 2D stereo-video data. In this paper, we describe the gesture capture system, the organization of database, the potential usages of the database and the contact point for the KUG database. We expect that this database would be very useful for the study of 2D/3D human gesture and its application.

Human face is one of the most common and useful keys to a person's identity. Although, a number of face recognition algorithms have been proposed, many researchers believe that the technology should be improved further in order to overcome the instability caused by variable illuminations, expressions, poses and accessories. To analyze these face recognition algorithm, it is indispensable to collect various data as much as possible. Face databases such as CMU PIE (USA), FERET (USA), AR Face DB (USA) and XM2VTS (UK) are the representative ones commonly used. However, many databases do not provide adequately annotated information of the pose angle, illumination angle, illumination color and ground-truth. Mostly, they do not include large enough number of images and video data taken under various environments. Furthermore, the faces on these databases have different characteristics from those of Asian. Thus, we have designed and constructed a Korean Face Database (KFDB) which includes not only images but also video clips, ground-truth information of facial feature points and descriptions of subjects and environment conditions so that it can be used for general purposes. In this paper, we present the KFDB which contains image and video data for 1920 subjects and has been constructed in 3 years (sessions). We also present recognition results by CM (Correlation Matching) and PCA (Principal Component Analysis) which are used as baseline algorithms upon CMU PIE and KFDB, so as to understand how recognition rate is changed by altering image taking conditions.

Visual interpretation of gestures can be useful in accomplishing natural human-robot interaction (HRI). Previous HRI research focused on issues such as hand gestures, sign language, and command gesture recognition. Automatic recognition of whole-body gestures is required in order for HRI to operate naturally. This presents a challenging problem, because describing and modeling meaningful gesture patterns from whole-body gestures is a complex task. This paper presents a new method for recognition of whole-body key gestures in HRI. A human subject is first described by a set of features, encoding the angular relationship between a dozen body parts in 3-D. A feature vector is then mapped to a codeword of hidden Markov models. In order to spot key gestures accurately, a sophisticated method of designing a transition gesture model is proposed. To reduce the states of the transition gesture model, model reduction which merges similar states based on data-dependent statistics and relative entropy is used. The experimental results demonstrate that the proposed method can be efficient and effective in HRI, for automatic recognition of whole-body key gestures from motion sequences

In this paper, we present a method of tracking and identifying persons in video images taken by a fixed camera situated at an entrance. In video sequences a person may be totally or partially occluded in a scene for some period of time. The proposed approach uses the appearance model for the identification of persons and the weighted temporal texture features. The weight is related to the size, duration as well as the number of persons adjacent to the target person. Most systems have built an appearance model for each person to solve occlusion problems. The appearance model contains certain information on the target person. We have compared the proposed method with other related methods using color and shape features, and analyzed the features' stability. Experimental results with various real video data sequences revealed that real time person tracking and recognition is possible with increased stability in video surveillance applications even under situations of occasional occlusion.

In order to raise a class discrimination power by the combination of multiple classifiers, the upper bound of Bayes error rate which is bounded by the conditional entropy of a class and decisions should be minimized. Based on the minimization of the upper bound of the Bayes error rate, Wang and Wong proposed only a tree dependence approximation scheme of a high-dimensional probability distribution composed of a class and patterns. This paper extends such a tree dependence approximation scheme to higher order dependency for improving the classification performance and thus optimally approximates the high-dimensional probability distribution with a product of low-dimensional distributions. And then, a new combination method by the proposed approximation scheme is presented and evaluated with classifiers recognizing unconstrained handwritten numerals.

In this paper, we propose a new method of removing glasses from a human frontal facial image. We first detect the regions occluded by the glasses and generate a natural looking facial image without glasses by recursive error compensation using PCA reconstruction. The resulting image has no trace of the glasses frame or of the reflection and shade caused by the glasses. The experimental results show that the proposed method provides an effective solution to the problem of glasses occlusion and we believe that this method can also be used to enhance the performance of face recognition systems.

Form document analysis is one of the most essential tasks in document analysis and recognition. One of the most fundamental and crucial tasks is the extraction of the reference lines which are contained in almost all form documents. This paper presents an efficient methodology for the complicated grey-level form image processing. We construct a non-orthogonal wavelet with adjustable rectangle supports and offer algorithms for the extraction of the reference lines based on the strip growth method using the multiresolution wavelet sub images. We have compared this system with the popular Hough transform (HT) based and the novel orthogonal wavelet based methods. As shown in the experiments, the proposed algorithm demonstrates high performance and fast speed for the complicated form images. This system is also effective for the form images with slight skew.

We present a method for detecting and tracking multiple people totally occluded or out of sight in a scene for some period of time in image sequences. Our approach is to use time weighted color information (i.e. the temporal color) for robust medium-term people tracking. The temporal color is the set of pairs of a color value and its associated weights. The weight is related to the size, duration and frequency of appearance of the color region, as well as the number of people adjacent to the target person. It assures our system to continuously track people moving in a group with occlusion. Most systems have built an appearance model for each person to solve occlusion problems. The appearance model contains certain information on the target person — color, shape, texture, position, velocity and face pattern. We use temporal color in the appearance model for the identification of the people occluded or out of sight in the scene upon their reappearance. Experimental results show that the temporal color is more stable than shape or intensity in various cases.

In this paper, we present a real-time pedestrian detection method in outdoor environments. It is necessary for pedestrian detection to implement obstacle and face detection which are major parts of a walking guidance system for the visually impaired. It detects foreground objects on the ground, discriminates pedestrians from other noninterest objects, and extracts candidate regions for face detection and recognition. For effective real-time pedestrian detection, we have developed a method using stereo-based segmentation and the SVM (Support Vector Machines), which works well particularly in binary classification problem (e.g. object detection). We used vertical edge features extracted from arms, legs and torso. In our experiments, test results on a large number of outdoor scenes demonstrated the effectiveness of the proposed pedestrian detection method.

In this paper, we present a survey on pattern recognition applications of Support Vector Machines (SVMs). Since SVMs show good generalization performance on many real-life data and the approach is properly motivated theoretically, it has been applied to wide range of applications. This paper describes a brief introduction of SVMs and summarizes its various pattern recognition applications.

This paper proposes a method for reconstructing partially damaged faces based on a morphable face model. Faces are modeled by linear combinations of prototypes of shape and texture. With the shape and texture information from an undamaged region only, we can estimate optimal coefficients for linear combinations of prototypes of shape and texture by simple projection for least-square minimization (LSM). Our experimental results show that reconstructed faces are very natural and plausible like real photos.

Only a few studies have been conducted on how to select multiple classifiers from the pool of available classifiers for showing good performance. The selection problem of classifiers on how to select or how many to select still remains an important issue. In this paper, provided that the number of selected classifiers is constrained in advance, a number of selection criteria are proposed and applied to the construction of multiple classifiers. All the sets of classifiers are examined by the selection criteria under the constraint of the number of selected classifiers, and then some of those sets are selected as the candidates of multiple classifier systems. The multiple classifier system candidates were evaluated by the experiments recognizing UCI handwritten numerals.

This paper concerns the modeling of character images by a Markov-like stochastic template that is constructed in the form of the pseudo 2D HMM. Being constructed directly from the image templates, the models can be composed in real time. Preliminary test results in digit recognition and Korean Hangul character spotting show the feasibility of the proposed method, especially in the context of incremental location of keywords.

Color-based object tracking is an important issue since color is an efficient and robust visual cue for characterizing one object from the other objects. Color segmentation is, however, suffered from color variations induced from irregular illumination variations and the viewing geometry of a camera. This paper proposes a reliable color modeling approach including intensity information in HSI color space using B-spline curves based on the fact that the color distribution of a single-colored object, even in a hue-saturation plane, is not invariant with respect to brightness variations in practical cameras. By using the approach, our color-based visual tracker is able to adapt to irregular illumination variations and abrupt changes of brightness. The approach is applied to real-time face tracking under various illumination conditions in a PC-based vision system.

In this paper we consider a hidden Markov mesh random field (HMMRF) for character recognition. The model consists of a "hidden" Markov mesh random field (MMRF) and an overlying probabilistic observation function of the MMRF. Just like the 1-D HMM, the hidden layer is characterized by the initial and the transition probability distributions, and the observation layer is defined by distribution functions for vector-quantized (VQ) observations. The HMMRF-based method consists of two phases: decoding and training. The decoding and the training algorithms are developed using dynamic programming and maximum likelihood estimation methods. To accelerate the computation in both phases, we employed a look-ahead scheme based on maximum marginal it a posteriori probability criterion for third-order HMMRF. Tested on a larget-set handwritten Korean Hangul character database, the model showed a promising result: up to 87.2% recognition rate with 8 state HMMRF and 128 VQ levels.

Automatic transformation of paper documents into electronic documents requires geometric document layout analysis at the first stage. However, variations in character font sizes, text line spacing, and document layout structures have made it difficult to design a general-purpose document layout analysis algorithm for many years. The use of some parameters has therefore been unavoidable in previous methods. The authors propose a parameter-free method for segmenting the document images into maximal homogeneous regions and identifying them as texts, images, tables, and ruling lines. A pyramidal quadtree structure is constructed for multiscale analysis and a periodicity measure is suggested to find a periodical attribute of text regions for page segmentation. To obtain robust page segmentation results, a confirmation procedure using texture analysis is applied to only ambiguous regions. Based on the proposed periodicity measure, multiscale analysis, and confirmation procedure, we could develop a robust method for geometric document layout analysis independent of character font sizes, text line spacing, and document layout structures. The proposed method was experimented with the document database from the University of Washington and the MediaTeam Document Database. The results of these tests have shown that the proposed method provides more accurate results than previous ones.

Medial axis transform (MAT) is a basic tool for shape analysis. However, in spite of its usefulness, it has some drawbacks, one of which is its instability under the boundary perturbation. We show that, although medial axis transform is unstable with respect to standard measures such as the Hausdorff distance, it is stable in a measure called relative Hausdorff distance for some "smoothed out" injective domains. In fact, we obtain an upper bound of the relative Hausdorff distance of the MAT of an injective domain with respect to the MAT of an arbitrary domain which is in small Hausdorff distance from the original injective domain. One consequence of the above result is that, by approximating a given domain with injective domains, we can extract the most "essential part" of the MAT within the prescribed error bound in Hausdorff distance. This introduces a new pruning strategy with precise error estimation. We illustrate our results with an example.

The cascade method, a way of calculating orientation components in an image, was developed to extract some important regions from a given image of a human face. By combining local orientation components and thresholding them, we construct five feature maps and a final composite map which is a linear combination of the five maps. As in human visual perception, the composite map operates like pre-attentive processing in the early stage of vision, and then shows robustness in selecting the most informative areas of images. For 50 non-normalized face images from the ORL database, it showed 91% detection accuracy which is the ratio of corresponding points between the feature maps of a whole image and the same maps of important regions in that image such as eyes, nose and mouth, etc.

This paper proposes a method for face reconstruction that makes use of only a small set of feature points. Faces can be modeled by forming linear combinations of prototypes of shape and texture information. With the shape and future information at the feature points alone, we can achieve only an approximation to the deformation required. In such an underdetermined condition, we find an optimal solution using a simple least square minimization method. As experimental results, we show well-reconstructed 2D faces even from a small number of feature points.

Most studies on combining multiple classifiers have mainly focused on how to combine their classification results and only a few studies have investigated on how to construct multiple classifier systems from available classifiers pool. In this paper, an information-theoretic strategy based on information theory model is proposed for constructing the multiple classifier systems. Provided that the number of classifiers in the multiple classifier systems is restricted in advance, this proposed strategy is applied to the classifiers pool and examines the possible sets of classifiers with the information-theoretic measure, and then it selects some sets of classifiers as the multiple classifier system candidates. The multiple classifier system candidates were evaluated together with the other sets of classifiers in the recognition of unconstrained handwritten numerals. The experimental results supported that the proposed strategy was a promising approach.

In order to process video data efficiently, a video segmentation technique through scene change detection must be employed. Many of advanced video applications require manipulations of compressed video signals. So, the scene change detection process is achieved by analyzing the video directly in the compressed domain, thereby avoiding the overhead of decompressing video into individual frames in the pixel domain. In this paper, we propose a fast scene change detection algorithm using direct feature extraction from MPEG compressed videos, and evaluate this technique using sample video data. This process was made possible by a new mathematical formulation for deriving the edge information directly from the discrete cosine transform coefficients.

Support vector machines (SVM) have been recently proposed for pattern recognition. Their basic property allows us to find a decision surface between two classes in terms of a hyperplane in a high dimensional space. In a multiclass recognition problem, SVM are used in the form of a combination of binary classifiers. However, SVM are unable to retrieve the top N matches, since they are designed to yield only one-the best match-in a multiclass problem. In other words, there is no proper similarity measurement for ordering all the classes in a given space using SVM. In this paper, we present an efficient method for the retrieval of the top N matches in a multiclass problem using SVM. For evaluation of the proposed method, we compared its result with that of a PCA algorithm in ranking the matches between classes.

We consider the problem of realizing associative memories via cellular neural networks (CNNs). We formulate the synthesis of CNN that can store given binary vectors with improved performance as a constrained optimization problem. Next, we convert the synthesis problem into a generalized eigenvalue problem, which can be efficiently solved by recently developed interior point methods. The validity of the proposed approach is illustrated by computer simulations.

Vergence movement enables human and vertebrates, having stereo vision, to perceive the depth of an interesting visual target fixated by both left and right eyes. To simulate this on a binocular robotic camera head, we propose a new control model for vergence movement using disparity flux. Experimental results showed that this model is efficient in controlling vergence movement in various environments. When the perception-action cycle is short enough to approach to the real-time frame rate, the precision of disparity flux increases, and then a more accurate control of vergence movements on the stereo robotic head is possible.

We propose two methods for converting complex multi-column document images into HTML documents, and a method for generating a structured table of contents (ToC) page based on the logical structure analysis of the document image. Experiments with various kinds of multi-column document images show that HTML documents corresponding to the paper documents can be generated in a visual layout, and that their structured table of contents page, with the hierarchically ordered section titles hyperlinked to the contents, can be also produced by the proposed methods.

Video text extraction is a core technique for multimedia applications such as news-on-demand (NOD) and digital libraries, and research about video text extraction have been conducted vigorously. In this paper, we propose an efficient method for extracting texts in MPEG compressed videos for content-based indexing. The proposed method makes the best use of 2-level DCT coefficients and macroblock type information in MPEG compressed video, and this method can be organized into three stages to increase overall performance; text frame detection, text region extraction, and character extraction. The main advantage of the proposed method is that it can avoid the overhead of decompressing video into individual frames in the pixel domain. We evaluated this method using various types of news video data.

With a surge of interest in OCR in the 1990s, a large number of handwriting or handprinting databases have been built one after another around the world. One problem that researches encounter today is that all the databases differ in various ways including the script qualities. The paper proposes a method for measuring handwriting qualities that can be used for comparison of databases and objective test for character recognizers. The key idea involved is classifying character samples into a number of groups each characterizing a set of qualities. In order to evaluate the proposed method, we carried out experiments on the KU-1 database.

Text-to-phoneme conversion is a common problem in speech processing. This can be done using a rule-based system or a neural network. In this paper we propose a solution to this problem using a modular hybrid system that uses basic rules to subdivide the original problem into easier tasks which are then solved by dedicated neural networks. Such a solution can be more rapidly constructed, and is easily extendable. A voting committee concept is used to enhance generalization abilities of the system.

We present a method for the detection and tracking of multiple people totally occluded or out of sight in a scene for some period of time in image sequences. Our approach is to use time weighted color information, i.e., the temporal color, for robust medium-term people tracking. It assures our system continuously tracks people moving in a group with occlusion. Experimental results show that the temporal color is more stable than shape or intensity when used in various cases.

We propose a new method independent of parameters for segmenting the document images into maximal homogeneous regions and identifying them as texts, images, tables and lines. A pyramidal quadtree structure is constructed for multiscale analysis and top-down approach, and a periodicity measure is suggested to find a periodical attribute of text regions. To obtain robust page segmentation results, a confirmation procedure using texture analysis is applied to only ambiguous regions. Experimental results with the document database from the University of Washington show that the proposed method works better than the previous ones.

In order to process video data efficiently, a video segmentation technique through scene change detection must be required. This is a fundamental operation used in many digital video applications such as digital libraries, video on demand (VOD), etc. Many of these advanced video applications require manipulations of compressed video signals. So, the scene change detection process is achieved by analyzing the video directly in the compressed domain, thereby avoiding the overhead of decompressing video into individual frames in the pixel domain. In this paper, we propose a fast scene change detection algorithm using direct feature extraction from MPEG compressed videos, and evaluate this technique using sample video data, First, we derive binary edge maps from the AC coefficients in blocks which were discrete cosine transformed. Second, we measure edge orientation, strength and offset using correlation between the AC coefficients in the derived binary edge maps. Finally, we match two consecutive frames using these two features (edge orientation and strength). This process was made possible by a new mathematical formulation for deriving the edge information directly from the discrete cosine transform (DCT) coefficients. We have shown that the proposed algorithm is faster or more accurate than the previously known scene change detection algorithms.

Although Behavior-Knowledge Space (BKS) method does not need any assumptions in combining multiple experts, it should build theoretically exponential storage spaces for storing and managing jointly observed K decisions from K experts. That is, combining K experts needs a (K+1)st-order probability distribution. However, it is well known that the distribution becomes unmanageable in storing and estimating, even for a small K. In order to overcome such weakness, it would be attractive to decompose the distribution into a number of component distributions and to approximate the distribution with a product of the component distributions. One of such previous works is to apply a conditional independence assumption to the distribution. Another work is to approximate the distribution with a product of only first-order tree dependencies or second-order distributions. In this paper, a dependency-based framework is proposed to optimality approximate a probability distribution with a product set of dth-order dependencies where 1<d

Presents a concept of an information capturing camera, which is an enhanced digital/video camera that can recognize characters in captured images and show relevant information, such as a translated version of the words in the image into another language. Also discussed are technical issues and possible approaches. Technical issues include detection of the text image regions, perspective distortion normalization, binarization of character images in unknown colors, recognition of omnifont and decorative characters, word recognition and noun compound translation. Finally, an international collaboration on the development of the required technology is proposed.

In order to raise a class discrimination power by combining multiple classifiers, the upper bound of a Bayes error rate bounded by the conditional entropy of a class variable and decision variables should be minimized. Wang and Wong (1979) proposed a tree dependence approximation scheme of a high order probability distribution composed of those variables, based on minimizing the upper bound. In addition to that, this paper presents an extended approximation scheme dealing with higher order dependency. Multiple classifiers recognizing unconstrained handwritten numerals were combined by the proposed approximation scheme based on the minimization of the Bayes error rate, and the high recognition rates were obtained by them.

A new method that uses a modified fractal dimension theory to segment a document image and to discriminate between Oriental and European languages is presented in this paper. Two types of techniques have been usually adopted in language discrimination: token matching and statistical analysis. A modified fractal feature is used to discriminate the distinct textual structure complexities of Oriental and European languages. Experiments show that this method is effective and reliable for processing the document image even if it is skewed or contains noise that can not be removed clearly.

Proposes an integrated image segmentation and recognition method using a new type of cascade neural network that has been is developed to train the spatial dependencies in connected handwritten numerals. This network was originally extended from a multilayer feedforward neural network in order to improve its discrimination and generalization power. To verify the performance of the proposed method, recognition experiments with the National Institute of Standards and Technology (NIST) numerals databases have been performed. The experimental results reveal that the proposed method has a higher discrimination and generalization power than previous integrated segmentation and recognition methods have had. Moreover, the network size of the proposed method is smaller than that of the previous methods.

A stochastic oscillator neural network (STONN) model of the Hopfield-type memory is proposed for the pattern segmentation tasks, that exploits temporal dynamics of the stochastic nonlinear oscillators. For an input pattern which is an overlapped superposition of several stored patterns the proposed model network is shown to be capable of segmenting out each pattern one after another as the network evolves its temporal dynamics. The temporal segmentation attains its optimal performance at an intermediate noise intensity and the performance becomes improved as the coupling strength between oscillators increases. A mechanism for the selective attention is also introduced in the STONN by controlling the level of noise applied to the most salient pattern and by adopting the inhibition-of-return into the patterns that have been segmented before.

We present an efficient video parsing method using shot boundary detection and camera operation analysis technique. In the shot boundary detection, the local color information and an adaptive time window is used. The local spatio-temporal images and multilayer perceptron are used for analyzing the camera operations. In order to verify the performance of the proposed video parsing method, experiments with video database have been carried out. Experimental results demonstrate the efficiency of the video parsing technique.

In general, nonlinear shape normalization methods for binary images have been used in order to compensate for the shape distortions of handwritten characters. However, in most document image analysis and recognition systems, a gray-scale image is first captured and digitized using a scanner or a video camera, then a binary image is extracted from the original gray-scale image using a certain extraction technique. This binarization process may remove some useful information of character images such as topological features, and introduce noises to character background. These errors are accumulated in nonlinear shape normalization step and transferred to the following feature extraction or recognition step. They may eventually cause incorrect recognition results. In this paper, we propose nonlinear shape normalization methods for gray-scale handwritten Oriental characters in order to minimize the loss of information caused by binarization and compensate for the shape distortions of characters. Two-dimensional linear interpolation technique has been extended to nonlinear space and the extended interpolation technique has been adopted in the proposed methods to enhance the quality of normalized images. In order to verify the efficiency of the proposed methods, the recognition rate, the processing time and the computational complexity of the proposed algorithms have been considered. The experimental results demonstrate that the proposed methods are efficient not only to compensate for the shape distortions of handwritten Oriental characters but also to maintain the information in gray-scale Oriental characters.

In this paper, we propose a practical scheme for multi-lingual, multi-font and multi-size large-set Oriental character recognition using a self-organizing hierarchical neural network classifier. In order to absorb the variation of the character shapes in multi-font and multi-size characters, a modified nonlinear shape normalization method based on dot density was introduced, and also to represent the different topological structures of multi-lingual characters effectively, a hierarchical feature extraction method was adopted. For coarse classification, a tree classifier and SOFM/LVQ based classifier which is composed of an adaptive SOFM coarse-classifier and an LVQ4 language-classifier were considered. For fine classification, a classifier based on LVQ4 learning algorithm has been developed. The experimental results revealed that the proposed scheme has the highest recognition rate of 98.27% for testing data with 7,320 kinds of multi-lingual classes and the time performance of more than 40 characters per second on 486DX-2 66MHz PC.

In this paper, an off-line recognition system based on multifeature and multilevel classification is presented for handwritten Chinese characters. Ten classes of multifeatures, such as peripheral shape features, stroke density features, and stroke direction features, are used in this system. The multilevel classification scheme consists of a group classifier and a five-level character classifier, where two new technologies, overlap clustering and Gaussian distribution selector are developed. Experiments have been conducted to recognize 5,401 daily-used Chinese characters. The recognition rate is about 90 percent for a unique candidate, and 98 percent for multichoice with 10 candidates.

Neural networks have been successfully applied to various pattern classification problems in terms of their learning ability, high discrimination power, and excellent generalization ability. However, for the case of classifying large-set and complex patterns, the greater part of conventional neural networks suffer from several difficulties such as the determination of the structure and size of the network, the computational complexity, and so on. In this paper, to cope with these difficulties, we propose a structurally adaptive intelligent neural tree (SAINT). The basic idea is to partition hierarchically input pattern space using a tree-structured network which is composed of subnetworks with topology-preserving mapping ability. The main advantage of SAINT is that it attempts to find automatically a network structure and size suitable for the classification of large-set and complex patterns through structure adaptation. Experimental results reveal that SAINT is very effective for the classification of large-set real world handwritten characters with high variations, as well as multilingual, multifont, and multisize large-set characters.

Proposes nonlinear shape normalization methods for gray-scale handwritten characters in order to minimize the loss of information caused by binarization and to compensate for the shape distortions of characters. A 2D linear interpolation technique has been extended to nonlinear space and the extended interpolation technique has been adopted in the proposed methods to enhance the quality of the normalized images. In order to verify the efficiency of the proposed methods, the recognition rate, the processing time and the computational complexity of the proposed algorithms have been considered. The experimental results indicate that the proposed methods are efficient not only for compensating for the shape distortions of handwritten characters but also for maintaining the information of gray-scale input characters.

We propose a new type of recurrent neural-network architecture, in which each output unit is connected to itself and is also fully connected to other output units and all hidden units. The proposed recurrent neural network differs from Jordan's and Elman's recurrent neural networks with respect to function and architecture, because it has been originally extended from being a mere multilayer feedforward neural network, to improve discrimination and generalization powers. We also prove the convergence properties of the learning algorithm in the proposed recurrent neural network, and analyze the performance of the proposed recurrent neural network by performing recognition experiments with the totally unconstrained handwritten numeric database of Concordia University, Montreal, Canada. Experimental results have confirmed that the proposed recurrent neural network improves discrimination and generalization powers in the recognition of visual patterns.

We propose a new methodology for off-line handwritten character recognition using a 2D hidden Markov mesh random field (HMMRF)-based statistical approach. In the HMMRF model for character recognition, the inputs to the model are assumed to be sequences of discrete symbols chosen from a finite alphabet. In the proposed methodology, the grey-level input image is first divided into nonoverlapping blocks with same size. Then, each block is encoded into a discrete symbol based on the local features of the block by using the vector quantizer. The HMMRF-based statistical approach necessitates two phases: the decoding phase and the training phase. In both phases we use the lookahead scheme based on a maximum, marginal a posteriori probability criterion for a third-order HMMRF model. In order to verify the performance of the proposed methodology for off-line handwritten character recognition, a large-set handwritten Hangul database was used. Experimental results revealed the viability of the HMMRF-based statistical approach on the task of off-line handwritten character recognition.

In this paper, we propose a new type of recurrent neural network architecture in which each output unit is connected with itself and fully-connected with other output units and all hidden units. The proposed recurrent neural network differs from Jordan's and Elman's recurrent neural networks in view of functions and architectures because it was originally extended from the multilayer feedforward neural network for improving the discrimination and generalization power. We also prove the convergence property of learning algorithm in the proposed recurrent neural network and analyze the performance of the proposed recurrent neural network by performing recognition experiments with the totally unconstrained handwritten numeral database of Concordia University of Canada. Experimental results confirmed that the proposed recurrent neural network improves the discrimination and generalization power in recognizing spatial patterns.

In this paper, we propose a new scheme for off-line recognition of totally unconstrained handwritten numerals using a simple multilayer cluster neural network trained with the backpropagation algorithm and show that the use of genetic algorithms avoids the problem of finding local minima in training the multilayer cluster neural network with gradient descent technique, and improves the recognition rates. In the proposed scheme, Kirsch masks are adopted for extracting feature vectors and a three-layer cluster neural network with five independent subnetworks is developed for classifying similar numerals efficiently. In order to verify the performance of the proposed multilayer cluster neural network, experiments with handwritten numeral database of Concordia University of Canada, that of Electro-Technical Laboratory of Japan, and that of Electronics and Telecommunications Research Institute of Korea were performed. For the case of determining the initial weights using a genetic algorithm, 97.10%, 99.12%, and 99.40% correct recognition rates were obtained, respectively.

Generally speaking, through the binarization of gray-scale images, useful information for the segmentation of touched or overlapped characters may be lost in many cases. If we analyze gray-scale images, however, specific topographic features and the variation of intensities can be observed in the character boundaries. In this paper, we propose a new methodology for character segmentation and recognition which makes the best use of the characteristics of gray-scale images. In the proposed methodology, the character segmentation regions are determined by using projection profiles and topographic features extracted from the gray-scale images. Then a nonlinear character segmentation path in each character segmentation region is found by using multi-stage graph search algorithm. Finally, in order to confirm the nonlinear character segmentation paths and recognition results, a recognition-based segmentation method is adopted. Through the experiments with various kinds of printed documents, it is convinced that the proposed methodology is very effective for the segmentation and recognition of touched and overlapped characters.

In this paper, we describe a parallel hardware implementation of handwritten character recognition system with neural network classifier on wavefront array processor (WAP) architecture. The WAP architecture needs no global clock for controlling the entire processors and each processing element can even be clocked differently. Thus, the clock skew becomes no longer problem and it is possible to construct massively parallel networks which are required for handwritten character recognition. We also present experimental results for the recognition of unconstrained handwritten numerals with the WAP architecture. The results were very impressive in view of real world application.

The approach in this paper focuses on the contextual postprocessing by selecting the most feasible word from multiple output strings of an OCR system. The correction is applied only when the selection fails. The selected word is confirmed by the collocation between the word and the adjacent words. The five functions applied in the system are (1) to select a word from candidate words, (2) to correct candidate words using a confusion matrix of syllables, (3) to combine two substrings to a word that spans two lines, (4) to guess unknown nouns, and (5) to confirm a selected word by the contextual information of adjacent words. To improve speed, we use syllable di-grams and viable-prefixes of Korean words. The experimental result shows that the two heuristics speed up the system more than 1,000 times in worst case. Our system improves the word recognition rate of the OCR system from 90.50% to 94.72%.

We propose a practical scheme for multilingual multi font, and multi size large set character recognition using self organizing neural network. In order to improve the performance of the proposed scheme, a nonlinear shape normalization based on dot density and three kinds of hierarchical features are introduced. For coarse classification, two kinds of classifiers are proposed. One is a hierarchical tree classifier, and the other is a SOFM/LVQ based classifier which is composed of an adaptive SOFM coarse classifier and LVQ4 language classifiers. For fine classification, an LVQ4 classifier has been adopted. In order to evaluate the performance of the proposed scheme, recognition experiments with 3,367,200 characters having 7320 different classes have been carried out on a 486 DX-2 66 MHz PC. Experimental results reveal that the proposed scheme using an adaptive SOFM coarse classifier, LVQ4 language classifiers, and LVQ4 fine classifiers has a high recognition rate of over 98.27% and a fast execution time of more than 40 characters per second.

The authors propose a new type of recurrent neural network for handwritten character recognition. The proposed recurrent neural network differs from Jordan and Elman recurrent neural networks in view of functions and architectures because it was originally extended from the multilayer feedforward neural network for improving discrimination and generalization power in recognizing handwritten characters. They also analyze the performance of the proposed recurrent neural network by performing recognition experiments with the totally unconstrained handwritten numeral database of Concordia University of Canada. The experimental results showed that the proposed recurrent neural network greatly improves the discrimination and generalization power.

In this paper, we propose a new scheme for multiresolution recognition of totally unconstrained handwritten numerals using wavelet transform and a simple multilayer cluster neural network. The proposed scheme consists of two stages: a feature extraction stage for extracting multiresolution features with wavelet transform, and a classification stage for classifying totally unconstrained handwritten numerals with a simple multilayer cluster neural network. In order to verify the performance of the proposed scheme, experiments with unconstrained handwritten numeral database of Concordia University of Canada, that of Electro-Technical Laboratory of Japan, and that of Electronics and Telecommunications Research Institute of Korea were performed. The error rates were 3.20%, 0.83%, and 0.75%, respectively. These results showed that the proposed scheme is very robust in terms of various writing styles and sizes.

Image shape restoration based on mathematical transformation is a successful approach to nonlinear distortions in computer vision, robot vision and pattern recognition. The key of this process is to find the distortion function and its inverse function. Usually, the distortion function is unknown or unclear. Even in the case where the function is known, it remains difficult to compute or estimate the parameters necessary for the restoration. To overcome this problem, Coons transformation utilizing boundary functions for the distorted images have been used to approximate the exact distortion function. The boundary functions are calculated using B-spline curve interpolation which is coincided with the necessary condition of major elements that constitute a Coons transformation.

In this paper, we propose an integrated segmentation and recognition method for recognizing connected handwritten characters with recurrent neural network. It has been developed to both integrate segmentation and recognition within a single recurrent neural network and recognize connected handwritten characters using the spatial dependencies in the images of connected handwritten characters. In order to verify the performance of the proposed method, experiments with the NIST database have been carried out and the performance of the proposed method has been compared with those of the previous integrated segmentation and recognition methods.

Generally speaking, through the binarization of gray-scale images, useful information for the segmentation of touching or overlapping characters may be lost. If we analyze gray-scale images, however, specific topographic features and the variation of intensity can be observed in the character boundaries. We believe that such kinds of clues obtained from gray-scale images should be useful for efficient character segmentation. In this paper, we propose a new methodology for character segmentation and recognition which makes the best use of the characteristics of gray-scale images. In the proposed methodology, the character segmentation regions are determined by using projection profiles and topographic features extracted form gray-scale images. Then the nonlinear character segmentation path in each character segmentation region is found by using multistage graph search algorithm. Finally, in order to confirm the character segmentation paths and recognition results, recognition based segmentation method is adopted.

In recent years, there have been some attempts to extend one-dimensional hidden Markov model (HMM) to two-dimensions. This paper presents a new statistical model for image modeling and recognition under the assumption that images can be represented by a third-order hidden Markov mesh random field (HMMRF) model. We focus on two major problems: image decoding and parameter estimation. A solution to these problems is derived from the scheme based on a maximum, marginal a posteriori probability criterion for the third-order HMMRF model. We also attempt to illustrate how theoretical results of HMMRF models can be applied to the problems of handwritten character recognition.

We present a new scheme for the restoration of faxed images degraded by both salient noise and additive white noise. We consider two fundamental aspects of faxed image restoration: modeling and the restoration algorithm. First, a model of a faxed image is presented. The model is based on an autoregressive Gauss-Markov random field and includes vertical and horizontal overlap effects. In particular, we concentrate on the nonsymmetric half plane causality. Second, the restoration of faxed images degraded by salient noise and additive white noise is considered by 2D Kalman filtering which provides an efficient recursive procedure. In order to illustrate the effectiveness of the proposed scheme, we present experimental results on the restoration of a faxed image.

Optical character recognition (OCR) traditionally applies to binary-valued imagery although text is always scanned and stored in gray scale. However, binarization of multivalued image may remove important topological information from characters and introduce noise to character background. In order to avoid this problem, it is indispensable to develop a method which can minimize the information loss due to binarization by extracting features directly from gray scale character images. In this paper, we propose a new method for the direct extraction of topographic features from gray scale character images. By comparing the proposed method with Wang and Pavlidis' method, we realized that the proposed method enhanced the performance of topographic feature extraction by computing the directions of principal curvature efficiently and prevented the extraction of unnecessary features. We also show that the proposed method is very effective for gray scale skeletonization compared to Levi and Montanari's method.>

In this paper, we propose a simple multilayer cluster neural network with five independent subnetworks for off-line recognition of totally unconstrained handwritten numerals. We also show that the use of genetic algorithms for avoiding the problem of finding local minima in training the multilayer cluster neural network with gradient descent technique reduces error rates.

In this paper, we propose a knowledge-based approach for locating destination address block (DAB) on handwritten Korean envelopes by merging and splitting method. DAB of the input envelope image is extracted by an iterative procedure based on the knowledge acquired from the statistical feature analysis of the various handwritten Korean envelopes. Experimental results reveal that the proposed approach is very effective for address block location on handwritten Korean envelopes.

Image shape restoration based on mathematical transformation is a successful approach to nonlinear distortions in computer vision, robot vision and pattern recognition. The key of this process is to find the distortion function and its inverse function. Usually, the distortion function is unknown or unclear. Even in the case of when the function is known, it remains difficult to compute or estimate the parameters necessary for the restoration. To overcome this problem, in this paper, boundary functions have been used to approximate the exact distortion function. In this manner, the boundary functions are calculated using B-spline curve interpolation which coincides with the necessary condition of major elements that constitute a Coons transformation. The performance of the proposed method with nonlinear distorted images has been demonstrated in experiments with interesting results.

Proposes efficient postprocessing algorithms for error correction in handwritten Hangul (Korean script) address and human name recognition. As the load on the character recognizer for the recognition of the administrative district part in addresses was reduced by restricting the candidate characters to be matched based on a hierarchical address lexicon, the processing speed and recognition rate were greatly improved. Also, the misrecognition results from the character recognizer were corrected by using efficient postprocessing algorithms based on backtracking. For the recognition of the human name part, misrecognition of human names could be effectively corrected by combining the a priori probability and the confusion probability of each character making up the human names.>

Recently, several nonlinear shape normalization methods have been proposed in order to compensate for shape distortions in large-set handwritten characters. The authors review these methods from the two points of view: feature projection and feature density equalization. The former makes a feature projection histogram by projecting a certain feature at each point into horizontal- or vertical-axis and the latter equalizes the feature densities of input image by re-sampling the feature projection histogram. Then, a quantitative evaluation for these methods has been made based on the following criteria: recognition rate, processing speed, computational complexity, and degree of variation.>

For the recognition of large-set handwritten characters, classification methods based on pattern matching have been commonly used, and good reference models play a very important role in achieving high performance in these methods. Learning vector quantization (LVQ) has been studied intensively to generate good reference models in speech recognition since 1986. However, the design of reference models based on LVQ has several drawbacks for the recognition of large-set handwritten characters. To cope with these, the authors propose a method for the optimal design of reference models using simulated annealing combined with an improved LVQ3 for the recognition of large-set handwritten characters. Experimental results reveal that the proposed method is superior to the conventional method based on averaging and other LVQ-based methods.>

An efficient recognition scheme for large-set handwritten characters is proposed in the framework of multiple stochastic models, in this case, first order hidden Markov models which can model stochastically the input pattern with numerous variations. In this scheme, after extracting four kinds of regional projection contours for an input pattern by using the regional projection contour transformation, four kinds of HMMs are constructed during the training phase based on the direction components of these contours. In the recognition phase, the four kinds of HMMs constructed in the training phase are combined to output the final recognition result for an input pattern.>

A VLSI architecture to implement the horizontal- vertical region integration (HVRI) algorithm has been designed. The HVRI algorithm transforms a multi-contour pattern into a unique outer contour. It is applicable to a wide range of areas such as image analysis, pattern recognition, etc. A linear array has been designed based on a canonical methodology which maps homogeneous dependence graphs into processor arrays. An N/2-element vector is used to process a pattern with a size of N/spl times/N. It can speed up the recognition process considerably with a time complexity of O(N) compared with O(N/sup 2/) when a uniprocessor is used.>

As a result of its central role in the preprocessing of image patterns, or because of its intrinsic appeal, the design of skeletonization algorithms has been a very active research area. However, few attempts have been made to evaluate the performance of different skeletonization algorithms. This paper presents the results of experiments to evaluate the performance of 20 skeletonization algorithms previously published in the literature. These algorithms have been implemented on the SUN 3/60 workstation in C and tested with a large variety of character patterns. A systematic comparison of these algorithms has been made based on the following criteria: reconstructibility, computation speed, similarity to the reference skeleton, quality of the skeleton, connectivity after skeletonization, and the degree of parallelism.

A comprehensive survey of thinning methodologies is presented. A wide range of thinning algorithms, including iterative deletion of pixels and nonpixel-based methods, is covered. Skeletonization algorithms based on medial axis and other distance transforms are not considered. An overview of the iterative thinning process and the pixel-deletion criteria needed to preserve the connectivity of the image pattern is given first. Thinning algorithms are then considered in terms of these criteria and their modes of operation. Nonpixel-based methods that usually produce a center line of the pattern directly in one pass without examining all the individual pixels are discussed. The algorithms are considered in great detail and scope, and the relationships among them are explored.>

In this paper, a model-based scheme for recognizing hand-drawn symbols in schematic diagrams using attributed graph (AG) matching in the absence of any information concerning their pose (translation, rotation and scale) is described. The process of AG matching proceeds as follows. First, an observed AG (AG O ) is constructed from single-pixel-width line-representation of an observed symbol. Second, the pose of the AG O is estimated in terms of translation, rotation and scale with respect to the model AGs (AG M s ). The search space is effectively pruned by introducing the concept of control vertex and applying geometrical constraints in an early stage. In this step, a small number of candidate AG M s are selected. Third, correspondences between components of the observed AG after normalization (AG O N ) and those of the AG M s are found for the given poses. Fourth, distance measures between the AG O N and the AG M s are calculated, based upon the correspondences. Finally, the AG O N is classified as the AG M with the minimum distance. Experimental results for hand-drawn symbols with and without templates show that using AG matching is very efficient and successful for translation-, rotation- and scale-invariant recognition of hand-drawn symbols in schematic diagrams.

A model-based scheme for classifying similar line-drawing patterns by attributed graph matching in the absence of any information concerning their pose (translation, rotation and scale) is proposed. Experimental results for two classes of line-drawing, each consisting of a number of similar patterns (hand-drawn electrical symbols and seal imprints), confirmed that the proposed scheme might be feasible for practical applications.>

A basic minimum-square-error (MSE) transform for point pattern matching and a fast computational method for its implementation are described. Currently, the method is being used successfully in a number of research projects. A computational analysis shows that it is faster than that of F.C.A. Groen et al.>