Moiré patterns, caused by frequency aliasing between fine repetitive structures and a camera sensor’s sampling process, have been a significant obstacle in various real-world applications, such as consumer photography and industrial defect inspection. With the advancements in deep learning algorithms, numerous studies—predominantly based on convolutional neural networks—have suggested various solutions to address this issue. Despite these efforts, existing approaches still struggle to effectively eliminate artifacts due to the diverse scales, orientations, and color shifts of moiré patterns, primarily because the constrained receptive field of CNN-based architectures limits their ability to capture the complex characteristics of moiré patterns. In this paper, we propose MZNet, a U-shaped network designed to bring images closer to a ‘Moiré-Zero’ state by effectively removing moiré patterns. It integrates three specialized components: Multi-Scale Dual Attention Block (MSDAB) for extracting and refining multi-scale features, Multi-Shape Large Kernel Convolution Block (MSLKB) for capturing diverse moiré structures, and Feature Fusion-Based Skip Connection for enhancing information flow. Together, these components enhance local texture restoration and large-scale artifact suppression. Experiments on benchmark datasets demonstrate that MZNet achieves state-of-the-art performance on high-resolution datasets and delivers competitive results on lower-resolution datasets.

Multi-modal understanding plays a crucial role in artificial intelligence by enabling models to jointly interpret inputs from different modalities. However, conventional approaches such as contrastive learning often struggle with modality discrepancies, leading to potential misalignments. In this paper, we propose a novel class anchor alignment approach that leverages class probability distributions for multi-modal representation learning. Our method, Class-anchor-ALigned generative Modeling (CALM), encodes class anchors as prompts to generate and align class probability distributions for each modality, enabling more effective alignment. Furthermore, we introduce a cross-modal probabilistic variational autoencoder to model uncertainty in the alignment, enhancing the ability to capture deeper relationships between modalities and data variations. Extensive experiments on four benchmark datasets demonstrate that our approach significantly outperforms state-of-the-art methods, especially in out-of-domain evaluations. This highlights its superior generalization capabilities in multi-modal representation learning.

Continual learning enables pre-trained generative vision-language models (VLMs) to incorporate knowledge from new tasks without retraining data from previous ones. Recent methods update a visual projector to translate visual information for new tasks, connecting pre-trained vision encoders with large language models. However, such adjustments may cause the models to prioritize visual inputs over language instructions, particularly learning tasks with repetitive types of textual instructions. To address the neglect of language instructions, we propose a novel framework that grounds the translation of visual information on instructions for language models. We introduce a mixture of visual projectors, each serving as a specialized visual-to-language translation expert based on the given instruction context to adapt to new tasks. To avoid using experts for irrelevant instruction contexts, we propose an expert recommendation strategy that reuses experts for tasks similar to those previously learned. Additionally, we introduce expert pruning to alleviate interference from the use of experts that cumulatively activated in previous tasks. Extensive experiments on diverse vision-language tasks demonstrate that our method outperforms existing continual learning approaches by generating instruction-following responses.

In robot task planning, large language models (LLMs) have shown significant promise in generating complex and long-horizon action sequences. However, it is observed that LLMs often produce responses that sound plausible but are not accurate. To address these problems, existing methods typically employ predefined error sets or external knowledge sources, requiring human efforts and computation resources. Recently, self-correction approaches have emerged, where LLM generates and refines plans, identifying errors by itself. Despite their effectiveness, they are more prone to failures in correction due to insufficient reasoning. In this paper, we introduce InversePrompt, a novel self-corrective task planning approach that leverages inverse prompting to enhance interpretability. Our method incorporates reasoning steps to provide clear, interpretable feedback. It generates inverse actions corresponding to the initially generated actions and verifies whether these inverse actions can restore the system to its original state, explicitly validating the logical coherence of the generated plans. The results on benchmark datasets show an average 16.3% higher success rate over existing LLM-based task planning methods. Our approach offers clearer justifications for feedback in real-world environments, resulting in more successful task completion than existing self-correction approaches across various scenarios.

Neural radiance fields (NeRF) represent multi-view images as 3D scenes, achieving a photo-realistic novel view synthesis quality. However, capturing multi-view images in real-world scenarios is not well aligned and often results in blur or noise. Deblur-NeRF, which uses kernel deformation to improve sharpness, is effective but the quantity of training blur samples and imbalance significantly affect the overall results. In this study, we propose neural radiance fields deblurring with active learning (NeRF-DA), focusing on high-quality blurred images for 3D scene modeling. NeRF-DA uses pool-based active learning with uncertainty estimation to improve model efficiency with a high-quality training set. Subsequently, we deblur the data using the trained model and proceed with NeRF training by selecting the best-sharpened images for querying. Experiments on both camera motion blur and defocus blur demonstrate that NeRF-DA significantly enhances the quality of the existing Deblur-NeRF.

High Dynamic Range (HDR) imaging seeks to enhance image quality by combining multiple Low Dynamic Range (LDR) images captured at varying exposure levels. Traditional deep learning approaches often employ reconstruction loss, but this method can lead to ambiguities in feature space during training. To address this issue, we present a new loss function, termed Gravitated Latent Space (GLS) loss, that leverages a metric tensor to introduce a form of virtual gravity within the latent space. This feature helps the model in overcoming saddle points more effectively. Easy to integrate, the GLS loss function fosters stable learning within a convex environment and demonstrates its performance in improving HDR image quality. Experimental data confirms that the proposed method outperforms existing state-of-the-art techniques in quantitative evaluations.

In the field of robot manipulation, learnable task planners are gaining attention, especially for long-horizon tasks such as cooking. However, existing methods that predominantly rely on symbolic representations suffer from limitations in generalization capabilities, particularly in handling unseen objects. Given that objects may vary in real-world environments, this limitation may constrain their practical applicability. To address this issue, we propose a novel task-planning framework that leverages a pretrained large language model (LLM) for environmental interpretation. Our proposed framework extracts semantic features directly from textual data, enabling the planner to accommodate unfamiliar objects. We further incorporate a transformer-based encoder-decoder framework to understand environmental attributes derived from the language model and generate sequential predictions in line with object-oriented subgoals. To validate the effectiveness of our model, we utilize a dataset focused on cooking recipes. Going a step further, we propose a method that automatically generates object-oriented data from natural language description using recurrent LLM, enhancing the framework to manage previously unseen targets as well. Our framework shows an average success rate of 95% when validated with test sets that involve unseen objects. By providing the automatically generated dataset to the framework, we achieve a significant 27% increase in success rate on unknown target recipes. We also provide evidence of the real-world viability of our planner by successfully deploying it on a robot platform.

In the era of data-driven technological advancements, deep learning still craves more training data. However, the high cost of data annotation and limited budgets pose significant challenges. To address this issue, active learning (AL) has emerged, and it gradually adds some informative samples to the training data by querying humans for annotation. Existing works have mainly focused on how to sample useful data based on the estimations of the model in the current cycle (time). However, models in different cycles have distinct knowledge and biases by training with the expanding dataset. Also, relying solely on a present bias is not necessarily the best choice in the real world, where the knowledge of the present model is distorted by labeling attacks or mislabeling situations. Here, we propose a novel AL approach that expands viewpoint and knowledge with the strong committee having long-range observation for seeking informative data. The committee is designed to reflect estimations of all previous and current models and sample data points by selectively aggregating model estimations. By exploiting various trajectories of multiple models and broadening knowledge, it can overcome limited perspectives and potential shortcomings of the current estimation. We validate the proposed approach under ideal and realistic scenarios with coarse-grained and fined-grained image classification tasks. In experimental results, the proposed method outperforms recent competitive methods for six settings, including realistic and ideal scenarios.

Neural Architecture Search (NAS) has emerged as a promising tool in the field of AutoML for designing more accurate and efficient architectures. The majority of NAS works employ a weight-sharing technique to reduce the search cost by sharing the weights of a supernet, which is a composite of all architectures produced from the search space. Nonetheless, this method has a significant drawback in that negative interference may arise when candidate architectures share the same weights. This issue becomes even more severe in multi-task searches, where a supernet is shared across tasks. To address this problem, we propose a task-aware nested search for multiple tasks that generates task-specific search spaces and architectures using a search-in-search approach consisting of space-search and architecture-search phases. In the space-search phase, we discover an optimal subspace in a task-aware manner by utilizing the proposed search space generator based on the global search space. On top of each subspace, we search for a promising architecture in the architecture-search phase. This method can mitigate search interference by adaptively sharing weights of the supernet by the generated subspace. The experimental results on various vision benchmarks (CityScapes, NYUv2, and Tiny-Taskonomy) show that the proposed method achieves outstanding performance over existing methods in terms of task accuracy, model parameters, and latency.

Deep neural networks suffer from catastrophic forgetting in continual learning, where they tend to lose information about previously learned tasks when optimizing a new incoming task. Recent strategies isolate the important parameters for previous tasks to retain old knowledge while learning the new task. However, using the fixed old knowledge might act as an obstacle to capturing novel representations. To overcome this limitation, we propose a framework that evolves the previously allocated parameters by absorbing the knowledge of the new task. The approach performs under two different networks. The base network learns knowledge of sequential tasks, and the sparsity-inducing hyper-network generates parameters for each time step for evolving old knowledge. The generated parameters transform old parameters of the base network to reflect the new knowledge. We design the hypernetwork to generate sparse parameters conditional to the task-specific information and the structural information of the base network. We evaluate the proposed approach on class-incremental and task-incremental learning scenarios for image classification and video action recognition tasks. Experimental results show that the proposed method consistently outperforms a large variety of continual learning approaches for those scenarios by evolving old knowledge.

Deep networks consume a large amount of memory by their nature. A natural question arises can we reduce that memory requirement whilst maintaining performance. In particular, in this work we address the problem of memory efficient learning for multiple tasks. To this end, we propose a novel network architecture producing multiple networks of different configurations, termed deep virtual networks (DVNs), for different tasks. Each DVN is specialized for a single task and structured hierarchically. The hierarchical structure, which contains multiple levels of hierarchy corresponding to different numbers of parameters, enables multiple inference for different memory budgets. The building block of a deep virtual network is based on a disjoint collection of parameters of a network, which we call a unit. The lowest level of hierarchy in a deep virtual network is a unit, and higher levels of hierarchy contain lower levels' units and other additional units. Given a budget on the number of parameters, a different level of a deep virtual network can be chosen to perform the task. A unit can be shared by different DVNs, allowing multiple DVNs in a single network. In addition, shared units provide assistance to the target task with additional knowledge learned from another tasks. This cooperative configuration of DVNs makes it possible to handle different tasks in a memory-aware manner. Our experiments show that the proposed method outperforms existing approaches for multiple tasks. Notably, ours is more efficient than others as it allows memory-aware inference for all tasks.

In this work, we consider the problem of instance-wise dynamic network model selection for multi-task learning. To this end, we propose an efficient approach to exploit a compact but accurate model in a backbone architecture for each instance of all tasks. The proposed method consists of an estimator and a selector. The estimator is based on a backbone architecture and structured hierarchically. It can produce multiple different network models of different configurations in a hierarchical structure. The selector chooses a model dynamically from a pool of candidate models given an input instance. The selector is a relatively small-size network consisting of a few layers, which estimates a probability distribution over the candidate models when an input instance of a task is given. Both estimator and selector are jointly trained in a unified learning framework in conjunction with a sampling-based learning strategy, without additional computation steps. We demonstrate the proposed approach for several image classification tasks compared to existing approaches performing model selection or learning multiple tasks. Experimental results show that our approach gives not only outstanding performance compared to other competitors but also the versatility to perform instance-wise model selection for multiple tasks.

Recently, there have been increasing demands to construct compact deep architectures to remove unnecessary redundancy and to improve the inference speed. While many recent works focus on reducing the redundancy by eliminating unneeded weight parameters, it is not possible to apply a single deep network for multiple devices with different resources. When a new device or circumstantial condition requires a new deep architecture, it is necessary to construct and train a new network from scratch. In this work, we propose a novel deep learning framework, called a nested sparse network, which exploits an n-in-1-type nested structure in a neural network. A nested sparse network consists of multiple levels of networks with a different sparsity ratio associated with each level, and higher level networks share parameters with lower level networks to enable stable nested learning. The proposed framework realizes a resource-aware versatile architecture as the same network can meet diverse resource requirements, i.e., anytime property. Moreover, the proposed nested network can learn different forms of knowledge in its internal networks at different levels, enabling multiple tasks using a single network, such as coarse-to-fine hierarchical classification. In order to train the proposed nested network, we propose efficient weight connection learning and channel and layer scheduling strategies. We evaluate our network in multiple tasks, including adaptive deep compression, knowledge distillation, and learning class hierarchy, and demonstrate that nested sparse networks perform competitively, but more efficiently, compared to existing methods.

Recently, finding the low-dimensional structure of high-dimensional data has gained much attention. Given a set of data points sampled from a single subspace or a union of subspaces, the goal is to learn or capture the underlying subspace structure of the data set. In this paper, we propose elastic-net subspace representation, a new subspace representation framework using elastic-net regularization of singular values. Due to the strong convexity enforced by elastic-net, the proposed method is more stable and robust in the presence of heavy corruptions compared with existing lasso-type rank minimization approaches. For discovering a single low-dimensional subspace, we propose a computationally efficient low-rank factorization algorithm, called FactEN, using a property of the nuclear norm and the augmented Lagrangian method. Then, ClustEN is proposed to handle the general case, in which the data samples are drawn from a union of multiple subspaces, for joint subspace clustering and estimation. The proposed algorithms are applied to a number of subspace representation problems to evaluate the robustness and efficiency under various noisy conditions, and experimental results show the benefits of the proposed method compared with existing methods.

Learning a low-dimensional structure plays an important role in computer vision. Recently, a new family of methods, such as l1 minimization and robust principal component analysis, has been proposed for low-rank matrix approximation problems and shown to be robust against outliers and missing data. But these methods often require heavy computational load and can fail to find a solution when highly corrupted data are presented. In this paper, an elastic-net regularization based low-rank matrix factorization method for subspace learning is proposed. The proposed method finds a robust solution efficiently by enforcing a strong convex constraint to improve the algorithm's stability while maintaining the low-rank property of the solution. It is shown that any stationary point of the proposed algorithm satisfies the Karush-Kuhn-Tucker optimality conditions. The proposed method is applied to a number of low-rank matrix approximation problems to demonstrate its efficiency in the presence of heavy corruptions and to show its effectiveness and robustness compared to the existing methods.

In this paper, we propose a novel regression method that can incorporate both positive and negative training data into a single regression framework. In detail, a leveraged kernel function for non-stationary Gaussian process regression is proposed. With this new kernel function, we can vary the correlation betwen two inputs in both positive and negative directions by adjusting leverage parameters. By using this property, the resulting leveraged non-stationary Gaussian process regression can anchor the regressor to the positive data while avoiding the negative data. We first prove the positive semi-definiteness of the leveraged kernel function using Bochner's theorem. Then, we apply the leveraged non-stationary Gaussian process regression to a real-time motion control problem. In this case, the positive data refer to what to do and the negative data indicate what not to do. The results show that the controller using both positive and negative data outperforms the controller using positive data only in terms of the collision rate given training sets of the same size.

This paper considers the problem of approximating a kernel matrix in an autoregressive Gaussian process regression (AR-GP) in the presence of measurement noises or natural errors for modeling complex motions of pedestrians in a crowded environment. While a number of methods have been proposed to robustly predict future motions of humans, it still remains as a difficult problem in the presence of measurement noises. This paper addresses this issue by proposing a structured low-rank matrix approximation method using nuclear-norm regularized l1-norm minimization in AR-GP for robust motion prediction of dynamic obstacles. The proposed method approximates a kernel matrix by finding an orthogonal basis using low-rank symmetric positive semi-definite matrix approximation assuming that a kernel matrix can be well represented by a small number of dominating basis vectors. The proposed method is suitable for predicting the motion of a pedestrian, such that it can be used for safe autonomous robot navigation in a crowded environment. The proposed method is applied to well-known regression and motion prediction problems to demonstrate its robustness and excellent performance compared to existing approaches.

Low-rank matrix approximation plays an important role in the area of computer vision and image processing. Most of the conventional low-rank matrix approximation methods are based on the l2 -norm (Frobenius norm) with principal component analysis (PCA) being the most popular among them. However, this can give a poor approximation for data contaminated by outliers (including missing data), because the l2 -norm exaggerates the negative effect of outliers. Recently, to overcome this problem, various methods based on the l1 -norm, such as robust PCA methods, have been proposed for low-rank matrix approximation. Despite the robustness of the methods, they require heavy computational effort and substantial memory for high-dimensional data, which is impractical for real-world problems. In this paper, we propose two efficient low-rank factorization methods based on the l1 -norm that find proper projection and coefficient matrices using the alternating rectified gradient method. The proposed methods are applied to a number of low-rank matrix approximation problems to demonstrate their efficiency and robustness. The experimental results show that our proposals are efficient in both execution time and reconstruction performance unlike other state-of-the-art methods.

In this paper, we propose a novel Gaussian process motion controller that can navigate through a crowded dynamic environment. The proposed motion controller predicts future trajectories of pedestrians using an autoregressive Gaussian process motion model (AR-GPMM) from the partially-observable egocentric view of a robot and controls a robot using an autoregressive Gaussian process motion controller (AR-GPMC) based on predicted pedestrian trajectories. The performance of the proposed method is extensively evaluated in simulation and validated experimentally using a Pioneer 3DX mobile robot with a Microsoft Kinect sensor. In particular, the proposed method shows over 68% improvement on the collision rate compared to a reactive planner and vector field histogram (VFH).

This paper considers the problem of modeling complex motions of pedestrians in a crowded environment. A number of methods have been proposed to predict the motion of a pedestrian or an object. However, it is still difficult to make a good prediction due to challenges, such as the complexity of pedestrian motions and outliers in a training set. This paper addresses these issues by proposing a robust autoregressive motion model based on Gaussian process regression using l1-norm based low-rank kernel matrix approximation, called PCGP-l1. The proposed method approximates a kernel matrix assuming that the kernel matrix can be well represented using a small number of dominating principal components, eliminating erroneous data. The proposed motion model is robust against outliers present in a training set and can reliably predict the motion of a pedestrian, such that it can be used by a robot for safe navigation in a crowded environment. The proposed method is applied to a number of regression and motion prediction problems to demonstrate its robustness and efficiency. The experimental results show that the proposed method considerably improves the motion prediction rate compared to other Gaussian process regression methods.

There is a growing interest in smart homes and predicting behaviors of inhabitants is a key element for the success of smart home services. In this paper, we propose two algorithms, DBN-ANN and DBN-R, based on the deep learning framework for predicting various activities in a home. We also address drawbacks of contrastive divergence, a widely used learning method for restricted Boltzmann machines, and propose an efficient online learning algorithm based on bootstrapping. From experiments using home activity datasets, we show that our proposed prediction algorithms outperform existing methods, such as a nonlinear SVM and k-means, in terms of prediction accuracy of newly activated sensors. In particular, DBN-R shows an accuracy of 43.9% (51.8%) for predicting newly activated sensors based on MIT home dataset 1 (dataset 2), while previous work based on the n-gram algorithm has shown an accuracy of 39% (43%) on the same dataset.