Existing realistic blur datasets provide insufficient variety in scenes and blur patterns to be trained, while expanding data diversity demands considerable time and effort due to complex dual-camera systems. To address the challenge, data augmentation can be an effective way to artificially increase data diversity. However, existing methods on this line are typically designed to estimate motions from a 2D perspective, e.g., estimating 2D non-uniform kernels disregarding 3D aspects of blur modeling, which leads to unrealistic motion patterns due to the fact that camera and object motions inherently arise in 3D space. In this paper, we propose a 3D-aware blur synthesizer capable of generating diverse and realistic blur images for blur data augmentation. Specifically, we estimate 3D camera positions within the motion blur interval, generate the corresponding scene images, and aggregate them to synthesize a realistic blur image. Since the 3D camera positions projected onto the 2D image plane inherently lie in 2D space, we can represent the 3D transformation as a combination of 2D transformation and projected 3D residual component. This allows for 3D transformation without requiring explicit depth measurements, as the 3D residual component is directly estimated via a neural network. Furthermore, our blur synthesizer allows for controllable blur data augmentation by modifying blur magnitude, direction, and scenes, resulting in diverse blur images. As a result, our method significantly improves deblurring performance, making it more practical for real-world scenarios.

Autofocus aims to accurately position the camera lens to bring the desired region of interest into focus. Conventional works search for the sharpest frame within the lens movement. However, sharpness measure in many real-world settings is ambiguous and may cause a focus hunting problem, where the lens continuously moves back and forth to search for the accurate position. To mitigate this problem, we introduce a simple yet powerful loss function, specifically designed to produce consistent outputs in autofocus systems. The proposed Focal Consistency Loss (FCL) allows auto-focus models to better learn the geometric cues relative to each initial position of the lens, significantly reducing distracting lens movement and enhancing the user experience when taking a photo. Furthermore, we improve autofocus stability by utilizing multiple consecutive frames in a practical way. Experimental results show the effectiveness of FCL in various practical scenarios, including multi-frame autofocus for both conventional and dual-pixel images.

As recent advances in mobile camera technology have enabled the capability to capture high-resolution images, such as 4K images, the demand for an efficient deblurring model handling large motion has increased. In this paper, we discover that the image residual errors, i.e., blur-sharp pixel differences, can be grouped into some categories according to their motion blur type and how complex their neighboring pixels are. Inspired by this, we decompose the deblurring (regression) task into blur pixel discretization (pixel-level blur classification) and discrete-to-continuous conversion (regression with blur class map) tasks. Specifically, we generate the discretized image residual errors by identifying the blur pixels and then transform them to a continuous form, which is computationally more efficient than naively solving the original regression problem with continuous values. Here, we found that the discretization result, i.e., blur segmentation map, remarkably exhibits visual similarity with the image residual errors. As a result, our efficient model shows comparable performance to state-of-the-art methods in realistic benchmarks, while our method is up to 10 times computationally more efficient.

In digital photography, autofocus is a key feature that aids high-quality image capture, and modern approaches use the phase patterns arising from dual-pixel sensors as important focus cues. However, dual-pixel data is prone to multiple error sources in its image capturing process, including lens shading or distortions due to the inherent optical characteristics of the lens. We observe that, while these degradations are hard to model using prior knowledge, they are correlated with the spatial position of the pixels within the image sensor area, and we propose a learning-based autofocus model with positional encodings (PE) to capture these patterns. Specifically, we introduce RoI-PE, which encodes the spatial position of our focusing region-of-interest (RoI) on the imaging plane. Learning with RoI-PE allows the model to be more robust to spatially-correlated degradations. In addition, we also propose to encode the current focal position of lens as lens-PE, which allows us to significantly reduce the computational complexity of the autofocus model. Experimental results clearly demonstrate the effectiveness of using the proposed position encodings for automatic focusing based on dual-pixel data.

Diffractive blur and low light levels are two fundamental challenges in producing high-quality photographs in under-display cameras (UDCs). In this paper, we incorporate phase masks on display panels to tackle both challenges. Our design inserts two phase masks, specifically two microlens arrays, in front of and behind a display panel. The first phase mask concentrates light on the locations where the display is transparent so that more light passes through the display, and the second phase mask reverts the effect of the first phase mask. We further optimize the folding height of each microlens to improve the quality of PSFs and suppress chromatic aberration. We evaluate our design using a physically-accurate simulator based on Fourier optics. The proposed design is able to double the light throughput while improving the invertibility of the PSFs. Lastly, we discuss the effect of our design on the display quality and show that implementation with polarization-dependent phase masks can leave the display quality uncompromised.

We propose an information-theoretic knowledge distillation approach for the compression of generative adversarial networks, which aims to maximize the mutual information between teacher and student networks via a variational optimization based on an energy-based model. Because the direct computation of the mutual information in continuous domains is intractable, our approach alternatively optimizes the student network by maximizing the variational lower bound of the mutual information. To achieve a tight lower bound, we introduce an energy-based model relying on a deep neural network to represent a flexible variational distribution that deals with high-dimensional images and consider spatial dependencies between pixels, effectively. Since the proposed method is a generic optimization algorithm, it can be conveniently incorporated into arbitrary generative adversarial networks and even dense prediction networks, e.g., image enhancement models. We demonstrate that the proposed algorithm achieves outstanding performance in model compression of generative adversarial networks consistently when combined with several existing models.

Under-display camera (UDC) technology is essential for full-screen display in smartphones and is achieved by removing the concept of drilling holes on display. However, this causes inevitable image degradation in the form of spatially variant blur and noise because of the opaque display in front of the camera. To address spatially variant blur and noise in UDC images, we propose a novel controllable image restoration algorithm utilizing pixel-wise UDC-specific kernel representation and a noise estimator. The kernel representation is derived from an elaborate optical model that reflects the effect of both normal and oblique light incidence. Also, noise-adaptive learning is introduced to control noise levels, which can be utilized to provide optimal results depending on the user preferences. The experiments showed that the proposed method achieved superior quantitative performance as well as higher perceptual quality on both a real-world dataset and a monitor-based aligned dataset compared to conventional image restoration algorithms.

We introduce a Large Scale Multi-Illuminant (LSMI) Dataset that contains 7,486 images, captured with three different cameras on more than 2,700 scenes with two or three illuminants. For each image in the dataset, the new dataset provides not only the pixel-wise ground truth illumination but also the chromaticity of each illuminant in the scene and the mixture ratio of illuminants per pixel. Images in our dataset are mostly captured with illuminants existing in the scene, and the ground truth illumination is computed by taking the difference between the images with different illumination combination. Therefore, our dataset captures natural composition in the real-world setting with wide field-of-view, providing more extensive dataset compared to existing datasets for multi-illumination white balance. As conventional single illuminant white balance algorithms cannot be directly applied, we also apply per-pixel DNN-based white balance algorithm and show its effectiveness against using patch-wise white balancing. We validate the benefits of our dataset through extensive analysis including a user-study, and expect the dataset to make meaningful contribution for future work in white balancing.

In this paper, an interactive medical image visualization system to support medical therapy has been introduced, where 3D organ model with the corresponding medical image is visualized interactively for diagnosis and surgical planning. To show effectiveness of the proposed system, 3D liver model generated from CT data has been utilized in consideration of its deformable characteristics by respiration as well as appearance. In addition, a hand gesture interface is applied on the graphical user interface for providing more natural and intuitive interactivity.

Six-degrees-of-freedom (6-DOF) trackers, which were mainly for professional computer applications, are now in demand by everyday consumer applications. With the requirements of consumer electronics in mind, we designed an optical 6-DOF tracker where a few photo-sensors can track the position and orientation of an LED cluster. The operating principle of the tracker is basically source localization by solving an inverse problem. We implemented a prototype system for a TV viewing environment, verified the feasibility of the operating principle, and evaluated the basic performance of the prototype system in terms of accuracy and speed. We also examined its application possibility to different environments, such as a tabletop computer, a tablet computer, and a mobile spatial interaction environment.

The independence of people who need help with daily activities will become of vital importance to all societies in the future. This paper addresses the problem of controlling the assistive home environment and emphasizes human-friendly human-machine interactions in an approach designed to achieve independence. To provide residents with an accessible, convenient, and cost-effective environment for independent living, we introduce a new service robot, categorized as a steward robot, as an intermediate agent between residents and their complex smart house environment. The learning capability and emotional interaction of the robot can make it more human-friendly in various tasks. A learning system enables the robot to provide customized services by accumulating knowledge of the user's behavioral patterns in daily activities. An emotional interaction system generates facial expressions to communicate with the user in a human-friendly manner. We have developed two types of a steward robot: a software type, which can be used everywhere via personal computing devices such as a PDA and a cellular phone, and a hardware type, which provides tangible services with physical interaction via two robotic arms and a mobile base.

To deal with data patterns with linguistic ambiguity and with probabilistic uncertainty in a single framework, we construct an interpretable probabilistic fuzzy rule-based system that requires less human intervention and less prior knowledge than other state of the art methods. Specifically, we present a new iterative fuzzy clustering algorithm that incorporates a supervisory scheme into an unsupervised fuzzy clustering process. The learning process starts in a fully unsupervised manner using fuzzy c-means (FCM) clustering algorithm and a cluster validity criterion, and then gradually constructs meaningful fuzzy partitions over the input space. The corresponding fuzzy rules with probabilities are obtained through an iterative learning process of selecting clusters with supervisory guidance based on the notions of cluster-pureness and class-separability. The proposed algorithm is tested first with synthetic data sets and benchmark data sets from the UCI Repository of Machine Learning Database and then, with real facial expression data and TV viewing data.

Recently, in service robotics area, increasing attention is being paid on interaction capability of robot for human-being as well as its task performing capability. In this paper, in particular, an automatic gesture generation methodology for conversational interaction of service robots is presented toward more human-friendly human-robot interaction. From the survey on the results in the psychology field, we first categorized the target gestures into the three types of gestures, which are basic, supplementary/emphasizing, and finishing/interconnective gestures with their corresponding unit gesture components. Then, a set of mapping rules have been extracted for gesture generation based on observing human behavioral patterns during conversation, by means of morpheme decomposition and analysis. From the given text input in Korean, the proposed system tries to generate robotic gestures, which consist of the head and the arm motions, by gesture selection and motion scheduling schemes. Finally, we discuss on the effectiveness of the proposed system with the simulated motions of robot as an initial attempt to apply in a practical system.

In this paper, we propose a new service agent, called a steward robot, which provides inhabitants with accessible, convenient, and cost effective interfaces as an intermediate agent between the user and a smart home environment. To implement more subtle emotional reaction of the agent, we adopt a novel emotional cue, sentiment relation, and address a problem of modeling the intensity and transition of emotion words, while previous researches have mainly focused on the selection of discrete emotion words from psychological point of view. We also discuss some issues of the proposed emotional model applying a virtual scenario to our Intelligent Sweet Home

Human's intention plays a key role in human-machine interaction as in the case of a robot serving for a handicapped person. The quality of a service robot will be much enhanced if the robot can infer the human's intension during the interaction process. In this paper, we propose a soft computing-based technique to read a user's intention using some multisensors-based approach. We have tested the technique by a scenario of 'serving a drink to the user'. With such force/torque or vision sensor, the robot can effectively infer the user's intention to drink the beverage or not to drink. As an application, this intention technique is employed for building a rehabilitation robot, called KARES II, to perform various human-friendly human-robot interaction.

This paper presents a novel application of fuzzy logic in the field of rehabilitation robots. This soft computing-based approach is utilized for the task of serving a drink to a person, to determine whether contact between the person's mouth and the drinking glass has occurred. The decision is based on force sensor data, which are pre-processed before being applied for decision-making. The design or decisionmaking is accomplished by means of force information as well as the first and second differential of the force. The results demonstrate that the detection algorithm is of high reliability and robustness under various environmental conditions.