With a surge of large-scale pre-trained models, parameter-efficient transfer learning (PETL) of large models has garnered significant attention. While promising, they commonly rely on two optimistic assumptions: 1) full access to the parameters of a PTM and 2) sufficient memory capacity to cache all intermediate activations for gradient computation. However, in most real-world applications, PTMs serve as black-box APIs or proprietary software without full parameter accessibility. Besides, it is hard to meet a large memory requirement for modern PTMs. This work proposes black-box visual prompting (BlackVIP), which efficiently adapts the PTMs without knowledge of their architectures or parameters. BlackVIP has two components: 1) Coordinator and 2) simultaneous perturbation stochastic approximation with gradient correction (SPSA-GC). The Coordinator designs input-dependent visual prompts, which allow the target PTM to adapt in the wild. SPSA-GC efficiently estimates the gradient of PTM to update Coordinator. Besides, we introduce a variant, BlackVIP-SE, which significantly reduces the runtime and computational cost of BlackVIP. Extensive experiments on 19 datasets demonstrate that BlackVIPs enable robust adaptation to diverse domains and tasks with minimal memory requirements. We further provide a theoretical analysis on the generalization of visual prompting methods by presenting their connection to the certified robustness of randomized smoothing, and presenting an empirical support for improved robustness.

Recent research increasingly integrates causal inference into deep learning models to enhance the explainability and robustness of medical applications. However, data scarcity remains a fundamental challenge due to privacy constraints and the high cost of data collection. This issue, compounded by complex variable dependencies and unobserved latent confounders, hinders the reliable estimation of causal effects. To address these challenges, we collect two real-world COVID-19 infection cluster datasets, including public health measures, from distinct distributions in collaboration with local governments, a medical university, and a hospital. We also propose a cut-off augmentation method that generates diverse feature-label pairs by slicing time-series sequences at different observation windows, effectively simulating partial observations common in real-world settings. We further introduce the Causal Effect Variational Transformer (CEVT), a Transformer-based model that captures temporal structure and addresses the difficulty of causal estimation under scarce data, complex dependencies, and latent confounding by modeling multiple treatments through an iterative conditioning mechanism. We validate the causal modeling capability of CEVT on synthetic datasets and demonstrate that, on two distinct COVID-19 datasets, it consistently outperforms baselines in infection prediction. Notably, the causal effects estimated by CEVT converge with findings from medical studies on infection control, reinforcing its reliability and underscoring its potential to inform public health decision-making.

Bayesian neural networks (BNNs) estimate the posterior distribution of model parameters and utilize posterior samples for Bayesian Model Averaging (BMA) in prediction. However, despite the crucial role of flatness in the loss landscape in improving the generalization of neural networks, its impact on BMA has been largely overlooked. In this work, we explore how posterior flatness influences BMA generalization and empirically demonstrate that \emph{(1) most approximate Bayesian inference methods fail to yield a flat posterior} and \emph{(2) BMA predictions, without considering posterior flatness, are less effective at improving generalization}. To address this, we propose Flat Posterior-aware Bayesian Model Averaging (FP-BMA), a novel training objective that explicitly encourages flat posteriors in a principled Bayesian manner. We also introduce a Flat Posterior-aware Bayesian Transfer Learning scheme that enhances generalization in downstream tasks. Empirically, we show that FP-BMA successfully captures flat posteriors, improving generalization performance.

The performance of vision-language models (VLMs), such as CLIP, in visual classification tasks, has been enhanced by leveraging semantic knowledge from large language models (LLMs), including GPT. Recent studies have shown that in zero-shot classification tasks, descriptors incorporating additional cues, high-level concepts, or even random characters often outperform those using only category names. In many classification tasks, while the top-1 accuracy may be relatively low, the top-5 accuracy is often significantly higher. This gap implies that most misclassifications occur among a few similar classes, highlighting the model's difficulty in distinguishing between classes with subtle differences. To address this challenge, we introduce a novel concept of comparative descriptors. These descriptors emphasize the unique features of a target class against its most similar classes, enhancing differentiation. By generating and integrating these comparative descriptors into the classification framework, we refine the semantic focus and improve classification accuracy. An additional filtering process ensures that these descriptors are closer to the image embeddings in the CLIP space, further enhancing performance. Our approach demonstrates improved accuracy and robustness in visual classification tasks by addressing the specific challenge of subtle inter-class differences. Code is available at ht tps: //github.com/hklee / Comparative-CLIP

With the surge of large-scale pre-trained models (PTMs), fine-tuning these models to numerous downstream tasks becomes a crucial problem. Consequently, parameter efficient transfer learning (PETL) of large models has grasped huge attention. While recent PETL methods showcase impressive performance, they rely on optimistic assumptions: 1) the entire parameter set of a PTM is available, and 2) a sufficiently large memory capacity for the fine-tuning is equipped. However, in most real-world applications, PTMs are served as a black-box API or proprietary software without explicit parameter accessibility. Besides, it is hard to meet a large memory requirement for modern PTMs. In this work, we propose black-box visual prompting (Black-VIP), which efficiently adapts the PTMs without knowledge about model architectures and parameters. Black-VIP has two components; 1) Coordinator and 2) simultaneous perturbation stochastic approximation with gradient correction (SPSA-GC). The Coordinator designs inputdependent image-shaped visual prompts, which improves few-shot adaptation and robustness on distribution/location shift. SPSA-GC efficiently estimates the gradient of a target model to update Coordinator. Extensive experiments on 16 datasets demonstrate that BlackVIP enables robust adaptation to diverse domains without accessing PTMs' parameters, with minimal memory requirements. Code: https://github.com/changdaeoh/BlackVIP

Deep neural networks (DNNs) have successfully classified EEG-based brain-computer interface (BCI) systems. However, recent studies have found that well-designed input samples, known as adversarial examples, can easily fool well-performed deep neural networks model with minor perturbations undetectable by a human. This paper proposes an efficient generative model named generative perturbation network (GPN), which can generate universal adversarial examples with the same architecture for non-targeted and targeted attacks. Furthermore, the proposed model can be efficiently extended to conditionally or simultaneously generate perturbations for various targets and victim models. Our experimental evaluation demonstrates that perturbations generated by the proposed model outperform previous approaches for crafting signal-agnostic perturbations. We demonstrate that the extended network for signal-specific methods also significantly reduces generation time while performing similarly. The transferability across classification networks of the proposed method is superior to the other methods, which shows our perturbations' high level of generality.

We present an object proposal method which utilizes the 3D data obtained from a depth sensor as well as the color information of images. Our object proposal method is designed to improve the performance of the object detection for a mobile robot equipped with a camera and a laser scanner. Compared to traditional object proposal methods using only 2D images, the proposed method provides much less number of candidate windows for object detection. We show less than 100 object proposal windows per image using the proposed method result in high recall tested on the public dataset. Our method presents object proposals in 3D space as well as in 2D image thus it can further be applied to following tasks for mobile robots such as 3D location and pose estimation of the target object after successful object detection. We validate our method using the real-world object detection dataset for outdoor mobile robots captured during the DRC Finals 2015 and the public dataset for comparison with the previous methods.

We present an outdoor photometric stereo method using images captured in a single day. We simulate a sky hemisphere for each image according to its GPS and timestamp, and parameterize the obtained sky hemisphere into a quadratic skylight and a Gaussian sunlight distribution. Unlike previous works which usually model outdoor illumination as a sum of constant ambient light and a distant point light, our method models natural illumination according to a popular sky model and thus provides sufficient constraints for shape reconstruction from one day images. We generate pixel profiles of uniformly sampled unit vectors for the corresponding time of captures and evaluate them using correlation with the actual pixel profiles. The estimated surface normal is refined by MRF optimization. We have tested our method to recover objects and scenes of various sizes in real-world outdoor daylight.

We propose a novel camera calibration method for defocused images using a smartphone under the assumption that the defocus blur is modeled as a convolution of a sharp image with a Gaussian point spread function (PSF). In contrast to existing calibration approaches which require well-focused images, the proposed method achieves accurate camera calibration with severely defocused images. This robustness to defocus is due to the proposed set of unidirectional binary patterns, which simplifies 2D Gaussian deconvolution to a 1D Gaussian deconvolution problem with multiple observations. By capturing the set of patterns consecutively displayed on a smartphone, we formulate the feature extraction as a deconvolution problem to estimate feature point locations in sub-pixel accuracy and the blur kernel in each location. We also compensate the error in camera parameters due to refraction of the glass panel of the display device. We evaluate the performance of the proposed method on synthetic and real data. Even under severe defocus, our method shows accurate camera calibration result.

We present a calibration method of a time-of-flight (ToF) sensor and a color camera pair to align the 3D measurements with the color image correctly. We have designed a 2.5D pattern board with irregularly placed holes to be accurately detected from low resolution depth images of a ToF camera as well as from high resolution color images. In order to improve the accuracy of the 3D measurements of a ToF camera, we propose to perform ray correction and range bias correction. We reset the transformation of the ToF sensor which transforms the radial distance into the scene depth in Cartesian coordinate through ray correction. Then we capture a planar scene from different depths to correct the distance error that is shown to be dependent not only on the distance but also on the pixel location. The range error profiles along the calibrated distance are classified according to their wiggling shapes and each cluster of profiles with similar shape are separately estimated using a B-spline function. The standard deviation of the remaining random noise is recorded as an uncertainty information of distance measurements. We show the performance of our calibration method quantitatively and qualitatively on various datasets, and validate the impact of our method by demonstrating an RGB-D shape refinement application.

Recently, many researchers have made efforts for accurate calibration of a Time-of-Flight camera to fully utilize its provided depth values. Yet most previous works focus mainly on intrinsic calibration by modeling its systematic errors and noises while extrinsic calibration is also an important factor when constructing sensor fusion system. In this paper, we present a calibration process that can correctly transfer the depth measurements onto the color image. We use 2.5D pattern so that sufficient reprojection error can be considered for both color and ToF cameras. The issues on obtaining the correct correspondences for this pattern are discussed. In the optimization stage, the depth constraint is also employed to ensure the depth measurements to lie on the pattern plane. The strengths of the proposed method over previous approaches are evaluated in several robotic applications which require precise ToF and camera calibration.