Post-training quantization (PTQ) is a promising solution for deploying large language models (LLMs) on resource-constrained devices. Early methods developed for small-scale networks, such as ResNet, rely on gradient-based optimization, which becomes impractical for hyper-scale LLMs with billions of parameters. While recently proposed backpropagation-free or transformation-based methods alleviate this issue, they ignore inter-layer interactions or use the naive nearest-rounding-based quantized weight assignment to save the heavy computational cost of weight optimization. In this paper, we introduce a novel backpropagation-free PTQ algorithm that optimizes quantized weights by considering inter-layer dependencies. The key innovation is the development of attention-aware Hessian matrices that capture inter-layer interactions within the attention module. Extensive experiments demonstrate that our approach not only outperforms existing weight quantization methods but also shows good synergy with conventional methods to suppress activation outliers, leading to state-of-the-art weight-activation quantization performance. The code will be available at https://github.com/SamsungLabs/BoA.

With the increasing complexity of generative AI models, post-training quantization (PTQ) has emerged as a promising solution for deploying hyper-scale models on edge devices such as mobile and TVs. Existing PTQ schemes, however, consume considerable time and resources, which could be a bottleneck in real situations where frequent model updates and multiple hyperparameter tunings are required. As a cost-effective alternative, learning-free PTQ schemes have been proposed. However, the performance is somewhat limited because they cannot consider the inter-layer dependency within the attention module, which is a significant feature of Transformers. In this paper, we thus propose a novel PTQ algorithm that balances accuracy and efficiency. The key idea of the proposed algorithm called aespa is to perform quantization layer-wise for efficiency while targeting attention-wise reconstruction to consider the cross-layer dependency. Through extensive experiments on various language models and complexity analysis, we demonstrate that aespa is accurate and efficient in quantizing Transformer models. The code will be available at https: //github.com/SamsungLabs/aespa.

Generalized zero-shot learning (GZSL) is a technique to train a deep learning model to identify unseen classes using the image attribute. In this paper, we put forth a new GZSL technique exploiting Vision Transformer (ViT) to maximize the attribute-related information contained in the image feature. In ViT, the entire image region is processed without the degradation of the image resolution and the local image information is preserved in patch features. To fully enjoy the benefits of ViT, we exploit patch features as well as the CLS feature in the extraction of the attribute-related image feature. In particular, we propose a novel attention-based module, called attribute attention module (AAM), to aggregate the attribute-related information in the patch features. From extensive experiments on benchmark datasets, we demonstrate that the proposed technique outperforms the state-of-the-art GZSL approaches by a large margin.

Generalized zero-shot learning (GZSL) is a technique to train a deep learning model to identify unseen classes using the attribute. In this paper, we put forth a new GZSL technique that improves the GZSL classification performance greatly. Key idea of the proposed approach, henceforth referred to as semantic feature extraction-based GZSL (SE-GZSL), is to use the semantic feature containing only attribute-related information in learning the relationship between the image and the attribute. In doing so, we can remove the interference, if any, caused by the attribute-irrelevant information contained in the image feature. To train a network extracting the semantic feature, we present two novel loss functions, 1) mutual information-based loss to capture all the attribute-related information in the image feature and 2) similarity-based loss to remove unwanted attribute-irrelevant information. From extensive experiments using various datasets, we show that the proposed SE-GZSL technique outperforms conventional GZSL approaches by a large margin.

Recently, chatbots have been deployed in health care in various ways such as providing educational information, and monitoring and triaging symptoms. However, they can be ineffective when they are designed without a careful consideration of the cultural context of the users, especially for marginalized groups. Chatbots designed without cultural understanding may result in loss of trust and disengagement of the user. In this paper, through an interview study, we attempt to understand how chatbots can be better designed for Black American communities within the context of COVID-19. Along with the interviews, we performed design activities with 18 Black Americans that allowed them to envision and design their own chatbot to address their needs and challenges during the pandemic. We report our findings on our participants’ needs for chatbots’ roles and features, and their challenges in using chatbots. We then present design implications for future chatbot design for the Black American population.

Generalized zero-shot learning (GZSL) is a technique to train a deep learning model to identify unseen classes. Conventionally, conditional generative models have been employed to generate training data for unseen classes from the attribute. In this paper, we propose a new conditional generative model that improves the GZSL performance greatly. In a nutshell, the proposed model, called conditional Wasserstein autoencoder (CWAE), minimizes the Wasserstein distance between the real and generated image feature distributions using an encoder-decoder architecture. From the extensive experiments on various benchmark datasets, we show that the proposed CWAE outperforms conventional generative models in terms of the GZSL classification performance.

Orthogonal least squares (OLS) is a classic algorithm for sparse recovery, function approximation, and subset selection. In this paper, we analyze the performance guarantee of the OLS algorithm. Specifically, we show that OLS guarantees the exact reconstruction of any K-sparse vector in K iterations, provided that a sensing matrix has unit ℓ2-norm columns and satisfies the restricted isometry property (RIP) of order K + 1 with δK+1K= {1/√K, K = 1, 1/√(K + 1/4), K = 2, 1/√(K + 1/16), K = 3, 1/√K, K ≥ 4}. Furthermore, we show that the proposed guarantee is optimal in the sense that if δK+1≥ CK, then there exists a counterexample for which OLS fails the recovery.

In this paper, we put forth a new joint sparse recovery algorithm called signal space matching pursuit (SSMP). The key idea of the proposed SSMP algorithm is to sequentially investigate the support of jointly sparse vectors to minimize the subspace distance to the residual space. Our performance guarantee analysis indicates that SSMP accurately reconstructs any row K-sparse matrix of rank r in the full row rank scenario if the sampling matrix A satisfies krank(A) ≥ K+1, which meets the fundamental minimum requirement on A to ensure exact recovery. We also show that SSMP guarantees exact reconstruction in at most K - r + [r/L] iterations, provided that A satisfies the restricted isometry property (RIP) of order L(K - r) + r + 1 %/L with δL(K-r)+r+1[7.8K]≤ 0.155. Furthermore, we show that under a suitable RIP condition, the reconstruction error of SSMP is upper bounded by a constant multiple of the noise power, which demonstrates the robustness of SSMP to measurement noise. Finally, from extensive numerical experiments, we show that SSMP outperforms conventional joint sparse recovery algorithms both in noiseless and noisy scenarios.

In this paper, we analyze the performance guarantee of the rank aware order recursive matching pursuit (RA-ORMP) algorithm in recovering a group of jointly sparse vectors. Specifically, we show that RAORMP accurately reconstructs any group of l linearly independent jointly K-sparse vectors, provided that a sampling matrix satisfies the restricted isometry property (RIP) of order K + 1 with δK+1K+1≥ √l/K.

Location awareness, providing the ability to identify the location of sensor, machine, vehicle, and wearable device, is a rapidly growing trend of hyper-connected society and one of the key ingredients for the Internet of Things (IoT) era. In order to make a proper reaction to the collected information from things, location information of things should be available at the data center. One challenge for the IoT networks is to identify the location map of whole nodes from partially observed distance information. The aim of this paper is to present an algorithm to recover the Euclidean distance matrix (and eventually the location map) from partially observed distance information. By casting the low-rank matrix completion problem into the unconstrained minimization problem in a Riemannian manifold in which a notion of differentiability can be defined, we solve the low-rank matrix completion problem using a modified conjugate gradient algorithm. From the convergence analysis, we show that localization in Riemannian manifold using conjugate gradient (LRM-CG) converges linearly to the original Euclidean distance matrix under the extended Wolfe's conditions. From the numerical experiments, we demonstrate that the proposed method, called LRM-CG, is effective in recovering the Euclidean distance matrix.

Joint sparse recovery aims to reconstruct multiple sparse signals having a common support using multiple measurement vectors (MMV). In this paper, we propose a robust joint sparse recovery algorithm, termed MMV multiple orthogonal least squares (MMV-MOLS). Owing to the novel identification rule that fully exploits the correlation between the measurement vectors, MMV-MOLS greatly improves the accuracy of the recovered signals over the conventional joint sparse recovery techniques. From the simulation results, we show that MMV-MOLS outperforms conventional joint sparse recovery algorithms, in both full row rank and rank deficient scenarios. In our analysis, we show that MMV-MOLS recovers any row K-sparse matrix accurately in the full row rank scenario with m = K + 1 measurements, which is, in fact, the minimum number of measurements to recover a row K-sparse matrix. In addition, we analyze the performance guarantee of the MMV-MOLS algorithm in the rank deficient scenario using the restricted isometry property (RIP).