This paper focuses on the challenge of machine unlearning, aiming to remove the influence of specific training data on machine learning models. Traditionally, the development of unlearning algorithms runs parallel with that of membership inference attacks (MIA), a type of privacy threat to determine whether a data instance was used for training. However, the two strands are intimately connected: one can view machine unlearning through the lens of MIA success with respect to removed data. Recognizing this connection, we propose a game-theoretic framework that integrates MIAs into the design of unlearning algorithms. Specifically, we model the unlearning problem as a Stackelberg game in which an unlearner strives to unlearn specific training data from a model, while an auditor employs MIAs to detect the traces of the ostensibly removed data. Adopting this adversarial perspective allows the utilization of new attack advancements, facilitating the design of unlearning algorithms. Our framework stands out in two ways. First, it takes an adversarial approach and proactively incorporates the attacks into the design of unlearning algorithms. Secondly, it uses implicit differentiation to obtain the gradients that limit the attacker's success, thus benefiting the process of unlearning. We present empirical results to demonstrate the effectiveness of the proposed approach for machine unlearning.

Building a conversational embodied agent to execute real-life tasks has been a long-standing yet quite challenging research goal, as it requires effective human-agent communication, multi-modal understanding, long-range sequential decision making, etc. Traditional symbolic methods have scaling and generalization issues, while end-to-end deep learning models suffer from data scarcity and high task complexity, and are often hard to explain. To benefit from both worlds, we propose JARVIS, a neuro-symbolic commonsense reasoning framework for modular, generalizable, and interpretable conversational embodied agents. First, it acquires symbolic representations by prompting large language models (LLMs) for language understanding and sub-goal planning, and by constructing semantic maps from visual observations. Then the symbolic module reasons for sub-goal planning and action generation based on task- and action-level common sense. Extensive experiments on the TEACh dataset validate the efficacy and efficiency of our JARVIS framework, which achieves state-of-the-art (SOTA) results on all three dialog-based embodied tasks, including Execution from Dialog History (EDH), Trajectory from Dialog (TfD), and Two-Agent Task Completion (TATC) (e.g., our method boosts the unseen Success Rate on EDH from 6.1% to 15.8%). Moreover, we systematically analyze the essential factors that affect the task performance and also demonstrate the superiority of our method in few-shot settings.

Deep learning for medical image classification needs large amounts of carefully labeled data with the aid of domain experts. However, data labeling is vulnerable to noises, which may degrade the accuracy of classifiers. Given the cost of medical data collection and annotation, it is highly desirable for methods that can effectively utilize noisy labeled data. In addition, efficiency and universality are essential for noisy label training, which requires further research.To address the lack of high-quality labeled medical data and meet algorithm efficiency requirements for clinical application, we propose a simple yet effective approach for multi-field medical images to utilize noisy data, named Pseudo-T correction. Specifically, we design a noisy label filter to divide the training data into clean and noisy samples. Then, we estimate a transition matrix that corrects model predictions based on the partitions of clean and noisy data samples. However, if the model overfits noisy data, noisy samples become more difficult to detect in the filtering step, resulting in inaccurate transition matrix estimation. Therefore, we employ gradient disparity as an effective criterion to decide whether or not to refine the transition matrix in the model's further training steps. The novel design enables us to build more accurate machine-learning models by leveraging noisy labels. We demonstrate that our method outperforms the state-of-the-art methods on three public medical datasets and achieves superior computational efficiency over the alternatives.

Yunchao Zhang, Zonglin Di, Kaiwen Zhou, Cihang Xie, Xin Wang. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

We propose to personalize a 2D human pose estimator given a set of test images of a person without using any manual annotations. While there is a significant advancement in human pose estimation, it is still very challenging for a model to generalize to different unknown environments and unseen persons. Instead of using a fixed model for every test case, we adapt our pose estimator during test time to exploit person-specific information. We first train our model on diverse data with both a supervised and a self-supervised pose estimation objectives jointly. We use a Transformer model to build a transformation between the self-supervised keypoints and the supervised keypoints. During test time, we personalize and adapt our model by fine-tuning with the self-supervised objective. The pose is then improved by transforming the updated self-supervised keypoints. We experiment with multiple datasets and show significant improvements on pose estimations with our self-supervised personalization. Project page with code is available at https://liyz15.github.io/TTP/.

Deep learning is revolutionizing the mapping industry. Under lightweight human curation, computer has generated almost half of the roads in Thailand on Open- StreetMap (OSM) using high resolution aerial imagery. Bing maps are displaying 125 million computer generated building polygons in the U.S. While tremendously more efficient than manual mapping, one cannot map out everything from the air. Especially for roads, a small prediction gap by image occlusion renders the entire road useless for routing. Misconnections can be more dangerous. Therefore computer-based mapping often requires local verifications, which is still labor intensive. In this paper, we propose to leverage crowdsourced GPS data to improve and support road extraction from aerial imagery. Through novel data augmentation, GPS rendering, and 1D transpose convolution techniques, we show almost 5% improvements over previous competition winning models, and much better robustness when predicting new areas without any new training data or domain adaptation.

Imbalanced classification using a support vector machine (SVM) is a normal but crucial problem in machine learning. Compared with binary classification, multiclass classification is much more complicated. Most existing studies on imbalanced classification using SVM focus on binary imbalanced classification; while only few of them look into imbalanced classification with multiple classes. Pre-clustering is a useful technique to prepare proper data from an imbalanced dataset for a classifier. It can be used to extract the feature of a dataset first and improve classification performance. Density peak based on Euclidean distance proves its effectiveness and generality in clustering. Motivated by this and the fact that the number of clusters is known in multi-class classification using a one-vs-rest strategy, we combine density peak clustering and SVM to propose a new pre-clustering method to perform effective imbalanced classification with multiple classes. Specifically, we transform a multi-class classification problem into several binary classification tasks. The results on 5 public datasets in terms of F-measure, G-mean and Area Under Curve (AUC) show its superiority over the original SVM and SVM with other methods including random under-sampling, Synthetic Minority Oversampling Technique, pre-clustering using K-Means and EasyEnsemble methods using either a one-vs-rest or one-vs-one strategy.

Imbalance classification is a common but critical problem in machine learning and artificial intelligence. Derived from structural risk minimization, a Support Vector Machine (SVM) enjoys great reputation in classification. However, the original SVM is not suitable for the imbalance classification and the existing modifications of SVM for this kind of problems fail to take the distribution of datasets into full consideration, thereby leading to some remarkable loss in their classification performance. Recently, an Adaptive Clustering by Fast Search and Find of Density Peaks (ADPclust) is proposed and performs well in finding cluster centroids in a sample space automatically and more reliably by using adaptive density peak detection and silhouette theory. Motivated by this, this work proposes an adaptive pre-clustering SVM (AP-SVM) such that the information of the original dataset distribution is well utilized to yield balanced sub-datasets for accurate and efficient classification. Specifically, AP-SVM clusters the majority into several groups given a dataset and then applies undersampling on every cluster to re-balance the dataset to be used in the SVM classification step. After experiments on 10 binary public datasets and evaluation using Area Under Curve (AUC), F-Measure, G-Mean, we well show the superiority of the proposed method over SVM, Synthetic Minority Over-sampling Technique algorithm (SMOTE), Undersampling-SVM (U-SVM), K-Means, Fuzzy C Means and EasyEnsemble.