Membership Inference Attack (MIA) aims to determine if a specific sample is present in the training dataset of a target machine learning model. Previous MIAs against fine-tuned Large Language Models (LLMs) either fail to address the unique challenges in the fine-tuned setting or rely on strong assumption of the training data distribution. This paper proposes a distribution-free MIA framework tailored for fine-tuned LLMs, named DF-MIA. We recognize that samples await to test can serve as a valuable reference dataset for fine-tuning reference models. By enhancing the signals of non-member samples within this reference dataset, we can achieve a more reliable and practical calibration of probabilities, improving the differentiation between members and non-members. Leveraging these insights, we have developed a two-stage framework that employs specially designed data augmentation and perturbation techniques to prioritize the significance of non-members and mitigate the influence of potential members within the reference dataset. We evaluate our method on three representative LLM models ranging from 1B to 8B on three datasets. The results demonstrate that the DF-MIA significantly enhances the performance of MIA.

Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both.Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives.For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful.Conversely, instructiontuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding.One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome.We propose a simple yet effective method that relies on Rejection Sampling for Continued Selfinstruction Tuning (RESET), which significantly outperforms vanilla MTL.Surprisingly, we find that less is more, as training RESET with high-quality, yet substantially smaller data (three-fold less) yields superior results.Our findings offer a better understanding of objective discrepancies in alignment training of LMs.

Open-domain question answering (OpenQA) research has grown rapidly in recent years. However, OpenQA usability evaluation in its real world applications is largely left under studied. In this paper, we evaluated the actual user experience of OpenQA model deployed in a large tech company’s production enterprise search portal. From qualitative query log analysis and user interviews, our preliminary findings are: 1) There exists a large number of “contingency answers” that cannot be simply evaluated against their face textual values, due to noisy source passages and ambiguous query intents from short keywords queries. 2) Contingency answers contribute to positive search experience for providing “information scents”. 3) Click-through-rate (CTR) is a good user-behavior metric to measure OpenQA result quality, despite the rare existence of “good abandonment”. This exploratory study reveals an often neglected gap between existing OpenQA research and its search engine applications that disconnects the offline research effort with online user experience. We call for reformulating OpenQA model objective beyond answer face value and developing new dataset and metrics for better evaluation protocols.

Natural scene text images captured by handheld devices usually cause low resolution (LR) problems, thus making sub-sequent detection and recognition tasks more challenging. To address this problem, LR text images are generally super-resolution (SR) processed first. In this paper, we propose a novel low-resolution text image super-resolution method. This method adopts the residual-in-residual dense network (RRDN) to extract deeper high-frequency features than the residual dense network (RDN). Then, enhances the spatial and channel features with an attention mechanism. According to the characteristics of the text, we added gradient loss to adversarial learning. Experiments show that our method performs well in both qualitative and quantitative aspects of the latest public text image super-resolution dataset. Similarly, the proposed super-resolution method for text images of natural scenes also achieves the latest results.

Character shape reconstruction in video is challenging due to low contrast, complex backgrounds and arbitrary orientation of characters. This work proposes an Improved Ring Radius Transform (IRRT) for reconstructing impaired characters through medial axis prediction. At first, the technique proposes a novel idea based on the Tangent Vector (TV) concept that identifies each actual pair of end pixels caused by gaps in impaired character components. Next, the actual direction to predict medial axis pixels using IRRT for each pair of end pixels is proposed with a new normal vector concept. The process of prediction repeats iteratively to find all the medial axis pixels for every gap in question. Further, medial axis pixels with their radii are used to reconstruct the shapes of impaired characters. The proposed technique is tested on benchmark datasets consisting of video, natural scenes, objects and multi-lingual data to demonstrate that it reconstructs shapes well, even for heterogeneous data. Comparative studies with different binarization and character recognition methods show that the proposed technique is effective, useful and outperforms existing methods.

Generative models for Information Retrieval, where ranking of documents is viewed as the task of generating a query from a document's language model, were very successful in various IR tasks in the past.However, with the advent of modern deep neural networks, attention has shifted to discriminative ranking functions that model the semantic similarity of documents and queries instead.Recently, deep generative models such as GPT2 and BART have been shown to be excellent text generators, but their effectiveness as rankers have not been demonstrated yet.In this work, we revisit the generative framework for information retrieval and show that our generative approaches are as effective as state-of-the-art semantic similarity-based discriminative models for the answer selection task.Additionally, we demonstrate the effectiveness of unlikelihood losses for IR.

The success of self-attention in NLP has led to recent applications in end-to-end encoder-decoder architectures for speech recognition. Separately, connectionist temporal classification (CTC) has matured as an alignment-free, non-autoregressive approach to sequence transduction, either by itself or in various multitask and decoding frameworks. We propose SAN-CTC, a deep, fully self-attentional network for CTC, and show it is tractable and competitive for end-to-end speech recognition. SAN-CTC trains quickly and outperforms existing CTC models and most encoder-decoder models, with character error rates (CERs) of 4.7% in 1 day on WSJ eval92 and 2.8% in 1 week on LibriSpeech test-clean, with a fixed architecture and one GPU. Similar improvements hold for WERs after LM decoding. We motivate the architecture for speech, evaluate position and down-sampling approaches, and explore how label alphabets (character, phoneme, subword) affect attention heads and performance.

Despite rapid advances in speech recognition, current models remain brittle to superficial perturbations to their inputs. Small amounts of noise can destroy the performance of an otherwise state-of-the-art model. To harden models against background noise, practitioners often perform data augmentation, adding artificially-noised examples to the training set, carrying over the original label. In this paper, we hypothesize that a clean example and its superficially perturbed counterparts shouldn't merely map to the same class - they should map to the same representation. We propose invariant-representation-learning (IRL): At each training iteration, for each training example, we sample a noisy counterpart. We then apply a penalty term to coerce matched representations at each layer (above some chosen layer). Our key results, demonstrated on the LibriSpeech dataset are the following: (i) IRL significantly reduces character error rates (CER) on both `clean' (3.3% vs 6.5%) and `other' (11.0% vs 18.1%) test sets; (ii) on several-of-domain noise settings (different from those seen during training) IRL's benefits are even more pronounced. Careful ablations confirm that our results are not simply due to shrinking activations at the chosen layers.

While deep convolutional neural networks (CNNs) have shown a great success in single-label image classification, it is important to note that real world images generally contain multiple labels, which could correspond to different objects, scenes, actions and attributes in an image. Traditional approaches to multi-label image classification learn independent classifiers for each category and employ ranking or thresholding on the classification results. These techniques, although working well, fail to explicitly exploit the label dependencies in an image. In this paper, we utilize recurrent neural networks (RNNs) to address this problem. Combined with CNNs, the proposed CNN-RNN framework learns a joint image-label embedding to characterize the semantic label dependency as well as the image-label relevance, and it can be trained end-to-end from scratch to integrate both information in a unified framework. Experimental results on public benchmark datasets demonstrate that the proposed architecture achieves better performance than the state-of-the-art multi-label classification models.

We present an approach that exploits hierarchical Recurrent Neural Networks (RNNs) to tackle the video captioning problem, i.e., generating one or multiple sentences to describe a realistic video. Our hierarchical framework contains a sentence generator and a paragraph generator. The sentence generator produces one simple short sentence that describes a specific short video interval. It exploits both temporal-and spatial-attention mechanisms to selectively focus on visual elements during generation. The paragraph generator captures the inter-sentence dependency by taking as input the sentential embedding produced by the sentence generator, combining it with the paragraph history, and outputting the new initial state for the sentence generator. We evaluate our approach on two large-scale benchmark datasets: YouTubeClips and TACoS-MultiLevel. The experiments demonstrate that our approach significantly outperforms the current state-of-the-art methods with BLEU@4 scores 0.499 and 0.305 respectively.

In this paper, we address the task of learning novel visual concepts, and their interactions with other concepts, from a few images with sentence descriptions. Using linguistic context and visual features, our method is able to efficiently hypothesize the semantic meaning of new words and add them to its word dictionary so that they can be used to describe images which contain these novel concepts. Our method has an image captioning module based on [38] with several improvements. In particular, we propose a transposed weight sharing scheme, which not only improves performance on image captioning, but also makes the model more suitable for the novel concept learning task. We propose methods to prevent overfitting the new concepts. In addition, three novel concept datasets are constructed for this new task, and are publicly available on the project page. In the experiments, we show that our method effectively learns novel visual concepts from a few examples without disturbing the previously learned concepts. The project page is: www.stat.ucla.edu/junhua. mao/projects/child_learning.html.

In this paper, we present the mQA model, which is able to answer questions about the content of an image. The answer can be a sentence, a phrase or a single word. Our model contains four components: a Long Short-Term Memory (LSTM) to extract the question representation, a Convolutional Neural Network (CNN) to extract the visual representation, an LSTM for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer. We construct a Freestyle Multilingual Image Question Answering (FM-IQA) dataset to train and evaluate our mQA model. It contains over 150,000 images and 310,000 freestyle Chinese question-answer pairs and their English translations. The quality of the generated answers of our mQA model on this dataset is evaluated by human judges through a Turing Test. Specifically, we mix the answers provided by humans and our model. The human judges need to distinguish our model from the human. They will also provide a score (i.e. 0, 1, 2, the larger the better) indicating the quality of the answer. We propose strategies to monitor the quality of this evaluation process. The experiments show that in 64.7% of cases, the human judges cannot distinguish our model from humans. The average score is 1.454 (1.918 for human). The details of this work, including the FM-IQA dataset, can be found on the project page: \url{http://idl.baidu.com/FM-IQA.html}.

Recurrent neural network language models (RNNLMs) have recently produced improvements on language processing tasks ranging from machine translation to word tagging and speech recognition. To date, however, the computational expense of RNNLMs has hampered their application to first pass decoding. In this paper, we show that by restricting the RNNLM calls to those words that receive a reasonable score according to a n-gram model, and by deploying a set of caches, we can reduce the cost of using an RNNLM in the first pass to that of using an additional n-gram model. We compare this scheme to lattice rescoring, and find that they produce comparable results for a Bing Voice search task. The best performance results from rescoring a lattice that is itself created with a RNNLM in the first pass.

Recurrent neural network (RNN) language models have proven to be successful to lower the perplexity and word error rate in automatic speech recognition (ASR). However, one challenge to adopt RNN language models is due to their heavy computational cost in training. In this paper, we propose two techniques to accelerate RNN training: 1) two stage class RNN and 2) parallel RNN training. In experiments on Microsoft internal short message dictation (SMD) data set, two stage class RNNs and parallel RNNs not only result in equal or lower WERs compared to original RNNs but also accelerate training by 2 and 10 times respectively. It is worth noting that two stage class RNN speedup can also be applied to test stage, which is essential to reduce the latency in real time ASR applications.

The growth of detection datasets and the multiple directions of object detection research provide both an unprecedented need and a great opportunity for a thorough evaluation of the current state of the field of categorical object detection. In this paper we strive to answer two key questions. First, where are we currently as a field: what have we done right, what still needs to be improved? Second, where should we be going in designing the next generation of object detectors? Inspired by the recent work of Hoiem et al. on the standard PASCAL VOC detection dataset, we perform a large-scale study on the Image Net Large Scale Visual Recognition Challenge (ILSVRC) data. First, we quantitatively demonstrate that this dataset provides many of the same detection challenges as the PASCAL VOC. Due to its scale of 1000 object categories, ILSVRC also provides an excellent test bed for understanding the performance of detectors as a function of several key properties of the object classes. We conduct a series of analyses looking at how different detection methods perform on a number of image-level and object-class-level properties such as texture, color, deformation, and clutter. We learn important lessons of the current object detection methods and propose a number of insights for designing the next generation object detectors.

Fuzzy interpolative reasoning plays an important role in fuzzy modelling as it not only helps to reduce the number of rules in a rule base, but also provides an inference mechanism for sparse rule bases. In interpolation, it is desirable to preserve piece-wise linearity as piece-wise linear results can thus be inferred from piece-wise linear rules and observations. This safely ensures the ignorance of non-characteristic points in performing interpolations. However, almost all existing fuzzy interpolative reasoning methods do not preserve piecewise linearity for general polygonal fuzzy sets. This paper, based on the work of, proposes a new interpolative method which preserves this property.

Fuzzy classification is one of the most important applications in fuzzy set and fuzzy-logic-related research. Its goal is to find a set of fuzzy rules that form a classification model. Most of the existing fuzzy rule induction methods (e.g., the fuzzy decision trees (FDTs) induction method) focus on searching rules consisting of triangular norms (t-norms) (i.e., and) only, but not triangular conorms (t-conorms) (or) explicitly. This may lead to the omission of generating important rules that involve t-conorms explicitly. This paper proposes a type of tree termed pattern trees (PTs) that makes use of different aggregations, including both t-norms and t-conorms. Like decision trees, PTs are an effective tool for classification applications. This paper discusses the difference between decision trees and PTs, and also shows that the subsethood-based method (SBM) and the weighted-subsethood-based method (WSBM) are two specific cases of PT induction. A novel PT induction method is proposed using similarity measure and fuzzy aggregations. The comparison to other classification methods including SBM, WSBM, C4.5, nearest neighbor, support vector machine, and FDT induction shows that: 1) PTs can obtain high accuracy rates in classifications; 2) PTs are robust to overfltting; and 3) PTs, especially simple pattern trees (SPTs), maintain compact tree structures.

Fuzzy interpolation does not only help to reduce the complexity of fuzzy models, but also makes inference in sparse rule-based systems possible. It has been successfully applied to systems control, but limited work exists for its applications to tasks like prediction and classification. Almost all fuzzy interpolation techniques in the literature make strong assumptions that there are two closest adjacent rules available to the observation, and that such rules must flank the observation for each attribute. Also, some interpolation approaches cannot handle fuzzy sets whose membership functions involve vertical slopes. To avoid such limitations and develop a more practical approach, this paper extends the work of Huang and Shen. The result enables both interpolation and extrapolation which involve multiple fuzzy rules, with each rule consisting of multiple antecedents. Two realistic applications, namely truck backer-upper control and computer activity prediction, are provided in this paper to demonstrate the utility of the extended approach. Experiment-based comparisons to the most commonly used Mamdani fuzzy reasoning mechanism, and to other existing fuzzy interpolation techniques are given to show the significance and potential of this research.

This paper proposes a new type of tree termed pattern trees. Like decision trees, pattern trees are an effective tool for classification applications. This paper discusses the difference between decision trees and pattern trees, and also shows that the subsethood based method and the weighted subsethood based method are two specific cases of pattern trees. A novel pattern tree induction method is proposed. The comparison to other classification methods including fuzzy decision tree induction shows that pattern trees can obtain higher accuracy rates in classifications. In addition, pattern trees are capable of generating patterns with good generality, while decision trees can easily fall into the trap of over-fitting.

Interpolative reasoning does not only help reduce the complexity of fuzzy models but also makes inference in sparse rule-based systems possible. This paper presents an interpolative reasoning method by means of scale and move transformations. It can be used to interpolate fuzzy rules involving complex polygon, Gaussian or other bell-shaped fuzzy membership functions. The method works by first constructing a new inference rule via manipulating two given adjacent rules, and then by using scale and move transformations to convert the intermediate inference results into the final derived conclusions. This method has three advantages thanks to the proposed transformations: 1) it can handle interpolation of multiple antecedent variables with simple computation; 2) it guarantees the uniqueness as well as normality and convexity of the resulting interpolated fuzzy sets; and 3) it suggests a variety of definitions for representative values, providing a degree of freedom to meet different requirements. Comparative experimental studies are provided to demonstrate the potential of this method

Fuzzy interpolation offers the potential to model problems with sparse rule bases, as opposed to dense rule bases deployed in traditional fuzzy systems. It thus supports the simplification of complex fuzzy models and facilitates inferences when only limited knowledge is available. This paper first introduces the general concept of representative values (RVs), and then uses it to present an interpolative reasoning method which can be used to interpolate fuzzy rules involving arbitrary polygonal fuzzy sets, by means of scale and move transformations. Various interpolation results over different RV implementations are illustrated to show the flexibility and diversity of this method. A realistic application shows that the interpolation-based inference can outperform the conventional inferences.

This paper generalises the previously proposed interpolative reasoning method to cover the interpolations involving the complex polygon, Gaussian or other bell-shaped fuzzy membership functions. This can be achieved by the generality of the proposed scale and move transformations. The method works by first constructing a new inference rule via manipulating the two given adjacent rules, and then by using scale and move transformations to convert the intermediate inference results into the final derived conclusions. This generalised method has two advantages, thanks to the elegantly proposed transformations: 1) It can easily handle the interpolation of multiple antecedent variables with simple computation; and 2) It guarantees the uniqueness as well as normality and convexity of the resulting interpolated fuzzy sets. Numerical examples are provided to demonstrate the use of this method.

Interpolative reasoning methods do not only help reduce the complexity of fuzzy models but also make inference in sparse-rule based systems possible. This paper presents an interpolative reasoning method by exploiting the center of gravity (COG) property of the fuzzy sets concerned. The method works by first constructing a new inference rule via manipulating two given adjacent rules, and then by using similarity information to convert the intermediate inference results into the final derived conclusion. Two transformation operations are introduced to support such reasoning, which allow the COG of a fuzzy set to remain unaltered before and after the transformation. Results of experimental comparisons are provided to reflect the success of this work.