Vision-and-Language Models (VLMs) have shown impressive capabilities on single-turn benchmarks, yet real-world applications often demand more intricate multi-turn dialogues. Existing multi-turn datasets (e.g, MMDU, ConvBench) only partially capture the breadth and depth of conversational scenarios encountered by users. In this work, we introduce MultiVerse, a novel multi-turn conversation benchmark featuring 647 dialogues - each averaging four turns - derived from a diverse set of 12 popular VLM evaluation benchmarks. With 484 tasks and 484 interaction goals, MultiVerse covers a wide range of topics, from factual knowledge and perception to advanced reasoning tasks such as mathematics and coding. To facilitate robust assessment, we propose a checklist-based evaluation method that leverages GPT-4o as the automated evaluator, measuring performance across 37 key aspects, including perceptual accuracy, linguistic clarity, and factual correctness. We evaluate 18 VLMs on MultiVerse, revealing that even the strongest models (e.g., GPT-4o) achieve only a 50% success rate in complex multi-turn conversations, highlighting the dataset's challenging nature. Notably, we find that providing full dialogue context significantly enhances performance for smaller or weaker models, emphasizing the importance of in-context learning. We believe MultiVerse is a landscape of evaluating multi-turn interaction abilities for VLMs.

Young-Jun Lee, Byungsoo Ko, Han-Gyu Kim, Jonghwan Hyeon, Ho-Jin Choi. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Anomaly segmentation on the urban landscape scene is an important task in autonomous driving. This process exploits a pre-trained semantic segmentation network to estimate anomalous regions. Anomaly segmentation approaches implemented with extra requirements such as out-of-domain data, extra network, or network retraining might increase the computational cost or degradation of segmentation performance. In this study, to exploit information from the segmentation network for more robust anomaly segmentation, we propose the use of pixel-wise logit variance, which tends to be small for anomalies as network outputs even logits without confidence. Additionally iterative background highlighting is proposed to robustly detect anomalous objects on the background, which is implemented by feeding the logits back into the linear classifier of the network. We achieved state-of-the-art performance among anomaly segmentation approaches without extra requirements, reaching relative average precision improvements of 21.7% on the Fishyscapes Lost&Found and 17.4% on the Fishyscapes Static compared to the state-of-the-art method. The code of this work is available at our Github repository (https://github.com/hagg30/LogitVar).

One of the main purposes of deep metric learning is to construct an embedding space that has well-generalized embeddings on both seen (training) classes and unseen (test) classes. Most existing works have tried to achieve this using different types of metric objectives and hard sample mining strategies with given training data. However, learning with only the training data can be overfitted to the seen classes, leading to the lack of generalization capability on unseen classes. To address this problem, we propose a simple regularizer called Proxy Synthesis that exploits synthetic classes for stronger generalization in deep metric learning. The proposed method generates synthetic embeddings and proxies that work as synthetic classes, and they mimic unseen classes when computing proxy-based losses. Proxy Synthesis derives an embedding space considering class relations and smooth decision boundaries for robustness on unseen classes. Our method is applicable to any proxy-based losses, including softmax and its variants. Extensive experiments on four famous benchmarks in image retrieval tasks demonstrate that Proxy Synthesis significantly boosts the performance of proxy-based losses and achieves state-of-the-art performance. Our implementation is available at github.com/navervision/proxy-synthesis.

The core of deep metric learning (DML) involves learning visual similarities in high-dimensional embedding space. One of the main challenges is to generalize from seen classes of training data to unseen classes of test data. Recent works have focused on exploiting past embeddings to increase the number of instances for the seen classes. Such methods achieve performance improvement via augmentation, while the strong focus on seen classes still remains. This can be undesirable for DML, where training and test data exhibit entirely different classes. In this work, we present a novel training strategy for DML called MemVir. Unlike previous works, MemVir memorizes both embedding features and class weights to utilize them as additional virtual classes. The exploitation of virtual classes not only utilizes augmented information for training but also alleviates a strong focus on seen classes for better generalization. Moreover, we embed the idea of curriculum learning by slowly adding virtual classes for a gradual increase in learning difficulty, which improves the learning stability as well as the final performance. MemVir can be easily applied to many existing loss functions without any modification. Extensive experimental results on famous benchmarks demonstrate the superiority of MemVir over state-of-the-art competitors. Code of MemVir is publicly available1.

In this paper, we improve the low-rank matrix completion algorithm by assuming that the data points lie in a union of low dimensional subspaces. We applied the self-expressiveness, which is a property of a dataset when the data points lie in a union of low dimensional subspaces, to the low-rank matrix completion. By considering self-expressiveness of low dimensional subspaces, the proposed low-rank matrix completion may perform well even with little information, leading to the robust completion on a dataset with high missing rate. In our experiments on movie rating datasets, the proposed model outperforms state-of-the-art matrix completion models. In clustering experiments conducted on MNIST dataset, the result indicates that our method closely recovers the subspaces of original dataset even with the high missing rate.

This paper proposes the rehabilitation treatment coach robot which will help at-home patients do their rehabilitation exercises at home without any professional trainers. The coach robot is designed to be cheap enough for patients to afford it. The robot suggests the rehabilitation program and corrects the posture of the patients during the exercise. The deep neural network is used for posture correction. Besides, the voice interface is applied for convenient interaction between robot and patients during the exercise. The emergency detection module is adopted which will inform doctors when emergency happens on patients. The emergency detection will be implemented using deep neural network on voice input and video input simultaneously. The detailed data collection plan for training deep neural network and performance evaluation plan are also provided in the paper.

The deep neural network (DNN) that models characteristics of general blood test (GBT) results was used in clinical opinions generation. The DNN that generates clinical opinions has the complex structure, which causes overfitting problem. The relatively small size of medical dataset also contributes to the occurrence of overfitting. In order to deal with overfitting, we apply two techniques that solve overfitting of DNN, which are dropout, and batch normalization. Dropout is inserted into the network in various ways in order to find out the optimal structure of the network. Batch normalization is also added in various ways for the same purpose. The experiment conducted on GBT dataset shows that DNNs with dropout and batch normalization outperform the simple DNN in generating clinical opinions for our GBT dataset. Besides, dropout shows slightly better performance compared to batch normalization.

Dropout is a technique widely used for preventing overfitting while training deep neural networks. However, applying dropout to a neural network typically increases the training time. This paper proposes a different dropout approach called controlled dropout that improves training speed by dropping units in a column-wise or row-wise manner on the matrices. In controlled dropout, a network is trained using compressed matrices of smaller size, which results in notable improvement of training speed. In the experiment on feed-forward neural networks for MNIST data set and convolutional neural networks for CIFAR-10 and SVHN data sets, our proposed method achieves faster training speed than conventional methods both on CPU and GPU, while exhibiting the same regularization performance as conventional dropout. Moreover, the improvement of training speed increases when the number of fully-connected layers increases. As the training process of neural network is an iterative process comprising forward propagation and backpropagation, speed improvement using controlled dropout would provide a significantly decreased training time.

We propose a pitch track correction technique using a sigmoidal objective function in a particle filtering framework. The conventional method for pitch track correction simply considers the peak locations of the autocorrelation functions as the pitch values and depends only on the longest reliable pitch streak. This constraint may induce pitch correction errors. The proposed approach uses particle filtering to further correctly track the pitch values. To apply the particle filtering algorithm to pitch track correction, a method for the importance weight computation using a sigmoidal function on foreground streams is proposed. The similarity between the the estimated pitch values and the subband signals are computed, and a sigmoid transfer function is used to convert it as a probability value. To verify the efficiency of our proposed approach, we carried out speech segregation experiments for mixtures of speech and various noise sources in various mixing signal-to-noise ratios (SNRs). With respect to several performance measures including SNR, energy loss ratio, and noise residue ratio of the segregated speech, the proposed method achieved superior performance compared to the conventional approach.