Previous methods evaluate reward models by testing them on a fixed pairwise ranking test set, but they typically do not provide performance information on each preference dimension. In this work, we address the evaluation challenge of reward models by probing preference representations. To confirm the effectiveness of this evaluation method, we construct a Multi-dimensional Reward Model Benchmark (MRMBench), a collection of six probing tasks for different preference dimensions. We design it to favor and encourage reward models that better capture preferences across different dimensions. Furthermore, we introduce an analysis method, inference-time probing, which identifies the dimensions used during the reward prediction and enhances its interpretability. Through extensive experiments, we find that MRMBench strongly correlates with LLM alignment performance, supporting it as a reliable reference for developing advanced reward models. By analyzing the evaluation results on MRMBench, we reveal that reward models struggle to simultaneously capture preferences across multiple dimensions, highlighting the potential of multi-objective optimization in reward modeling. Furthermore, our results demonstrate that the proposed inference-time probing method provides a reliable metric for assessing the confidence of reward predictions, leading to improved alignment of large language models.

Chen Xu, Xiaoqian Liu, Xiaowen Liu, Qingxuan Sun, Yuhao Zhang, Murun Yang, Qianqian Dong, Tom Ko, Mingxuan Wang, Tong Xiao, Anxiang Ma, Jingbo Zhu. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Previous work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is on the quality of vision models.In this work, we investigate the impact of vision models on MMT.Given the fact that Transformer is becoming popular in computer vision, we experiment with various strong models (such as Vision Transformer) and enhanced features (such as object-detection and image captioning).We develop a selective attention model to study the patch-level contribution of an image in MMT.On detailed probing tasks, we find that stronger vision models are helpful for learning translation from the visual modality.Our results also suggest the need of carefully examining MMT models, especially when current benchmarks are small-scale and biased.Our code could be found at https: //github.com/libeineu/fairseq_mmt.

Because of load fluctuating, the performance of service-based system(SBS) in the cloud environment may deviate from service level agreement(SLA). In the cloud environment, it is important to dynamically allocate resource for SBS according to the predicted system load, so as to satisfy global SLA constraint and minimize resource cost. By the analysis of complex business logic in SBS and the feature of dynamic resource allocation problem, this paper models the dynamic resource allocation problem as composite optimization problem and proposes the cost optimization oriented dynamic resource allocation model. Then this paper applies genetic algorithm to solve the dynamic resource allocation model so as to improve the resolving efficiency. Finally, the approach proposed in this paper is evaluated and compared with some related algorithms. It reveals very encouraging results in terms of the quality of resource allocation.

How to classify automatically Web information that grows explosive is becoming an imminent problem needed to be resolved. Based on site structure, we propose, in this paper, a new mechanism of automatic classification of Web information, which downloads Web pages within a Web site, records the hyperlinks among Web pages, catches the site structure, extracts the classifying system of the site itself, and then links categorizing information with the correspondent position in the site structure. Therefore automatic classification of Web information can be realized through matching the positions of categorizing information with the positions of Web pages. Experiments show that such classification based on site structure works more accurately and efficiently.