Large language models (LLMs) have shown impressive capabilities across various natural language tasks. However, evaluating their alignment with human preferences remains a challenge. To this end, we propose a comprehensive human evaluation framework to assess LLMs’ proficiency in following instructions on diverse real-world tasks. We construct a hierarchical task tree encompassing seven major areas covering over 200 categories and over 800 tasks, which covers diverse capabilities such as question answering, reasoning, multi-turn dialogue, and text generation, to evaluate LLMs in a comprehensive and in-depth manner. We also design detailed evaluation standards and processes to facilitate consistent, unbiased judgments from human evaluators. A test set of over 3,000 instances is released, spanning different difficulty levels and knowledge domains. Our work provides a standardized methodology to evaluate human alignment in LLMs for both English and Chinese. We also analyze the feasibility of automating parts of evaluation with a strong LLM (GPT-4). Our framework supports a thorough assessment of LLMs as they are integrated into real-world applications. We have made publicly available the task tree, TencentLLMEval dataset, and evaluation methodology which have been demonstrated as effective in assessing the performance of Tencent Hunyuan LLMs. By doing so, we aim to facilitate the benchmarking of advances in the development of safe and human-aligned LLMs.

Most commercial industrial recommender systems have built their closed feedback loops. Though it is helpful in item recommendation and model training, the closed feedback loop may lead to the so-called bias problems, including the position bias, selection bias and popularity bias. The recommendation models trained with biased may hurt the user experiences by recommending homogenous items. How to control the biases in the closed feedback loop has become one of major challenges in modern recommender systems. This talk discusses the counterfactual learning technologies for tackling the bias problem in recommendation.

Recommender system is one of the most successful machine learning technologies for commerce. However, it can reinforce the closed feedback loop problem, where the recommender system generates items to users, then the further recommendation model is trained with the data that users' feedback to the items. Such self-reinforcing pattern can cause data bias problems. There are several debiasing methods, inverse-propensity-scoring (IPS) is a practical one for industry product. Since it is relatively easy to reweight training samples, and ameliorate the distribution shift problem. However,because of deterministic policy problem and confoundings in real-world data, it is hard to predict propensity score accurately. Inspired by the sample reweight work for robust deep learning, we propose a novel influence function based method for recommendation modeling, and analyze how the influence function corrects the bias. In the experiments, our proposed method achieves better performance against the state-of-the-art approaches.

Classifier combination is an effective method to improve the recognition accuracy of a biometric system. It has been applied to many practical biometric systems and achieved excellent performance. However, there is little literature involving theoretical analysis on the effectiveness of classifier combination. In this paper, we investigate classifiers combined with the max and min rules. In particular, we compute the recognition performance of each combined classifier, and illustrate the condition in which the combined classifier outperforms the original unimodal classifier. We focus our study on personal verification, where the input pattern is classified into one of two categories, the genuine or the impostor. For simplicity, we further assume that the matching score produced by the original classifier follows a normal distribution and the outputs of different classifiers are independent and identically distributed. Randomly-generated data are employed to test our conclusion. The influence of finite samples is explored at the same time. Moreover, an iris recognition system, which adopts multiple snapshots to identify a subject, is introduced as a practical application of the above discussions.