n the agricultural domain, the deployment of large language models (LLMs) is hindered by the lack of training data and evaluation benchmarks. To mitigate this issue, we propose AgriEval, the first comprehensive Chinese agricultural benchmark with three main characteristics: (1) Comprehensive Capability Evaluation. AgriEval covers six major agriculture categories and 29 subcategories within agriculture, addressing four core cognitive scenarios—memorization, understanding, inference, and generation. (2) High-Quality Data. The dataset is curated from university-level examinations and assignments, providing a natural and robust benchmark for assessing the capacity of LLMs to apply knowledge and make expert-like decisions. (3) Diverse Formats and Extensive Scale. AgriEval comprises 14,697 multiple-choice questions and 2,167 open-ended question-and-answer questions, establishing it as the most extensive agricultural benchmark available to date. We also present comprehensive experimental results over 51 open-source and commercial LLMs. The experimental results reveal that most existing LLMs struggle to achieve 60 percent accuracy, underscoring the developmental potential in agricultural LLMs. Additionally, we conduct extensive experiments to investigate factors influencing model performance and propose strategies for enhancement.

Multi-modal Large Language Models (MLLMs) integrating images, text, and speech can provide farmers with accurate diagnoses and treatment of pests and diseases, enhancing agricultural efficiency and sustainability. However, existing benchmarks lack comprehensive evaluations, particularly in multi-level reasoning, making it challenging to identify model limitations. To address this issue, we introduce Agri-CM^3, an expert-validated benchmark assessing MLLMs’ understanding and reasoning in agricultural management. It includes 3,939 images and 15,901 multi-level multiple-choice questions with detailed explanations. Evaluations of 45 MLLMs reveal significant gaps. Even GPT-4o achieves only 63.64% accuracy, falling short in fine-grained reasoning tasks. Analysis across three reasoning levels and seven compositional abilities highlights key challenges in accuracy and cognitive understanding. Our study provides insights for advancing MLLMs in agricultural management, driving their development and application. Code and data are available at https://github.com/HIT-Kwoo/Agri-CM3.

Multi-hop question answering faces substantial challenges due to data sparsity, which increases the likelihood of language models learning spurious patterns.To address this issue, prior research has focused on diversifying question generation through content planning and varied expression.However, these approaches often emphasize generating simple questions and neglect the integration of essential knowledge, such as relevant sentences within documents.This paper introduces the Knowledge Composition Sampling (KCS), an innovative framework designed to expand the diversity of generated multi-hop questions by sampling varied knowledge compositions within a given context.KCS models the knowledge composition selection as a sentence-level conditional prediction task and utilizes a probabilistic contrastive loss to predict the next most relevant piece of knowledge.During inference, we employ a stochastic decoding strategy to effectively balance accuracy and diversity.Compared to competitive baselines, our KCS improves the overall accuracy of knowledge composition selection by 3.9%, and its application for data augmentation yields improvements on HotpotQA and 2Wiki-MultihopQA datasets.

Dialogue System for Medical Diagnosis (DSMD) based on reinforcement learning (RL) can simulate patient-doctor interactions, playing a crucial role in clinical diagnosis. However, due to the complexity of disease etiology, DSMD faces the challenges of low efficiency in diagnostic evidence search. Moreover, solely RL-based DSMS, without the constraints of professional medical knowledge, often generates irrational, meaningless, or even erroneous symptom inquiries, leading to poor interpretability of diagnostic path and high misdiagnosis rates. To address these issues, we propose anEvidence-based dialogue system with highlyInterpretableReasoning path forAutomaticDiagnosis (EIRAD) grounded in medical knowledge graph (MKG). Specifically, our automated diagnostic model captures key symptoms for suspected diseases by explicitly leveraging the topology of MKG, enhancing the interpretability and accuracy of diagnosis. To expedite the retrieval of factual evidence, we develop two mechanisms: 1) Mapping mechanism between the entity set of MKG and DSMD's diagnostic evidence and diseases. According to the patient's symptoms, EIRAD prunes irrelevant disease and symptom nodes from the MKG, which can truncate the invalid action of RL-based DSMD. 2) Reward Mechanism of integrating the effectiveness of symptom inquiry and the accuracy of disease diagnosis. The comprehensive reward system is suitable for intelligent consultation, which can effectively drive DSMD to accelerate evidence collection. Experimental results demonstrate that our model significantly outperforms competitive benchmark methods in symptom inquiry efficiency and diagnostic accuracy.

With the rise of intelligent medical assistance, the Dialogue System for Medical Diagnosis(DSMD) guided by reinforcement learning(RL) has gained much attention. However, currently available medical dialogue datasets suffer from insufficient diagnostic evidence caused by sparse symptoms, making it difficult to reproduce the evidence-based process of doctors in differential diagnosis and disease confirmation. Moreover, purely data-driven RL often involves extensive and blind trial-and-error, leading to inquiries about irrelevant symptoms to the patient’s chief complaints in limited dialogue turns, further exacerbating the issue of inadequate diagnostic evidence. To enhance the quantity and effectiveness of potential symptom collection in DSMD, we first construct a more comprehensive medical dialogue dataset CMD based on electronic medical records. The diversity of diseases and symptoms mentioned in the dialogue context of CMD surpasses that of existing public datasets. Furthermore, to enhance the efficiency of diagnostic evidence collection in DSMD, inspired by the logic of symptom inquiries in doctor-patient interactions, we combine experiential diagnostic knowledge with a specialized medical knowledge graph to constrain the inquiry of symptoms via RL, eliminating the introduction of symptoms unrelated to the patient. Experimental results demonstrate that our model significantly outperforms competitive benchmark methods in terms of diagnostic accuracy and the efficiency of symptom inquiries. Our codes and the CMD dataset are available at https://github.com/YanPioneer/EBAD.

Classification has been commonly used in many data mining projects in the financial service industry. For instance, to predict collectability of accounts receivable, a binary class label is created based on whether a payment is received within a certain period. However, optimization of the classifier does not necessarily lead to maximization of return on investment (ROI), since maximization of the true positive rate is often different from maximization of the collectable amount which determines the ROI under a fixed budget constraint. The typical cost sensitive learning does not solve this problem either since it involves an unknown opportunity cost due to the budget constraint. Learning the ranks of collectable amount would ultimately solve the problem, but it tries to tackle an unnecessarily difficult problem and often results in poorer results for our specific target. We propose a new algorithm that uses gradient descent to directly optimize the related monetary measure under the budget constraint and thus maximizes the ROI. By comparison with several classification, regression, and ranking algorithms, we demonstrate the new algorithm's substantial improvement of the financial impact on our clients in the financial service industry.

Machine learning techniques have been used to predict customer behavior in telecommunications industry. Typically, several data sources, including historical usage, billing, payment, network, customer service, and demographic data, can be used in a predictive model. However, in some cases, e.g., in the prepaid customer segment, there is often little data available except for the call detail record (CDR) data. In this paper, we tackle this challenging problem, using significantly delayed CDR data as the primary data source to predict customer behavior. We extract calling links, i.e., who called whom, from the CDR data, and propose several distance measures based on calling links. We demonstrate that, by using information derived from the calling links alone as inputs to a neural network model, an acceptable accuracy for predicting churn (customer switching from one service provider to another) can be achieved. Calling links can also be used to identify calling communities, which may be used for targeted marketing campaigns and help predict acceptance of marketing offers.

A lift curve, with the true positive rate on the y-axis and the customer pull (or contact) rate on the x-axis, is often used to depict the model performance in many data mining applications, especially in the area of customer relationship management (CRM). Typically, these applications concern only the model accuracy at a relatively small pull or contact/intervention rate of the whole customer base, which is predetermined by a budget constraint for the project, e.g., how many customers can be contacted every month. In this paper, we address the important problem of enhancing the lift (true positive rate) at a specified pull rate. We propose two distinct algorithms, which are applicable to different scenarios. In particular, when the binary class label of the training set is extracted from a continuous variable, we can optimize a training objective which takes into account the specified pull rate rather than the class prior, based on the often ignored continuous variable. In those cases where only the binary class label is available during training, we propose a constrained optimization algorithm to maximize the true positive rate related to a specific decision threshold at which the specified pull rate is achieved. We applied both algorithms to our projects of predicting defection (decline in account value) of mutual fund accounts for two major U.S. mutual fund companies and achieved substantial enhancement of the lift at the specified pull rate.

In order to effectively use machine learning algorithms, e.g., neural networks, for the analysis of survival data, the correct treatment of censored data is crucial. The concordance index (CI) is a typical metric for quantifying the predictive ability of a survival model. We propose a new algorithm that directly uses the CI as the objective function to train a model, which predicts whether an event will eventually occur or not. Directly optimizing the CI allows the model to make complete use of the information from both censored and non-censored observations. In particular, we approximate the CI via a differentiable function so that gradient-based methods can be used to train the model. We applied the new algorithm to predict the eventual recurrence of prostate cancer following radical prostatectomy. Compared with the traditional Cox proportional hazards model and several other algorithms based on neural networks and support vector machines, our algorithm achieves a significant improvement in being able to identify high-risk and low-risk groups of patients.

We propose a new learning method for discrete space statistical classifiers. Similar to Chow and Liu (1968) and Cheeseman (1983), we cast classification/inference within the more general framework of estimating the joint probability mass function (p.m.f.) for the (feature vector, class label) pair. Cheeseman's proposal to build the maximum entropy (ME) joint p.m.f. consistent with general lower-order probability constraints is in principle powerful, allowing general dependencies between features. However, enormous learning complexity has severely limited the use of this approach. Alternative models such as Bayesian networks (BNs) require explicit determination of conditional independencies. These may be difficult to assess given limited data. Here we propose an approximate ME method, which, like previous methods, incorporates general constraints while retaining quite tractable learning. The new method restricts joint p.m.f. support during learning to a small subset of the full feature space. Classification gains are realized over dependence trees, tree-augmented naive Bayes networks, BNs trained by the Kutato algorithm, and multilayer perceptrons. Extensions to more general inference problems are indicated. We also propose a novel exact inference method when there are several missing features.

We propose a new method for learning a general statistical inference engine, operating on discrete and mixed discrete/continuous feature spaces. Such a model allows inference on any of the discrete features, given values for the remaining features. Applications are, e.g., to medical diagnosis with multiple possible diseases, fault diagnosis, information retrieval, and imputation in databases. Bayesian networks (BN's) are versatile tools that possess this inference capability. However, BN's require explicit specification of conditional independencies, which may be difficult to assess given limited data. Alternatively, Cheeseman proposed finding the maximum entropy (ME) joint probability mass function (pmf) consistent with arbitrary lower order probability constraints. This approach is in principle powerful and does not require explicit expression of conditional independence. However, until now, the huge learning complexity has severely limited the use of this approach. Here we propose an approximate ME method, which also encodes arbitrary low-order constraints but while retaining quite tractable learning. Our method uses a restriction of joint pmf support (during learning) to a subset of the feature space. Results on the University of California-Irvine repository reveal performance gains over several BN approaches and over multilayer perceptrons.

We develop new rules for combining estimates obtained from each classifier in an ensemble. A variety of combination techniques have been previously suggested, including averaging probability estimates, as well as hard voting schemes. We introduce a critic associated with each classifier, whose objective is to predict the classifier's errors. Since the critic only tackles a two-class problem, its predictions are generally more reliable than those of the classifier, and thus can be used as the basis for our suggested improved combination rules. While previous techniques are only effective when the individual classifier error rate is p<0.5, the new approach is successful, as proved under an independence assumption, even when this condition is violated-in particular, so long as p+q<1, with q the critic's error rate. More generally, critic-driven combining achieves consistent, substantial performance improvement over alternative methods, on a number of benchmark data sets.

In this work, we propose neural network inversion of a backward predictor as a technique for multi-step prediction of dynamic time series. It may be difficult to train a large network to capture the correlation that exists in some dynamic time series represented by small data sets. The new approach combines an estimate obtained from a forward predictor with an estimate obtained by inverting a backward predictor to more efficiently capture the correlation and to achieve more accurate predictions. Inversion allows us to make causal use of prediction backward in time. Also a new regularization method is developed to make neural network inversion less ill-posed. Experimental results on two benchmark time series demonstrate the new approach's significant improvement over standard forward prediction, given comparable complexity.

We propose a new method for learning discrete space statistical classifiers. We cast classification/inference within the more general framework of estimating the joint probability mass function (PMF) for the (feature vector, class label) pair. The proposal of Cheeseman (1983) to construct the maximum entropy (ME) joint PMF consistent with general lower order probability constraints has been severely limited by its huge learning complexity. Alternatives such as Bayesian networks require explicit specification of conditional independencies. Here we reconsider the ME problem, propose an approximate method which encodes arbitrary low order constraints, while retaining quite tractable learning. The new method approximates the joint feature PMF during learning on a sub-grid of the full feature space. Extensions to more general inference problems are indicated.