Software development is shifting from traditional programming to AI-integrated applications that leverage generative AI and large language models (LLMs) during runtime. However, integrating LLMs remains complex, requiring developers to manually craft prompts and process outputs. Existing tools attempt to assist with prompt engineering, but often introduce additional complexity. This paper presents Meaning-Typed Programming (MTP) , a novel paradigm that abstracts LLM integration through intuitive language-level constructs. By leveraging the inherent semantic richness of code, MTP automates prompt generation and response handling without additional developer effort. We introduce the (1) by operator for seamless LLM invocation, (2) MT-IR , a meaning-based intermediate representation for semantic extraction, and (3) MT-Runtime , an automated system for managing LLM interactions. We implement MTP in Jac , a programming language that supersets Python, and find that MTP significantly reduces coding complexity while maintaining accuracy and efficiency. MTP significantly reduces development complexity, lines of code modifications needed, and costs while improving run-time performance and maintaining or exceeding the accuracy of existing approaches. Our user study shows that developers using MTP completed tasks 3.2× faster with 45% fewer lines of code compared to existing frameworks. Moreover, MTP demonstrates resilience even when up to 50% of naming conventions are degraded, demonstrating robustness to suboptimal code. MTP is developed as part of the Jaseci open-source project, and is available under the module byLLM .

Social media like forums and microblogs have accumulated a huge amount of user generated content (UGC) containing human knowledge.Currently, most of UGC is listed as a whole or in pre-defined categories.This "list-based" approach is simple, but hinders users from browsing and learning knowledge of certain topics effectively.To address this problem, we propose a hierarchical entity-based approach for structuralizing UGC in social media.By using a large-scale entity repository, we design a three-step framework to organize UGC in a novel hierarchical structure called "cluster entity tree (CET)".With Yahoo!Answers as a test case, we conduct experiments and the results show the effectiveness of our framework in constructing CET.We further evaluate the performance of CET on UGC organization in both user and system aspects.From a user aspect, our user study demonstrates that, with CET-based structure, users perform significantly better in knowledge learning than using traditional list-based approach.From a system aspect, CET substantially boosts the performance of two information retrieval models (i.e., vector space model and query likelihood language model).

Community Question Answering (CQA) attracts increasing volume of research on question retrieval, high quality content discovery and experts finding. However, few studies are focused on community per se of CQA services and also provide an in-depth analysis of them. This paper aims to enrich our knowledge on two of these CQA services, namely Yahoo! Answers and Baidu Zhidao through reviewing their communities, comparing similarities and differences of the two communities, together with analyzing their influence on solving questions. Six data sets are employed for comparative analysis. In this paper: (1) We analyze the social network structures of Yahoo! Answers and Baidu Zhidao; (2) We compare the the social community characteristics of top contributors; (3) We reveal the behaviors of users in different categories in these two portals; (4) We reveal temporal trends of these characteristics; (5) We find that the community of Yahoo! Answers and Baidu Zhidao complement each other in efficiency and effectiveness of answering questions.

This paper investigates a ground-breaking incorporation of question category to Question Routing (QR) in Community Question Answering (CQA) services. The incorporation of question category was designed to estimate answerer expertise for routing questions to potential answerers. Two category-sensitive Language Models (LMs) were developed with large-scale real world data sets being experimented. Results demonstrated that higher accuracies of routing questions with lower computational costs were achieved, relative to traditional Query Likelihood LM (QLLM), state-of-the-art Cluster-Based LM (CBLM) and the mixture of Latent Dirichlet Allocation and QLLM (LDALM).

In this paper, we investigate the novel problem of automatic question identification in the microblog environment. It contains two steps: detecting tweets that contain questions (we call them "interrogative tweets") and extracting the tweets which really seek information or ask for help (so called "qweets") from interrogative tweets. To detect interrogative tweets, both traditional rule-based approach and state-of-the-art learning-based method are employed. To extract qweets, context features like short urls and Tweet-specific features like Retweets are elaborately selected for classification. We conduct an empirical study with sampled one hour's English tweets and report our experimental results for question identification on Twitter.

Community question answering services (CQA), which provides a platform for people with diverse backgrounds to share information and knowledge, has become an increasingly popular research topic recently as made popular by sites such as Yahoo! Answers1, answerbag2, zhidao3, etc. Question retrieval (QR) in CQA can automatically find the most relevant and recent questions that have been solved by other users. Current QR approaches typically consider using diverse retrieval models, but they fail to analyze users' intention. User intentions such as finding facts, interacting with others, seeking reasons, etc. reflect what the users really want to know. Hence, we propose to integrate user intention analysis into QR. Firstly, we classify questions into different and multiple types of users' intentions. Another practical problem is that there naturally exist some preferences among the possible questions types. The more relevant type should be ranked higher than types which are not so relevant. Therefore, we propose to utilize a novel label ranking method, which is a machine learning algorithm that aims to predict a ranking among all the possible labels, to perform question classification. Secondly, based on the result of question classification, we integrate user intentions with translation-based language models to explore whether a user's intention does help to improve the performance. We conduct a series of experiments with Yahoo data, and the experimental results demonstrate that our proposed improved question retrieval can indeed enhance the performance of traditional question retrieval model.

Community Question Answering (CQA) service provides a platform for increasing number of users to ask and answer for their own needs but unanswered questions still exist within a fixed period. To address this, the paper aims to route questions to the right answerers who have a top rank in accordance of their previous answering performance. In order to rank the answerers, we propose a framework called Question Routing (QR) which consists of four phases: (1) performance profiling, (2) expertise estimation, (3) availability estimation, and (4) answerer ranking. Applying the framework, we conduct experiments with Yahoo! Answers dataset and the results demonstrate that on average each of 1,713 testing questions obtains at least one answer if it is routed to the top 20 ranked answerers.

With the rapid growth of the Internet, more and more people interact with their friends in online social networks like Facebook. Current online social networks have designed some strategies to protect users' privacy, but they are not stringent enough. Some public information of profile or relationship can be utilized to infer users' private information. Online social networks usually contain little public available information of users (labeled data) but with a large number of hidden ones (unlabeled data). Recently, Semi-Supervised Learning (SSL), which has the advantage of utilizing fewer labeled data to achieve better performance compared to classical Supervised Learning, attracts much attention from the web research community with a massive set of unlabeled data. In our paper, we focus on the privacy issue of online social networks, which is a hot and dynamic research topic. More specifically, we propose a novel SSL framework that can be used to exploit security issues in online social networks. We first introduce the general SSL framework and outline two exploit models with associated strategies within it, e.g., graph-based models and co-training model. Finally, we conduct a series of experiments on real-world data from Facebook and StudiVZ to evaluate the effectiveness of this SSL exploit framework. Experimental results demonstrate that our approaches can accurately infer sensitive information of online users and more effective compared to previous models.

User evaluations of dialogs from a spoken dialog system (SDS) can be directly used to gauge the system's performance. However, it is costly to obtain manual evaluations of a large corpus of dialogs. Semi-supervised learning (SSL) provides a possible solution. This process learns from a small amount of manually labeled data, together with a large amount of unlabeled data, and can later be used to perform automatic labeling. We conduct comparative experiments among SSL approaches, classical regression and supervised learning in evaluation of dialogs from CMU's Let's Go Bus Information System. Two typical SSL methods, namely co-training and semi-supervised support vector machine (S3VM), are found to outperform the other approaches in automatically predicting user evaluations of unseen dialogs in the case of low training rate.

This paper presents an initial attempt at the use of crowd-sourcing for collection of user judgments on spoken dialog systems (SDSs). This is implemented on Amazon Mechanical Turk (MTurk), where a Requester can design a human intelligence task (HIT) to be performed by a large number of Workers efficiently and cost-effectively. We describe a design methodology for two types of HITs - the first targets at fast rating of a large number of dialogs regarding some dimensions of the SDS's performance and the second aims to assess the reliability of Workers on MTurk through the variability in ratings across different Workers. A set of approval rules are also designed to control the quality of ratings from MTurk. At the end of the collection work, user judgments for about 8,000 dialogs rated by around 700Workers are collected in 45 days. We observe reasonable consistency between the manual MTurk ratings and an automatic categorization of dialogs in terms of task completion, which partially verifies the reliability of the approved ratings from MTurk. From the second type of HITs, we also observe moderate inter-rater agreement for ratings in task completion which provides support for the utilization of MTurk as a judgments collection platform. Further research on the exploration of SDS evaluation models could be developed based on the collected corpus.

Developing accurate models to automatically predict user satisfaction about the overall quality of a Spoken Dialog System (SDS) is highly desirable for SDS evaluation. In the original PARADISE framework, a linear regression model is trained using measures drawn from rated dialogs as predictors with user satisfaction as the target. In this paper, we extend PARADISE by introducing a collaborative filtering (CF) model for user satisfaction prediction and its corresponding extension. This prediction model is drawn from the idea of CF in recommendation systems, which uses information from near neighbors of an unrated dialog to predict its user satisfaction. We also present the methodology of collecting user judgments on SDS quality with crowdsourcing through Amazon Mechanical Turk. Experimental results show that the CF approaches could distinctly improve the prediction accuracy of user satisfaction.

Development of spoken dialog systems (SDSs) can be facilitated by better evaluation methods. Previous methods seldom consider the efficiency of the system, which is important to users. We study the problem of evaluating SDSs and propose a new framework by generalizing states from utterances of dialogs to build finite state machine (FSM). These states can be regarded as efficiency measurement of SDSs. The FSM framework models dialogs as paths in an FSM to combine efficiency measurement with regression models. The proposed FSM framework can be applied in conjunction with regression models to improve evaluation accuracy. We compare our FSM framework combined with three regression models in several experiments. We obtain promising results on a collection of dialogs from the “Let's Go!” system, with our approach outperforming regression models.