We present Auto-BenchmarkCard, a workflow for generating validated descriptions of AI benchmarks. Benchmark documentation is often incomplete or inconsistent, making it difficult to interpret and compare benchmarks across tasks or domains. Auto-BenchmarkCard addresses this gap by combining multi-agent data extraction from heterogeneous sources (e.g., Hugging Face, Unitxt, academic papers) with LLM-driven synthesis. A validation phase evaluates factual accuracy through atomic entailment scoring using the FactReasoner tool. This workflow has the potential to promote transparency, comparability, and reusability in AI benchmark reporting, enabling researchers and practitioners to better navigate and evaluate benchmark choices.

We present Risk Atlas Nexus, an open source system for governing AI risks. The system unifies several risk classification frameworks through a common ontology. Given an AI application use case (called an intent), the system estimates risks and associated mitigations that are linked to identified risks. The tool is designed to be incorporated in AI governance workflows where recommendations can be translated to business controls to cover risks arising from AI use in firms.

Javier Carnerero-Cano, Massimiliano Pronesti, Radu Marinescu, Tigran T. Tchrakian, James Barry, Jasmina Gajcin, Yufang Hou, Alessandra Pascale, Elizabeth M. Daly. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Bringing a new AI system into a production environment involves multiple different stakeholders such as business owners, risk officer, ethics officers approving the AI System for a specific usage. Governance frameworks typically include multiple manual steps, including curating information needed to assess risks and reviewing outcomes to identify appropriate actions and governance strategies. We demo a human-in-the-loop automation system that takes a natural language description of an intended use case for an AI system in order to create semi-structured governance information, recommend the most appropriate model for that use case, prioritise risks to be evaluated, automatically running those evaluations and finally storing these results for auditing, reporting and future recommendations. As a result we increase transparency to stakeholders and provide valuable information to aid in decision making when assessing risks associated with an AI solution.

We present EvalAssist, a framework that simplifies the LLM- as-a-judge workflow. The system provides an online criteria development environment, where users can interactively build, test, and share custom evaluation criteria in a structured and portable format. A library of LLM based evaluators is made available that incorporates various algorithmic innovations such as token-probability based judgement, positional bias checking, and certainty estimation that help to engender trust in the evaluation process. We have computed extensive benchmarks and also deployed the system internally in our organization with several hundreds of users.

Risk-averse Constrained Reinforcement Learning (RaCRL) aims to learn policies that minimise the likelihood of rare and catastrophic constraint violations caused by an environment’s inherent randomness. In general, risk-aversion leads to conservative exploration of the environment which typically results in converging to sub-optimal policies that fail to adequately maximise reward or, in some cases, fail to achieve the goal. In this paper, we propose an exploration-based approach for RaCRL called Optimistic Risk-averse Actor Critic (ORAC), which constructs an exploratory policy by maximising a local upper confidence bound of the state-action reward value function whilst minimising a local lower confidence bound of the risk-averse state-action cost value function. Specifically, at each step, the weighting assigned to the cost value is increased or decreased if it exceeds or falls below the safety constraint value. This way the policy is encouraged to explore uncertain regions of the environment to discover high reward states whilst still satisfying the safety constraints. Our experimental results demonstrate that the ORAC approach prevents convergence to sub-optimal policies and improves significantly the reward-cost trade-off in various continuous control tasks such as Safety-Gymnasium and a complex building energy management environment CityLearn.

Erik Miehling, Michael Desmond, Karthikeyan Natesan Ramamurthy, Elizabeth M. Daly, Kush R. Varshney, Eitan Farchi, Pierre Dognin, Jesus Rios, Djallel Bouneffouf, Miao Liu, Prasanna Sattigeri. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Large language models (LLMs) are powerful tools capable of handling diverse tasks. Comparing and selecting appropriate LLMs for specific tasks requires systematic evaluation methods, as models exhibit varying capabilities across different domains. However, finding suitable benchmarks is difficult given the many available options. This complexity not only increases the risk of benchmark misuse and misinterpretation but also demands substantial effort from LLM users, seeking the most suitable benchmarks for their specific needs. To address these issues, we introduce BenchmarkCards, an intuitive and validated documentation framework that standardizes critical benchmark attributes such as objectives, methodologies, data sources, and limitations. Through user studies involving benchmark creators and users, we show that BenchmarkCards can simplify benchmark selection and enhance transparency, facilitating informed decision-making in evaluating LLMs.Data & Code:github.com/SokolAnn/BenchmarkCards huggingface.co/datasets/ASokol/BenchmarkCards

User flow diagram for EvalAssist in the direct assessment evaluation, illustrating criteria definition, test data input, annotation, AI evaluator selection, result review, iterative adjustments, and criteria export for dataset-wide evaluation via SDK.

Bias mitigation algorithms differ in their definition of bias and how they go about achieving that objective. Bias mitigation algorithms impact different cohorts differently and allowing end users and data scientists to understand the impact of these differences in order to make informed choices is a relatively unexplored domain. This demonstration presents an interactive bias mitigation pipeline that allows users to understand the cohorts impacted by their algorithm choice and provide feedback in order to provide a bias mitigated pipeline that most aligns with their goals.

Retrieval-augmented generation (RAG) has emerged as a promising solution to mitigate the limitations of large language models (LLMs), such as hallucinations and outdated information. However, it remains unclear how LLMs handle knowledge conflicts arising from different augmented retrieved passages, especially when these passages originate from the same source and have equal trustworthiness. In this work, we conduct a comprehensive evaluation of LLM-generated answers to questions that have varying answers based on contradictory passages from Wikipedia, a dataset widely regarded as a high-quality pre-training resource for most LLMs. Specifically, we introduce WikiContradict, a benchmark consisting of 253 high-quality, human-annotated instances designed to assess the performance of LLMs in providing a complete perspective on conflicts from the retrieved documents, rather than choosing one answer over another, when augmented with retrieved passages containing real-world knowledge conflicts. We benchmark a diverse range of both closed and open-source LLMs under different QA scenarios, including RAG with a single passage, and RAG with 2 contradictory passages. Through rigorous human evaluations on a subset of WikiContradict instances involving 5 LLMs and over 3,500 judgements, we shed light on the behaviour and limitations of these models. For instance, when provided with two passages containing contradictory facts, all models struggle to generate answers that accurately reflect the conflicting nature of the context, especially for implicit conflicts requiring reasoning. Since human evaluation is costly, wealso introduce an automated model that estimates LLM performance using a strong open-source language model, achieving an F-score of 0.8. Using this automated metric, we evaluate more than 1,500 answers from seven LLMs across all WikiContradict instances.

A well-defined reward function is crucial for successful training of an reinforcement learning (RL) agent. However, defining a suitable reward function is a notoriously challenging task, especially in complex, multi-objective environments. Developers often have to resort to starting with an initial, potentially misspecified reward function, and iteratively adjusting its parameters, based on observed learned behavior. In this work, we aim to automate this process by proposing ITERS, an iterative reward shaping approach using human feedback for mitigating the effects of a misspecified reward function. Our approach allows the user to provide trajectory-level feedback on agent’s behavior during training, which can be integrated as a reward shaping signal in the following training iteration. We also allow the user to provide explanations of their feedback, which are used to augment the feedback and reduce user effort and feedback frequency. We evaluate ITERS in three environments and show that it can successfully correct misspecified reward functions.

We present AutoDOViz, an interactive user interface for automated decision optimization (AutoDO) using reinforcement learning (RL). Decision optimization (DO) has classically being practiced by dedicated DO researchers [43] where experts need to spend long periods of time fine tuning a solution through trial-and-error. AutoML pipeline search has sought to make it easier for a data scientist to find the best machine learning pipeline by leveraging automation to search and tune the solution. More recently, these advances have been applied to the domain of AutoDO [36], with a similar goal to find the best reinforcement learning pipeline through algorithm selection and parameter tuning. However, Decision Optimization requires significantly more complex problem specification when compared to an ML problem. AutoDOViz seeks to lower the barrier of entry for data scientists in problem specification for reinforcement learning problems, leverage the benefits of AutoDO algorithms for RL pipeline search and finally, create visualizations and policy insights in order to facilitate the typical interactive nature when communicating problem formulation and solution proposals between DO experts and domain experts. In this paper, we report our findings from semi-structured expert interviews with DO practitioners as well as business consultants, leading to design requirements for human-centered automation for DO with RL. We evaluate a system implementation with data scientists and find that they are significantly more open to engage in DO after using our proposed solution. AutoDOViz further increases trust in RL agent models and makes the automated training and evaluation process more comprehensible. As shown for other automation in ML tasks [33, 59], we also conclude automation of RL for DO can benefit from user and vice-versa when the interface promotes human-in-the-loop.

Cookies are designed to enable more accurate identification and tracking of user behavior, in turn allowing for more personalized ads and better performing ad campaigns. Given the additional information that is recorded, questions related to privacy and fairness naturally arise. How does a user's consent decision influence how much the system can learn about their demographic and tastes? Is the impact of a user's consent decision on the recommender system's ability to learn about their latent attributes uniform across demographics? We investigate these questions in the context of an engagement-driven recommender system using simulation. We empirically demonstrate that when consent rates exhibit demographic-dependence, user consent has a disparate impact on the recommender agent's ability to estimate users' latent attributes. In particular, we find that when consent rates are demographic-dependent, a user disagreeing to share their cookie may counter-intuitively cause the recommender agent to know more about the user than if the user agreed to share their cookie. Furthermore, the gap in base consent rates across demographics serves as an amplifier: users from the lower consent rate demographic who agree to cookie sharing generally experience higher estimation errors than the same users from the higher consent rate demographic, and conversely for users who choose to disagree to cookie sharing, with these differences increasing in consent rate gap. We discuss the need for new notions of fairness that encourage consistency between a user's privacy decisions and the system's ability to estimate their latent attributes.

Understanding the differences between machine learning (ML) models is of interest in scenarios ranging from choosing amongst a set of competing models, to updating a deployed model with new training data. In these cases, we wish to go beyond differences in overall metrics such as accuracy to identify where in the feature space do the differences occur. We formalize this problem of model differencing as one of predicting a dissimilarity function of two ML models’ outputs, subject to the representation of the differences being human-interpretable. Our solution is to learn a Joint Surrogate Tree (JST), which is composed of two conjoined decision tree surrogates for the two models. A JST provides an intuitive representation of differences and places the changes in the context of the models’ decision logic. Context is important as it helps users to map differences to an underlying mental model of an AI system. We also propose a refinement procedure to increase the precision of a JST. We demonstrate, through an empirical evaluation, that such contextual differencing is concise and can be achieved with no loss in fidelity over naive approaches.

In real-world applications when deploying Machine Learning (ML) models, initial model development includes close analysis of the model results and behavior by a data scientist. Once trained, however, models may need to be retrained with new data or updated to adhere to new rules or regulations. This presents two challenges. First, how to communicate how a model is making its decisions before and after retraining, and second how to support model editing to take into account new requirements. To address these needs, we built AIMEE (AI Model Explorer and Editor), a tool created to address these challenges by providing interactive methods to explain, visualize, and modify model decision boundaries using rules. Rules should benefit model builders by providing a layer of abstraction for understanding and manipulating the model and reduces the need to modify individual rows of data directly. To evaluate if this was the case, we conducted a pair of user studies totaling 23 participants to evaluate AIMEE's rules-based approach for model explainability and editing. We found that participants correctly interpreted rules and report on their perspectives of how rules are beneficial (and not), ways that rules could support collaboration, and provide a usability evaluation of the tool.

Over the last few years, incremental machine learning for streaming data has gained significant attention due to the need to learn from a constantly evolving stream of data without the need to store it. The advent of big data has further fuelled research on developing systems that can cope with continuously changing data streams and tackle the challenges associated with historical data requirements. The problem of time series forecasting has been studied using varied approaches like neural networks, ensemble methods, decision trees and rules, support vector machines to name a few. However, neural network models have gained particular attention in dealing with changing data distribution due to their generalization abilities. In this paper, we propose a prequential framework named PS-PGSM which involves incrementally training the base models and online Regions of Competence (ROC) computation followed by selection of the best forecaster for the task of time series forecasting using saliency maps. We build upon the state-of-the-art approach named OS-PGSM (Online Model Selection using Performance Gradient based Saliency Maps) in which the model training and ROC computation is performed offline. Past research has demonstrated that a set of different models enables specialization for each model compared to a single forecasting model which is particularly useful when predicting for an evolving time series sequence. Our approach uses saliency maps for prequential calculation of ROC for each model to find the best forecaster based on the performance of each model. We evaluate the proposed approach against OS-PGSM, as well as against previous best performing model by first conducting preliminary experiments on 10 real-world time series datasets and then using 2 real-world big datasets to showcase its applicability to big data. Experimental results not only validate the effectiveness of our approach for big data but also demonstrate superior performance in terms of prediction accuracy and computational time efficiency while also handling concept drift.

Machine learning technologies are increasingly being applied in many different domains in the real world. As autonomous machines and black-box algorithms begin making decisions previously entrusted to humans, great academic and public interest has been spurred to provide explanations that allow users to understand the decision-making process of the machine learning model. Besides explanations, Interactive Machine Learning (IML) seeks to leverage user feedback to iterate on an ML solution to correct errors and align decisions with those of the users. Despite the rise in explainable AI (XAI) and Interactive Machine Learning (IML) research, the links between interactivity, explanations, and trust have not been comprehensively studied in the machine learning literature. Thus, in this study, we develop and evaluate an explanation-driven interactive machine learning (XIML) system with the Tic-Tac-Toe game as a use case to understand how a XIML mechanism improves users’ satisfaction with the machine learning system. We explore different modalities to support user feedback through visual or rules-based corrections. Our online user study (n = 199) supports the hypothesis that allowing interactivity within this XIML system causes participants to be more satisfied with the system, while visual explanations play a less prominent (and somewhat unexpected) role. Finally, we leverage a user-centric evaluation framework to create a comprehensive structural model to clarify how subjective system aspects, which represent participants’ perceptions of the implemented interaction and visualization mechanisms, mediate the influence of these mechanisms on the system’s user experience.

Interpretable and explainable machine learning has seen a recent surge of interest. We focus on safety as a key motivation behind the surge and make the relationship between interpretability and safety more quantitative. Toward assessing safety, we introduce the concept of maximum deviation via an optimization problem to find the largest deviation of a supervised learning model from a reference model regarded as safe. We then show how interpretability facilitates this safety assessment. For models including decision trees, generalized linear and additive models, the maximum deviation can be computed exactly and efficiently. For tree ensembles, which are not regarded as interpretable, discrete optimization techniques can still provide informative bounds. For a broader class of piecewise Lipschitz functions, we leverage the multi-armed bandit literature to show that interpretability produces tighter (regret) bounds on the maximum deviation. We present case studies, including one on mortgage approval, to illustrate our methods and the insights about models that may be obtained from deviation maximization.

AI solutions are heavily dependant on the quality and accuracy of the input training data, however the training data may not always fully reflect the most up-to-date policy landscape or may be missing business logic. The advances in explainability have opened the possibility of allowing users to interact with interpretable explanations of ML predictions in order to inject modifications or constraints that more accurately reflect current realities of the system. In this paper, we present a solution which leverages the predictive power of ML models while allowing the user to specify modifications to decision boundaries. Our interactive overlay approach achieves this goal without requiring model retraining, making it appropriate for systems that need to apply instant changes to their decision making. We demonstrate that user feedback rules can be layered with the ML predictions to provide immediate changes which in turn supports learning with less data.

We consider the problem of distinguishing two machine learning (ML) models built for the same task in a human-interpretable way. As models can fail or succeed in different ways, classical accuracy metrics may mask crucial qualitative differences. This problem arises in a few contexts. In business applications with periodically retrained models, an updated model may deviate from its predecessor for some segments without a change in overall accuracy. In automated ML systems, where several ML pipelines are generated, the top pipelines have comparable accuracy but may have more subtle differences. We present a method for interpretable comparison of binary classification models by approximating them with Boolean decision rules. We introduce stabilization conditions that allow for the two rule sets to be more directly comparable. A method is proposed to compare two rule sets based on their statistical and semantic similarity by solving assignment problems and highlighting changes. An empirical evaluation on several benchmark datasets illustrates the insights that may be obtained and shows that artificially induced changes can be reliably recovered by our method.

Finding a shortest path in a graph is at the core of many combinatorial search problems. A closely related problem refers to counting the number of shortest paths between two nodes. Such problems are solvable in polynomial time in the size of the graph. However, more realistic problem formulations could additionally specify constraints to satisfy. We study the problem of counting the shortest paths that are vertex disjoint and can satisfy additional constraints. Specifically, we look at the problems of counting vertex-disjoint shortest paths in edge-colored graphs, counting vertex-disjoint shortest paths with directional constraints, and counting vertex-disjoint shortest paths between multiple source-target pairs. We give a detailed theoretical analysis, and show formally that all of these three counting problems are NP-complete in general.

Recommender systems are used with increasing frequency in a wide variety of domains ranging from e- commerce to tourism, healthcare and online learning. However, the interaction with these systems generally tends to be limited to shallow feedback, such as providing ratings or filtering. Allowing users to interact with the recommender systems in a conversational environment brings opportunities in which the preferences can effectively be elicited from the users while the users can feel more in control of the whole process. However, when the existing non-interactive recommender systems are considered, it may not be easy to build an interactive layer directly on top of them. This is because there is already a great deal of modelling and work invested in the underlying algorithm and the system itself. Enabling interaction could mean rebuilding the whole solution from scratch, as the current design may not be able to consume preferences and information learnt from the user interaction online. In this paper, we propose the Interactive Recommender Framework, which converts non-interactive recommender solutions to conversational recommenders. We demonstrate how Interactive Recommender Framework can successfully enable interactivity on top of non-interactive recommender systems by integrating it into two different recommender algorithms from literature, and validate our solution through offline simulation experiments and online user studies.

Chatbots or conversational recommenders have gained increasing popularity as a new paradigm for Recommender Systems (RS). Prior work on RS showed that providing explanations can improve transparency and trust, which are critical for the adoption of RS. Their interactive and engaging nature makes conversational recommenders a natural platform to not only provide recommendations but also justify the recommendations through explanations. The recent surge of interest inexplainable AI enables diverse styles of justification, and also invites questions on how styles of justification impact user perception. In this article, we explore the effect of “why” justifications and “why not” justifications on users’ perceptions of explainability and trust. We developed and tested a movie-recommendation chatbot that provides users with different types of justifications for the recommended items. Our online experiment ( n = 310) demonstrates that the “why” justifications (but not the “why not” justifications) have a significant impact on users’ perception of the conversational recommender. Particularly, “why” justifications increase users’ perception of system transparency, which impacts perceived control, trusting beliefs and in turn influences users’ willingness to depend on the system’s advice. Finally, we discuss the design implications for decision-assisting chatbots.

Career goals represent a special case for recommender systems and require considering both short and long term goals. Recommendations must represent a trade off between relevance to the user, achievability and aspirational goals to move the user forward in their career. Users may have different motivations and concerns when looking for a new long term goal, so involving the user in the recommender process becomes all the more important than in other domains. Additionally, the cost to the user of making a bad decision is much higher than investing two hours in watching a movie they don't like or listening to an unappealing song. As a result, we feel career recommendations is a unique opportunity to truly engage the user in an interactive recommender as we believe they will invest the cognitive load. In this paper, we present an interactive career goal recommender framework that leverages the power of dialogue to allow the user interactively improve the recommendations and bring their own preferences to the system. The underlying recommendation algorithm is a novel solution that suggests both short and long term goals through utilizing the sequential patterns extracted from career trajectories that are enhanced with features of the supporting user profiles. The effectiveness of the proposed solution is demonstrated with extensive experiments on two real world data sets.

Recent research focuses beyond recommendation accuracy, towards human factors that influence the acceptance of recommendations, such as user satisfaction, trust, transparency and sense of control. We present a generic interactive recommender framework that can add interaction functionalities to non-interactive recommender systems. We take advantage of dialogue systems to interact with the user and we design a middleware layer to provide the interaction functions, such as providing explanations for the recommendations, managing users' preferences learnt from dialogue, preference elicitation and refining recommendations based on learnt preferences.

Multi-modal journey planning, which allows multiple types of transport within a single trip, is becoming increasingly popular, due to a strong practical interest and an increasing availability of data. In real life, transport networks feature uncertainty. Yet, most approaches assume a deterministic environment, making plans more prone to failures such as missed connections and major delays in the arrival. This paper presents an approach to computing optimal contingent plans in multi-modal journey planning. The problem is modeled as a search in an and/or state space. We describe search enhancements used on top of the AO* algorithm. Enhancements include admissible heuristics, multiple types of pruning that preserve the completeness and the optimality, and a hybrid search approach with a deterministic and a nondeterministic search. We demonstrate an NP-hardness result, with the hardness stemming from the dynamically changing distributions of the travel time random variables. We perform a detailed empirical analysis on realistic transport networks from cities such as Montpellier, Rome and Dublin. The results demonstrate the effectiveness of our algorithmic contributions, and the benefits of contingent plans as compared to standard sequential plans, when the arrival and departure times of buses are characterized by uncertainty.

Sellers work together as a team on sales opportunities, using their expertise in different roles to increase the probability of a win. These roles include managing the relationship with the client, overall architecture support or deep knowledge of a particular product depending on the seller's expertise, and the current opportunity requirements. Forming the right team for an incoming opportunity is vital and depends on several factors including understanding the required roles for the opportunity and assigning the right person to fulfill these roles, taking into consideration the seller's social network. In this paper, we present the Opportunity Team Builder solution, which supports sellers in this work by dividing the process into the following sub-tasks; identifying the required roles for the opportunity based on the products that the client is interested in, recommending the best people to fulfill these roles, and providing a win probability figure to guide users in team formation. This supports the sellers in forming the bestfitting team for current opportunity dynamics. Each task in the solution is implemented as a model using historical data from previous sales opportunities. Models work in coordination with each other to ultimately maximize the probability of win over loss. The solution not only recommends the best person to join a team taking into account a combination of inferred skills and social relationships, but also the predicted impact the person can have on the overall performance of the team. We present how the whole solution is realized with an intelligent user interface enabling interaction with the user throughout the team formation process. Substantial experiments with real world data show that win/loss prediction is performed accurately and the Opportunity Team Builder solution can recommend teams that achieve a higher win probability.

Optimal heuristic search has been successful in many domains, including journey planning, route planning and puzzle solving. Existing work typically assumes that the cost of each action can easily be obtained. However, in many problems, the exact edge cost is expensive to compute. Existing search algorithms face a significant performance bottleneck, due to an excessive overhead associated with dynamically calculating exact edge costs. We present DEA*, an algorithm for problems with expensive edge cost computations. DEA* combines heuristic edge cost evaluations with delayed node expansions, reducing the number of exact edge computations. We formally prove that DEA* is optimal and it is efficient with respect to the number of exact edge cost computations. We empirically evaluate DEA* on multiple-worker routing problems where the exact edge cost is calculated by invoking an external multi-modal journey planning engine. The results demonstrate the effectiveness of our ideas in reducing the computational time and improving the solving ability. In addition, we show the advantages of DEA* in domain-independent planning, where we simulate that accurate edge costs are expensive to compute.

In this paper we tackle the recently proposed problem of hidden streams. In many situations, the data stream that we are interested in, is not directly accessible. Instead, part of the data can be accessed only through applying filters (e.g. keyword filtering). In fact this is the case of the most discussed social stream today, Twitter. The problem in this case is how to retrieve as many relevant documents as possible by applying the most appropriate set of filters to the original stream and, at the same time, respect a number of constrains (e.g. maximum number of filters that can be applied). In this work we introduce a search approach on a dynamic filter space. We utilize heterogeneous filters (not only keywords) making no assumptions about the attributes of the individual filters. We advance current research by considering realistically hard constraints based on real-world scenarios that require tracking of multiple dynamic topics. We demonstrate the effectiveness of our approaches on a set of topics of static and dynamic nature. The development of the approach was motivated by a real application. Our system is deployed in Dublin City's Traffic Management Center and allows the city officers to analyze large sources of heterogeneous data and identify events related to traffic as well as emergencies.

Efficiently detecting conversation threads from a pool of messages, such as social network chats, emails, comments to posts, news etc., is relevant for various applications, including Web Marketing, Information Retrieval and Digital Forensics. Existing approaches focus on text similarity using keywords as features that are strongly dependent on the dataset. Therefore, dealing with new corpora requires further costly analyses conducted by experts to find out new relevant features. This paper introduces a novel method to detect threads from any type of conversational texts overcoming the issue of previously determining specific features for each dataset. To automatically determine the relevant features of messages we map each message into a three dimensional representation based on its semantic content, the social interactions in terms of sender/recipients and its timestamp; then clustering is used to detect conversation threads. In addition, we propose a supervised approach to detect conversation threads that builds a classification model which combines the above extracted features for predicting whether a pair of messages belongs to the same thread or not. Our model harnesses the distance measure of a message to a cluster representing a thread to capture the probability that a message is part of that same thread. We present our experimental results on seven datasets, pertaining to different types of messages, and demonstrate the effectiveness of our method in the detection of conversation threads, clearly outperforming the state of the art and yielding an improvement of up to a 19%.

Optimal multi-modal journey planning under uncertainty is a challenging problem, due in part to an increased branching factor generated by nondeterministic actions. Deterministic search, which ignores all uncertainty, can be much faster, but deterministic plans lack correctness and optimality guarantees in the uncertainty-aware domain.

Real-time traffic awareness applications are playing an ever increasing role understanding and tackling traffic congestion in cities. First-hand accounts from drivers witnessing an incident is an invaluable source of information for traffic managers. Nowadays, drivers increasingly contact control rooms through social media to report on journey times, accidents or road weather conditions. These new interactions allow traffic controllers to engage users, and in particular to query them for information rather than passively collecting it. Querying participants presents the challenge of which users to probe for updates about a specific situation. In order to maximise the probability of a user responding and the accuracy of the information, we propose a strategy which takes into account the engagement levels of the user, the mobility profile and the reputation of the user. We provide an analysis of a real-world user corpus of Twitter users contributing updates to LiveDrive, a Dublin based traffic radio station.

Knowledge on the Web relies heavily on multi-relational representations, such as RDF and Schema.org. Automatically extracting knowledge from documents and linking existing databases are common approaches to construct multi-relational data. Complementary to such approaches, there is still a strong demand for manually encoding human expert knowledge. For example, human annotation is necessary for constructing a common-sense knowledge base, which stores facts implicitly shared in a community, because such knowledge rarely appears in documents. As human annotation is both tedious and costly, an important research challenge is how to best use limited human resources, whiles maximizing the quality of the resulting dataset. In this paper, we formalize the problem of dataset construction as active learning problems and present the Active Multi-relational Data Construction (AMDC) method. AMDC repeatedly interleaves multi-relational learning and expert input acquisition, allowing us to acquire helpful labels for data construction. Experiments on real datasets demonstrate that our solution increases the number of positive triples by a factor of 2.28 to 17.0, and that the predictive performance of the multi-relational model in AMDC achieves the highest or comparable to the best performance throughout the data construction process.

Recommender systems can be employed to assist users in complex decision making processes. This paper presents a multi-criteria housing recommender system which takes into account not just features of a home, such as rent, but also the transportation links to user specified locations. First, we describe an efficient multi-hop journey time calculator. Second, we introduce a mechanism to find the optimal solutions for multi-criteria evaluation, where a balanced trade-off between the target goals is found. Finally, we present a user study to demonstrate the potential of such a system.

The advent of real-time traffic streaming offers users the opportunity to visualise current traffic conditions and congestion information. However, real-time information highlighting the underlying reason for tail-backs remains largely unexplored. Broken traffic lights, an accident, a large concert, or road-works reveal important information for citizens and traffic operators alike. Providing such information in real-time requires intelligent mechanisms and user interfaces in order to (i) harness heterogeneous data sources (volume, velocity, variety, veracity) and (ii) make derived knowledge consumable so users can visualize traffic conditions and congestion information making better routing decisions while travelling. This work focuses on surfacing relevant information and explaining the underlying reasons behind traffic conditions. To this end, static data from event providers, planned road works together with dynamically emerging events such as a traffic accidents, localized weather conditions or unplanned obstructions are captured through social media to provide users real-time feedback to highlight the causes of traffic congestion.

In this paper, we propose to improve time management in the enterprise by providing users interactive visualizations of how they are spending their time. Through an interview study (n=21) in a multi-national corporation, we were able to determine the data available for visualizations and the value of a number of general visualizations of employees' calendar data. We develop implications for design in improving personal time management.

We present the system design and rational for a novel social microcalendar called Timely. Our system has been inspired by previous research on calendaring and popular social network applications, in particular microblogging. Timely provides an open, social space for enterprise users to share their events, socialize, and discover what else is going on in their network and beyond. A detailed analysis of the events shared by users during the site's first 47 days reveals that users willingly share their time commitments despite an existing culture of restricted calendars.

The ever blurring line between online interactions and physical encounters presents an interesting challenge when recommending events. Events created on social networking sites may have ambiguous location scope. The location information provided may be fuzzy or non existent and additionally the reach and radius of interest in the event can vary greatly. In this work, we identify four categories of events: global, location dependent and socially independent, socially dependent and location independent, and location and socially dependent. We classify events from an organizations internal event management service where the location of the event is unknown, but the location of the attendees are known in order to improve scoping of event recommendations. Our results, investigate the impact of ignoring location properties when recommending events using classic collaborative filtering techniques. Additionally, once global and socially independent events are identified, they can be used to provide recommendations to new users, addressing the cold-start problem.

Social networking sites have begun to be used in the enterprise as a method of connecting employees. Recommender systems may be used to recommend social contacts in order to increase user engagement, encourage collaboration and facilitate expertise discovery. This paper evaluates the effects of four recommendation algorithms on the network as a whole and the social structure. We demonstrate that depending on the basis of the recommendation algorithm the effects on the network vary greatly and their potential impact should be understood. It is hoped this research can be used as guidance for future recommendation algorithms.

The success of social media has resulted in an information overload problem, where users are faced with hundreds of new contributions, edits and communications at every visit. A prime example of this in social networks is the news or activity feeds, where the actions (friending, commenting, photo sharing, etc) of friends on the network are presented to users in order to inform them of the network activity. In this work we endeavour to reduce the burden on individuals of identifying interesting updates in social network news feeds by automatically identifying and recommending relevant items to individuals where item relevance is based on the observed interactions of the individual with the social network. The results of our offline study show that combining short term interest models, exploiting previous viewing behavior of users, and long-term models, exploiting previous viewing of network actions, was the best predictor of feed item relevance.

The ldquowisdom of the crowdsrdquo is a concept used to describe the utility of harnessing group behaviour, where user opinion evolves over time and the opinion of the masses collectively demonstrates wisdom. Web 2.0 is a new medium where users are not just consumers, but are also contributors. By contributing content to the system, users become part of the network and relationships between users and content can be derived. Example applications are collaborative bookmarking networks such as delicious and file sharing applications such as YouTube and Flickr. These networks rely on user contributed content, described and classified using tags. The wealth of user generated content can be hard to navigate and search due to difficulties in comparing documents with similar tags and the application of traditional information retrieval scoring techniques are limited. Evaluating the time evolving interests of users may be used to derive quality of content. In this paper, we propose a technique to rank documents based on reputation. The reputation is a combination of the number of bookmarkers, the reputation of the bookmarking user and the time dynamics of the document. Experimental results and analysis are presented on a large collaborative IBM bookmarking network called Dogear.

Message delivery in sparse mobile ad hoc networks (MANETs) is difficult due to the fact that the network graph is rarely (if ever) connected. A key challenge is to find a route that can provide good delivery performance and low end-to-end delay in a disconnected network graph where nodes may move freely. We cast this challenge as an information flow problem in a social network. This paper presents social network analysis metrics that may be used to support a novel and practical forwarding solution to provide efficient message delivery in disconnected delay-tolerant MANETs. These metrics are based on social analysis of a node's past interactions and consists of three locally evaluated components: a node's "betweenness" centrality (calculated using ego networks) and a node's social 'similarity' to the destination node and a node's tie strength relationship with the destination node. We present simulations using three real trace data sets to demonstrate that by combining these metrics delivery performance may be achieved close to epidemic routing but with significantly reduced overhead. Additionally, we show improved performance when compared to PRoPHET routing.

Message delivery in sparse Mobile Ad hoc Networks (MANETs) is difficult due to the fact that the network graph is rarely (if ever) connected. A key challenge is to find a route that can provide good delivery performance and low end-to-end delay in a disconnected network graph where nodes may move freely. This paper presents a multidisciplinary solution based on the consideration of the so-called small world dynamics which have been proposed for economy and social studies and have recently revealed to be a successful approach to be exploited for characterising information propagation in wireless networks. To this purpose, some bridge nodes are identified based on their centrality characteristics, i.e., on their capability to broker information exchange among otherwise disconnected nodes. Due to the complexity of the centrality metrics in populated networks the concept of ego networks is exploited where nodes are not required to exchange information about the entire network topology, but only locally available information is considered. Then SimBet Routing is proposed which exploits the exchange of pre-estimated "betweenness' centrality metrics and locally determined social "similarity' to the destination node. We present simulations using real trace data to demonstrate that SimBet Routing results in delivery performance close to Epidemic Routing but with significantly reduced overhead. Additionally, we show that SimBet Routing outperforms PRoPHET Routing, particularly when the sending and receiving nodes have low connectivity.

Searching and organization of peers are fundamental challenges in P2P networks. Unstructured networks, such as Gnutella, inefficiently use broadcast searches and random neighbors. Structured networks are similarly inefficient, as they generally rely on globally unique identifiers (GUIDs) which are assigned irrespective of content, which prevents fuzzy semantic searches. In both types of network search, neighbors establish trust between themselves, regardless of whether or not their content is likely to satisfy searches. We present the idea of using context-based profiles to describe peers. This enables self-organizing clusters of similar peers. A profile represents a peer's expertise based on content and responsiveness. By refining the search process using these profiles, more efficient directed searches are possible. Moreover, expertise provides a basis for trust establishment.

A recently developed technique for variable-rate vector quantizer (VQ) design by P.A. Chou et al. (see IEEE Trans. Inf. Theory, vol.35, no.2, p.299-315, 1989) has been applied to both memoryless and predictive VQ of images. This technique, called pruned tree-structured vector quantization (PTSVQ), uses variable-depth encoders that are tree-structured and thus have very low design and search complexity. PTSVQ is applied to a series of medical images, and gains over full-search VQ of up to 3.78 dB in the signal-to-noise-ratio (SNR) are measured. On still images from the USC database, gains of up to 1.63 dB in the peak SNR are realized for predictive PTSVQ over predictive full search VQ, resulting in high image quality at 0.51 bits per pixel.>

The authors investigate a novel vector-quantized compression strategy that provides guaranteed image quality with variable bit rate and show its applicability to packet-switched networks. This preliminary study indicates that significant improvements in both picture quality and SNR value can be achieved. The technique can be refined by reducing the number of overhead bits for each block and by separating coding schemes and high/low detail thresholds for each block size.>