Email responses often contain items---such as a file or a hyperlink to an external document---that are attached to or included inline in the body of the message. Analysis of an enterprise email corpus reveals that 35% of the time when users include these items as part of their response, the attachable item is already present in their inbox or sent folder. A modern email client can proactively retrieve relevant attachable items from the user's past emails based on the context of the current conversation, and recommend them for inclusion, to reduce the time and effort involved in composing the response. In this paper, we propose a weakly supervised learning framework for recommending attachable items to the user. As email search systems are commonly available, we constrain the recommendation task to formulating effective search queries from the context of the conversations. The query is submitted to an existing IR system to retrieve relevant items for attachment. We also present a novel strategy for generating labels from an email corpus---without the need for manual annotations---that can be used to train and evaluate the query formulation model. In addition, we describe a deep convolutional neural network that demonstrates satisfactory performance on this query formulation task when evaluated on the publicly available Avocado dataset and a proprietary dataset of internal emails obtained through an employee participation program.

A key problem in crowdsourcing is the aggregation of judgments of proportions. For example, workers might be presented with a news article or an image, and be asked to identify the proportion of each topic, sentiment, object, or colour present in it. These varying judgments then need to be aggregated to form a consensus view of the document’s or image’s contents. Often, however, these judgments are skewed by workers who provide judgments randomly. Such spammers make the cost of acquiring judgments more expensive and degrade the accuracy of the aggregation. For such cases, we provide a new Bayesian framework for aggregating these responses (expressed in the form of categorical distributions) that for the first time accounts for spammers. We elicit 796 judgments about proportions of objects and coloursin images. Experimental results show comparable aggregation accuracy when 60% of the workers are spammers, as other state of the art approaches do when there are no spammers.

Over the past few years, Domestic Heating Automation Systems (DHASs) that optimize the domestic space heating control process with minimum user input, utilizing appropriate occupancy prediction technology, have emerged as commercial products (e.g., the smart thermostats from Nest and Honeywell). At the same time, many houses are being equipped with, potentially grid-connected, Intermittent Energy Resources (IERs), such as rooftop photovoltaic systems and/or small wind turbine generators. Now, in many regions of the world, such houses can sell energy to the grid but at a lower price than the price of buying it. In this context, and given the anticipated increase in electrification of heating, the next generation DHASs need to incorporate Advanced Economic Control (AEC). Such AEC can exploit the energy buffer that heating loads provide, in order to shift the consumption of electricity-based heating systems to follow the intermittent energy generation of the house. By so doing, the energy imported from the grid can be minimized and considerable monetary gains for the household can be achieved, without affecting the occupants’ schedule. These benefits can be amplified still further in domestic coalitions, where a number of houses come together and share their IER generation to minimize their cumulative grid energy import. Given the above, in this work we extend a state-of-the-art DHAS, to propose AdaHeat+, a practical DHAS, that, for the first time, incorporates AEC. Our work is applicable to both individual houses and domestic coalitions and comes complete with an allocation mechanism to share the coalition gains. Importantly, we propose an effective heuristic heating schedule planning approach for collective AEC that (i) has a complexity that scales in a linear and parallelizable manner with the coalition size, and (ii) enables AdaHeat+ to handle the distinct preferences, in balancing heating cost and thermal discomfort, of the households. Our approach relies on stochastic IER power output predictions. In this context, we propose a simple and effective formulation for the site-specific calibration of such predictions based on adaptive Gaussian process modeling. Finally, we demonstrate the effectiveness of AdaHeat+ through real data evaluation, to show that collective AEC can improve heating cost-efficiency by up to 60%, compared to independent AEC (and even more when compared to no-AEC).

Many aspects of the design of efficient crowdsourcing processes, such as defining worker’s bonuses, fair prices and time limits of the tasks, involve knowledge of the likely duration of the task at hand. In this work we introduce a new time–sensitive Bayesian aggregation method that simultaneously estimates a task’s duration and obtains reliable aggregations of crowdsourced judgments. Our method, called BCCTime, uses latent variables to represent the uncertainty about the workers’ completion time, the tasks’ duration and the workers’ accuracy. To relate the quality of a judgment to the time a worker spends on a task, our model assumes that each task is completed within a latent time window within which all workers with a propensity to genuinely attempt the labelling task (i.e., no spammers) are expected to submit their judgments. In contrast, workers with a lower propensity to valid labelling, such as spammers, bots or lazy labellers, are assumed to perform tasks considerably faster or slower than the time required by normal workers. Specifically, we use efficient message-passing Bayesian inference to learn approximate posterior probabilities of (i) the confusion matrix of each worker, (ii) the propensity to valid labelling of each worker, (iii) the unbiased duration of each task and (iv) the true label of each task. Using two real- world public datasets for entity linking tasks, we show that BCCTime produces up to 11% more accurate classifications and up to 100% more informative estimates of a task’s duration compared to state–of–the–art methods.

We present an open-source toolkit that allows the easy comparison of the performance of active learning methods over a series of datasets. The toolkit allows such strategies to be constructed by combining a judgement aggregation model, task selection method and worker selection method.The toolkit also provides a user interface which allows researchers to gain insight into worker performance and task classification at runtime.

Social media has led to the democratisation of opinion sharing. A wealth of information about public opinions, current events, and authors' insights into specific topics can be gained by understanding the text written by users. However, there is a wide variation in the language used by different authors in different contexts on the web. This diversity in language makes interpretation an extremely challenging task. Crowdsourcing presents an opportunity to interpret the sentiment, or topic, of free-text. However, the subjectivity and bias of human interpreters raise challenges in inferring the semantics expressed by the text. To overcome this problem, we present a novel Bayesian approach to language understanding that relies on aggregated crowdsourced judgements. Our model encodes the relationships between labels and text features in documents, such as tweets, web articles, and blog posts, accounting for the varying reliability of human labellers. It allows inference of annotations that scales to arbitrarily large pools of documents. Our evaluation using two challenging crowdsourcing datasets shows that by efficiently exploiting language models learnt from aggregated crowdsourced labels, we can provide up to 25% improved classifications when only a small portion, less than 4% of documents has been labelled. Compared to the six state-of-the-art methods, we reduce by up to 67% the number of crowd responses required to achieve comparable accuracy. Our method was a joint winner of the CrowdFlower - CrowdScale 2013 Shared Task challenge at the conference on Human Computation and Crowdsourcing (HCOMP 2013).

This paper addresses the problem of extracting accurate labels from crowdsourced datasets, a key challenge in crowdsourcing. Prior work has focused on modeling the reliability of individual workers, for instance, by way of confusion matrices, and using these latent traits to estimate the true labels more accurately. However, this strategy becomes ineffective when there are too few labels per worker to reliably estimate their quality. To mitigate this issue, we propose a novel community-based Bayesian label aggregation model, CommunityBCC, which assumes that crowd workers conform to a few different types, where each type represents a group of workers with similar confusion matrices. We assume that each worker belongs to a certain community, where the worker's confusion matrix is similar to (a perturbation of) the community's confusion matrix. Our model can then learn a set of key latent features: (i) the confusion matrix of each community, (ii) the community membership of each user, and (iii) the aggregated label of each item. We compare the performance of our model against established aggregation methods on a number of large-scale, real-world crowdsourcing datasets. Our experimental results show that our CommunityBCC model consistently outperforms state-of-the-art label aggregation methods, requiring, on average, 50% less data to pass the 90% accuracy mark.

Understanding the dynamics of a crowdsourcing application and controlling the quality of the data it generates is challenging, partly due to the lack of tools to do so. Provenance is a domain-independent means to represent what happened in an application, which can help verify data and infer their quality. It can also reveal the processes that led to a data item and the interactions of contributors with it. Provenance patterns can manifest real-world phenomena such as a significant interest in a piece of content, providing an indication of its quality, or even issues such as undesirable interactions within a group of contributors. This paper presents an application-independent methodology for analyzing provenance graphs, constructed from provenance records, to learn about such patterns and to use them for assessing some key properties of crowdsourced data, such as their quality, in an automated manner. Validating this method on the provenance records of CollabMap, an online crowdsourcing mapping application, we demonstrated an accuracy level of over 95% for the trust classification of data generated by the crowd therein.

Many crowdsourcing applications require spatial data modelling to make sense of location-based observations provided by multiple users. In this context, we propose a new spatial function modelling approach to address the problem of fusing multiple spatial observations reported by possibly untrustworthy users in the domains of participatory sensing and crowdsourcing applications. Specifically, we use a heteroskedastic Gaussian process model to incorporate user trust modelling into Bayesian spatial regression. In particular, by training the model with the reports gathered from the crowd, we are able to estimate the spatial function at any location of interest and also learn the level of trustworthiness of each user. We show that our method outperforms other standard homoskedastic and heteroskedastic Gaussian processes by up to 23% on a crowdsourced radiation dataset collected during the 2011 Fukushima earthquake in Japan. We also show that our method is able to improve the quality of spatial predictions on synthetic data by up to 70% and is robust in settings of up to 30% presence of untrustworthy users within the crowd.

In this article we consider the special abilities needed by agents for assessing trust based on inference and reasoning. We analyze the case in which it is possible to infer trust towards unknown counterparts by reasoning on abstract classes or categories of agents shaped in a concrete application domain. We present a scenario of interacting agents providing a computational model implementing different strategies to assess trust. Assuming a medical domain, categories, including both competencies and dispositions of possible trustees, are exploited to infer trust towards possibly unknown counterparts. The proposed approach for the cognitive assessment of trust relies on agents' abilities to analyze heterogeneous information sources along different dimensions. Trust is inferred based on specific observable properties (manifesta), namely explicitly readable signals indicating internal features (krypta) regulating agents' behavior and effectiveness on specific tasks. Simulative experiments evaluate the performance of trusting agents adopting different strategies to delegate tasks to possibly unknown trustees, while experimental results show the relevance of this kind of cognitive ability in the case of open multiagent systems.

In this paper we present some results obtained through an experimental evaluation of tangible user interfaces (TUIs), comparing their novel interaction paradigms with more conventional interfaces, such as a joypad and a keyboard. Our main goal is to make a formal assessment of TUIs in robotics through a rigorous and extensive experimental evaluation. Firstly, we identified the main benefits of TUIs for robot teleoperation in a urban search and rescue task. Secondly, we provide an evaluation framework to allow for an effective comparison of tangible interfaces with other input devices.

Typical solutions for agents assessing trust relies on the circulation of information on the individual level, i.e. reputational images, subjective experiences, statistical analysis, etc. This work presents an alternative approach, inspired to the cognitive heuristics enabling humans to reason at a categorial level. The approach is envisaged as a crucial ability for agents in order to: (1) estimate trustworthiness of unknown trustees based on an ascribed membership to categories; (2) learn a series of emergent relations between trustees observable properties and their effective abilities to fulfill tasks in situated conditions. On such a basis, categorization is provided to recognize signs (Manifesta) through which hidden capabilities (Kripta) can be inferred. Learning is provided to refine reasoning attitudes needed to ascribe tasks to categories. A series of architectures combining categorization abilities, individual experiences and context awareness are evaluated and compared in simulated experiments.

Tangible user interfaces (TUIs) exhibit innovative interaction paradigms, for example through motion sensing, tactile feedback, or gesturing. Whilst wide spread in human-computer interaction, their value in robotic systems has still to be assessed. In this paper, we present some results obtained through an extensive experimental evaluation of motion sensing interaction paradigms implemented on TUIs for the teleoperation of ground robots. Once analyzed the collected data, in order to evince relevant properties of TUIs, we provide a detailed discussion of our results in terms of mission-related performance, environment conditions, robot operation degree, and human cognitive effort. Our belief is that such a study represents a valuable step towards a formal assessment of tangible interfaces in robotics.