International audience

International audience

This paper explores and offers guidance on a specific and relevant problem in task design for crowdsourcing: how to formulate a complex question used to classify a set of items. In micro-task markets, classification is still among the most popular tasks. We situate our work in the context of information retrieval and multi-predicate classification, i.e., classifying a set of items based on a set of conditions. Our experiments cover a wide range of tasks and domains, and also consider crowd workers alone and in tandem with machine learning classifiers. We provide empirical evidence into how the resulting classification performance is affected by different predicate formulation strategies, emphasizing the importance of predicate formulation as a task design dimension in crowdsourcing.

This paper reports on the challenges and lessons we learned while running controlled experiments in crowdsourcing platforms. Crowdsourcing is becoming an attractive technique to engage a diverse and large pool of subjects in experimental research, allowing researchers to achieve levels of scale and completion times that would otherwise not be feasible in lab settings. However, the scale and flexibility comes at the cost of multiple and sometimes unknown sources of bias and confounding factors that arise from technical limitations of crowdsourcing platforms and from the challenges of running controlled experiments in the “wild”. In this paper, we take our experience in running systematic evaluations of task design as a motivating example to explore, describe, and quantify the potential impact of running uncontrolled crowdsourcing experiments and derive possible coping strategies. Among the challenges identified, we can mention sampling bias, controlling the assignment of subjects to experimental conditions, learning effects, and reliability of crowdsourcing results. According to our empirical studies, the impact of potential biases and confounding factors can amount to a 38% loss in the utility of the data collected in uncontrolled settings; and it can significantly change the outcome of experiments. These issues ultimately inspired us to implement CrowdHub, a system that sits on top of major crowdsourcing platforms and allows researchers and practitioners to run controlled crowdsourcing projects.

Text classification is one of the most common goals of machine learning (ML) projects, and also one of the most frequent human intelligence tasks in crowdsourcing platforms. ML has mixed success in such tasks depending on the nature of the problem, while crowd-based classification has proven to be surprisingly effective, but can be expensive. Recently, hybrid text classification algorithms, combining human computation and machine learning, have been proposed to improve accuracy and reduce costs. One way to do so is to have ML highlight or emphasize portions of text that it believes to be more relevant to the decision. Humans can then rely only on this text or read the entire text if the highlighted information is insufficient. In this paper, we investigate if and under what conditions highlighting selected parts of the text can (or cannot) improve classification cost and/or accuracy, and in general how it affects the process and outcome of the human intelligence tasks. We study this through a series of crowdsourcing experiments running over different datasets and with task designs imposing different cognitive demands. Our findings suggest that highlighting is effective in reducing classification effort but does not improve accuracy - and in fact, low-quality highlighting can decrease it.

Systematic literature reviews (SLRs) are one of the most common and useful form of scientific research and publication. Tens of thousands of SLRs are published each year, and this rate is growing across all fields of science. Performing an accurate, complete and unbiased SLR is however a difficult and expensive endeavor. This is true in general for all phases of a literature review, and in particular for the paper screening phase, where authors filter a set of potentially in-scope papers based on a number of exclusion criteria. To address the problem, in recent years the research community has began to explore the use of the crowd to allow for a faster, accurate, cheaper and unbiased screening of papers. Initial results show that crowdsourcing can be effective, even for relatively complex reviews.

Literature reviews allow scientists to stand on the shoulders of giants, showing promising directions, summarizing progress, and pointing out existing challenges in research. At the same time conducting a systematic literature review is a laborious and consequently expensive process. In the last decade, there have been several studies on crowdsourcing in literature reviews. This paper explores the feasibility of crowdsourcing for facilitating the literature review process in terms of results, time and effort, and identifies which crowdsourcing strategies provide the best results based on the budget available. In particular we focus on the screening phase of the literature review process and we contribute and assess strategies for running crowdsourcing tasks that are efficient in terms of budget and classification error. Finally, we present our findings based on experiments run on Crowdflower.

Crowdsourcing (CS) is the outsourcing of a unit of work to a crowd of people via an open call for contributions. Thanks to the availability of online CS platforms, such as Amazon Mechanical Turk or CrowdFlower, the practice has experienced a tremendous growth over the past few years and demonstrated its viability in a variety of fields, such as data collection and analysis or human computation. Yet it is also increasingly struggling with the inherent limitations of these platforms: each platform has its own logic of how to crowdsource work (e.g., marketplace or contest), there is only very little support for structured work (work that requires the coordination of multiple tasks), and it is hard to integrate crowdsourced tasks into state-of-the-art business process management (BPM) or information systems. We attack these three shortcomings by (1) developing a flexible CS platform (we call it Crowd Computer , or CC) that allows one to program custom CS logics for individual and structured tasks, (2) devising a BPMN--based modeling language that allows one to program CC intuitively, (3) equipping the language with a dedicated visual editor, and (4) implementing CC on top of standard BPM technology that can easily be integrated into existing software and processes. We demonstrate the effectiveness of the approach with a case study on the crowd-based mining of mashup model patterns.

Despite the common claim by mashup platforms that they enable end-users to develop their own software, in practice end-users still don't develop their own mashups, as the highly technical or inexistent user bases of today's mashup platforms testify. The key shortcoming of current platforms is their general-purpose nature, that privileges expressive power over intuitiveness. In our prior work, we have demonstrated that a domain-specific mashup approach, which privileges intuitiveness over expressive power, has much more potential to enable end-user development (EUD). The problem is that developing mashup platforms—domain-specific or not—is complex and time consuming . In addition, domain-specific mashup platforms by their very nature target only a small user basis, that is, the experts of the target domain, which makes their development not sustainable if it is not adequately supported and automated. With this article, we aim to make the development of custom, domain-specific mashup platforms cost-effective. We describe a mashup tool development kit (MDK) that is able to automatically generate a mashup platform (comprising custom mashup and component description languages and design-time and runtime environments) from a conceptual design and to provision it as a service . We equip the kit with a dedicated development methodology and demonstrate the applicability and viability of the approach with the help of two case studies.

This paper presents a model and a tool that allows Web users to define, execute, and manage lifecycles for any artifact available on the Web. In the paper we show the need for lifecycle management of Web artifacts, and we show in particular why it is important that non-programmers are also able to do this. We then discuss why current models do not allow this, and we present a model and a system implementation that achieves lifecycle management for any URI-identifiable and accessible object. The most challenging parts of the work lie in the definition of a simple but universal model and system (and in particular in allowing universality and simplicity to coexist) and in the ability to hide from the lifecycle modeler the complexity intrinsic in having to access and manage a variety of resources, which differ in nature, in the operations that are allowed on them, and in the protocols and data formats required to access them.

The rapid growth of social networking sites and web communities have motivated web sites to expose their APIs to external developers who create mashups by assembling existing functionalities. Current APIs, however, aim toward developers with programming expertise; they are not directly usable by wider class of users who do not have programming background, but would nevertheless like to build their own mashups. To address this need, we propose a spreadsheet-based Web mashups development framework, which enables users to develop mashups in the popular spreadsheet environment. First, we provide a mechanism that makes structured data first class values of spreadsheet cells. Second, we propose a new component model that can be used to develop fairly sophisticated mashups, involving joining data sources and keeping spreadsheet data up to date. Third, to simplify mashup development, we provide a collection of spreadsheet-based mashup patterns that captures common Web data access and spreadsheet presentation functionalities. Users can reuse and customize these patterns to build spreadsheet-based Web mashups instead of developing them from scratch. Fourth, we enable users to manipulate structured data presented on spreadsheet in a drag-and-drop fashion. Finally, we have developed and tested a proof-of-concept prototype to demonstrate the utility of the proposed framework.

Business processes (BPs) are central to the operation of both public and private organizations. A business process is a set of coordinated tasks and activities to achieve a business objective or goal. Given the importance of BPs to overall efficiency and effectiveness, the competitiveness of organizations hinges on continuous BP improvement. In the nineties, the focus of BP improvement was on automation: workflow management systems (WfMSs) and other middleware technologies were used to reduce cost and improve efficiency by providing better system integration and automated enactment of operational business processes. Recently, the focus of business process has expanded to monitoring, analysis and understanding of business processes, and such techniques are incorporated in business process management systems (BPMSs).

Understanding the business (interaction) protocol supported by a service is very important for both clients and service providers: it allows developers to know how to write clients that interact with a service, and it allows development tools and runtime middleware to deliver functionality that simplifies the service development lifecycle. It also greatly facilitates the monitoring, visualization, and aggregation of interaction data. This paper presents an approach for discovering protocol definitions from real-world service interaction logs. It first describes the challenges in protocol discovery in such a context. Then, it presents a novel discovery algorithm, which is widely applicable, robust to different kinds of imperfections often present in realworld service logs, and able to derive protocols of small sizes, also thanks to heuristics. As finding the most precise and the smallest model is algorithmically not feasible from imperfect service logs, finally, the paper presents an approach to refine the discovered protocol via user interaction, to compensate for possible imprecision introduced in the discovered model. The approach has been implemented and experimental results show its viability on both synthetic and real-world datasets.

In service-oriented architectures, everything is a service and everyone is a service provider. Web services (or simply services) are loosely coupled software components that are published, discovered, and invoked across the Web. As the use of Web service grows, in order to correctly interact with them, it is important to understand the business protocols that provide clients with the information on how to interact with services. In dynamic Web service environments, service providers need to constantly adapt their business protocols for reflecting the restrictions and requirements proposed by new applications, new business strategies, and new laws, or for fixing problems found in the protocol definition. However, the effective management of such a protocol evolution raises critical problems: one of the most critical issues is how to handle instances running under the old protocol when it has been changed. Simple solutions, such as aborting them or allowing them to continue to run according to the old protocol, can be considered, but they are inapplicable for many reasons (for example, the loss of work already done and the critical nature of work). In this article, we present a framework that supports service managers in managing the business protocol evolution by providing several features, such as a variety of protocol change impact analyses automatically determining which ongoing instances can be migrated to the new version of protocol, and data mining techniques inferring interaction patterns used for classifying ongoing instances migrateable to the new protocol. To support the protocol evolution process, we have also developed database-backed GUI tools on top of our existing system. The proposed approach and tools can help service managers in managing the evolution of ongoing instances when the business protocols of services with which they are interacting have changed.

This paper discusses process analysis issues, especially in the context of outsourced processes. In particular, we describe a process warehousing solution developed at HP along with the challenges we had to face and the lessons we learned in implementing and deploying it.

This paper deals with the problem of discovering protocol models by analyzing real-world interaction logs. There are several scenarios where protocol discovery is useful and needed: (i) In practice, the protocol definition may not be available. This can happen for many reasons, e.g., the service has been developed using a bottom-up approach, by simply SOAP-ifying an existing application; (ii) even when the protocol model is available, protocol discovery is important as we may want to verify if the designed protocol model is what is actually being supported by the implementation and, if not, what are the differences. An instance of this problem involves discovering if the service is compliant with the protocol specification required by some domain-specific standardization body or industry consortium.

In service-oriented computing, a conversation is a sequence of message exchanges between two or more services to achieve a certain goal, for example to order and pay for goods. A business protocol of a service is a specification of the possible conversations that a service can have with its partners. Motivated by the goal of facilitating the scalable development and maintenance of service oriented applications, especially in light of the many benefits of protocols, we have developed ServiceMosaic (servicemosaic.isima.fr), a platform for Web services life-cycle management. ServiceMosaic is an interactive and model-driven CASE tool for managing Web service interactions, which consists of two broad modules: protocol discovery and protocol management.

In today's Web, many functionality-wise similar Web services are offered through heterogeneous interfaces (operation definitions) and business protocols (ordering constraints defined on legal operation invocation sequences). The typical approach to enable interoperation in such a heterogeneous setting is through developing adapters. There have been approaches for classifying possible mismatches between service interfaces and business protocols to facilitate adapter development. However, the hard job is that of identifying, given two service specifications, the actual mismatches between their interfaces and business protocols. In this paper we present novel techniques and a tool that provides semi-automated support for identifying and resolution of mismatches between service interfaces and protocols, and for generating adapter specification. We make the following main contributions: (i) we identify mismatches between service interfaces, which leads to finding mismatches of type of signature, merge/split, and extra/missing messages; (ii) we identify all ordering mismatches between service protocols and generate a tree, called mismatch tree, for mismatches that require developers' input for their resolution. In addition, we provide semi-automated support in analyzing the mismatch tree to help in resolving such mismatches. We have implemented the approach in a tool inside IBM WID (WebSphere Integration Developer). Our experiments with some real-world case studies show the viability of the proposed approach. The methods and tool are significant in that they considerably simplify the problem of adapting services so that interoperation is possible.

The development of user interfaces (UIs) is one of the most time-consuming aspects in software development. In this context, the lack of proper reuse mechanisms for UIs is increasingly becoming manifest, especially as software development is more and more moving toward composite applications. In this paper we propose a framework for the integration of stand-alone modules or applications, where integration occurs at the presentation layer. Hence, the final goal is to reduce the effort required for UI development by maximizing reus.

There is an increasing demand in making web user interfaces richer and more interactive. Currently, there are two approaches aiming at improving web user interfaces. First, downloaded code in form of Java Applet or ActiveX can be executed in browsers. And more recently, AJAX (Asynchronous JavaScript + XML) leverages browsers' JavaScript engine to render user interfaces without reloading pages. Both approaches have some weaknesses.In this paper, we present an alternative approach to creating highly interactive web user interfaces. Our approach is based on the Extensible User Interface Protocol (XUP), a SOAP-based protocol for communicating events and incremental user interface updates on the web. On top of XUP, we have built a web user interface development framework, OpenXUP, consisting of a thin client and a server toolkit which offers a set of event-driven APIs. The framework allows for the rapid development of highly interactive web applications and services.

Web services, and more in general service-oriented architectures (SOAs), are emerging as the technologies and architectures of choice for implementing distributed systems and performing application integration within and across companies boundaries. In this article we describe Web services from an evolutionary perspective, with an emphasis on the utilization for enterprise application integration and service-oriented architectures. The article also covers basic middleware problems and shows how the solutions to these problems have finally evolved into what we call today Web services.

As IT systems become more and more complex and as business operations become increasingly automated, there is a growing need from business managers to have better control on business operations and on how these are aligned with business goals. This paper describes iBOM, a platform for business operation management developed by HP that allows users to i) analyze operations from a business perspective and manage them based on business goals; ii) define business metrics, perform intelligent analysis on them to understand causes of undesired metric values, and predict future values; iii) optimize operations to improve business metrics. A key aspect is that all this functionality is readily available almost at the click of the mouse. The description of the work proceeds from some specific requirements to the solution developed to address them. We also show that the platform is indeed general, as demonstrated by subsequent deployment domains other than finance.

A scalable approach to trust negotiation is required in Web service environments that have large and dynamic requester populations. We introduce Trust-Serv, a model-driven trust negotiation framework for Web services. The framework employs a model for trust negotiation that is based on state machines, extended with security abstractions. Our policy model supports lifecycle management, an important trait in the dynamic environments that characterize Web services. In particular, we provide a set of change operations to modify policies, and migration strategies that permit ongoing negotiations to be migrated to new policies without being disrupted. Experimental results show the performance benefit of these strategies. The proposed approach has been implemented as a container-centric mechanism that is transparent to the Web services and to the developers of Web services, simplifying Web service development and management as well as enabling scalable deployments.

Web services are emerging as a promising technology for the automation of inter-organizational interactions. As technology matures and the foundations of Web services become more solid, users will start to demand tools that facilitate the service development lifecycle. It is only when such tools become available that novel technologies become applied and enter the mainstream, since the complexity, cost and time necessary to deploy and manage solutions is dramatically reduced. In this paper, we present a framework and a tool that support the model-driven development of Web services. The idea consists in identifying key Web services abstractions, in addition to those of basic Web services standards, that enable the description of service policies and properties that are useful in practice. In this paper, we focus on service protocols, and specifically on conversation and trust negotiation protocols. These protocols are modeled by means of graphical tools and high-level languages so that they are easy to specify, understand, and evolve. The tools also support the automatic generation of service implementation skeletons based on these abstractions, manage the entire service lifecycle, and provide run-time support to verify that the interaction among clients and services occur in compliance with the specified policies.

Business-to-business (B2B) e-commerce is emerging as a new market with tremendous potential. Organizations are trying to link services across organizational boundaries in order to electronically trade goods and services. Standards such as RosettaNet, CBL, ED1, OB1, and cXML, describe how electronic B2B interactions should be carried on so that dynamic trade partnerships can be established and transactions can be executed across organizations. While the development of standards is a fundamental step towards enabling e-business, the problem of linking B2B interactions with internal business processes is still a challenge. In addition, as the industry standards evolve continuously based on changing needs, organizations have to adopt new standards quickly. In this paper we describe how workflow technology can be extended in order to support B2B interactions and to link them with the internal workflows. The proposed framework can be used to speed up both the development of new business processes that support B2B interaction standards and the enhancement of the existing business processes by the addition of B2B interaction capability.

Supply-chain management is a crucial activity in every company. Surprisingly, today, most of the supply-chain activities are carried out manually, and IT support is often limited to having a set of (disconnected) data repositories. In addition, business-to-business (B2B) communications are performed via phone, fax or e-mail. Increasing the operational efficiency of the supply chain results in huge savings and is the key to remaining competitive or even gaining a competitive advantage. Furthermore, a more efficient supply chain also enables revenue growth, which is often impossible to sustain with the current manual operations. In this paper, we discuss the requirements and challenges for e-business applications that support supply-chain management. Then, we propose an architecture that meets the requirements and enables solutions that deliver results quickly and that evolve with the business and IT environment. Both the requirements and the architecture are the results of several different types of supply-chain automation projects in which we have been involved.

No abstract available.

Although workflow management systems are most applicable when an organization follows standard business processes and routines, any of these processes faces the need for handling exceptions, i.e., asynchronous and anomalous situations that fall outside the normal control flow. In this paper we concentrate upon anomalous situtations that, although unusual, are part of the semantics of workflow applications, and should be specified and monitored coherently; in most real-life applications, such exceptions affect a significant fraction of workflow cases. However, very few workflow management systems are integrated with a highly expressive language for specifying this kind of exception and with a system component capable of handling it. We present Chimera-Exc, a language for the specification of exceptions for workflows based on detached active rules, and then describe the architecture of a system, called FAR, that implements Chimera-Exc and integrates it with a commercial workflow management system and database server. We discuss the main issues that were solved by our implementation, and report on the performance of FAR. We also discuss design criteria for exceptions in light of the formal properties of their execution. Finally, we focus on the portability of FAR on its unbundling to a generic architecture with detached active rules.