Robustness of reasoning remains a significant challenge for large language models, and addressing it is essential for the practical applicability of AI-driven reasoning systems. We introduce Semantic Self-Verification (SSV), a novel approach that addresses the key challenge in combining language models with the rigor of logical solvers: to accurately formulate the reasoning problem from natural language to the formal language of the solver. SSV uses a consistency-based approach to produce strong abstract formalizations of problems using concrete instantiations that are generated by the model and verified by the solver. In addition to significantly advancing the overall reasoning accuracy over the state-of-the-art, a key novelty that this approach presents is a feature of verification that has near-perfect precision over a significant coverage of cases, as we demonstrate on open reasoning benchmarks. We propose such *near-certain reasoning* as a new approach to reduce the need for manual verification in many cases, taking us closer to more dependable and autonomous AI reasoning systems.

Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, Firoj Alam. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Incorporating conversational context and knowledge into dialogue generation models has been essential for improving the quality of the generated responses.The context, comprising utterances from previous dialogue exchanges, is used as a source of content for response generation and as a means of selecting external knowledge.However, to avoid introducing irrelevant content, it is key to enable fine-grained scoring of context and knowledge.In this paper, we present a novel approach to context and knowledge weighting as an integral part of model training.We guide the model training through a Contextual Knowledge Learning (CKL) process which involves Latent Vectors for context and knowledge, respectively.CKL Latent Vectors capture the relationship between context, knowledge, and responses through weak supervision and enable differential weighting of context utterances and knowledge sentences during the training process.Experiments with two standard datasets and human evaluation demonstrate that CKL leads to a significant improvement compared with the performance of six strong baseline models and shows robustness with regard to reduced sizes of training sets.

Over the past eight decades, computer science has advanced as a field, and the computing profession has matured by establishing professional codes of conduct, fostering best practices, and establishing industry standards to support the proliferation of technologies and services. Research and applications of digital computation continue to change all aspects of human endeavor through new waves of innovation. While it is clear that different research advances fuel innovation, the ways they come together to make an impact vary. In contrast to highly regulated sectors such as pharma, medicine and law, the process of transforming research into widely deployed technologies is not regulated. We reflect on collective practices, from discovery by scientists and engineers to market delivery by entrepreneurs, industry leaders, and practitioners. We consider ecosystem changes that are required to sustain the transformational effects of new technologies and enable new practices to take root. Every such transformation ruptures in the existing socio-technical fabric and requires a concerted effort to remedy this through effective policies and regulations. Computing experts are involved in all phases and must match the transformational power of their innovation with the highest standard of professional conduct. We highlight the principles of responsible innovation and discuss three waves of digital innovation. We use wide and uncontrolled generative AI deployments to illustrate risks from the implosion of digital media due to contamination of digital records, removal of human agency, and risk to an individual's personhood.

Systematic reviews are a comprehensive and parameterised form of literature review, found in most disciplines, that involve exhaustive analyses and rigorous interpretation of prior literature. Performing systematic reviews, however, can involve repetitive and laborious work in order to reach reliable standards. Strict guidelines and availability of published reviews make the task amenable to computerised assistance and automation using text mining, information extraction, and machine learning techniques. However, it is unclear which aspects of this Work Task are best suited for such support. This paper describes a three-month ethnographic study and CognitiveWork Analysis of the systematic reviews performed by a medical research group. Our findings show that the IR aspects of systematic reviews involve many tasks at two separate levels: 1) taxonomic organisation of documents and sub-document elements in relation to topic queries and domain-specific resources, and 2) extraction methods for structured summaries from the classified resources. This provides the basis for future work designing search tools with localised optimization and subtask automation to support specific phases of the process.

Web sites and online services increasingly engage with users through live chats to provide support, advice, and offers. Such approaches require reliable methods to predict the user’s intent and make an informed decision when and how to intervene during an active session. Prior work on predicting purchase intent involved clickstream data mining and feature construction in an ad-hoc manner with a moderate success (AUC 0.70 range). We demonstrate the use of the consumer Purchase Decision Model (PDM) and a principled way of constructing features predictive of the purchase intent. We show that the Logistic Regression (LR) classifiers, trained with multi-action motifs, perform on par with the state-of-the-art LSTM sequence model achieving comparable AUC (0.95 vs 0.96) and performing better for the sparse purchase sessions, with higher recall (0.85 vs 0.61) and higher F1 score (0.73 vs 0.66). While LSTM performs better than LR in terms of weighted averages of F1, precision, and recall, it requires 7 times longer to train and offers no insights about the predictive model in terms of the user actions and the purchase decision stages. The LR predictors are robust and effective in simulating real-time interventions, achieving F1 of 0.84 and AUC of 0.85 after observing only 50% of an active session. For non-purchase sessions that leaves room for live intervention, on average within 8 actions before the session ends.

E-commerce platforms facilitate sales of products while product vendors engage in Social Media Activities (SMA) to drive E-commerce Platform Activities (EPA) of consumers, enticing them to search, browse and buy products. The frequency and timing of SMA are expected to affect levels of EPA, increasing the number of brand related queries, clickthrough, and purchase orders. This paper applies cross-sectional data analysis to explore such beliefs and demonstrates weak-to-moderate correlations between daily SMA and EPA volumes. Further correlation analysis, using 30-day rolling windows, shows a high variability in correlation of SMA-EPA pairs and calls into question the predictive potential of SMA in relation to EPA. Considering the moderate correlation of selected SMA and EPA pairs (e.g., Post-Orders), we investigate whether SMA features can predict changes in the EPA levels, instead of precise EPA daily volumes. We define such levels in terms of EPA distribution quantiles (2, 3, and 5 levels) over training data. We formulate the EPA quantile predictions as a multi-class categorization problem. The experiments with Random Forest and Logistic Regression show a varied success, performing better than random for the top quantiles of purchase orders and for the lowest quantile of search and clickthrough activities. Similar results are obtained when predicting multi-day cumulative EPA levels (1, 3, and 7 days). Our results have considerable practical implications but, most importantly, urge the common beliefs to be re-examined, seeking a stronger evidence of SMA effects on EPA.

Improvements of entity-relationship (E-R) search techniques have been hampered by a lack of test collections, particularly for complex queries involving multiple entities and relationships. In this paper we describe a method for generating E-R test queries to support comprehensive E-R search experiments. Queries and relevance judgments are created from content that exists in a tabular form where columns represent entity types and the table structure implies one or more relationships among the entities. Editorial work involves creating natural language queries based on relationships represented by the entries in the table. We have publicly released the RELink test collection comprising 600 queries and relevance judgments obtained from a sample of Wikipedia List-of-lists-of-lists tables. The latter comprise tuples of entities that are extracted from columns and labelled by corresponding entity types and relationships they represent. In order to facilitate research in complex E-R retrieval, we have created and released as open source the RELink Framework that includes Apache Lucene indexing and search specifically tailored to E-R retrieval. RELink includes entity and relationship indexing based on the ClueWeb-09-B Web collection with FACC1 text span annotations linked to Wikipedia entities. With ready to use search resources and a comprehensive test collection, we support community in pursuing E-R research at scale.

Recent research has explored the use of Knowledge Bases (KBs) to represent documents as subgraphs of a KB concept graph and define metrics to characterize semantic relatedness of documents in terms of properties of the document concept graphs. However, none of the studies so far have examined to what degree such metrics capture a user-perceived relatedness of documents. Considering the users' explanations of how pairs of documents are related, the aim is to identify concepts in a KB graph that express the same notion of document relatedness. Our algorithm generates paths through the KB graph that originate from the terms in two documents. KB concepts where these paths intersect capture the semantic relatedness of the two starting terms and therefore the two documents. We consider how such intersecting concepts relate to the concepts in the users' explanations. The higher the users' concepts appear in the ranked list of intersecting concepts, the better the method in capturing the users' notion of document relatedness. Our experiments show that our approach outperforms a simpler graph method that uses properties of the concept nodes alone.

Good alignment and repetition of objects across presentation slides can facilitate visual processing and contribute to audience understanding. However, creating and maintaining such consistency during slide design is difficult. To solve this problem, we present two complementary tools: (1) StyleSnap, which increases the alignment and repetition of objects by adaptively clustering object edge positions and allowing parallel editing of all objects snapped to the same spatial extent; and (2) FlashFormat, which infers the least-general generalization of editing examples and applies it throughout the selected range. In user studies of repetitive styling task performance, StyleSnap and FlashFormat were 4-5 times and 2-3 times faster respectively than conventional editing. Both use a mixed-initiative approach to improve the consistency of slide decks and generalize to any situations involving direct editing across disjoint visual spaces.

Recent advances in Programming by Example (PBE) have supported new applications to text editing, but existing approaches are limited to simple text strings. In this paper we address transformations in richly formatted documents, using an approach based on the idea of least general generalizations from inductive inference, which avoids the scalability issues faced by state-of-the-art PBE methods. We describe a novel domain specific language (DSL) that expresses transformations over XML structures describing richly formatted content, and a synthesis algorithm that generates a minimal program with respect to a natural subsumption ordering in our DSL. We present experimental results on tasks collected from online help forums, showing an average of 4.17 examples required for task completion.

HCI is increasingly interested in amateurism, but the wider literature suggests that the amateur is a complex and distinctive phenomenon. An interview study reveals the nature of the amateur in the digital age. Even though operating non-professionally at a micro-scale, amateur musicians employ a plethora of online services to sustain local fanbases, reach out to new fans, collaborate internationally, and actively promote both digital and material products. Our findings lead to recommendations for event-oriented promotion tools; community-oriented analytics; tangible and embedded products; and limited-edition digital experiences. We conclude that HCI needs to recognise the amateur as an important class of user, one who is serious about their leisure, and who is also distinct from the professional as from the novice and hobbyist.

Prolific adoption of digital media across scientific fields has led to inevitable transformation of a traditional lab book into an electronic lab notebook (ELN). Research so far has focussed on designing ELN prototypes and learning from their limited deployments. At the same time, a variety of commercially available ELNs have been adopted by industrial and academic laboratories. That provides opportunities for situated research and a deeper understanding of the role that ELNs assumes as an integral part of a scientific environment. In this paper we present a study of ELN design that has emerged as scientists appropriated commercial off-the-shelf note-taking software and adapted it to their work. Through in-situ observations we analysed the interplay between the technology and emerging practices. Our study reveals a tension that is intrinsic to the digital nature of ELNs: a conflict between the flexibility, fluidity, and low threshold for modifying digital records and the requirement for persistence and consistency. This led to refined requirements and design considerations for ELNs.

Our research is concerned with designing support for ubiquitous access, organization, and interpretation of digital assets that users produce and store across multiple devices and computing platforms. Through a co-design study with scientists we identified specific aspects of conceptual maps they wish to create for their projects in order to make sense of diverse collections of digital media related to their individual and collective work. By the use of a technology probe called DeskPiles we observed the mechanisms involved in such a process. We identified the practice of using sub-document elements to compose the views of digital collections and diverse linking strategies to express semantic relations among concepts and ideas and to enable access to supporting digital assets. Our findings reveal the need to expand traditional information architectures and include (1) an extended range of content reference options and (2) transformation services to enable extraction, conversion, and linking of digital content. We advocate a granular and multi-format representation of content as the basis for reuse and ubiquitous access of digital media in diverse multi-device environments.

Marketing campaigns using social media services aim to exploit social connections to propagate messages to potential customers. However, social activities often give rise to multiple network structures and some may be more effective in achieving the communication objectives than others. This led us to investigate a problem: given an observed sequence of messages and a social network that includes individuals involved in messaging, does the network structure 'explain' the observed propagation. To facilitate this investigation, we designed a method for encoding propagation events relative to the structure of a given network. The resulting transmission codes capture both the temporal and the structural characteristics of the propagation. We analyze the codes for maximal repeats and k-common sub strings to uncover dynamic network motifs within the propagation trace. By considering the dynamic motifs and the connected graph components, we can determine how the propagation events relate to the specific network. As a case study, we applied our method to rumor topics in Twitter and analyzed their propagation trails relative to the 'follower' network. The study demonstrates the computational feasibility of our approach and illustrates the use of dynamic motifs to reason about the impact of follower relationship rumor propagation in Twitter.

Internet advertisers reach millions of customers through practices that real time tracking of users' online activities. The tracking is conducted by third party ad services engaged by the Web sites to facilitate marketing campaigns. Previous research has investigated tracking practices and tracking agencies associated with popular Web sites. Here we investigate the network properties of the third party referral structures that facilitate gathering of user information for the delivery of personalized ads. By considering third party domains associated with the top ten search results for a diverse set of queries, we arrived at the networks of third party domains in four search markets. We show a consistent network structure across markets, with a dominant connected component that, on average, includes 92.8% of network vertices and 99.8% of the connecting edges. There is 99.5% chance that a user will become tracked by all top 10 trackers within 30 clicks on search results. Finally, the third party networks exhibit properties of the small world networks. This implies a high-level global and local efficiency in spreading the user information and delivering targeted ads.

Information retrieval systems require human contributed relevance labels for their training and evaluation. Increasingly such labels are collected under the anonymous, uncontrolled conditions of crowdsourcing, leading to varied output quality. While a range of quality assurance and control techniques have now been developed to reduce noise during or after task completion, little is known about the workers themselves and possible relationships between workers' characteristics and the quality of their work. In this paper, we ask how do the relatively well or poorly-performing crowds, working under specific task conditions, actually look like in terms of worker characteristics, such as demographics or personality traits. Our findings show that the face of a crowd is in fact indicative of the quality of their work.

Scientific communities have long been concerned with the design and implementation of effective infrastructures for data access and collaborative scientific work. Recent studies have shown an increase in collaborative data generation and reuse. However, further improvements require a deeper understanding of the social and technological circumstances under which they emerge. To that effect we conduct in-situ observation study of a Nano-photonics Research Centre. We consider the artifact ecology that evolved from the Centre's common experimentation and data platform, the scientific practices, and the intricate interactions with digital artifacts that arise from the researchers' activities. We uncover the use of progress summaries for collaborative data interpretation and knowledge sharing. By studying this reputable collaborative scientific environment we (1) identified the factors that led to its functional and effective artifact ecology and (2) propose expansions of tools and services to improve it further. The latter include effective support for contextual search, browsing, and flexible viewing of information artifacts based on relevant parameters and properties.

We propose a mathematical framework for query selection as a mechanism for reducing the cost of constructing information retrieval test collections. In particular, our mathematical formulation explicitly models the uncertainty in the retrieval effectiveness metrics that is introduced by the absence of relevance judgments. Since the optimization problem is computationally intractable, we devise an adaptive query selection algorithm, referred to as Adaptive, that provides an approximate solution. Adaptive selects queries iteratively and assumes that no relevance judgments are available for the query under consideration. Once a query is selected, the associated relevance assessments are acquired and then used to aid the selection of subsequent queries. We demonstrate the effectiveness of the algorithm on two TREC test collections as well as a test collection of an online search engine with 1000 queries. Our experimental results show that the queries chosen by Adaptive produce reliable performance ranking of systems. The ranking is better correlated with the actual systems ranking than the rankings produced by queries that were selected using the considered baseline methods.

We introduce "Facet-Streams", a hybrid interactive surface for co-located collaborative product search on a tabletop. Facet-Streams combines techniques of information visualization with tangible and multi-touch interaction to materialize collaborative search on a tabletop. It harnesses the expressive power of facets and Boolean logic without exposing users to complex formal notations. Two user studies reveal how Facet-Streams unifies visual and tangible expressivity with simplicity in interaction, supports different strategies and collaboration styles, and turns product search into a fun and social experience.

We consider the problem of optimally allocating a fixed budget to construct a test collection with associated relevance judgements, such that it can (i) accurately evaluate the relative performance of the participating systems, and (ii) generalize to new, previously unseen systems. We propose a two stage approach. For a given set of queries, we adopt the traditional pooling method and use a portion of the budget to evaluate a set of documents retrieved by the participating systems. Next, we analyze the relevance judgments to prioritize the queries and remaining pooled documents for further relevance assessments. The query prioritization is formulated as a convex optimization problem, thereby permitting efficient solution and providing a flexible framework to incorporate various constraints. Query-document pairs with the highest priority scores are evaluated using the remaining budget. We evaluate our resource optimization approach on the TREC 2004 Robust track collection. We demonstrate that our optimization techniques are cost efficient and yield a significant improvement in the reusability of the test collections.

Crowdsourcing platforms offer unprecedented opportunities for creating evaluation benchmarks, but suffer from varied output quality from crowd workers who possess different levels of competence and aspiration. This raises new challenges for quality control and requires an in-depth understanding of how workers' characteristics relate to the quality of their work.

The evaluation of information retrieval (IR) systems over special collections, such as large book repositories, is out of reach of traditional methods that rely upon editorial relevance judgments. Increasingly, the use of crowdsourcing to collect relevance labels has been regarded as a viable alternative that scales with modest costs. However, crowdsourcing suffers from undesirable worker practices and low quality contributions. In this paper we investigate the design and implementation of effective crowdsourcing tasks in the context of book search evaluation. We observe the impact of aspects of the Human Intelligence Task (HIT) design on the quality of relevance labels provided by the crowd. We assess the output in terms of label agreement with a gold standard data set and observe the effect of the crowdsourced relevance judgments on the resulting system rankings. This enables us to observe the effect of crowdsourcing on the entire IR evaluation process. Using the test set and experimental runs from the INEX 2010 Book Track, we find that varying the HIT design, and the pooling and document ordering strategies leads to considerable differences in agreement with the gold set labels. We then observe the impact of the crowdsourced relevance label sets on the relative system rankings using four IR performance metrics. System rankings based on MAP and Bpref remain less affected by different label sets while the [email protected] and [email protected] lead to dramatically different system rankings, especially for labels acquired from HITs with weaker quality controls. Overall, we find that crowdsourcing can be an effective tool for the evaluation of IR systems, provided that care is taken when designing the HITs.

With the popularity of social media sites, digital content is increasingly stored and managed online. At the same time, the desktop and local storage continues to provide a personal environment in which users perform their daily tasks. Thus, to accomplish their tasks, users need to continuously switch between local and remote resources and applications, often carrying the burden of coordinating and synchronizing these in a consistent way. In this demonstration, we describe a system, called ScholarLynk, that bridges the local and online worlds and allows users to manage both local and online resources in a uniform way and in collaboration with others.

In this paper we present a case study of a longitudinal in-situ observation that involves a new social application for mobile communication. Our study demonstrates the need for an adaptive approach to planning, design, and implementation that is responsive to emerging social and infrastructure conditions. This represents a shift from traditional longitudinal studies that observe prototype systems with fixed sets of affordances. In the case of mobile and social applications there is a complex interaction between the social dynamics, the new technology, and the mobile infrastructure. Exploratory research thus requires approaches that can deal with such complex conditions. That includes a high level of prototype plasticity to ensure adoption and sustained use that is needed for longitudinal in-situ research. The social aspects dictate specific forms of instrumentation to enable observation of social interactions and mechanisms to inject the new technology into an existing social and communication ecosystem. Our study demonstrates the evolving use of complementary techniques and in-situ modifications of the prototype to support longitudinal observations in a real setting.

User experience in social media involves rich interactions with the media content and other participants in the community. In order to support such communities, it is important to understand the factors that drive the users' engagement. In this paper we show how to define statistical models of different complexity to describe patterns of song listening in an online music community. First, we adapt the LDA model to capture music taste from listening activities across users and identify both the groups of songs associated with the specific taste and the groups of listeners who share the same taste. Second, we define a graphical model that takes into account listening sessions and captures the listening moods of users in the community. Our session model leads to groups of songs and groups of listeners with similar behavior across listening sessions and enables faster inference when compared to the LDA model. Our experiments with the data from an online media site demonstrate that the session model is better in terms of the perplexity compared to two other models: the LDA-based taste model that does not incorporate cross-session information and a baseline model that does not use latent groupings of songs.

Could people use tagging to manage day-to-day work in their personal computing environment? Could tagging be sufficiently generic and lightweight to support diverse ways of working and, perhaps, support new and efficient practices for managing applications and accessing documents? We investigate these issues by implementing the TAGtivity system that enables users to tag resources in the context of their ongoing work. We deployed TAGtivity and studied users' tagging practices in their actual work places over a three week period. Our analysis of interviews and logs reveals that affordances of the TAGtivity system supported users in a variety of information and activity management tasks. These include new practices for managing emerging activities and ephemeral information and accessing documents across application data silos.

Standard approaches to evaluating and comparing information retrieval systems compute simple averages of performance statistics across individual topics to measure the overall system performance. However, topics vary in their ability to differentiate among systems based on their retrieval performance. At the same time, systems that perform well on discriminative queries demonstrate notable qualities that should be reflected in the systems' evaluation and ranking. This motivated research on alternative performance measures that are sensitive to the discriminative value of topics and the performance consistency of systems. In this paper we provide a mathematical formulation of a performance measure that postulates the dependence between the system and topic characteristics. We propose the Generalized Adaptive-Weight Mean (GAWM) measure and show how it can be computed as a fixed point of a function for which the Brouwer Fixed Point Theorem applies. This guarantees the existence of a scoring scheme that satisfies the starting axioms and can be used for ranking of both systems and topics. We apply our method to TREC experiments and compare the GAWM with the standard averages used in TREC.

Knowledge sharing communities, such as Wikipedia or Yahoo! Answers, add greatly to the wealth of information available on the Web. They represent complex social ecosystems that rely on user paricipation and the quality of users' contributions to prosper. However, quality is harder to achieve when knowledge sharing is facilitated through a high degree of personal interactions. The individuals' objectives may change from knowledge sharing to socializing, with a profound impact on the community and the value it delivers to the broader population of Web users. In this paper we provide new insights into the types of content that is shared through Community Question Answering (CQA) services. We demonstrate an approach that combines in-depth content analysis with social network analysis techniques. We adapted the Undirected Inductive Coding method to analyze samples of user questions and arrive at a comprehensive typology of the user intent. In our analysis we focused on two types of intent, social vs. non-social, and define measures of social engagement to characterize the users' participation and content contributions. Our approach is applicable to a broad class of online communities and can be used to monitor the dynamics of community ecosystems.

Growing interest in online collections of digital books and video content motivates the development and optimization of adequate retrieval systems. However, traditional methods for collecting relevance assessments to tune system performance are challenged by the nature of digital items in such collections, where assessors are faced with a considerable effort to review and assess content by extensive reading, browsing, and within-document searching. The extra strain is caused by the length and cohesion of the digital item and the dispersion of topics within it. We propose a method for the collective gathering of relevance assessments using a social game model to instigate participants' engagement. The game provides incentives for assessors to follow a predefined review procedure and makes provisions for the quality control of the collected relevance judgments. We discuss the approach in detail, and present the results of a pilot study conducted on a book corpus to validate the approach. Our analysis reveals intricate relationships between the affordances of the system, the incentives of the social game, and the behavior of the assessors. We show that the proposed game design achieves two designated goals: the incentive structure motivates endurance in assessors and the review process encourages truthful assessment.

Community Question Answering (cQA) services, such as Yahoo! Answers and MSN QnA, facilitate knowledge sharing through question answering by an online community of users. These services include incentive mechanisms to entice participation and self-regulate the quality of the content contributed by the users. In order to encourage quality contributions, community members are asked to nominate the ‘best’ among the answers provided to a question. The service then awards extra points to the author who provided the winning answer and to the voters who cast their vote for that answer. The best answers are typically selected by plurality voting, a scheme that is simple, yet vulnerable to random voting and collusion. We propose a weighted voting method that incorporates information about the voters’ behavior. It assigns a score to each voter that captures the level of agreement with other voters. It uses the voter scores to aggregate the votes and determine the best answer. The mathematical formulation leads to the application of the Brouwer Fixed Point Theorem which guarantees the existence of a voter scoring function that satisfies the starting axiom. We demonstrate the robustness of our approach through simulations and analysis of real cQA service data.

Social networking promises individuals new dimensions of freedom to interact, associate, and give expression to their talents. Recently, systems such as Mechanical Turk have started to facilitate self-organizing collaboration on work-related tasks. Such developments raise interesting questions. Is it possible to create (and sustain) businesses that do not have traditional, formal structure - without traditional "employees"? Can we find and organize (and optimize) talent on the web for task-oriented work - spontaneously and efficiently? How do people relate to one another in possibly evanescent workgroups? One aspect of the challenge in the Social Workspace is understanding and modeling the user behavior and the economic basis for creating, preserving, and exchanging value in the marketplace when workgroup identity, orientations to property, recruiting and managing appropriate talent are not organized under traditional company structures. Another aspect is the technology needed to support virtual organizations and work. The panel will discuss trends in social work and the evolving (scientific) basis of our understanding of new models of workers and organizations.

We present a social information retrieval (SIR) model comprising the social network of actors (e.g., authors, publishers, consumers), the graph representing relations in data (e.g., publications), and the links between the social and data network that reflect activities in the network such as search, authoring, annotation, etc. Building on this hybrid network, we describe relevance in terms of the trust propagated through the network and rendered onto a given item. In particular, relevance is a function of the approval votes from the associated sub-graph and the reputation of the sub-graph nodes. We explore a model that differentiates between approval from actors who are perceived authorities by the user and the approval by a wider community, representing the popular opinion.

In this paper, we consider the task of estimating query effectiveness, i.e., assessment of the retrieval system performance in absence of the user relevance judgments. In our approach we model the score associated with each document in the result set as a Gaussian random variable. The mean and the variance of each document score can then be used to estimate the probability that a document will be ranked above another one and thus calculate the expected rank of the document in the ranked list. We propose to measure the effectiveness of the system performance by comparing the predicted and actual ranks of the retrieved documents. In our experiments we consider two retrieval models and five document scoring methods and evaluate their impact on the proposed estimation measures. Our experiments with standardized data sets that include document relevance judgments and the task of predicting the relative query effectiveness show that the expected rank metric is robust to variations in document scoring and retrieval algorithms.

Building on the Internet (IP) platforms available on mobile devices and personal computers (PCs), we implemented a service called weConnect that facilitates broadcast of content via dedicated and personal media channels. weConnect includes simple tools for integrating content into media 'mixes' and provides 'content viewers' for ubiquitous access to weConnect channels via mobile, desktop, and other IP enabled devices. In this paper we present an exploratory user study based on deployment of the weConnect service among individuals in close relationships. The study focuses on the user perception and experience with the 'always-on' channels, delivering personalized content. We started with images, text, and animations as familiar and easily accessible media. We observed a high level of reciprocity in creating and exchanging expressive content and a need for persistence, reuse, and notification of content delivery. The users voiced their enthusiasm for receiving personalized media across mobile and desktop devices. The always-on nature of the weConnect channels raised new requirements for the service design to assist content producers with creating streams of personalized media efficiently and to enable recipients to view the content flexibly.

On-line community services such as Live QnA and Yahoo! Answers enable their members to ask questions and have them answered by the community. The questions are labelled by the users to facilitate search, navigation, and recommendations. In this paper we provide an in-depth analysis of the question labelling practices by contrasting the use of community generated tags in the Live QnA service with the use of topic categories from a fixed taxonomy in the Yahoo! Answers service. We found that community tagging is related to higher levels of social interactions amongst users. Analysis of the most frequently used community tags reveals that active users may establish strong social ties around specific tags. Furthermore, the discriminative value of individual community tags can be low since the corresponding questions may cover a variety of topics. Thus, appropriate care needs to be taken when designing search, browsing, and recommender features for question discovery.

Improving the classification of newsgroup messages through social network analysis.

This paper reports on an exploratory study of a mobile communication prototype mGuide that enables continuous and data rich messaging across the mobile and desktop platform, augmented by complementary Web services. The study focuses on design aspects of the mobile interface and factors that influence the perceived value and usability of such integrated mobile applications. We show how the complexity of image and voice messaging within a location focused scenario can be successfully mediated by careful user interface design, without detracting from utility and experience. More importantly, our study confirms that the integration of personal communication and Web based information services increases the perceived value of mobile applications. Besides the optimized user interface on the device itself, it is the seamless integration with the user's environment that increases the utility of mobile devices and services. We thus propose design recommendations for Web augmented mobile applications and promote a design optimization approach that encompasses flexible integration with Web-based services.

Web sites are often organized into several regions, each dedicated to a specific topic or serving a particular function. From a user's perspective, these regions typically form coherent sets of pages characterized by a distinct navigation structure and page layout-we refer to them as subsites. In this paper we propose to characterize Web site structure as a collection of subsites and devise a method for detecting subsites and entry points for subsite navigation. In our approach we use a new model for representing Web site structure called Link Structure Graph (LSG). The LSG captures a complete hyperlink structure of a Web site and models link associations reflected in the page layout. We analyze a sample of Web sites and compare the LSG based approach to commonly used statistics for Web graph analysis. We demonstrate that LSG approach reveals site properties that are beyond the reach of standard site models. Furthermore, we devise a method for evaluating the performance of subsite detection algorithms and provide evaluation guidelines.

This article is concerned with the problem of improving software products and investigates how to base that process on solid empirical foundations. Our key contribution is a contextual method that provides a means of identifying new features to support discovered and currently unsupported ways of working and a means of evaluating the usefulness of proposed features. Standard methods of discovery and evaluation, such as interviews and usability testing, gather some of the necessary data but fall short of covering important aspects. The shortcomings of these approaches are overcome by applying an integrated and iterative method for collecting and interpreting data about product usage in context. This article demonstrates its effectiveness when applied to the discovery and evaluation of new features for standard Web clients.

There is a growing interest in estimating the effectiveness of search. Two approaches are typically considered: examining the search queries and examining the retrieved document sets. In this paper, we take the latter approach. We use four measures to characterize the retrieved document sets and estimate the quality of search. These measures are (i) the clustering tendency as measured by the Cox-Lewis statistic, (ii) the sensitivity to document perturbation, (iii) the sensitivity to query perturbation and (iv) the local intrinsic dimensionality. We present experimental results for the task of ranking 200 queries according to the search effectiveness over the TREC (discs 4 and 5) dataset. Our ranking of queries is compared with the ranking based on the average precision using the Kendall t statistic. The best individual estimator is the sensitivity to document perturbation and yields Kendall t of 0.521. When combined with the clustering tendency based on the Cox-Lewis statistic and the query perturbation measure, it results in Kendall t of 0.562 which to our knowledge is the highest correlation with the average precision reported to date.

The growth of digital information increases the need to build better techniques for automatically storing, organizing and retrieving it. Much of this information is textual in nature and existing representation models struggle to deal with the high dimensionality of the resulting feature space. Techniques like latent semantic indexing address, to some degree, the problem of high dimensionality in information retrieval. However, promising alternatives, like random mapping (RM), have yet to be completely studied in this context. In this paper, we show that despite the attention RM has received in other applications, in the case of text retrieval it is outperformed not only by principal component analysis (PCA) and independent component analysis (ICA) but also by a simple noise reduction algorithm.

Routine user activities on the Web result in the revisitation of Web sites and pages. Standard browser applications provide limited support for this type of habitual behaviour. They typically expose lists of visited URLs that are automatically recorded by the system or manually created by the user, such as bookmarks. Studies have shown that these approaches are not successful in supporting routine user activities. Informed by our user research, we designed a browser feature that automatically exposes candidate URLs for revisitation by the user. In this paper, we describe and evaluate the algorithms that we use to model the user's habitual behaviour. We demonstrate how a structured navigation history model facilitates the discovery of relevant usage patterns and supports predictive algorithms that are applicable to relatively short personal navigation histories.

This paper explores feature scoring and selection based on weights from linear classification models. It investigates how these methods combine with various learning models. Our comparative analysis includes three learning algorithms: Naïve Bayes, Perceptron, and Support Vector Machines (SVM) in combination with three feature weighting methods: Odds Ratio, Information Gain, and weights from linear models, the linear SVM and Perceptron. Experiments show that feature selection using weights from linear SVMs yields better classification performance than other feature weighting methods when combined with the three explored learning algorithms. The results support the conjecture that it is the sophistication of the feature weighting method rather than its apparent compatibility with the learning algorithm that improves classification performance.

This paper presents the design and user evaluation of SmartBack, a feature that complements the standard Back button by enabling users to jump directly to key pages in their navigation session, making common navigation activities more efficient. Defining key pages was informed by the findings of a user study that involved detailed monitoring of Web usage and analysis of Web browsing in terms of navigation trails. The pages accessible through SmartBack are determined automatically based on the structure of the user's navigation trails or page association with specific user's activities, such as search or browsing bookmarked sites. We discuss implementation decisions and present results of a usability study in which we deployed the SmartBack prototype and monitored usage for a month in both corporate and home settings. The results show that the feature brings qualitative improvement to the browsing experience of individuals who use it.

WebScout is a system that creates a personal archive of Web pages seen by the user and a rich record of the user's navigation, including various types of user and system generated annotations. We explore how this rich archive can be used to provide support for user navigation, in particular, for revisitation of pages within a navigation session. We describe the WebScout SessionNavigator feature that enhances the current browser functionality by providing both sequential and graph representation of the user navigation. It introduces the concept of a WebTrail which designates a sequence of navigation steps, started by a particular event, such as search, or explicit specification of a URL by typing into the address bar, or executing a link from a bookmark list. We present details of a user study that explores how users perceive and remember their navigation on the Web.

MS Read is a prototype application implemented as an extension of the Web Browser that creates an evolving model of the users topic of interest. It uses that model to analyze documents that are accessed while searching and browsing the Web. In the presented version of MS Read the model is used to highlight topic related terminology in the documents.