Open-domain question answering (OpenQA) research has grown rapidly in recent years. However, OpenQA usability evaluation in its real world applications is largely left under studied. In this paper, we evaluated the actual user experience of OpenQA model deployed in a large tech company’s production enterprise search portal. From qualitative query log analysis and user interviews, our preliminary findings are: 1) There exists a large number of “contingency answers” that cannot be simply evaluated against their face textual values, due to noisy source passages and ambiguous query intents from short keywords queries. 2) Contingency answers contribute to positive search experience for providing “information scents”. 3) Click-through-rate (CTR) is a good user-behavior metric to measure OpenQA result quality, despite the rare existence of “good abandonment”. This exploratory study reveals an often neglected gap between existing OpenQA research and its search engine applications that disconnects the offline research effort with online user experience. We call for reformulating OpenQA model objective beyond answer face value and developing new dataset and metrics for better evaluation protocols.

Conversation double pivots recommend target items related to a source item, based on co-mentions of source and target items in online forums. We deployed several variants on the drupal.org site that supports the Drupal open source community, and assessed them through clickthrough rates. A similarity metric based on correlation of mentions rather than mere co-occurrence reduced the problem of over-recommending the most popular modules, but additional corrections for recency and uniqueness of mentions were not helpful. Detection of more module mentions in conversations dramatically improved the quality of recommendations, even though the detection algorithm then had more false positives. Recommendations based on conversation co-mention were more effective than those based on co-installation, because co-installation data only led to recommendations of complementary modules and not substitutes. Recommendations based on co-mention were more effective than those based on text similarity matching for navigating from the most popular modules, but less effective than text matching for less popular modules.

Many sites on the web offer collaborative databases that catalog items such as bands, events, products, or software modules. Conversation pivots allow readers to navigate from pages about these items to conversations about them on the same site or elsewhere on the Internet. Double pivots allow readers to navigate from item pages to pages about other items mentioned in the same conversations. Using text mining techniques specific to the collection it is possible to find references to collected items in online conversations. We implemented conversation pivots for the CPAN archive of Perl modules, and for Drupal.org, the reference site for the Drupal content management system.