Practical summarization systems are expected to produce summaries of varying lengths, per user needs.While a couple of early summarization benchmarks tested systems across multiple summary lengths, this practice was mostly abandoned due to the assumed cost of producing reference summaries of multiple lengths.In this paper, we raise the research question of whether reference summaries of a single length can be used to reliably evaluate system summaries of multiple lengths.For that, we have analyzed a couple of datasets as a case study, using several variants of the ROUGE metric that are standard in summarization evaluation.Our findings indicate that the evaluation protocol in question is indeed competitive.This result paves the way to practically evaluating varying-length summaries with simple, possibly existing, summarization benchmarks.

In this study we aim to explore users' behavior when assessing search results relevance based on the hypothesis of categorical thinking. To investigate how users categories search engine results, we perform several experiments where users are asked to group a list of 20 search results into several categories, while attaching a relevance judgment to each formed category. Moreover, to determine how users change their minds over time, each experiment was repeated three times under the same conditions, with a gap of one month between rounds. The results show that on average users form 4–5 categories. Within each round the size of a category decreases with the relevance of a category. To measure the agreement between the search engine's ranking and the users’ relevance judgments, we defined two novel similarity measures, the average concordance and the MinMax swap ratio. Similarity is shown to be the highest for the third round as the users' opinion stabilizes. Qualitative analysis uncovered some interesting points that users tended to categories results by type and reliability of their source, and particularly, found commercial sites less trustworthy, and attached high relevance to Wikipedia when their prior domain knowledge was limited.

Domain experts are skilled in buliding a narrow ontology that reflects their subfield of expertise based on their work experience and personal beliefs. We call this type of ontology a single‐viewpoint ontology. There can be a variety of such single viewpoint ontologies that represent a wide spectrum of subfields and expert opinions on the domain. However, to have a complete formal vocabulary for the domain they need to be linked and unified into a multiviewpoint model while having the subjective viewpoint statements marked and distinguished from the objectively true statements. In this study, we propose and implement a two‐phase methodology for multiviewpoint ontology construction by nonexpert users. The proposed methodology was implemented for the domain of the effect of diet on health. A large‐scale crowdsourcing experiment was conducted with about 750 ontological statements to determine whether each of these statements is objectively true, viewpoint, or erroneous. Typically, in crowdsourcing experiments the workers are asked for their personal opinions on the given subject. However, in our case their ability to objectively assess others' opinions was examined as well. Our results show substantially higher accuracy in classification for the objective assessment approach compared to the results based on personal opinions.

We present the first systematic study of the influence of time on user judgements for rankings and relevance grades of web search engine results. The goal of this study is to evaluate the change in user assessment of search results and explore how users' judgements change. To this end, we conducted a large‐scale user study with 86 participants who evaluated 2 different queries and 4 diverse result sets twice with an interval of 2 months. To analyze the results we investigate whether 2 types of patterns of user behavior from the theory of categorical thinking hold for the case of evaluation of search results: (a) coarseness and (b) locality. To quantify these patterns we devised 2 new measures of change in user judgements and distinguish between local (when users swap between close ranks and relevance values) and nonlocal changes. Two types of judgements were considered in this study: (a) relevance on a 4‐point scale, and (b) ranking on a 10‐point scale without ties. We found that users tend to change their judgements of the results over time in about 50% of cases for relevance and in 85% of cases for ranking. However, the majority of these changes were local.

Focused retrieval (a.k.a., passage retrieval) is important at its own right and as an intermediate step in question answering systems. We present a new Web-based collection for focused retrieval. The document corpus is the Category A of the ClueWeb12 collection. Forty-nine queries from the educational domain were created. The $100$ documents most highly ranked for each query by a highly effective learning-to-rank method were judged for relevance using crowdsourcing. All sentences in the relevant documents were judged for relevance.

Blogs that cite academic articles have emerged as a potential source of alternative impact metrics for the visibility of the blogged articles. Nevertheless, to evaluate more fully the value of blog citations, it is necessary to investigate whether research blogs focus on particular types of articles or give new perspectives on scientific discourse. Therefore, we studied the characteristics of peer‐reviewed references in blogs and the typical content of blog posts to gain insight into bloggers' motivations. The sample consisted of 391 blog posts from 2010 to 2012 in Researchblogging.org's health category. The bloggers mostly cited recent research articles or reviews from top multidisciplinary and general medical journals. Using content analysis methods, we created a general classification scheme for blog post content with 10 major topic categories, each with several subcategories. The results suggest that health research bloggers rarely self‐cite and that the vast majority of their blog posts (90%) include a general discussion of the issue covered in the article, with more than one quarter providing health‐related advice based on the article(s) covered. These factors suggest a genuine attempt to engage with a wider, nonacademic audience. Nevertheless, almost 30% of the posts included some criticism of the issues being discussed.

Journal‐based citations are an important source of data for impact indices. However, the impact of journal articles extends beyond formal scholarly discourse. Measuring online scholarly impact calls for new indices, complementary to the older ones. This article examines a possible alternative metric source, blog posts aggregated at ResearchBlogging.org, which discuss peer‐reviewed articles and provide full bibliographic references. Articles reviewed in these blogs therefore receive “blog citations.” We hypothesized that articles receiving blog citations close to their publication time receive more journal citations later than the articles in the same journal published in the same year that did not receive such blog citations. Statistically significant evidence for articles published in 2009 and 2010 support this hypothesis for seven of 12 journals (58%) in 2009 and 13 of 19 journals (68%) in 2010. We suggest, based on these results, that blog citations can be used as an alternative metric source.

Users’ preferences for folders versus tags was studied in 2 working environments where both options were available to them. In the Gmail study, we informed 75 participants about both folder‐labeling and tag‐labeling, observed their storage behavior after 1 month, and asked them to estimate the proportions of different retrieval options in their behavior. In the Windows 7 study, we informed 23 participants about tags and asked them to tag all their files for 2 weeks, followed by a period of 5 weeks of free choice between the 2 methods. Their storage and retrieval habits were tested prior to the learning session and, after 7 weeks, using special classification recording software and a retrieval‐habits questionnaire. A controlled retrieval task and an in‐depth interview were conducted. Results of both studies show a strong preference for folders over tags for both storage and retrieval. In the minority of cases where tags were used for storage, participants typically used a single tag per information item. Moreover, when multiple classification was used for storage, it was only marginally used for retrieval. The controlled retrieval task showed lower success rates and slower retrieval speeds for tag use. Possible reasons for participants’ preferences are discussed.

In this study, we consider the structure and linking strategy of Hebrew websites of several nonprofit organizations. Because nonprofit organizations differ from commercial, educational, or governmental sectors, it is important to understand the ways they utilize the web. To the best of our knowledge, the linking structure of nonprofit organizations has not been previously studied. We surveyed websites of 54 nonprofit organizations in Israel; most of these sites have at least 100 volunteers. We compared their orientation and contents and we built their linking map. We divided the organizations into four main groups: economic aid and citizen rights organizations, health aid organizations, organizations supporting families and individuals with special needs, and organizations for women and children. We found that the number of links inside the special needs group is much higher than in the other groups. We tried to explain this behavior by considering the data obtained from the site‐linking graph. The value of our results is in defining and testing a method to investigate a group of nonprofit organizations, using a case study of Israeli organizations.

This experiment studied the impact of various task phrasings on the search process. Eighty‐eight searchers performed four web search tasks prescribed by the researchers. Each task was linked to an existing target web page, containing a piece of text that served as the basis for the task. A matching phrasing was a task whose wording matched the text of the target page. A nonmatching phrasing was synonymous with the matching phrasing, but had no match with the target page. Searchers received tasks for both types in English and in Hebrew. The search process was logged. The findings confirm that task phrasing shapes the search process and outcome, and also user satisfaction. Each search stage—retrieval of the target page, visiting the target page, and finding the target answer—was associated with different phenomena; for example, target page retrieval was negatively affected by persistence in search patterns (e.g., use of phrases), user‐originated keywords, shorter queries, and omitting key keywords from the queries. Searchers were easily driven away from the top‐ranked target pages by lower‐ranked pages with title tags matching the queries. Some searchers created consistently longer queries than other searchers, regardless of the task length. Several consistent behavior patterns that characterized the Hebrew language were uncovered, including the use of keyword modifications (replacing infinitive forms with nouns), omitting prefixes and articles, and preferences for the common language. The success self‐assessment also depended on whether the wording of the answer matched the task phrasing.

Abstract In this article, we describe the results of an experiment designed to understand the effects of background information and social interaction on image tagging. The participants in the experiment were asked to tag 12 preselected images of Jewish cultural heritage. The users were partitioned into three groups: the first group saw only the images with no additional information whatsoever, the second group saw the images plus a short, descriptive title, and the third group saw the images, the titles, and the URL of the page in which the image appeared. In the first stage of the experiment, each user tagged the images without seeing the tags provided by the other users. In the second stage, the users saw the tags assigned by others and were encouraged to interact. Results show that after the social interaction phase, the tag sets converged and the popular tags became even more popular. Although in all cases the total number of assigned tags increased after the social interaction phase, the number of distinct tags decreased in most cases. When viewing the image only, in some cases the users were not able to correctly identify what they saw in some of the pictures, but they overcame the initial difficulties after interaction. We conclude from this experiment that social interaction may lead to convergence in tagging and that the “wisdom of the crowds” helps overcome the difficulties due to the lack of information.

Abstract We describe the results of an experiment designed to study user preferences for different orderings of search results from three major search engines. In the experiment, 65 users were asked to choose the best ordering from two different orderings of the same set of search results: Each pair consisted of the search engine's original top‐10 ordering and a synthetic ordering created from the same top‐10 results retrieved by the search engine. This process was repeated for 12 queries and nine different synthetic orderings. The results show that there is a slight overall preference for the search engines' original orderings, but the preference is rarely significant. Users' choice of the “best” result from each of the different orderings indicates that placement on the page (i.e., whether the result appears near the top) is the most important factor used in determining the quality of the result, not the actual content displayed in the top‐10 snippets. In addition to the placement bias, we detected a small bias due to the reputation of the sites appearing in the search results.

Abstract The universe of information has been enriched by the creation of the World Wide Web, which has become an indispensible source for research. Since this source is growing at an enormous speed, an in‐depth look of its performance to create a method for its evaluation has become necessary; however, growth is not the only process that influences the evolution of the Web. During their lifetime, Web pages may change their content and links to/from other Web pages, be duplicated or moved to a different URL, be removed from the Web either temporarily or permanently, and be temporarily inaccessible due to server and/or communication failures. To obtain a better understanding of these processes, we developed a method for tracking topics on the Web for long periods of time, without the need to employ a crawler and relying only on publicly available resources. The multiple data‐collection methods used allow us to discover new pages related to the topic, to identify changes to existing pages, and to detect previously existing pages that have been removed or whose content is not relevant anymore to the specified topic. The method is demonstrated through monitoring Web pages that contain the term “informetrics” for a period of 8 years. The data‐collection method also allowed us to analyze the dynamic changes in search engine coverage, illustrated here on Google—the search engine used for the longest period of time for data collection in this project.

The research objective of this work is to develop a general framework that incorporates collaborative social tagging with a novel ontology scheme conveying multiple perspectives. We propose a framework where multiple users tag the same object (an image in our case), and an ontology is extended based on these tags while being tolerant about different points of view. We are not aware of any other work that attempted to devise such an environment and to study its dynamics. The proposed framework characterizes the underlying processes for controlled collaborative development of a multi-perspective ontology and its application to improve image annotation, searching and browsing. Our case study experiment with a set of selected annotated images indicates the soundness of the proposed ontological model.

The opening book is an important component of a chess engine, and thus computer chess programmers have been developing automated methods to improve the quality of their books. For chess, which has a very rich opening theory, large databases of high-quality games can be used as the basis of an opening book, from which statistics relating to move choices from given positions can be collected. In order to find out whether the opening books used by modern chess engines in machine versus machine competitions are ``comparable'' to those used by chess players in human versus human competitions, we carried out analysis on 26 test positions using statistics from two opening books one compiled from humans' games and the other from machines' games. Our analysis using several nonparametric measures, shows that, overall, there is a strong association between humans' and machines' choices of opening moves when using a book to guide their choices.

Abstract Children are increasingly using the Web. Cognitive theory tells us that directory structures are especially suited for information retrieval by children; however, empirical results show that they prefer keyword searching. One of the reasons for these findings could be that the directory structures and terminology are created by grown‐ups. Using a card‐sorting method and an enveloping system, we simulated the structure of a directory. Our goal was to try to understand what browsable, hierarchical subject categories children create when suggested terms are supplied and they are free to add or delete terms. Twelve groups of four children each (fourth and fifth graders) participated in our exploratory study. The initial terminology presented to the children was based on names of categories used in popular directories, in the sections on Arts, Television, Music, Cinema, and Celebrities. The children were allowed to introduce additional cards and change the terms appearing on the 61 cards. Findings show that the different groups reached reasonable consensus; the majority of the category names used by existing directories were acceptable by them and only a small minority of the terms caused confusion. Our recommendation is to include children in the design process of directories, not only in designing the interface but also in designing the content structure as well.

Abstract In this study, we investigate the similarities and differences between rankings of search results by users and search engines. Sixty‐seven students took part in a 3‐week‐long experiment, during which they were asked to identify and rank the top 10 documents from the set of URLs that were retrieved by three major search engines (Google, MSN Search, and Yahoo!) for 12 selected queries. The URLs and accompanying snippets were displayed in random order, without disclosing which search engine(s) retrieved any specific URL for the query. We computed the similarity of the rankings of the users and search engines using four nonparametric correlation measures in [0,1] that complement each other. The findings show that the similarities between the users' choices and the rankings of the search engines are low. We examined the effects of the presentation order of the results, and of the thinking styles of the participants. Presentation order influences the rankings, but overall the results indicate that there is no “average user,” and even if the users have the same basic knowledge of a topic, they evaluate information in their own context, which is influenced by cognitive, affective, and physical factors. This is the first large‐scale experiment in which users were asked to rank the results of identical queries. The analysis of the experimental results demonstrates the potential for personalized search.

Abstract The World Wide Web has become one of our more important information sources, and commercial search engines are the major tools for locating information; however, it is not enough for a Web page to be indexed by the search engines—it also must rank high on relevant queries. One of the parameters involved in ranking is the number and quality of links pointing to the page, based on the assumption that links convey appreciation for a page. This article presents the results of a content analysis of the links to two top pages retrieved by Google for the query “jew” as of July 2004: the “jew” entry on the free online encyclopedia Wikipedia, and the home page of “Jew Watch,” a highly anti‐Semitic site. The top results for the query “jew” gained public attention in April 2004, when it was noticed that the “Jew Watch” homepage ranked number 1. From this point on, both sides engaged in “Googlebombing” (i.e., increasing the number of links pointing to these pages). The results of the study show that most of the links to these pages come from blogs and discussion links, and the number of links pointing to these pages in appreciation of their content is extremely small. These findings have implications for ranking algorithms based on link counts, and emphasize the huge difference between Web links and citations in the scientific community.

Abstract The present paper analyzes the changes that occurred to a set of Web pages related to “informetrics” over a period of 5 years between June 1998 and June 2003. Four times during this time span, in 1998, 1999, 2002, and 2003, we monitored previously located pages and searched for new ones related to the topic. Thus, we were able to study the growth of the topic, while analyzing the rates of change and disappearance. The results indicate that modification, disappearance, and resurfacing cannot be ignored when studying the structure and development of the Web.

Abstract This study introduces methods for evaluating search engine performance over a time period. Several measures are defined, which as a whole describe search engine functionality over time. The necessary setup for such studies is described, and the use of these measures is illustrated through a specific example. The set of measures introduced here may serve as a guideline for the search engines for testing and improving their functionality. We recommend setting up a standard suite of measures for evaluating search engine performance.

This article addresses the question of whether the Web can serve as an information source for research. Specifically, it analyzes by way of content analysis the Web pages retrieved by the major search engines on a particular date (June 7, 1998), as a result of the query “informetrics OR informetric.” In 807 out of the 942 retrieved pages, the search terms were mentioned in the context of information science. Over 70% of the pages contained only indirect information on the topic, in the form of hypertext links and bibliographical references without annotation. The bibliographical references extracted from the Web pages were analyzed, and lists of most productive authors, most cited authors, works, and sources were compiled. The list of references obtained from the Web was also compared to data retrieved from commercial databases. For most cases, the list of references extracted from the Web outperformed the commercial, bibliographic databases. The results of these comparisons indicate that valuable, freely available data is hidden in the Web waiting to be extracted from the millions of Web pages.

The purpose of this study was to examine and compare use of the Internet among various sectors of the faculty, in order to verify the influence of a number of parameters on this use, e.g.: (1) The field and research interests of the faculty members; (2) formal training in the use of the internet via courses, workshops, etc.; (3) self-instruction in the use of the Internet by means of manuals, how-to books, etc.; (4) general use and knowledge of computers; and (5) perceived need for the information this network can provide. Questionnaires were sent to faculty members in all departments and professional schools of The Hebrew University of Jerusalem, a total population of 918 for both the pilot project and the main study. Results indicated that Internet use is consistently higher among faculty members in the sciences and agriculture (“Sciagr”) than among those in the humanities or social sciences (“Humsoc”), suggesting that, among other things, the connectivity level of the Humsoc group may be lowr than that of the Sciagr group. The Humsoc group also showed a higher demand for courses in Internet use. Suggestions for raising the level of Internet use among the various disciplines of the faculty are included. © 1997 John Wiley & Sons, Inc.

Article Non-cryptographic fault-tolerant computing in constant number of rounds of interaction Share on Authors: J. Bar-Ilan Hebrew University Hebrew UniversityView Profile , D. Beaver Harvard University Harvard UniversityView Profile Authors Info & Claims PODC '89: Proceedings of the eighth annual ACM Symposium on Principles of distributed computingJune 1989 Pages 201–209https://doi.org/10.1145/72981.72995Online:01 June 1989Publication History 163citation1,190DownloadsMetricsTotal Citations163Total Downloads1,190Last 12 Months94Last 6 weeks18 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access