People frequently use search engines in the lead-up to elections, and the results they encounter can influence their voting decisions. In this study, we analyzed the stances of search results from Google and Bing in Germany related to the 2024 European Parliament elections, as well as the types of sources presented to users. We collected 760 search results for 38 political queries and had jurors assess their stances, with each result being evaluated by five jurors. The findings reveal that public authority and journalistic pages dominate the search results, while political party pages are the least represented source category. Furthermore, we found that Google and Bing predominantly display neutral search results, with neutral stances being more common in Google than in Bing. Additionally, in both search engines, search results that indicate a political leaning tend to align more closely with left-leaning parties than with right-leaning ones. While acknowledging that the findings may be influenced by how the queries were formulated, the results highlight the importance of using multiple search engines to access diverse political viewpoints. They also raise questions about whether both agreeing and disagreeing results should be displayed for controversial topics.

The Result Assessment Tool (RAT) is an open-source software toolkit for conducting research with results from commercial search engines and other web-based information retrieval systems. Conducting such research is challenging due to the “black-box” nature of these systems and limited data access. RAT addresses this by providing an integrated software environment that unifies modules for study design, automated data collection, manual assessment in a dedicated interface, and automated analysis via an extensible classifier framework. The software is designed to assist with various research tasks, including comparative assessments of result quality, investigations of source variety, and content analysis. It emphasizes transparency in methodologies, reproducibility of outcomes, and responsible data collection.

Abstract Search engines, such as Google, have a considerable impact on society. Therefore, undesirable consequences, such as retrieving incorrect search results, pose a risk to users. Although previous research has reported the adverse outcomes of web search, little is known about how search engine users evaluate those outcomes. In this study, we show which aspects of web search are perceived as risky using a sample (N = 3884) representative of the German Internet population. We found that many participants are often concerned with adverse consequences immediately appearing on the search engine result page. For example, 45.2% of respondents are concerned about retrieving incorrect information. In contrast, consequences with a delayed impact are rarely perceived as a risk. Moreover, participants' experiences with adverse consequences are directly related to their risk perception. Our results demonstrate that people perceive risks related to web search. In addition to our study, there is a need for more independent research on the possible detrimental outcomes of web search to monitor and mitigate risks. Apart from risks for individuals, search engines with a massive number of users have an extraordinary impact on society; therefore, the acceptable risks of web search should be discussed.

Searching for medical information is both a common and important activity since it influences decisions people make about their healthcare. Using search engine optimization (SEO), content producers seek to increase the visibility of their content. SEO is more likely to be practiced by commercially motivated content producers such as pharmaceutical companies than by non-commercial providers such as governmental bodies. In this study, we ask whether content quality correlates with the presence or absence of SEO measures on a web page. We conducted a user study in which N = 61 participants comprising laypeople as well as experts in health information assessment evaluated health-related web pages classified as either optimized or non-optimized. The subjects rated the expertise of non-optimized web pages as higher than the expertise of optimized pages, justifying their appraisal by the more competent and reputable appearance of non-optimized pages. In addition, comments about the website operators of the non-optimized pages were exclusively positive, while optimized pages tended to receive positive as well as negative assessments. We found no differences between the ratings of laypeople and experts. Since non-optimized, but high-quality content may be outranked by optimized content of lower quality, trusted sources should be prioritized in rankings.

Abstract This opinion piece takes Google's response to the so‐called COVID‐19 infodemic, as a starting point to argue for the need to consider societal relevance as a complement to other types of relevance. The authors maintain that if information science wants to be a discipline at the forefront of research on relevance, search engines, and their use, then the information science research community needs to address itself to the challenges and conditions that commercial search engines create in. The article concludes with a tentative list of related research topics.

Abstract According to recent studies, search engine users have little knowledge of Google's business model. In addition, users cannot sufficiently distinguish organic results from advertisements, resulting in result selections under false assumptions. Following on from that, this study examines how users' understanding of search‐based advertising influences their viewing and selection behavior on desktop computer and smartphone. To investigate this, we used a mixed methods approach (n = 100) consisting of a pre‐study interview, an eye‐tracking experiment, and a post‐study questionnaire. We show that participants with a low level of knowledge on search advertising are more likely to click on ads than subjects with a high level of knowledge. Moreover, subjects with little knowledge show less willingness to scroll down to organic results. Regarding the device, there are significant differences in viewing behavior. These can be attributed to the influence of the direct visibility of search results on both devices tested: Ads that were ranked on top received significantly more visual attention on the small screen than the top ranked ads on the large screen. The results call for a clearer labeling of advertisements and for the promotion of users' information literacy. Future studies should investigate the motivations of searchers when clicking on ads.

This representative study of German search engine users (N = 1,000) focuses on the ability of users to distinguish between organic results and advertisements on Google results pages. We combine questions about Google's business with task‐based studies in which users were asked to distinguish between ads and organic results in screenshots of results pages. We find that only a small percentage of users can reliably distinguish between ads and organic results, and that user knowledge of Google's business model is very limited. We conclude that ads are insufficiently labelled as such, and that many users may click on ads assuming that they are selecting organic results.

The purpose of this half-day workshop is to bring together academic and industry interactive information retrieval (IIR) researchers with an interest in evaluation methodologies. The workshop articulates contemporary challenges in the investigation of IIR and invites user- and system-oriented researchers to work collaboratively to address these challenges by combining user- and system-centered methodologies in meaningful ways. We anticipate that this workshop will initiate productive knowledge exchange and partnerships that can respond to the increasing user, task, system, and contextual complexity of the IIR field.

Search engine retrieval effectiveness studies are usually small scale, using only limited query samples. Furthermore, queries are selected by the researchers. We address these issues by taking a random representative sample of 1,000 informational and 1,000 navigational queries from a major German search engine and comparing Google's and Bing's results based on this sample. Jurors were found through crowdsourcing, and data were collected using specialized software, the Relevance Assessment Tool (RAT). We found that although Google outperforms Bing in both query types, the difference in the performance for informational queries was rather low. However, for navigational queries, Google found the correct answer in 95.3% of cases, whereas Bing only found the correct answer 76.6% of the time. We conclude that search engine performance on navigational queries is of great importance, because users in this case can clearly identify queries that have returned correct results. So, performance on this query type may contribute to explaining user satisfaction with search engines.

The purpose of this article is to test the reliability of query intents derived from queries, either by the user who entered the query or by another juror. We report the findings of three studies. First, we conducted a large‐scale classification study (~50,000 queries) using a crowdsourcing approach. Next, we used clickthrough data from a search engine log and validated the judgments given by the jurors from the crowdsourcing study. Finally, we conducted an online survey on a commercial search engine's portal. Because we used the same queries for all three studies, we also were able to compare the results and the effectiveness of the different approaches. We found that neither the crowdsourcing approach, using jurors who classified queries originating from other users, nor the questionnaire approach, using searchers who were asked about their own query that they just entered into a Web search engine, led to satisfying results. This leads us to conclude that there was little understanding of the classification tasks, even though both groups of jurors were given detailed instructions. Although we used manual classification, our research also has important implications for automatic classification. We must question the success of approaches using automatic classification and comparing its performance to a baseline from human jurors.

This paper aims to review the fiercely discussed question of whether the ranking of Wikipedia articles in search engines is justified by the quality of the articles. After an overview of current research on information quality in Wikipedia, a summary of the extended discussion on the quality of encyclopedic entries in general is given. On this basis, a heuristic method for evaluating Wikipedia entries is developed and applied to Wikipedia articles that scored highly in a search engine retrieval effectiveness test and compared with the relevance judgment of jurors. In all search engines tested, Wikipedia results are unanimously judged better by the jurors than other results on the corresponding results position. Relevance judgments often roughly correspond with the results from the heuristic evaluation. Cases in which high relevance judgments are not in accordance with the comparatively low score from the heuristic evaluation are interpreted as an indicator of a high degree of trust in Wikipedia. One of the systemic shortcomings of Wikipedia lies in its necessarily incoherent user model. A further tuning of the suggested criteria catalog, for instance, the different weighing of the supplied criteria, could serve as a starting point for a user model differentiated evaluation of Wikipedia articles. Approved methods of quality evaluation of reference works are applied to Wikipedia articles and integrated with the question of search engine evaluation.