Multimodal deep-learning models power interactive video retrieval by ranking keyframes in response to textual queries. Despite these advances, users must still browse ranked candidates manually to locate a target. Keyframe arrangement within the search grid highly affects browsing effectiveness and user efficiency, yet remains underexplored. We report a study with 49 participants evaluating seven keyframe layouts for the Visual Known-Item Search task. Beyond efficiency and accuracy, we relate browsing phenomena, such as overlooks, to layout characteristics. Our results show that a video-grouped layout is the most efficient, while a four-column, rank-preserving grid achieves the highest accuracy. Sorted grids reveal potential and trade-offs, enabling rapid scanning of uninteresting regions but down-ranking relevant targets to less prominent positions, delaying first arrival times and increasing overlooks. These findings motivate hybrid designs that preserve positions of top-ranked items while sorting or grouping the remainder, and offer guidance for searching in grids beyond video retrieval.

Behavioral patterns captured in embeddings learned from interaction data are pivotal across various stages of production recommender systems. However, in the initial retrieval stage, practitioners face an inherent tradeoff between embedding expressiveness and the scalability and latency of serving components, resulting in the need for representations that are both compact and expressive. To address this challenge, we propose a training strategy for learning high-dimensional sparse embedding layers in place of conventional dense ones, balancing efficiency, representational expressiveness, and interpretability. To demonstrate our approach, we modified the production-grade collaborative filtering autoencoder ELSA, achieving up to 10× reduction in embedding size with no loss of recommendation accuracy, and up to 100× reduction with only a 2.5% loss. Moreover, the active embedding dimensions reveal an interpretable inverted-index structure that segments items in a way directly aligned with the model's latent space, thereby enabling integration of segment-level recommendation functionality (e.g., 2D homepage layouts) within the candidate retrieval model itself. Source codes, additional results, as well as a live demo are available at https://github.com/zombak79/compressed\_elsa.

Recommender systems (RS) rely on interaction data between users and items to generate effective results. Originally, RS aimed solely at predicting items’ relevance, but additional (beyond-relevance) quality criteria gained increased attention over time. Objectives such as diversity, novelty, fairness, or serendipity are nowadays at the center of RS research and also among the core components in production systems. Naturally, to properly steer toward such objectives, the system has to gain an understanding of how the users perceive these objectives, to what extent they require them in the recommendations, and how they evaluate the sufficiency of the results w.r.t. these objectives. However, so far, there is no publicly available dataset that would capture all the necessary knowledge. This results in a half-blind algorithmic design and evaluation, where the importance of individual objectives or the metrics for their evaluation cannot be validated from the users’ perspective. To address this issue, we present SM-RS 2.0, an expansion of the original single- and multi-objective recommendations dataset. The dataset links the self-declared propensity toward individual objectives with impressions, item selections, and explicit evaluation of individual quality criteria. Together with the dataset, we also distribute an evaluation framework containing six rather unique tasks that are rarely available to conduct on existing RS datasets. These include impression-aware click prediction, predicting propensity toward individual objectives, construction of proportional recommendations, and predicting the user-perceived fulfillment of individual objectives as well as their overall satisfaction. The dataset is available at https://osf.io/wsakx .

Multi-objective recommender systems (MORS) aim to optimize multiple criteria while generating recommendations, such as relevance, novelty, diversity, or exploration.These algorithms are based on the assumption that an operationalization of these criteria (i.e., translating abstract goals into measurable metrics), will reflect how users perceive them.Nevertheless, such beliefs are rarely rigorously evaluated, which can lead to a mismatch between algorithmic goals and user satisfaction.Moreover, if users are allowed to control the RS via their propensities towards such objectives, the misconceptions may further impact users' trust and engagement.To characterize this problem, we conduct a large user study focusing on recommender systems in two domains: books and movies.Part of the study is focused on how users perceive different recommendation objectives, which we compared with well-established metrics aiming at the same objectives.We found that despite such metrics correlating to some extent with users' perceptions, the mapping is far from perfect.Moreover, we also report on conceptual-level differences in users' understanding of RS objectives and how this affects the results.Study data are available from https://osf.io/2n9mf/.

Figure 1: Overview of SAGEA: The training phase (left) consists of: (a) training the ELSA collaborative autoencoder on individual user interactions to learn dense user embeddings, and (b) training a Top-𝑘 sparse autoencoder (SAE) to reconstruct these embeddings via a disentangled, sparse representation.The disentangled representation can be understood as a dictionary of activated concepts (such as preference for "Old music", "Love", "Dark humor", etc., as highlighted in the example).During inference (right), individual group members are encoded into sparse representations via the ELSA encoder E 𝐸𝐿𝑆𝐴 extended with the SAE encoder E 𝑆𝐴𝐸 .These are aggregated using a fairness-preserving strategy to form a group-level sparse embedding.The SAE decoder (D 𝑆𝐴𝐸 ) transforms this back into a dense embedding, which is then decoded by the ELSA decoder (D 𝐸𝐿𝑆𝐴 ) to generate group-wise relevance scores.

Deep learning has revolutionized multimedia retrieval, yet effectively searching within large video collections remains a complex challenge. This paper focuses on the design and evaluation of known-item search systems, leveraging the strengths of CLIP-based deep neural networks for ranking. At events like the Video Browser Showdown, these models have shown promise in effectively ranking the video frames. While ranking models can be pre-selected automatically based on a benchmark collection, the selection of an optimal browsing interface, crucial for refining top-ranked items, is complex and heavily influenced by user behavior. Our study addresses this by presenting an eye tracking-based analysis of user interaction with different image grid layouts. This approach offers novel insights into search patterns and user preferences, particularly examining the trade-off between displaying fewer but larger images versus more but smaller images. Our findings reveal a preference for grids with fewer images and detail how image similarity and grid position affect user search behavior. These results not only enhance our understanding of effective video retrieval interface design but also set the stage for future advancements in the field.

We derive a novel graph-based interpretation of linear autoencoder models easer, slim, and their approximate variants. Contrary to popular belief, we reveal that the weights of these models should not be interpreted as dichotomic item similarity but merely as its magnitude. Consequently, we propose a simple modification that considerably improves retrieval ability in sparse domains and yields interpretable inference with negative inputs, as demonstrated by both offline and online experiments. Experiment codes and extended results are available at https://osf.io/bjmuv/.

Recommender systems (RS) rely on interaction data between users and items to generate effective results. Historically, RS aimed to deliver the most consistent (i.e., accurate) items to the trained user profiles. However, the attention towards additional (beyond-accuracy) quality criteria has increased tremendously in recent years. Both the research and applied models are being optimized for diversity, novelty, or fairness, to name a few. Naturally, the proper functioning of such optimization methods depends on the knowledge of users' propensities towards interacting with recommendations having certain quality criteria. However, so far, no dataset that captures such propensities exists. To bridge this research gap, we present SM-RS (single-objective + multi-objective recommendations dataset) that links users' self-declared propensity toward relevance, novelty, and diversity criteria with impressions and corresponding item selections. After presenting the dataset's collection procedure and basic statistics, we propose three tasks that are rarely available to conduct using existing RS datasets: impressions-aware click prediction, users' propensity scores prediction, and construction of recommendations proportional to the users' propensity scores. For each task, we also provide detailed evaluation procedures and competitive baselines. The dataset is available at https://osf.io/hkzje/.

Abstract In data science and content-based retrieval, we find many domain-specific techniques that employ a data processing pipeline with two fundamental steps. First, data entities are represented by some visualizations, while in the second step, the visualizations are used with a machine learning model to extract deep features. Deep convolutional neural networks (DCNN) became the standard and reliable choice. The purpose of using DCNN is either a specific classification task or just a deep feature representation of visual data for additional processing (e.g., similarity search). Whereas the deep feature extraction is a domain-agnostic step in the pipeline (inference of an arbitrary visual input), the visualization design itself is domain-dependent and ad hoc for every use case. In this paper, we survey and analyze many instances of data visualizations used with deep learning models (mostly DCNN) for domain-specific tasks. Based on the analysis, we synthesize a taxonomy that provides a systematic overview of visualization techniques suitable for usage with the models. The aim of the taxonomy is to enable the future generalization of the visualization design process to become completely domain-agnostic, leading to the automation of the entire feature extraction pipeline. As the ultimate goal, such an automated pipeline could lead to universal deep feature data representations for content-based retrieval.

Improvements in the recommender systems (RS) domain are not possible without a thorough way to evaluate and compare newly proposed approaches. User studies represent a viable alternative to online and offline evaluation schemes, but despite their numerous benefits, they are only rarely used. One of the main reasons behind this fact is that preparing a user study from scratch involves a lot of extra work on top of a simple algorithm proposal. To simplify this task, we propose EasyStudy, a modular framework built on the credo “Make simple things fast and hard things possible”. It features ready-to-use datasets, preference elicitation methods, incrementally tuned baseline algorithms, study flow plugins, and evaluation metrics. As a result, a simple study comparing several RS can be deployed with just a few clicks, while more complex study designs can still benefit from a range of reusable components, such as preference elicitation. Overall, EasyStudy dramatically decreases the gap between the laboriousness of offline evaluation vs. user studies and, therefore, may contribute towards the more reliable and insightful user-centric evaluation of next-generation RS. The project repository is available from https://bit.ly/easy-study-repo.

Multi-objective recommender systems (MORS) provide suggestions to users according to multiple (and possibly conflicting) goals. When a system optimizes its results at the individual-user level, it tailors them on a user’s propensity towards the different objectives. Hence, the capability to understand users’ fine-grained needs towards each goal is crucial. In this paper, we present the results of a user study in which we monitored the way users interacted with recommended items, as well as their self-proclaimed propensities towards relevance, novelty, and diversity objectives. The study was divided into several sessions, where users evaluated recommendation lists originating from a relevance-only single-objective baseline as well as MORS. We show that, despite MORS-based recommendations attracting fewer selections, their presence in the early sessions are crucial for users’ satisfaction in the later stages. Surprisingly, the self-proclaimed willingness of users to interact with novel and diverse items is not always reflected in the recommendations they accept. Post-study questionnaires provide insights on how to deal with this matter, suggesting that MORS-based results should be accompanied by elements that allow users to understand the recommendations, so as to facilitate the choice of whether a recommendation should be accepted or not. Detailed study results are available at https://bit.ly/looks-can-be-deceiving-repo.

In the field of recommender systems, shallow autoencoders have recently gained significant attention. One of the most highly acclaimed shallow autoencoders is easer, favored for its competitive recommendation accuracy and simultaneous simplicity. However, the poor scalability of easer (both in time and especially in memory) severely restricts its use in production environments with vast item sets. In this paper, we propose a hyperefficient factorization technique for sparse approximate inversion of the data-Gram matrix used in easer. The resulting autoencoder, sansa, is an end-to-end sparse solution with prescribable density and almost arbitrarily low memory requirements — even for training. As such, sansa allows us to effortlessly scale the concept of easer to millions of items and beyond.

Modern online platforms for user modeling and recommendation require complex data infrastructures to collect and process data. Some of this data has to be kept to later be used in batches to train personalization models. However, since user activity data can be generated at very fast rates it is also useful to have algorithms able to process data streams online, in real time. Given the continuous and potentially fast change of content, context and user preferences or intents, stream-based models, and their synchronization with batch models can be extremely challenging. Therefore, it is important to investigate methods able to transparently and continuously adapt to the inherent dynamics of user interactions, preferably over long periods of time. Models able to continuously learn from such flows of data are gaining attention in the recommender systems community, and are being increasingly deployed in online platforms. However, many challenges associated with learning from streams need further investigation.

Going beyond accuracy in the evaluation of a recommender system is an aspect that is receiving more and more attention. Among the many perspectives that can be considered, the impact of presentation bias is of central importance. Under presentation bias, the attention of the users to the items in a recommendation list changes, thus affecting their possibility to be considered and the effectiveness of a model. Page-wise within-subject studies are widely employed in the recommender systems literature to compare algorithms by displaying their results in parallel. However, no study has ever been performed to assess the impact of presentation bias in this context. In this paper, we characterize how presentation bias affects different layout options, which present the results in column- or row-wise fashion. Concretely, we present a user study where six layout variants are proposed to the users in a page-wise within-subject setting, so as to evaluate their perception of the displayed recommendations. Results show that presentation bias impacts users clicking behavior (low-level feedback), but not so much the perceived performance of a recommender system (high-level feedback). Source codes and raw results are available at https://tinyurl.com/PresBiasSIGIR2023.

Group Recommender Systems (GRSs), unlike recommendations for individuals, provide suggestions for groups of people. Clearly, many activities are often experienced by a group rather than an individual (visiting a restaurant, traveling, watching a movie, etc.) hence the requirement for such systems. The topic is gradually receiving more and more attention, with an increased number of papers published at significant venues, which is enabled by the predominance of online social platforms that allow their users to interact in groups, as well as to plan group activities. However, the research area lacks certain ground rules, such as basic evaluation agreements. We believe this is one of the main obstacles to make advances in the research area, and to enable researchers to compare and continue each others’ works. In other words, setting the basic evaluation agreements is a stepping-stone towards reproducible Group Recommenders research. The goal of this tutorial is to tackle this problem, by providing the basic principles of the GRSs offline evaluation approaches.

The main source of knowledge utilized in recommender systems (RS) is users’ feedback. While the usage of implicit feedback (i.e. user’s behavior statistics) is gaining in prominence, the explicit feedback (i.e. user’s ratings) remain an important data source. This is true especially for domains, where evaluation of an object does not require an extensive usage and users are well motivated to do so (e.g., video-on-demand services or library archives).

Modern online systems for user modeling and recommendation need to continuously deal with complex data streams generated by users at very fast rates. This can be overwhelming for systems and algorithms designed to train recommendation models in batches, given the continuous and potentially fast change of content, context and user preferences or intents. Therefore, it is important to investigate methods able to transparently and continuously adapt to the inherent dynamics of user interactions, preferably for long periods of time. Online models that continuously learn from such flows of data are gaining attention in the recommender systems community, given their natural ability to deal with data generated in dynamic, complex environments. User modeling and personalization can particularly benefit from algorithms capable of maintaining models incrementally and online.

The main focus of our work is the problem of multiple objectives optimization (MOO) while providing a final list of recommendations to the user. Currently, system designers can tune MOO by setting importance of individual objectives, usually in some kind of weighted average setting. However, this does not have to translate into the presence of such objectives in the final results. In contrast, in our work we would like to allow system designers or end-users to directly quantify the required relative ratios of individual objectives in the resulting recommendations, e.g., the final results should have 60% relevance, 30% diversity and 10% novelty. If individual objectives are transformed to represent quality on the same scale, these result conditioning expressions may greatly contribute towards recommendations tuneability and explainability as well as user's control over recommendations.

Abstract Decision-making in our everyday lives is surrounded by visually important information. Fashion, housing, dating, food or travel are just a few examples. At the same time, most commonly used tools for information retrieval operate on relational and text-based search models which are well understood by end users, but unable to directly cover visual information contained in images or videos. Researcher communities have been trying to reveal the semantics of multimedia in the last decades with ever-improving results, dominated by the success of deep learning. However, this does not close the gap to relational retrieval model on its own and often rather solves a very specialized task like assigning one of pre-defined classes to each object within a closed application ecosystem. Retrieval models based on these novel techniques are difficult to integrate in existing application-agnostic environments built around relational databases, and therefore, they are not so widely used in the industry. In this paper, we address the problem of closing the gap between visual information retrieval and relational database model. We propose and formalize a model for discovering candidates for new relational attributes by analysis of available visual content. We design and implement a system architecture supporting the attribute extraction, suggestion and acceptance processes. We apply the solution in the context of e-commerce and show how it can be seamlessly integrated with SQL environments widely used in the industry. At last, we evaluate the system in a user study and discuss the obtained results.

In this paper, we focus on the problems of fair aggregation of recommender systems (RS) and over-exposure of users with insignificant recommendations. While fair aggregation of diverse RS may contribute to both calibration and diversity challenges, some recently proposed methods suffer from repeating the same set of recommendations to the user over and over again. However, it may be difficult to distinguish between situations when users ignore recommendations because they are irrelevant or because they did not notice them. In order to cope with these challenges, we propose an innovative off-line RS evaluation methodology based on the noticeability of recommended items. We further propose a Fuzzy D'Hondt's algorithm with personalized implicit negative feedback attribution (FDHondtINF). The algorithm is designed to provide a fair ordering of candidate items coming from multiple individual RS, while considering also the objects previously ignored by the current user. FDHondtINF was evaluated off-line along with other aggregation methods and individual RS on MovieLens 1M dataset. The algorithm performs especially well in situations when the recommended items are less noticeable, or when a sequence of multiple recommendations for the same user model is given.

Recommender systems are now part of our daily life more than ever and most users are confronted with some form of recommendation on a daily basis. As users of such systems, we don't need to actively seek for new content, but let it be comfortably recommended to us instead. One of the important parts of our lives that is yet to be covered in this way is the domain of cooking. A traditional dilemma of a person, who is currently in the process of shopping for food is "What else should I buy, so that I can cook something new?" In another words, the person either has to look for novel recipes upfront (which does not have to correspond with available ingredients in the shop), or buy ingredients intuitively (which does not have to correspond with recipes).

Searching for memorized images in large datasets (known-item search) is a challenging task due to a limited effectiveness of retrieval models as well as limited ability of users to formulate suitable queries and choose an appropriate search strategy. A popular option to approach the task is to automatically detect semantic concepts and rely on interactive specification of keywords during the search session. Nonetheless, employed instances of such search models are often set arbitrarily in existing KIS systems as comprehensive evaluations with reals users are time demanding. This paper envisions and investigates an option to simulate keyword queries in a selected "toy'' (yet competitive) keyword search model relying on a deep image classification network. Specifically, two properties of such keyword-based model are experimentally investigated with our known-item search benchmark dataset: which output transformation and ranking models are effective for the utilized classification model and whether there are some options for simulations of keyword queries. In addition to the main objective, the paper inspects also the effect of interactive query reformulations for the considered keyword search model.

In this demo paper, we present a prototype web application of a product search engine of a fashion e-shop. Although e-shop products consist of full-text description, relational attributes (e.g., price, type, size, color, etc.) as well as visual information (product photo), traditional search engines in e-shops only provide full-text and relational attributes for product filtering. In our retrieval model, we incorporate also the visual information into the search by extracting visual-semantic features using deep convolutional neural networks. Furthermore, visual exploration of the product space using the visual-semantic features (multi-example queries) is used to dynamically discover latent visual attributes that could enhance the original relational schema by fuzzy attributes (e.g., a floral pattern in product). In the demo, we show how these latent attributes could be used to recommend the user preferred products and even outfits (e.g., shoes, bag, jacket) that fit a certain visual style.

This paper describes the concept and some preliminary experiments of extension of the sitIT.cz portal – the social network of the ICT specialists in Czech Republic. SitIT.cz interconnects ICT specialists and offers effective search according to several types of structured – machine readable – profiles. It is intended to support technology transfer, sharing information, building teams and to enable discussions, especially targeted to increase of competitiveness of R&D in ICT. Main stimulus for sitIT.cz extension came from acquaintance with software development which often needs extensive testing – not only technological but also from users' point of view. Extension can make network more attractive for both developers and users. We propose new structured profiles that can be used for evaluating software and allowing developers obtain as valuable feedback as possible. One of main outcomes of this system is creating baseline knowledge by humans for further comparison and/or training.

In this paper we summarize efforts of our research group on web semantization - process of increasing the degree of automation of web processing – and some of its applications. We present several methods for mining textual information and assisted annotations as we believe this should be the first steps towards the semantic web. Then several methods for processing the gathered data are described. The proposed methods mainly aim on modeling user and his/her preferences and then helping them with reaching their goals. We focus on the complex environment of the web semantization as a wider context where both tools providing and consuming semantical data are placed.