Recent advances in large language models have enabled the development of viable generative retrieval systems. Instead of a traditional document ranking, generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval. Yet, the established evaluation methodology for ranking-based ad hoc retrieval is not suited for the reliable and reproducible evaluation of generated responses. To lay a foundation for developing new evaluation methods for generative retrieval systems, we survey the relevant literature from the fields of information retrieval and natural language processing, identify search tasks and system architectures in generative retrieval, develop a new user model, and study its operationalization.

Conditional generative models such as DALL-E and Stable Diffusion generate images based on a user-defined text, the prompt. Finding and refining prompts that produce a desired image has become the art of prompt engineering. Generative models do not provide a built-in retrieval model for a user’s information need expressed through prompts. In light of an extensive literature review, we reframe prompt engineering for generative models as interactive text-based retrieval on a novel kind of “infinite index”. We apply these insights for the first time in a case study on image generation for game design with an expert. Finally, we envision how active learning may help to guide the retrieval of generated images.

We integrate irdatasets, ir_measures, and PyTerrier with TIRA in the Information Retrieval Experiment Platform (TIREx) to promote more standardized, reproducible, scalable, and even blinded retrieval experiments. Standardization is achieved when a retrieval approach implements PyTerrier's interfaces and the input and output of an experiment are compatible with ir_datasets and ir_measures. However, none of this is a must for reproducibility and scalability, as TIRA can run any dockerized software locally or remotely in a cloud-native execution environment. Version control and caching ensure efficient (re)execution. TIRA allows for blind evaluation when an experiment runs on a remote server or cloud not under the control of the experimenter. The test data and ground truth are then hidden from public access, and the retrieval software has to process them in a sandbox that prevents data leaks.

This paper focuses on the automated analysis of motion patterns for relating an individual’s behaviour with its pain experience. Reliable, automated behaviour analysis can improve video observation considerably, i.e. by lessening the work load of human operators, decreasing human error and by increasing anonymity and privacy. Possible applications are observation of traffic, public places and public transport. A new potential application is the early detection of pathologically relevant events, e.g. animal diseases (e.g. colics in horses), the automated post-surgical pain assessment of animals and similar applications. The challenge in the horses scenario is that they are flight animals that can not afford much of visible pain behaviour.Our approach is built on top of state of the art methods for object detection and tracking. From the object motion we derive motion patterns and respective features which we analyse by machine-learning methods. In the this paper we will present atypical behaviour detection (i.e. pain estimation) in animal videos, for which we have acquired a large video database. It could be shown that the condition of the horse can be analysed and classified by means of local histograms.