Since 2010,ARTigohas been leveraging games with a purpose (GWAPs) to engage citizen scientists in the tagging of artworks. In these games, two players collaboratively annotate images under time constraints. This paper introducesARTigoas an easily customizable framework for image labeling tailored to elicit specific domain knowledge. It also outlines a two-pronged evaluation strategy to assess the impact ofARTigo’s front-end design on visual aesthetics, usability, and cognitive workload, as well as on users' tagging behavior. Comparisons betweenARTigoversion$\boldsymbol {1}$, developed from 2010 to 2013, and version$\boldsymbol {2}$, developed in 2022, indicate significant improvements: the System Usability Scale rating improved from gradeBtoA-, and the Short Visual Aesthetics of Websites Inventory rating from 4.42 to 5.34 points ($\boldsymbol {p < 0.05}$). Moreover, the graphical user interface in version$\boldsymbol {2}$promotes more frequent and complex tagging, thus collecting a similar amount of tags with higher entropy in fewer rounds—which is especially beneficial for GWAP with smaller, active communities.

Iconography refers to the methodical study and interpretation of thematic content in the visual arts, distinguishing it, e.g., from purely formal or aesthetic considerations. In iconographic studies, Iconclass is a widely used taxonomy that encapsulates historical, biblical, and literary themes, among others. However, given the hierarchical nature and inherent complexity of such a taxonomy, it is highly desirable to use automated methods for (Iconclass-based) image classification. Previous studies either focused narrowly on certain subsets of narratives or failed to exploit Iconclass’s hierarchical structure. In this paper, we propose a novel approach for Hierarchical Multi-label Classification (HMC) of iconographic concepts in images. We present three strategies, including Language Models (LMs), for the generation of textual image descriptions using keywords extracted from Iconclass. These descriptions are utilized to pre-train a Vision-Language Model (VLM) based on a newly introduced data set of 477,569 images with more than 20,000 Iconclass concepts, far more than considered in previous studies. Furthermore, we present five approaches to multi-label classification, including a novel transformer decoder that leverages hierarchical information from the Iconclass taxonomy. Experimental results show the superiority of this approach over reasonable baselines.

Gesture as language of non-verbal communication has been theoretically established since the 17th century. However, its relevance for the visual arts has been expressed only sporadically. This may be primarily due to the sheer overwhelming amount of data that traditionally had to be processed by hand. With the steady progress of digitization, though, a growing number of historical artifacts have been indexed and made available to the public, creating a need for automatic retrieval of art-historical motifs with similar body constellations or poses. Since the domain of art differs significantly from existing real-world data sets for human pose estimation due to its style variance, this presents new challenges. In this paper, we propose a novel approach to estimate human poses in art-historical images. In contrast to previous work that attempts to bridge the domain gap with pre-trained models or through style transfer, we suggest semi-supervised learning for both object and keypoint detection. Furthermore, we introduce a novel domain-specific art data set that includes both bounding box and keypoint annotations of human figures. Our approach achieves significantly better results than methods that use pre-trained models or style transfer.

In this paper, we introduce iART: an open Web platform for art-historical research that facilitates the process of comparative vision. The system integrates various machine learning techniques for keyword- and content-based image retrieval as well as category formation via clustering. An intuitive GUI supports users to define queries and explore results. By using a state-of-the-art cross-modal deep learning approach, it is possible to search for concepts that were not previously detected by trained classification models. Art-historical objects from large, openly licensed collections such as Amsterdam Rijksmuseum and Wikidata are made available to users.