Abstract Analysis of cultural heritages, particularly their construction years, provides new insights into human history. However, due to natural disasters, wars, material deterioration, and human errors, records documenting the construction years of many artifacts have often been lost. Historians and experts can estimate construction years within specific ranges using chemical-based analysis technologies or extensive historical research. Given the vast number of collected artifacts, applying these conventional methods to every artifact is impractical. To address this challenge, we developed a deep neural network model designed for Buddha statues to estimate an artifact’s construction year from its image. One major challenge in this task is the heterogeneity of the labels: the training samples include both precise construction years and possible ranges (e.g., a dynasty or a century) estimated by historians. To unify these heterogeneous labels during training, we represent them as probabilistic distributions. In our previous work Qian et al. (2021), we assumed that the ambiguity in heterogeneous construction year labels followed a Gaussian distribution, assigning the highest likelihood to the midpoint of the designated time range. However, this assumption does not always hold. In this paper, we propose representing heterogeneous construction year labels as a uniform distribution, assigning equal probability to all points within the designated time range. Based on this label representation, we designed a semi-supervised learning loss function to leverage both labeled and unlabeled samples during training. Our experimental results demonstrate that our method achieves a mean absolute error of 34.3 years on a test set consisting of Buddha statues constructed between 400 and 1403. These results are further analyzed in two ways. First, we compared our model’s performance to the image quality BRISQUE score, revealing a correlation between higher image quality and lower prediction error rates. Second, we validated our predictions with experts, assessing the level of agreement with our model, the challenges in determining construction years, and identifying features of interest in the artifacts.

Abstract The use of AI systems for managing large-scale cultural heritage artifacts has become possible due to the rise of digitization. To classify such content, machine learning is typically used, where contextual information is important to structure the data. One way to capture context is through a knowledge graph. In this study, we propose a newd graph neural networks, we can improve artwork classification by utilizing the relationships between entities of the knowledge graph. Our experiments demonstrate that this approach achieves state-of-the-art results on multiple classification tasks across three datasets (SemArt paintings, Buddha statues, and Ukiyo-e woodblock prints). Moreover, our approach is effective in dealing with unbalanced data and we explore the use of both graph attention mechanisms and focal loss functions.

The rise of digitization of cultural documents offers large-scale contents, opening the road for development of AI systems in order to preserve, search, and deliver cultural heritage. To organize such cultural content also means to classify them, a task that is very familiar to modern computer science. Contextual information is often the key to structure such real world data, and we propose to use it in form of a knowledge graph. Such a knowledge graph, combined with content analysis, enhances the notion of proximity between artworks so it improves the performances in classification tasks. In this paper, we propose a novel use of a knowledge graph, that is constructed on annotated data and pseudo-labeled data. With label propagation, we boost artwork classification by training a model using a graph convolutional network, relying on the relationships between entities of the knowledge graph. Following a transductive learning framework, our experiments show that relying on a knowledge graph modeling the relations between labeled data and unlabeled data allows to achieve state-of-the-art results on multiple classification tasks on a dataset of paintings, and on a dataset of Buddha statues. Additionally, we show state-of-the-art results for the difficult case of dealing with unbalanced data, with the limitation of disregarding classes with extremely low degrees in the knowledge graph.

In this paper, we propose a full pipeline of analysis of a large corpus about a century of public meeting in historical Australian news papers, from construction to visual exploration. The corpus construction method is based on image processing and OCR. We digitize and transcribe texts of the specific topic of public meeting. Experiments show that our proposed method achieves a F-score of 87.8% for corpus construction. As a result, we built a content search tool for temporal and semantic content analysis.

Automatic art analysis aims to classify and retrieve artistic representations from a collection of images by using computer vision and machine learning techniques. In this work, we propose to enhance visual representations from neural networks with contextual artistic information. Whereas visual representations are able to capture information about the content and the style of an artwork, our proposed context-aware embeddings additionally encode relationships between different artistic attributes, such as author, school, or historical period. We design two different approaches for using context in automatic art analysis. In the first one, contextual data is obtained through a multi-task learning model, in which several attributes are trained together to find visual relationships between elements. In the second approach, context is obtained through an art-specific knowledge graph, which encodes relationships between artistic attributes. An exhaustive evaluation of both of our models in several art analysis problems, such as author identification, type classification, or cross-modal retrieval, show that performance is improved by up to 7.3% in art classification and 37.24% in retrieval when context-aware embeddings are used.

We introduce BUDA.ART, a system designed to assist researchers in Art History, to explore and analyze an archive of pictures of Buddha statues. The system combines different CBIR and classical retrieval techniques to assemble 2D pictures, 3D statue scans and meta-data, that is focused on the Buddha facial characteristics. We build the system from an archive of 50,000 Buddhism pictures, identify unique Buddha statues, extract contextual information, and provide specific facial embedding to first index the archive. The system allows for mobile, on-site search, and to explore similarities of statues in the archive. In addition, we provide search visualization and 3D analysis of the statues.

Advances in multimedia analysis, enables access and indexing of huge and rich video content. However, delivering this heterogeneous content remains a challenge. We proposes the use of multiplex networks to combine both textual and visual semantic cues extracted from TV news videos for interactive search refinement on heterogeneous data. After preprocessing, the indexed videos can be queried, for which result space is summarized in a visual cloud helping query refinement with contextual information by combining visual and textual cues. We leverage on properties of multiplex networks to extract a hierarchy of layers, which not only contextualizes results, but also guides the construction of our visual cloud, and its interactions.

One task key to the analysis of large multimedia archive over time is to dynamically monitor the activity of concepts and entities with their interactions. This is helpful to analyze threads of topics over news archives (how stories unfold), or to monitor evolutions and development of social groups. Dynamic graph modeling is a powerful tool to capture these interactions over time, while visualization and finding communities still remain difficult, especially with a high density of links. We propose to extract the backbone of dynamic graphs in order to ease community detection and guide the exploration of trends evolution. Through the graph structure, we interactively coordinate node-link diagrams, Sankey diagrams, time series, and animations in order to extract patterns and follow community behavior. We illustrate our system with the exploration of the role of soccer in 6 years of TV/radio magazines in France, and the role of North Korea in about 10 years of Japanese news.

Politicians in the same political party often share the same views on social issues and legislative agendas. By mining patterns in TV news co-appearances and Twitter followers, in this paper we estimate political leanings (left / right) of unknown individuals, and detect outlier politicians who have views different from their colleagues in the same party, from a graph signal processing (GSP) perspective. Specifically, we first construct a similarity graph with politicians as nodes, where a positive edge connects two politicians with sizable shared Twitter followers, and a negative edge connects two politicians appearing in the same TV news segment (and thus likely take opposite stands on the same issue). Given a graph with both positive and negative edges, we propose a new graph-signal smoothness prior based on a constructed generalized graph Laplacian matrix that is guaranteed to be positive semi-definite. We formulate a graph-signal restoration problem that can be solved in closed form. Experimental results show that political leanings of unknown individuals can be reliably estimated and outlier politicians can be detected.

Multimedia data is by nature heterogeneous, conveying semantic information through multiple cues. Text analysis of closed captions already brought us understanding of the spoken information. Today's advances in computer vision now enable us to look for relevant semantic information from the visual content of real-world archives. Combining these two levels of extracted information to make sense of an archive still remains a challenge. Multiplex net- works, which model multiple families of interactions in a graph, can capture and combine both sources of semantics. We can leverage on these objects to extract hierarchies and integrate them in an interactive heterogeneous "visual cloud". Inspired by word clouds, these clouds allow to grasp visual and textual semantic information captured from a multimedia collection all at once. The interaction then enables direct access to the relevant video. We demonstrate our system with the exploration of a Japanese news archive.

Visual analytics helps analytical reasoning and exploration of complex systems, for which it combines the means of interactive visualization, with the power of data analytics. The recent progress in computer vision techniques opens wide applications in real world video archives. Particularly, recent advances in face detection and recognition have been put under the spotlight. The applications of such techniques are often concern intelligence, or peer recognition in photo posted in social networks. We propose to combine those two domains by demonstrating a visual exploration of over a decade of the news program from the Japanese broadcaster NHK News 7. We derive social networks from face detection and tracking of this large dataset. With the help of a little domain knowledge, we monitor the activity of political public figures and explore the archive. This allows understanding and comparison of the politico-media scene presented by NHK under different Prime Minister's governance. The social networks are interactive, and also allow to explore the multimedia database and explore its video content.

The rich nature of news makes it a classic subject of visual analytics research. Such analysis is often based on rich textual data. However, we want to test how much we can understand the news from video information through face detection and tracking. Towards this goal, we propose a visual analytics system and discuss its design and implementation to support media experts in understanding political interactions in an archive of 12 years of the Japanese public broadcaster NHK's News 7 program. After identifying the tasks and abstraction required for our analysis, we construct links from face detection and tracking to derive multiple political networks. Our proposed design embeds this rich data into a visual analytics framework that presents four levels of abstraction: time period, network, timeline, and face-tracks within video. We present how the exploration of the archive with our system results in good understanding of the Japanese politico-media scene during these 12 years while finding evidence of “presidentialization” of the media.

Abstract A multiplex network has links of different types, allowing it to express many overlapping types of relationships. A core task in network analysis is to evaluate and understand group cohesion; that is, to explain why groups of elements belong together based on the underlying structure of the network. We present Detangler, a system that supports visual analysis of group cohesion in multiplex networks through dual linked views. These views feature new data abstractions derived from the original multiplex network: the substrate network and the catalyst network. We contribute two novel techniques that allow the user to analyze the complex structure of the multiplex network without the extreme visual clutter that would result from simply showing it directly. The harmonized layout visual encoding technique provides spatial stability between the substrate and catalyst views. The pivot brushing interaction technique supports linked highlighting between the views based on computations in the underlying multiplex network to leapfrog between subsets of catalysts and substrates. We present results from the motivating application domain of annotated news documents with a usage scenario and preliminary expert feedback. A second usage scenario presents group cohesion analysis of the social network of the early American independence movement.

The analysis and exploration of a social network depends on the type of relations at play. Borgatti had proposed a type taxonomy organizing relations in four possible categories. Homophily (similarity) relationships form an important category where relations occur when entities of the network link whenever they exhibit similar behaviors. Examples are networks of coauthor, where homophily between two persons follows from co-authorship; or network of actors having played under the supervision of the same movie director, for instance. Homophily is often embodied through a bipartite network where entities of a given type A (authors, movie directors) connect through entities of a different type B (papers, actors). A common strategy is then to project this bipartite graph onto a single-type network with entities of a same type A, possibly weighting edges based on how the type A entities interact with the type B entities underlying the edge. The resulting single-type network can then be studied using standard techniques such as community detection using edge density, or the computation of various centrality indices. This paper revisits this type of approach and introduces a homogeneity measure inspired form past work by Burt. Two entities of type B interact when they both induce a same edge between two entities of type A. The homogeneity of a subgroup thus depends on how intensely and how equally interactions occur between entities of type B giving rise to the subgroup. The measure thus differentiates between subgroups of type A exhibiting similar topologies depending on the interaction patterns of the underlying entities of type B. The method is validated using two widely used datasets. A first example looks at authors of the IEEE InfoVis Conference (InfoVis 2007 Contest). A second example looks at homophily relations between movie actors that have played under the direction of a same director (IMDB).

Exploring document collections remains a focus of research. This task can be tackled using various techniques, typically ranking documents according to a relevance index or grouping documents based on various clustering algorithms. The task complexity produces results of varying quality that inevitably carry noise. Users must be careful when interpreting document relevance or groupings. We address this problem by computing cohesion measures for a group of documents confirming/infirming whether it can be trusted to form a semantically cohesive unit. The index is inspired from past work in social network analysis (SNA) and illustrates how document exploration can benefit from SNA techniques.

The emergence of very large hierarchies that result from the increase in available data raises many problems of visualization and navigation. On data sets of such scale, classical graph drawing methods do not take advantage of certain human cognitive skills such as shape recognition. These cognitive skills could make it easier to remember the global structure of the data. In this paper, we propose a method that is based on the use of nested irregular shapes. We name it GosperMap as we rely on the use of a Gosper Curve to generate these shapes. By employing human perception mechanisms that were developed by handling, for example, cartographic maps, this technique facilitates the visualization and navigation of a hierarchy. An algorithm has been designed to preserve region containment according to the hierarchy and to set the leaves' sizes proportionally to a property, in such a way that the size of nonleaf regions corresponds to the sum of their children's sizes. Moreover, the input ordering of the hierarchy's nodes is preserved, i.e., the areas that represent two consecutive children of a node in the hierarchy are adjacent to one another. This property is especially useful because it guarantees some stability in our algorithm. We illustrate our technique by providing visualization examples of the repartition of tax money in the US over time. Furthermore, we validate the use of the GosperMap in a professional documentation context and show the stability and ease of memorization for this type of map.