This study examines how modalities can be introduced into Euler diagrams within the broader field of diagrammatic reasoning. To represent modality visually, we adapt notation from C.S. Peirce’s gamma graphs. A key requirement in this framework is distinguishing between two modal scopes: de re, which applies to the properties of objects, and de dicto, which applies to propositions or states of affairs. Our method achieves this distinction by positioning the modal notation either inside or outside the rectangle representing the world in an Euler diagram. This approach enables ontology visualization to capture concepts such as essentiality and disposition, which are otherwise difficult to express in existing frameworks. We further demonstrate its applicability with domain examples: applying de re to identify essential genes in an organism and de dicto to describe developmental states at specific time points. These findings shed light on a pathway toward new systems of diagrammatic logic and visualization methods that support appropriate specification in ontologies.

Abstract Human-generated captions for photographs, particularly snapshots, have been extensively collected in recent AI research. They play a crucial role in the development of systems capable of multimodal information processing that combines vision and language. Recognizing that diagrams may serve a distinct function in thinking and communication compared to photographs, we shifted our focus from snapshot photographs to diagrams. We provided humans with text-free diagrams and collected data on the captions they generated. The diagrams were sourced from AI2D-RST, a subset of AI2D. This subset annotates the AI2D image dataset of diagrams from elementary school science textbooks with types of diagrams. We mosaicked all textual elements within the diagram images to ensure that human annotators focused solely on the diagram’s visual content when writing a sentence about what the image expresses. For the 831 images in our dataset, we obtained caption data from at least three individuals per image. To the best of our knowledge, this dataset is the first collection of caption data specifically for diagrams.

Solving complex consistency checking tasks in natural languages is hard and requires sophisticated specialist expertise. The similar task of finding bugs in information systems can be large-scale and is often conducted with some visualisation of the data. Visualisation, therefore, could also be a useful tool when consistency checking in real world applications, such as in the case of ontology engineering. Previous experiments suggest that node-link visualisation, such as SOVA, are more effective than node-link-region visualisation, such as concept diagrams, in consistency checking tasks. In this study, we found that this tendency was not affected even in an alternative setting where multiple concept diagrams were used. Our findings have implications for the way in which information is presented visually: single (merged) visualisations are effective for these types of tasks.

Vicarious decisions are made on behalf of others that are not for the decision-makers themselves, but for the satisfaction of the others. They are often observed in interactive situations in the real-world, such as investment trusts in an outsourced agency (planners) and its clients (sponsors). We challenged the question of whether planners really could follow the intent of sponsors’ orders in making vicarious risk decisions. We designed and conducted an online experiment in which pairs of persons interacted with each other in the role of either sponsor or planner. Our results showed that planners adjusted the number of gambling or risky choices according to the sponsor’s orders, but did not take actions that reflected the sponsor’s risk preferences; nonetheless, sponsor’s satisfaction to the planner’s choice was substantially high. These findings shed light on the interaction design of how deeply vicarious agents (whether human or robot) should follow the client’s thoughts in collaborative tasks.

Ontologies describe complex world knowledge in that they consist of hierarchical relations, such as is-a, which can be expressed by quantifiers or sets, and various binary relations, which can be expressed by links or networks. Should hierarchical relations be distinguished from other binary relations as essentially different ones in building cognitively accessible systems of ontologies? In this study, two kinds of ontology visualizations, a network-based visualization (SOVA) and a set-based visualization (concept diagrams), are empirically compared in the case of consistency checking. Participants were presented with one diagram and then asked to answer the question of whether the meaning of the diagram was contradictory. Our results showed that SOVA is more effective than concept diagrams, suggesting that to represent hierarchical and binary relations of ontologies in a way based on networks suits human cognition when checking ontologies' consistencies.

We compared participant performance and brain activation changes during a syllogism-solving task with and without Euler diagrams, using functional magnetic resonance imaging (fMRI). Our experiment showed that when Euler diagrams were present, (i) response times in the task were significantly shorter than those in the usual reasoning task comprising only sentences, and (ii) the magnitude of activation in the left middle frontal gyrus (near BA 10), left inferior PFC (near BA 47), and left dorsal PFC (BA 6) was reduced. Result (i) provides evidence for the occurrence of cognitive offloading even when participants handle information of both sentences and diagrams in reasoning tasks. Result (ii) suggests that complex processes of inferences can be replaced by simple diagram manipulation. It is argued that cognitive details that are not fully specified by behavioral studies can be made salient using neuroscientific methods.