This paper explores the applicability of Convergent Cross Mapping (CCM) and its extension, Time Delay Convergent Cross Mapping (TDCCM), to assess the causal relationships between Bitcoin, the S&P 500 index, and gold. Unlike conventional causality analysis methods, such as Granger causality or transfer entropy, CCM accounts for non-separable, weakly connected dynamic systems, and TDCCM explicitly incorporates time lags during cross-mapping, enabling the detection of complex causal relationships in systems with shared nonlinear behavior. This makes it particularly suitable for financial time series that often exhibit chaotic and nonlinear dynamics, particularly during periods of market instability. We integrate TDCCM with simplex projection and sequential locally weighted global linear map (S-map) algorithms, applying a sliding window approach to identify short time intervals characterized by high levels of nonlinearity and chaoticity. Using this approach, we uncovered a strong causal relationship between Bitcoin and the S&P 500 index during the onset of the COVID-19 pandemic. Our analysis reveals a bidirectional causal relationship between Bitcoin and the S&P 500 index, highlighting their interconnectedness during periods of heightened economic uncertainty. Furthermore, we find a unidirectional causal influence of Bitcoin on gold, reflecting Bitcoin’s evolving role as a macroeconomic indicator and its growing relevance as an alternative store of value. These findings provide insight into the dynamics between cryptocurrencies and traditional financial markets, particularly during periods of global economic disruption. • We use Time Delay Convergent Cross Mapping (TD-CCM) to identify and quantify lagged causal interactions between financial time series (Bitcoin, Gold, and the S&P 500 index), which is a more recent and less explored method for Time Series causality. • We report a comprehensive and replicable methodology to apply TD-CCM to non-linear TS, based on combining the S-Map and Prediction Decay algorithm with a sliding window technique, while validating with surrogate analysis. • We show evidence of strong causal influence from Bitcoin to Gold and bidirectional causality between Bitcoin and the S&P 500 Index during significant economic events like the COVID-19 pandemic.

In the recent advancements in application of deep learning to time series forecasting, focus has shifted from training transformers end-to-end to efficiently leveraging the predictive capabilities of Large Language Models (LLMs). Models that encode the time series data to interact with a frozen LLM backbone have been shown to outperform transformers on all benchmark datasets. However, their efficiency on complex datasets, which do not show clear seasonality or trend, remains an open question. In this work, we seek to evaluate the performance of reprogrammed LLMs on the Bitcoin price chart, a financial time series known for its complexity and high volatility. We propose effective methods to improve the performance of Time-LLM, a State-of-the-art (SOTA) method, on such a time series. First, we propose structural improvements to Time-LLM. Second, we suggest an efficient way to handle the non-stationarity of the dataset. Finally, we propose an efficient method for passing additional financial information to the LLM. Our results demonstrate a 50% improvement on the average percentage loss and a 5% increase on accuracy of our adapted Time-LLM architecture on Bitcoin data when compared to SOTA models, including the original Time-LLM model. This highlights the impact on forecast accuracy of domain-specific decision making in data processing and feature selection.

The development of a real-world Unmanned Aircraft System (UAS) Traffic Management (UTM) system to ensure the safe integration of Unmanned Aerial Vehicles (UAVs) in low altitude airspace, has recently generated novel research challenges. A key problem is the development of Pre-Flight Conflict Detection and Resolution (CDR) methods that provide collision-free flight paths to all UAVs before their takeoff. Such problem can be represented as a Multi-Agent Path Finding (MAPF) problem. Currently, most MAPF methods assume that the UTM system is a centralized entity in charge of CDR. However, recent discussions on UTM suggest that such centralized control might not be practical or desirable. Therefore, we explore Pre-Flight CDR methods where independent UAS Service Providers (UASSPs) with their own interests, communicate with each other to resolve conflicts among their UAV operations—without centralized UTM directives. We propose a novel MAPF model that supports the decentralized resolution of conflicts, whereby different ‘agents’, here UASSPs, manage their UAV operations. We present two approaches: (1) a prioritization approach and (2) a simple yet practical pairwise negotiation approach where UASSPs agents determine an agreement to solve conflicts between their UAV operations. We evaluate the performance of our proposed approaches with simulation scenarios based on a consultancy study of predicted UAV traffic for delivery services in Sendai, Japan, 2030. We demonstrate that our negotiation approach improves the “fairness” between UASSPs, i.e. the distribution of costs between UASSPs in terms of total delays and rejected operations due to replanning is more balanced when compared to the prioritization approach.

Narrative Generation has attracted significant interest as a novel application of Automated Planning techniques. However, the vast amount of narrative material available opens the way to the use of Deep Learning techniques. In this paper, we explore the feasibility of narrative generation through self-supervised learning, using sequence embedding techniques or auto-encoders to produce narrative sequences. We use datasets of well-formed plots generated by a narrative planning approach, using pre-existing, published, narrative planning domains, to train generative models. Our experiments demonstrate the ability of generative sequence models to produce narrative plots with similar structure to those obtained with planning techniques, but with significant plot novelty in comparison with the training set. Most importantly, generated plots share structural properties associated with narrative quality measures used in Planning-based methods. As plan-based structures account for a higher level of causality and narrative consistency, this suggests that our approach is able to extend a set of narratives with novel sequences that display the same high-level narrative properties. Unlike methods developed to extend sets of textual narratives, ours operates at the level of plot structure. Thus, it has the potential to be used across various media for plots of significant complexity, being initially limited to training and generation operating in the same narrative genre.

Hyperbolic ordinal embedding (HOE) represents entities as points in hyperbolic space so that they agree as well as possible with given constraints in the form of entity $i$ is more similar to entity $j$ than to entity $k$. It has been experimentally shown that HOE can obtain representations of hierarchical data such as a knowledge base and a citation network effectively, owing to hyperbolic space’s exponential growth property. However, its theoretical analysis has been limited to ideal noiseless settings, and its generalization error in compensation for hyperbolic space’s exponential representation ability has not been guaranteed. The difficulty is that existing generalization error bound derivations for ordinal embedding based on the Gramian matrix are not applicable in HOE, since hyperbolic space is not inner-product space. In this paper, through our novel characterization of HOE with decomposed Lorentz Gramian matrices, we provide a generalization error bound of HOE for the first time, which is at most exponential with respect to the embedding space’s radius. Our comparison between the bounds of HOE and Euclidean ordinal embedding shows that HOE’s generalization error comes at a reasonable cost considering its exponential representation ability.

Graph embedding, which represents real-world entities in a mathematical space, has enabled numerous applications such as analyzing natural languages, social networks, biochemical networks, and knowledge bases.It has been experimentally shown that graph embedding in hyperbolic space can represent hierarchical tree-like data more effectively than embedding in linear space, owing to hyperbolic space's exponential growth property. However, since the theoretical comparison has been limited to ideal noiseless settings, the potential for the hyperbolic space's property to worsen the generalization error for practical data has not been analyzed.In this paper, we provide a generalization error bound applicable for graph embedding both in linear and hyperbolic spaces under various negative sampling settings that appear in graph embedding. Our bound states that error is polynomial and exponential with respect to the embedding space's radius in linear and hyperbolic spaces, respectively, which implies that hyperbolic space's exponential growth property worsens the error.Using our bound, we clarify the data size condition on which graph embedding in hyperbolic space can represent a tree better than in Euclidean space by discussing the bias-variance trade-off.Our bound also shows that imbalanced data distribution, which often appears in graph embedding, can worsen the error.

This paper focuses on the mapping of natural language sentences in written stories to a structured knowledge representation. This process yields an exponential explosion of instance combinations since each sentence may contain a set of ambiguous terms, each one giving place to a set of instance candidates. The selection of the best combination of instances is a structured classification problem that yields a high-demanding combinatorial optimization problem which, in this paper, is approached by a novel and efficient formulation of a genetic algorithm, which is able to exploit the conditional independence among variables, while improving the parallel scalability. The automatic rating of the resulting set of instance combinations, i.e., possible text interpretations, demands an exhaustive exploitation of the state-of-the-art resources in natural language processing to feed the system with pieces of evidence to be fused by the proposed framework. In this sense, a mapping framework able to reason with uncertainty, to integrate supervision and evidence from external sources, was adopted. To improve the generalization capacity while learning from a limited amount of annotated data, a new constrained learning algorithm for Bayesian networks is introduced. This algorithm bounds the search space through a set of constraints which encode information on mutually exclusive values. The mapping of natural language utterances to a structured knowledge representation is important in the context of game construction, e.g., in an RPG setting, as it alleviates the manual knowledge acquisition bottleneck. The effectiveness of the proposed algorithm is evaluated on a set of three stories, yielding nine experiments. Our mapping framework yields performance gains in predicting the most likely structured representations of sentences when compared with a baseline algorithm.

The visualization of procedural knowledge from textual documents using 3D animation may be a way to improve understanding. We are interested in applying this approach to documents relating to patient education for bariatric surgery: a domain with challenging textual documents describing behavior recommendations that contain few procedural steps and leave much commonsense knowledge unspecified. In this work we look at how to automatically capture knowledge from a range of differently phrased recommendations and use that with implicit knowledge about compliance and violation, such that the recommendations can be visualized using 3D animations. Our solution is an end-to-end system that automates this process via: analysis of input recommendations to uncover their conditional structure; the use of commonsense knowledge and deontic logic to generate compliance and violation rules; and mapping of this knowledge to update a default knowledge base, which is used to generate appropriate sequences of visualizations. In this paper we overview this approach and demonstrate its potential.

In this article, we discuss an innovative media entertainment application called Interactive Movietelling. As an offspring of Interactive Storytelling applied to movies, we propose to integrate narrative generation through artificial intelligence (AI) planning with video processing and modeling to construct filmic variants starting from the baseline content. The integration is possible thanks to content description using semantic attributes pertaining to intermediate-level concepts shared between video processing and planning levels. The output is a recombination of segments taken from the input movie performed so as to convey an alternative plot. User tests on the prototype proved how promising Interactive Movietelling might be, even if it was designed at a proof of concept level. Possible improvements that are suggested here lead to many challenging research issues.

Despite recent progress in plan-based narrative generation, one major limitation is that systems tend to produce a single plotline whose progression entirely determines the narrative experience. However, for certain narrative genres such as serial dramas and soaps, multiple interleaved subplots are expected by the audience, as this tends to be the norm in real-world, human-authored narratives. Current narrative generation techniques have overlooked this important requirement, something which could improve the perceived quality of generated stories. To this end, we have developed a flexible plan-based approach to multiplot narrative generation, that successfully generates narratives conforming to different subplot profiles, in terms of the number of subplots interleaved and the relative time spent on each presentation. We have identified specific challenges such as: distribution of virtual characters across subplots; length of each subplot presentation; and transitioning between subplots.

Narrative generation represents an application domain for AI planning where plan quality is related to properties such as shape of plan trajectory. In our work we have developed a plan-based approach to narrative generation that uses character relationships as a key determinant in controlling plan shape (relationships are key in genres such as serial dramas and soaps). Our approach is implemented in a demonstration Interactive Narrative, called NetworkING, set in the medical drama genre. The system features a user-friendly mechanism for specifying relationships between virtual characters, via a social network and real-time visualisation of generated narratives on a 3D stage.

User interaction with Embodied Conversational Agents (ECA) should involve a significant affective component to achieve realism in communication. This aspect has been studied through different frameworks describing the relationship between user and ECA, for instance alignment, rapport and empathy. We conducted an experiment to explore how an ECA's non-verbal expression can be controlled to respond to a single affective dimension generated by users as input. Our system is based on the mapping of a high-level affective dimension, approach/avoidance, onto a new ECA control mechanism in which Action Units (AU) are activated through a neural network. Since 'approach' has been associated to prefrontal cortex activation, we use a measure of prefrontal cortex left-asymmetry through fNIRS as a single input signal representing the user's attitude towards the ECA. We carried out the experiment with 10 subjects, who have been instructed to express a positive mental attitude towards the ECA. In return, the ECA facial expression would reflect the perceived attitude under a neurofeedback paradigm. Our results suggest that users are able to successfully interact with the ECA and perceive its response as consistent and realistic, both in terms of ECA responsiveness and in terms of relevance of facial expressions. From a system perspective, the empirical calibration of the network supports a progressive recruitment of various AUs, which provides a principled description of the ECA response and its intensity. Our findings suggest that complex ECA facial expressions can be successfully aligned with one high-level affective dimension. Furthermore, this use of a single dimension as input could support experiments in the fine-tuning of AU activation or their personalization to user preferred modalities.

Neurofeedback systems can be modeled as closed-loop control systems with negative feedback. However, little work to date has investigated the potential of this representation in gaining a better understanding of the actual dynamics of neurofeedback towards explaining subjects' performance. In this paper, we analyze neurofeedback training data through a PID control model. We first show that PID model fitting can produce curves that are qualitatively aligned to the measured BCI signal. Secondly, we examine how brain activity during neurofeedback can be related to common characteristics of control systems. For this, we formalized a pre-existing neurofeedback EEG experiment using a Simulink®model that captures both the neural activity and the external algorithm that was utilized to generate the feedback signal. We then used a regression model to fit individual trial data to PID coefficients for the control model. Our results suggest that successful trials tend to be associated to higher average values of Ki, which represents the error-reducing component of the PID controller. It hints that convergence in successful neurofeedback is progressive but complete in approaching the target.

Functional near-infrared spectroscopy (fNIRS) holds increasing potential for Brain-Computer Interfaces (BCI) due to its portability, ease of application, robustness to movement artifacts, and relatively low cost. The use of fNIRS to support the development of affective BCI has received comparatively less attention, despite the role played by the prefrontal cortex in affective control, and the appropriateness of fNIRS to measure prefrontal activity. We present an active, fNIRS-based neurofeedback (NF) interface, which uses differential changes in oxygenation between the left and right sides of the dorsolateral prefrontal cortex to operationalize BCI input. The system is activated by users generating a state of anger, which has been previously linked to increased left prefrontal asymmetry. We have incorporated this NF interface into an experimental platform adapted from a virtual 3D narrative, in which users can express anger at a virtual character perceived as evil, causing the character to disappear progressively. Eleven subjects used the system and were able to successfully perform NF despite minimal training. Extensive analysis confirms that success was associated with the intent to express anger. This has positive implications for the design of affective BCI based on prefrontal asymmetry.

Research suggests that browsing clinical guidelines in a linear format is difficult for users. One national producer of clinical guidelines (HAS, the French National Authority for Health) has recently developed a new document format designed to improve accessibility. It is a limited-depth hypertext structurally constrained so that all information lies within two clicks of a central index ('reco2clics'). In the following paper, we introduce an authoring tool which converts full-length clinical guidelines to the 'reco2clics' format. Alongside routine editorial operations, this tool supports dynamic document restructuring, a complex operation using text segmentation algorithms and deontic analysis.

The Art Exhibition of ACM Multimedia 2013 has attracted significant work from a variety of digital artists collaborating with research institutions. We have endeavored to select exhibits that achieved an interesting balance between technology and artistic intent. The techniques underpinning these artworks are relevant to several technical tracks of the conference, in particular those dealing with human-centered and interactive media. We briefly review how the various installations revisit current topics in Multimedia research, focusing more specifically on their approach to dynamic content generation, user experience, multimodality, and affective interfaces. Once again, the unique blend of technology and user experience does not limit itself to showcasing recent advances in interactive media, and should be of interest to all conference participants as well as the general public the exhibition space will be open to.

Professional users of medical information often report difficulties when attempting to locate specific information in lengthy documents. Sometimes these difficulties can be attributed to poorly specified section titles which fail to advertise relevant content. In this paper we describe preliminary work on a software plug-in for a document engineering environment that will assist authors when they formulate section-level headings. We describe two different algorithms which can be used to generate section titles. We compare the performance of these algorithms and correlate our experimental results with an evaluation of title quality performed by domain experts.

Advanced interactive visualization such as in virtual environments and ubiquitous interaction paradigms pose new challenges and opportunities in considering real-time responses to subliminal cues. In this paper, we propose a synthetic reality platform that, combined with psychophysiological recordings, enables us to study in realtime the effects of various subliminal cues. We endeavor to integrate various aspects known to be relevant to implicit perception. The context is of consumer experience and choice of an artifact where the generation of subliminal perception through an intelligent 3D interface controls the spatio-temporal aspects of the information displayed and of the emergent narrative. One novel contribution of this work is the programmable nature of the interface that exploits known perceptive phenomena (e.g. masking, crowding and change blindness) to generate subliminal perception.

Previous Interactive Storytelling systems have been designed to allow active user intervention in an unfolding story, using established multi-modal interactive techniques to influence narrative development. In this paper we instead explore the use of a form of passive interaction where users' affective responses, measured by physiological proxies, drive a process of narrative adaptation. We introduce a system that implements a passive interaction loop as part of narrative generation, monitoring users' physiological responses to an on-going narrative visualization and using these to adapt the subsequent development of character relationships, narrative focus and pacing. Idiomatic cinematographic techniques applied to the visualization utilize existing theories of establishing characteristic emotional tone and viewer expectations to foster additional user response. Experimental results support the applicability of filmic emotional theories in a non-film visual realization, demonstrating significant appropriate user physiological response to narrative events and "emotional cues". The subsequent narrative adaptation provides a variation of viewing experience with no loss of narrative comprehension.

The dominant interaction paradigm in Interactive Storytelling (IS) systems so far has been active interventions by the user by means of a variety of modalities. PINTER is an IS system that uses physiological inputs - surface electromyography (EMG) and galvanic skin response (GSR) [1] - as a form of passive interaction, opening up the possibility of the use of traditional filmic techniques [2, 3] to implement IS without requiring immersion-breaking interactive responses. The goal of this demonstration is to illustrate the ways in which passive interaction combined with filmic visualisation, dialogue and music, and a plan-based narrative generation approach can form a new basis for an adaptive interactive narrative.

Previous attempts at developing immersive versions of game engines have faced difficulties in achieving both overall high performance and preserving reusability of software developments. In this paper, we present a high-level VR middleware based on one of the most successful commercial game engines: the Unreal® Engine 3.0 (UE3). We describe a VR framework implemented as an extension to the Unreal® Development Kit (UDK) supporting CAVE"-like installations. Our approach relies on a distributed architecture reinforced by specific replication patterns to synchronize the user's point of view and interactions within a multi-screen installation. Our performance benchmarks indicated that our immersive port does not affect the game engine performance, even with complex real-time applications, such as fast-paced multiplayer First Person Shooter (FPS) games or high-resolution graphical environments with 2M+ polygons. A user study also demonstrated the capacity of our VR middleware to elicit high spatial presence while maintaining low cybersickness effects. With free distribution, we believe such a platform can support future Entertainment and VR research.

This paper describes an experiment comparinng immersive and non-immersive gaming using a state-of-the-art first person shooter game (FPS) in which we analyse user experience and performance through a combination of in-game metrics, questionaires and subjective reports. Our results show an overwhelming subjective preference for the immersive version despite a decrease in performance attributed to a more realistic aimming mechanism. Interaction metrics suggest that users took full advantage of the immmersive context rather than simply transposing their desktop gaming skills.

In this paper we present an evaluation of an affective multimodal fusion approach utilizing dimensional representations of emotion. The evaluation uses physiological signals as a reference measure of users' emotional states. Surface electromyography (EMG) and galvanic skin response (GSR) signals are known to be correlated with specific dimensions of emotion (Pleasure and Arousal) and are compared here to real time continuous values of these dimensions obtained from affective multimodal fusion. The results (both qualitative and quantitative) suggest that the particular multimodal fusion approach described is consistent with physiological indicators of emotion, constituting a first positive evaluation of the approach.

We present a novel approach to the automatic generation of filmic variants within an implemented Video-Based Storytelling (VBS) system that successfully integrates video segmentation with stochastically controlled re-ordering techniques and narrative generation via AI planning. We have introduced flexibility into the video recombination process by sequencing video shots in a way that maintains local video consistency and this is combined with exploitation of shot polysemy to enable shot reuse in a range of valid semantic contexts. Results of evaluations on output narratives using a shared set of video data show consistency in terms of local video sequences and global causality with no loss of generative power.

Currently, automatic generation of filmic variants faces a number of key technical issues and thus it usually resorts to the shooting of multiple versions of alternative scenes. However, recent advancements in video analysis has made this objective feasible, though semantic consistency must be somehow preserved. This demo presents a video-based storytelling (VBS) system that successfully integrates video processing with narrative generation by means of a shared semantic description. The novel filmic variants are constructed through a flexible video recombination process that takes advantage of the polysemy of baseline video segments. The short output video clips shown in this demo prove how the generated narratives are semantically consistent while keeping generative power intact.

Previous work has established that specific linguistic markers present in specialised medical documents (clinical guidelines) can be used to support their automatic structuring within a document engineering environment. This technique is commonly used by the French Health Authority (la Haute Autorite de Sante) during elaboration of clinical guidelines to improve the quality of the final document. In this paper, we explore the readability of clinical guidelines. We discuss a structural measure of document readability that exploits the ratio between these linguistic markers (deontic structures) and the remainder of the text. We describe an experiment in which a corpus of 10 French clinical guidelines is scored for structural readability. We correlate these scores with measures of textual cohesion (computed using latent semantic analysis) and the results of a readability survey performed by a panel of domain experts. Our results suggest an association between the density of deontic structures in a clinical guideline and its overall readability. This implies that certain generic readability measures can henceforth be utilised in our document engineering environment.

Whilst narrative representations have played a prominent role in AI research, there has been a renewed interest in the topic with the development of interactive narratives. A typical approach aims at generating narratives from baseline action representations, most often using planning techniques. However, this research has developed empirically, often as an application of planning. In this paper, we explore a more rigorous formalisation of narrative concepts, both at the action level and at the plot level. Our aim is to investigate how to bridge the gap between action descriptions and narrative concepts, by considering the latter from the perspective of resource consumption and causality. We propose to use Linear Logic, often introduced as a logic of resources, for it provides, through linear implication, a better description of causality than in Classical and Intuitionistic Logic. Besides advances in the fundamental principles of narrative formalisation, this approach can support the formal validation of scenario description as a preliminary step to their implementation via other computational formalisms.

Virtual characters are at the epicentre of Interactive Storytelling systems and in recent years multiple AI planning approaches have been described to specify their autonomous behaviour. This demonstrator provides an overview of our novel approach to the definition of virtual characters aimed at achieving a balance between character autonomy and global plot structure which proposes the notion of a character's Point of View. Additionally, the demonstrator offers the active spectator the ability to discover the story described from the perspective of a number of different characters. We present our fully-implemented Interactive Narrative based on Shakespeare's Merchant of Venice. The system, which features a novel AI planning approach to story generation, can generate very different stories depending on the Point of View adopted and support dynamic modification of the story world which results in different story consequences.

In the paper we present a prototype of video-based storytelling that is able to generate multiple story variants from a baseline video. The video content for the system is generated by an adaptation of forefront video summarisation techniques that decompose the video into a number of Logical Story Units (LSU) representing sequences of contiguous and interconnected shots sharing a common semantic thread. Alternative storylines are generated using AI Planning techniques and these are used to direct the combination of elementary LSU for output. We report early results from experiments with the prototype in which the reordering of video shots on the basis of their high-level semantics produces trailers giving the illusion of different storylines.

The Entertainment potential of Virtual Reality is yet to be fully realised.In recent years, this potential has been described through the Holodeck™ metaphor, without however addressing the issue of content creation and gameplay.Recent progress in Interactive Narrative technology makes it possible to envision immersive systems.Yet, little is known about the usability of such systems or which paradigms should be adopted for gameplay and interaction.We report user experiments carried out with a fully immersive Interactive Narrative system based on a CAVE-like system, which explore two interactivity paradigms for user involvement (Actor and Ghost).Our results confirm the potential of immersive Interactive Narratives in terms of performance but also of user acceptance.

Game programmers rely on artificial intelligence techniques to encode characters' behaviors initially specified by game designers. Although significant efforts have been made to assist their collaboration, the formalization of behaviors remains a time-consuming process during the early stages of game development. We propose an authoring tool allowing game designers to formalize, visualize, modify, and validate game level solutions in the form of automatically generated storyboards. This system uses planning techniques to produce a level solution consistent with gameplay constraints. The main planning agent corresponds to the player character, and the system uses the game actions as planning operators and level objectives as goals to plan the level solutions. Generated solutions are presented as 2-D storyboards similar to comic strips. We present in this paper the first version of a fully implemented prototype as well as examples of generated storyboards, adapted from the original design documents of the blockbuster game Hitman.

We have seen ten years of the application of AI planning to the problem of narrative generation in Interactive Storytelling (IS). In that time planning has emerged as the dominant technology and has featured in a number of prototype systems. Nevertheless key issues remain, such as how best to control the shape of the narrative that is generated (e.g., by using narrative control knowledge , i.e., knowledge about narrative features that enhance user experience) and also how best to provide support for real-time interactive performance in order to scale up to more realistic sized systems. Recent progress in planning technology has opened up new avenues for IS and we have developed a novel approach to narrative generation that builds on this. Our approach is to specify narrative control knowledge for a given story world using state trajectory constraints and then to treat these state constraints as landmarks and to use them to decompose narrative generation in order to address scalability issues and the goal of real-time performance in larger story domains. This approach to narrative generation is fully implemented in an interactive narrative based on the “Merchant of Venice.” The contribution of the work lies both in our novel use of state constraints to specify narrative control knowledge for interactive storytelling and also our development of an approach to narrative generation that exploits such constraints. In the article we show how the use of state constraints can provide a unified perspective on important problems faced in IS.

In this paper, the authors extend previous work on the automatic structuring of medical documents using content analysis. Their long-term objective is to take advantage of specific rhetoric markers encountered in specialized medical documents (clinical guidelines) to automatically structure free text according to its role in the document. This should enable to generate multiple views of the same document depending on the target audience, generate document summaries, as well as facilitating knowledge extraction from text. The authors have established in previous work that the structure of clinical guidelines could be refined through the identification of a limited set of deontic operators. They now propose to extend this approach by analyzing the text delimited by these operators using Rhetorical Structure Theory. The emphasis on causality and time in RST proves a powerful complement to the recognition of deontic structures while retaining the same philosophy of high-level recognition of sentence structure, which can be converted into applicationspecific mark-ups. Throughout the paper, the authors illustrate their findings through results produced by the automatic processing of English guidelines for the management of hypertension and Alzheimer disease.

We present a physical multimodal conversational Companion in the area of health and fitness. Conversational spoken dialogues using physical agents provide a potential interface for applications which are aimed at motivating and supporting users. Open source software called jNabServer, which enables spoken and multimodal interaction with Nabaztag/tag wireless rabbits, is presented together with other software architecture solutions applied in the development of the Companion. We also present how the Companion manages interaction with the combination of a dialogue manager and a cognitive model. Index Terms: spoken dialogue systems, multimodality, software architectures, physical agent interfaces

In this paper, we describe an Augmented Reality Art installation, which reacts to user behaviour using Multimodal analysis of affective signals. The installation features a virtual tree, whose growth is influenced by the perceived emotional response from spectators. The system implements a 'magic mirror' paradigm (using a large-screen display or projection system) and is based on the ARToolkit with extended representations for scene graphs. The system relies on a PAD dimensional model of affect to support the fusion of different affective modalities, while also supporting the representation of affective responses that relate to aesthetic impressions. The influence of affective input on the visual component is achieved by mapping affective data to an L-System governing virtual tree behaviour. We have performed an early evaluation of the system, both from the technical perspective and in terms of user experience. Post-hoc questionnaires were generally consistent with data from multimodal affective processing, and users rated the overall experience as positive and enjoyable, regardless of how proactive they were in their interaction with the installation.

3D graphics systems increasingly rely on sophisticated event systems derived from collision detection mechanisms, which support the discretisation of Physics as well as high-level programming and scripting. By contrast, augmented reality systems have not yet adopted this approach. We describe the development of a high-level event system on top of the ARToolkit environment incorporating the ODE Physics engine. We first define a typology of events encompassing interactions between virtual objects as well as interactions involving markers. We then describe how these events can be recognised in real-time from elementary collisions detected by the ODE Physics engine. We conclude by discussing examples of high-level event recognitions and how they can support the development of applications.

The development of complex interactive 3D systems raises the need for representations supporting more abstract descriptions of world objects, their behaviour and the world dynamics. The inclusion of Artificial Intelligence representations and their use within 3D graphic worlds face both fundamental and technical issues due to the difference in representational logic between computer graphics and knowledge-based systems. We present a framework for such an integration illustrated by a first prototype.

In this paper, we describe a small-scale, yet complete, integration of a real-time immersive interactive storytelling system. While significant progress has been achieved in recent years on the individual component technologies of interactive storytelling, the main objective of this work is to investigate the concept of interactive storytelling in a fully immersive context. We describe each individual component of immersive interactive storytelling from a technical perspective. We have used a commercial game engine as a development environment, supporting real-time visualisation as well as the inclusion of Artificial Intelligence components controlling virtual actors. This visualisation engine has been ported to an immersive setting using dedicated software and hardware supporting real-time stereoscopic visualisation. The hardware platform is built around a 4-sided CAVE-like immersive display operated by a PC-cluster. The interactive storytelling engine is constituted by a planning system based on characters motivations and emotional states. The user can interact with the virtual world using multimodal interaction. We illustrate the system's behaviour on the implementation of excerpts from Madame Bovary, a classic XIXth century novel, and demonstrate the ability for the user to play the role of one of the characters and influence the unfolding of the story by his actions.

In this paper, we discuss how a cognitive concept, causality, can be used for the conceptual underpinning of Virtual Reality Art installations. Causality plays an important role in our construction of reality and, as such, it makes sense to use it as a principle to define VR experiences. We have developed a VR platform using cognitive data on causal perception to create artificial event co-occurrences in virtual worlds, which can be perceived as possible outcomes for user actions. After a preliminary validation of this technology by user experiments, it has been used to implement prototypes of artistic installations by two different artists. We describe the technical approach behind the elicitation of causal perception in virtual reality, and illustrate its use through the two artistic installations being developed with this new VR platform.

In this paper, we describe a new approach to the creation of virtual environments, which uses qualitative physics to implement object behaviour. We adopted Qualitative Process Theory as a qualitative reasoning formalism, due to its representational properties (e.g., its orientation towards process ontologies and its explicit formulation of process' pre-conditions). The system we describe is developed using a game engine and takes advantage of its event-based system to integrate qualitative process simulation in an interactive fashion. We use a virtual kitchen as a test environment. In this virtual world, we have implemented various behaviours: physical object behaviour, complex device behaviour (appliances) and "alternative" (i.e. non-realistic) behaviours, which can all be simulated in user real-time. After a presentation of the system architecture and its implementation, we discuss example results from the prototype. This approach has potential applications in simulation and training, as well as in entertainment and digital arts. This work also constitutes a test case for the integration of an Artificial Intelligence technique into 3D user interfaces.

We introduce a novel approach to the creation of Virtual Reality Art installations, which supports the design of alternative worlds, in which laws of Physics can be redefined to induce new user experiences. To implement this concept of "Alternative Reality", we have used Artificial Intelligence techniques to support the definition of the virtual environment behaviour, an approach inspired by Qualitative Reasoning systems. Besides the redefinition of physical laws, we have developed mechanisms for eliciting causal relations between events, as causality plays an important part in users' perception of virtual worlds. Our pilot installation is a CAVETM-like system incorporating a state-of-the-art computer game engine as visualisation software, which has-been ported to this immersive display. The event-based system underlying the game engine is used to bypass the native Physics engine and replace it with our Alternative Reality software. A first prototype has been fully implemented, the Alternative Reality modules totaling over 100,000 lines of C++ code. We present early results obtained with this approach, illustrated with examples taken from two artistic briefs, developed by digital artists associated to this research.

In this paper, we describe a mixed reality system based on a "magic mirror" model, in which the user's image is captured in real time by a video camera, extracted from his/her background and mixed with a 3D graphic model of a virtual image including the synthetic characters taking part in the story. The resulting image is projected on a large screen facing the user, who sees his/her own image embedded in the virtual stage with the synthetic actors. The graphic component of the mixed reality world is based on a game engine, Unreal Tournament 2003. This engine not only performs graphic rendering and character animation but incorporates a new version of our previously described storytelling engine. A single 2D camera facing the user analyses the image in real-time by segmenting the user's contours.

No abstract available.

In this paper, we describe the development of a virtual human to be used for training applications in the field of cardiac emergencies. The system integrates AI techniques for simulating medical conditions (shock states) with a realistic visual simulation of the patient in a 3D environment representing an ER roomIt uses qualitative simulation of the cardio-vascular system to generate clinical syndromes and simulate the consequences of the trainees therapeutic interventions. The use of knowledge-based simulation provides a strong basis to integrate the behavioural aspects with the graphical appearance of the patient in the virtual ER. This also supports the creation of an emotional atmosphere increasing the realism of the training system

We present a new framework for the use of virtual reality (VR) in engineering design for configuration applications. Traditional VR systems support the visual exploration of a design solution but do not assist the user in exploring alternative solutions based on domain knowledge. Extending previous work in the area of intelligent virtual environment, we propose an intelligent configuration system based on constraint logic programming (CLP), integrated in a real-time 3D graphic environment. This type of integration facilitates the expression of design knowledge in the VE and enables the user to interactively solve and/or refine a spatial configuration problem. In the system described, the user can visually explore configurations, but his interaction with objects of the configuration problem triggers new cycles of constraint propagation from the modified configuration to produce a new compatible solution.

We present a new approach to computer-aided training in medicine based on the visualisation of simulated clinical conditions using a virtual patient. Our system uses a knowledge-based model of cardiac physiology to generate realistic emergency situations (shock states). The overall results of the simulations are presented through a realistic 3D virtual patient whose behaviour and appearance are consistent with the clinical data generated. After introducing the principles behind the simulation and the system's architecture, we describe the process through which physiological parameters are mapped to the visual appearance of the 3D patient and discuss representative results from the system.

No abstract available.

Virtual Reality Art involves the design of artificial worlds that offer new experiences to spectators. An important aspect for the development of VR Art installations is the principled definition of behaviour for the environment as a whole, which would facilitate experiments with alternative laws of physics, time, and causality. We describe the first results of an ongoing project dedicated to the development of software tools for the use of Intelligent Virtual Environments in VR Art. Using the architecture of a state-of-the-art game engine, we have developed Artificial Intelligence techniques that support the definition of alternative laws of physics. After discussing the principles behind alternative reality we describe two complementary modes of description for alternative behaviour: qualitative physics and causal simulation. This is illustrated by examples integrated into the virtual environment.

Abstract In this paper, we describe a method for implementing the behaviour of artificial actors in the context of interactive storytelling. We have developed a fully implemented prototype based on the Unreal Tournament™ game engine, and carried experiments with a simple sitcom‐like scenario. We discuss the central role of artificial actors in interactive storytelling and how real‐time generation of their behaviour participates in the creation of a dynamic storyline. We follow previous work describing the behaviour of artificial actors through AI planning formalisms, and adapt it to the context of narrative representation. In this context, the narrative equivalent of a character's behaviour consists of its role. The set of possible roles for a given actor is represented as a hierarchical task network (HTN). The system uses HTN planning to dynamically generate the character roles, by interleaving planning and execution, which supports dynamic interaction between actors, as well as user intervention in the unfolding plot. Finally, we present several examples of short plots and situations generated by the system from the dynamic interaction of artificial actors. Copyright © 2002 John Wiley & Sons, Ltd.

In this paper, we describe a method for implementing AI-based animation of artificial actors in the context of interactive storytelling. We have developed a fully implemented prototype based on the Unreal/sup TM/ game engine and carried out experiments with a simple sitcom-like scenario. We discuss the central role of artificial actors in interactive storytelling and how real-time generation of their behaviour participates to the creation of a dynamic storyline. We follow previous work describing the behaviour of artificial actors through AI planning formalisms, and adapt it to the context of narrative representation. The set of all possible behaviours, accounting for various instantiations of a basic plot, can be represented through an AND/OR graph. A real-time variant of the AO* algorithm can be used to interleave planning and action, thus allowing characters to interact between themselves and with the user. Finally, we present several examples of short plots and situations generated by the system from the dynamic interaction of artificial actors.

We describe the development of a virtual art gallery based on a game engine's technology. In the specific case of painting, the development of a virtual gallery comprises both the design of the spatial layout and the editing of high-quality textures from original images. Besides recreating the original gallery atmosphere, virtual painting galleries enable new aesthetic experiences, such as the construction of 3D installations. This ability to interact with high-resolution images should be furthered in the near future, as the next generation of game engines will support larger texture files and sophisticated mapping effects.

Content based indexing and retrieval of images and video requires a proper semantic description for image content. The paper discusses the mapping of high level, application specific features to the visual primitives that are accessible through image processing techniques. A major difficulty is that there are currently no established methodologies for describing the contents of an image in terms of semantic features, and we suggest that semiotic approaches could be adapted to the task of image description in limited areas of expertise. After reviewing current trends in the mapping of high level to low level features, we present preliminary results that suggest a possible strategy for mapping semiotic descriptions to image processing primitives.