• A new concept and framework of the Resilient Cognitive Twin (RCT). • A new generative approach with three enabling mechanisms. • An evaluation case study of a city flooding resilience system. Improving the resilience of an engineering system can be addressed from both a physical engineered system (PES) and its virtual replica--digital twin (DT) system. On the one hand, a DT can enhance the resilience of its PES by enabling real-time monitoring/proactive intervention and predictive maintenance. On the other hand, its deep integration with PES also increases the complexity of the overall system--both virtual and physical. This interdependence can transform single-sided risks into overall system-wide vulnerabilities, potentially undermining--even compromising--the resilience of the PES itself. To address this challenge, this paper proposes a new concept and framework of the Resilient Cognitive Twin (RCT), that supports generative reconfiguration and bidirectional feedback across the human–cyber–physical continuum. The framework is supported by a generative approach with three new enabling mechanisms: (1) a requirement decomposition and recompositing mechanism with edge-cloud-centre collaboration for forming a unified yet loosely coupled and low-level foundation of RCT; (2) a dynamic evolution mechanism enabling right-time co-evolution of data-model-service relationships for a changing environment and stakeholder needs; and (3) a generative cognitive mechanism for situational awareness and decision-making, responding to changing situations with proper and resilient services and their coordination/scheduling. The proposed framework and enabling approach are validated through an urban flood resilient DT system, demonstrating its capability to enhance resilience in complex, distributed environments, paving the way for Human-Cyber-Physical Systems and future industrial and societal applications.

Abstract To understand how individual user features affect one’s emotional changes in response to stimuli or changes in a situated external environment, it is necessary to measure how user features affect one’s behavioral changes (patterns of changes) in the use of keyboard, mouse, and touchscreen devices (KMT dynamics), as the measurable behavioral changes are related to both user features and the user’s emotional states. Our central research questions are: (1) which user features have a significant impact on behavior such as KMT dynamics? (2) how do these user features impact behavior from qualitative and quantitative perspectives? and (3) what is an effective framework to support personalized emotion recognition? To answer these questions, we conducted a questionnaire survey with 107 participants to investigate how, and to what extent, user profile features can influence keyboard, mouse, and touchscreen usage in dynamics terms (KMT dynamics). We then performed data analysis to explore the relationships between an initial set of user feature candidates and an initial set of KMT dynamics feature candidates (12 identified from a literature review). Based on the identified relationships, we identified seven important user features and seven important KMT dynamics features for further KMT-based emotion recognition studies. The seven key user features are: gender, KMT proficiency, language proficiency, hand using habit, software proficiency, task urgency, and emotion, while the seven important KMT dynamics features are: typing speed, typing error rate, pressure value of key/strength of pressing touchscreen, mouse (finger or pencil) movement speed, clicking speed, clicking strength, and keystroke speed. Finally, we proposed a new emotion recognition framework incorporating the identified key user and KMT dynamics features as a guiding model for further research in this field.

As communications are increasingly taking place virtually, the ability to present well online is becoming an indispensable skill. Online speakers are facing unique challenges in engaging with remote audiences. However, there has been a lack of evidence-based analytical systems for people to comprehensively evaluate online speeches and further discover possibilities for improvement. This paper introduces SpeechMirror, a visual analytics system facilitating reflection on a speech based on insights from a collection of online speeches. The system estimates the impact of different speech techniques on effectiveness and applies them to a speech to give users awareness of the performance of speech techniques. A similarity recommendation approach based on speech factors or script content supports guided exploration to expand knowledge of presentation evidence and accelerate the discovery of speech delivery possibilities. SpeechMirror provides intuitive visualizations and interactions for users to understand speech factors. Among them, SpeechTwin, a novel multimodal visual summary of speech, supports rapid understanding of critical speech factors and comparison of different speech samples, and SpeechPlayer augments the speech video by integrating visualization of the speaker's body language with interaction, for focused analysis. The system utilizes visualizations suited to the distinct nature of different speech factors for user comprehension. The proposed system and visualization techniques were evaluated with domain experts and amateurs, demonstrating usability for users with low visualization literacy and its efficacy in assisting users to develop insights for potential improvement.

Sketching is an intuitive and simple way to depict sciences with various object form and appearance characteristics. In the past few years, widely available touchscreen devices have increasingly made sketch-based human-AI co-creation applications popular. One key issue of sketch-oriented interaction is to prepare input sketches efficiently by non-professionals because it is usually difficult and time-consuming to draw an ideal sketch with appropriate outlines and rich details, especially for novice users with no sketching skills. Thus, sketching brings great obstacles for sketch applications in daily life. On the other hand, hand-drawn sketches are scarce and hard to collect. Given the fact that there are several large-scale sketch datasets providing sketch data resources, but they usually have a limited number of objects and categories in sketch, and do not support users to collect new sketch materials according to their personal preferences. In addition, few sketch-related applications support the reuse of existing sketch elements. Thus, knowing how to extract sketches from existing drawings and effectively re-use them in interactive scene sketch composition will provide an elegant way for sketch-based image retrieval (SBIR) applications, which are widely used in various touch screen devices. In this study, we first conduct a study on current SBIR to better understand the main requirements and challenges in sketch-oriented applications. Then we develop the SketchMaker as an interactive sketch extraction and composition system to help users generate scene sketches via reusing object sketches in existing scene sketches with minimal manual intervention. Moreover, we demonstrate how SBIR improves from composited scene sketches to verify the performance of our interactive sketch processing system. We also include a sketch-based video localization task as an alternative application of our sketch composition scheme. Our pilot study shows that our system is effective and efficient, and provides a way to promote practical applications of sketches.

Peking Opera as a branch of Chinese traditional cultures and arts has a very distinct colourful facial make-up for all actors in the stage performance. Such make-up is stylised in nonverbal symbolic semantics which all combined together to form the painted faces to describe and symbolise the background, the characteristic and the emotional status of specific roles. A study of Peking Opera Painted Faces (POPF) was taken as an example to see how information and meanings can be effectively expressed through the change of facial expressions based on the facial motion within natural and emotional aspects. The study found that POPF provides exaggerated features of facial motion through images, and the symbolic semantics of POPF provides a high-level expression of human facial information. The study has presented and proved a creative structure of information analysis and expression based on POPF to improve the understanding of human facial motion and emotion.

Three-dimensional (3D) body scanning technology opens opportunities for virtual try-on and automatic made-to-measure apparel design. This paper proposes a new feature-based parametric method for modeling human body shape from scanned point clouds of a 3D body scanner [TC]2. The human body model consists of two layers: the skeleton and the cross sections of each body part. First, a simple skeleton model from the body scanner [TC]2system has been improved by adding and adjusting the position of joints in order to better address some fit issues related to body shape changes such as spinal bending. Second, an automatic approach to extracting semantic features for cross sections has been developed based on the body hierarchy. For each cross section, it is described by a set of key points which can be fit with a closed cardinal spline. According to the point distribution in point clouds, an extraction method of key points on cross sections has been studied and developed. Third, this paper presents an interpolation approach to fitting the key points on a cross section to a cardinal spline, in which different tension parameters are tested and optimized to represent simple deformations of body shape. Finally, a connection approach of body parts is proposed by sharing a boundary curve. The proposed method has been tested with the developed virtual human model (VHM) system which is robust and easier to use. The model can also be imported in a CAD environment for other applications.

With the rapid growth of video resources, techniques for efficient organization of video clips are becoming appealing in the multimedia domain. In this article, a sketch-based approach is proposed to intuitively organize video clips by: (1) enhancing their narrations using sketch annotations and (2) structurizing the organization process by gesture-based free-form sketching on touch devices. There are two main contributions of this work. The first is a sketch graph, a novel representation for the narrative structure of video clips to facilitate content organization. The second is a method to perform context-aware sketch recommendation scalable to large video collections, enabling common users to easily organize sketch annotations. A prototype system integrating the proposed approach was evaluated on the basis of five different aspects concerning its performance and usability. Two sketch searching experiments showed that the proposed context-aware sketch recommendation outperforms, in terms of accuracy and scalability, two state-of-the-art sketch searching methods. Moreover, a user study showed that the sketch graph is consistently preferred over traditional representations such as keywords and keyframes. The second user study showed that the proposed approach is applicable in those scenarios where the video annotator and organizer were the same person. The third user study showed that, for video content organization, using sketch graph users took on average 1/3 less time than using a mass-market tool Movie Maker and took on average 1/4 less time than using a state-of-the-art sketch alternative. These results demonstrated that the proposed sketch graph approach is a promising video organization tool.

The creative process is often inspired by natural and cultural worlds and further supported by the immediacy of the presentation, the ease of manipulation, and the possibilities of tracing the development of ideas or revisiting them in order to explore other possible routes. The use of design sketches or drawings to represent design ideas is crucial to the creative process. Sketch-based design technology has the potential to create a seamless transition from conceptual design to a manufactured product. This talk will present an overview on 2D sketch-based interface and modelling pipelines: stroke filtering and processing, 2D tidy-up, visual perception, 3D understanding and modelling. The talk will also review 3D sketch-based interfaces for modelling and interaction: 3D sketching with VR perceptions, 3D sketch with gesture tracking and mixed design environments. The talk will finally discuss the challenges ahead and implications in conceptual design.

This research explored a new direction of improving collaborative design by performance measurement. More specifically, a novel 3-dimensional performance measurement model is developed and the purpose of this model is to help project managers improve team collaboration by indicating strengths and weaknesses of team members during the project development process. Considering the complexity of collaborative design work, a multiple criteria model is proposed to reflect the design dynamics, which highlights five performance indicators: efficiency, effectiveness, collaboration, management skills and innovation. These five indicators are mostly influenced by role-based performance measurement criteria (the second dimension). Design and development process (time) is also considered (the third dimension). This 3D model allows all involved design participants to measure work performance at any time during the product development process. In order to develop this model, the role-based task analysis and industrial survey methods were utilized. Three groups of role-based product design and development performance measurement criteria were identified for measuring design performance of the top managers, middle managers and individual designers in a project team. A 3-dimensional performance measurement method was proposed to calculate final performance scores from a performance measurement matrix. The proposed model was evaluated as a tool which can support project managers to reduce potential design and collaboration risks and increase confidence in decision-making process. The model has been discussed on implementing in a web-based application for measuring design performance throughout the product design and development process.

Abstract This paper presents an online personalised non‐photorealistic rendering (NPR) technique for 3D models generated from interactively sketched input. This technique has been integrated into a sketch‐based modelling system. It lets users interact with computers by drawing naturally, without specifying the number, order, or direction of strokes. After sketches are interpreted as 3D objects, they can be rendered with personalised drawing styles so that the reconstructed 3D model can be presented in a sketchy style similar in appearance to what have been drawn for the 3D model. This technique captures the user's drawing style without using template or prior knowledge of the sketching style. The personalised rendering style can be applied to both visible and initially invisible geometry. The rendering strokes are intelligently selected from the input sketches and mapped to edges of the 3D object. In addition, non‐geometric information such as surface textures can be added to the recognised object in different sketching modes. This will integrate sketch‐based incremental 3D modelling and NPR into conceptual design.