Sketching is one of the oldest techniques humans use to express themselves. We sketch to visualize concepts, externalize memory, and communicate ideas. However, we barely use sketching to interact with computers. Given how naturally sketching comes to humans, we believe untapped potential exists in being able to simply draw commands onto a user interface. In this paper, we present results of an elicitation study about expressing common operations in spreadsheets through sketching. Spreadsheets are an interesting class of applications because they are widely used, support complex data and operations, and are available on touch-enabled devices. Our results show that despite considerable variation in syntactic details, participants gravitate towards recurring patterns (e.g., enclosures and arrows, examples and cross-references, and temporal sequences of strokes). The sketch patterns we identified can be a first step towards developing interpreters of sketched commands, and thus enable new means of interacting with spreadsheets and other applications.

This study describes an AI model by leveraging advanced Convolutional Neural Networks (CNNs) to recognize affective states in real-world sports settings, particularly tennis matches. In contrast to prior studies that primarily utilized data acquired from actors and rudimentary statistical methods, the present research emphasizes the analysis of bodily expressions in real-life contexts, aiming for a more naturalistic representation of human emotions. Our CNN-based models demonstrate an accuracy rate of up to 68.9%, outperforming or matching human observers in many instances. Intriguingly, both the machine learning models and human observers exhibited a shared propensity to more effectively identify negative affective states, which may be attributed to the more intense and straightforward expression of these states. These results not only advance the state of the art in affective state recognition but also pave the way for broader applications, including in healthcare and automotive safety sectors, thereby constituting a significant advancement in the development of sophisticated and universally applicable emotional recognition systems.

Users must pay a fee depending on resource consumption when using smart contracts on the Ethereum blockchain. As even the most basic operations cost several dollars under moderate network load, developers may actively reduce user-paid fees by optimizing the smart contract resource consumption (’gas costs’). Previous works suggested patterns and tools supporting developers in gas cost optimization, but up to now a comprehensive analysis of their real-world impact is missing. Another gap is the maintenance and evolution support for smart contracts leveraging the publicly available usage data. We propose high-level gas cost profiles and review which profiles are considered in the existing literature. Additionally, we sampled around 68,000 smart contract interactions from three years, analyzed them using the gas cost profiles, and compare the findings to the current focus in literature. In our data set, external code, storage, and the transaction base fee are first-level cost drivers in terms of absolute gas usage, but contract deployment becomes also costly when considering the average gas usage per transaction. Our analysis also shows that plenty of previous work focused cost categories barely influencing resource consumption.

With the increased availability of solutions using Artificial Intelligence and Machine Learning, more and more business processes are based on technical components delivering probabilistic results. A prominent examples are applications from the Internet of Things that heavily rely on sensor information and data stream processing. Another trend that is gaining more traction is the use of No- and Low-Code-Platforms to create applications. Such approaches focus on defining the business logic via business process modeling and automatically create a corresponding executable application. We argue that using components based on Artificial Intelligence and Machine Learning in such applications requires to handle uncertainty resulting from probabilistic results accordingly. This means to introduce, e.g., fallback mechanisms if results delivered from composing using Artificial Intelligence err into modeled business processes. In this position paper, we discuss scenarios, arising problems, and potential solutions.

Augmented reality changes the way we perceive reality and how we interact with computers. However, we argue that to create augmented reality solutions, we need to rethink the way we develop software. In this paper, we review the state of the art in software engineering for augmented reality applications, derive open questions, and define a research agenda. For this purpose, we consider different engineering phases and evaluate conventional techniques regarding their applicability for AR development. In requirements engineering, we found the integration of AR experts and the associated collaboration between actors to be of key aspect in the development process. Additionally, requirements about the physical world must be considered, which in turn has a huge impact on UI design. The relevance of the physical environment is not yet sufficiently addressed in applicable techniques, which also applies to current implementation frameworks and tools, complicating the AR development process. When evaluating AR software iterations, we found interaction testing and test automation to have great potential, although they have not yet been sufficiently researched. Our paper contributes to AR research by revealing current core challenges within the AR development process and formulating explicit research questions that should be considered by future research.

Abstract To gain a common understanding of an application’s layouts, dialogs and interaction flows, development teams often sketch user interface (UI). Nowadays, they must also define multi-touch gestures, but tools for sketching UIs often lack support for custom gestures and typically just integrate a basic predefined gesture set, which might not suffice to specifically tailor the interaction to the desired use cases. Furthermore, sketching can be enhanced with digital means, but it remains unclear whether digital sketching is actually beneficial when designing gesture-based applications. We extended the AugIR, a digital sketching environment, with GestureCards, a hybrid gesture notation, to allow software engineers to define custom gestures when sketching UIs. We evaluated our approach in a user study contrasting digital and analog sketching of gesture-based UIs.

Describing gestures in detail has various advantages for project teams: communication is simplified, interaction concepts are documented, and technical decisions are supported. Common gesture notations focus on textual or graphical elements only, but we argue that hybrid approaches have various advantages, especially because some gesture traits are easier to describe with text and others with arrows or icons. We present GestureCards, a hybrid gesture notation mixing graphical and textual elements we developed to describe multi-touch gestures. To evaluate our approach, we compared how users perceive and are affected by different notations. First, we compared GestureCards with a textual notation and observed advantages in terms of speed, correctness, and confidence. Second, we asked participants to compare GestureCards, a textual, and a graphical notation and rate them. The results indicate that the participants' perception depends on the gesture, but GestureCards received consistently good ratings. Third, we monitored several participants working with GestureCards solving practical development tasks for gesture-based applications and they felt well supported by GestureCards.

Observing semantic dependencies in large and heterogeneous networks is a critical task, since it is quite difficult to find the actual source of a malfunction in the case of an error. Dependencies might exist between many network nodes and among multiple hops in paths. If those dependency structures are unknown, debugging errors gets quite difficult. Since CPS and other large networks change at runtime and consists of custom software and hardware, as well as components off-the-shelf, it is necessary to be able to not only include own components in approaches to detect dependencies between nodes. In this paper we present an extension to the Information Flow Monitor approach. Our goal is that this approach should be able to handle unalterable blackbox nodes. This is quite challenging, since the IFM originally requires each network node to be compliant with the IFM protocol.

Cyber-Physical Systems (CPS) are interconnected systems, that adapt to their environment. They are quite challenging to engineer and to test, because the its interconnected and networked structures changes at runtime. Dependencies and influencing between nodes in the network might be difficult to test, because there are many hidden impacts on not directly connected nodes. In this paper, we present our experimentation environment which we use to simulate different CPSs. The focus of the development of this environment is to be able to rapidly generate, deploy and change software and network connections within the CPS and to observe resulting impacts on the network.

Program comprehension is an important skill for programmers - extending and debugging existing source code is part of the daily routine. Syntax highlighting is one of the most common tools used to support developers in understanding algorithms. However, most research on code highlighting is more than 20 years old, when programmers used a completely different tool chain. Newer results on the effect of syntax highlighting as used in modern Integrated Development Environments (IDEs) are inconclusive.

Developing CPS means uniting multiple engineering disciplines: mechanical, electrical and software engineering. Each of them provides a set of models, guidelines and processes crucial for a projects success. Consequently, CPS development is heavily influenced by processes known in either engineering discipline. There is currently no coherent software engineering approach to develop CPS, making it very hard to align the aforementioned domain experts in a controlled and repeatable process. This paper takes a step towards filling this gap by providing the insights during the development of a CPS in the form of an experience report. Based on our findings we validated the use of the Double TwinPeaks model and contribute to the field of software engineering for CPS the Matrix of Granularity to assess the components of a CPS in terms of customizability, risk and constraints.

Cyber-Physical Systems are distributed, heterogeneous, decentralized and loosely coupled networks in which individual systems measure physical processes, exchange information, and influence processes. Sensors measure these physical processes, while aggregators process them and actuators perform resulting actions. Decisions are often based on sensor data collected by other systems. Furthermore, the aggregators also interchange information and use them to derive own decisions. Decisions must be comprehensible. However, this is only the case if all data dependencies are known. Due to the size of these networks, their loose coupling and their dynamic behavior, decisions made by a system are not always easy to understand. If an error occurs in the system, the error source must be identified. It must be known on which data a decision was based. However, since the decision can be based on information from other nodes, the search for the error source is not a trivial task. Keep in mind, that dependent nodes can have dependencies themselves as well. We present the Information Flow Monitor (IFM) that collects information about semantic data dependencies in dynamic networks. The collected dependency information is provided at a central network location. Subsequently, semantic dependencies between information can be visualized.

Cyber-physical systems are context-aware networked systems containing sensors, aggregators and actuators. Raw sensor data and aggregated information are spread among the network and processed in multiple nodes to result in an action, which is possibly executed at a different physical location in the network. Due to the flexible topology and the emergent features of CPS, decisions within the network that trigger actions are not always trivial to understand. These decisions are based on raw sensor data which are not clearly visible at the location of their execution. This can be problematic if decisions have to be justifiable. It becomes even more difficult if decisions has been flawed, and it is not ascertainable why they were made. In order to determine the reasons for incorrect decisions in the network, the dependencies and input values of a decision must be known. Without this information, debugging is quite difficult. In this paper, we present the Information Flow Monitor, which can capture and visualize dependencies between nodes and information in CPS.

Connecting digital information systems with real world objects and processes is the core of the digital transformation. New business models and opportunities thrive on the various options that the resulting information offers. Cyber-Physical Systems (CPS) are the most prominent incarnation and – in contrast to various other aspects of the digital transformation – really new: sensors and actors allow information systems to monitor the real world and will profoundly change most markets and business domains. However, developing CPS involves different specialists and various technical challenges. Software and hardware engineers, network specialists, and data scientists have to work hand in hand and combine their specialties to incorporate the different perspectives into one team. We present EngCPS, an engineering approach to develop CPS that enhances classic software engineering methods with CPS-specific extensions.

Testing is a crucial activity to maintain high quality standards and achieve long-lasting economic success. However, integrating voice control into apps prevents the use of test automation tools. Testing speech-based interaction is still manual work: repetitive, time-consuming, and error-prone. Consequently, a need for specialized software engineering methods exists. We present an approach to test automation for speech-based applications by integrating an audio injection test interface into the Android operating system. Our approach enables developers to integrate voice control into automatically executed user acceptance tests using the Calabash test automation framework.

Touch- and gesture-based interfaces are common in applications for mobile devices. By evolving into mass market products, smartphones and tablets created an increased need for specialized software engineering methods. To ensure high quality applications, constant and efficient testing is crucial in software development. However, testing mobile applications is still cumbersome, time-consuming and error-prone. One reason is the devices' focus on touch-based interaction - gestures cannot be easily incorporated into automated application tests. We present an extension to the popular Calabash testing framework solving this problem by allowing to describe gestures with a formal language in tests scripts.