Advances in emerging technologies, such as on-body mechanical actuators and electrical muscle stimulation, have allowed computers to take control over our bodies. This presents opportunities as well as challenges, raising fundamental questions about agency and the role of our bodies when interacting with technology. To advance this research field as a whole, we brought together expert perspectives in a week-long seminar to articulate the grand challenges that should be tackled when it comes to the design of computers’ control over our bodies. These grand challenges span technical, design, user, and ethical aspects. By articulating these grand challenges, we aim to begin initiating a research agenda that positions bodily control not only as a technical feature but as a central, experiential, and ethical concern for future human–computer interaction endeavors.

Recent research across mathematical problem solving, proof assistant programming and multimodal jailbreaking documents a striking finding: when (multimodal) language model tackle a suite of tasks with multiple attempts per task – succeeding if any attempt is correct – then the negative log of the average success rate scales a power law in the number of attempts. In this work, we identify an apparent puzzle: a simple mathematical calculation predicts that on each problem, the failure rate should fall exponentially with the number of attempts. We confirm this prediction empirically, raising a question: from where does aggregate polynomial scaling emerge? We then answer this question by demonstrating per-problem exponential scaling can be made consistent with aggregate polynomial scaling if the distribution of single-attempt success probabilities is heavy tailed such that a small fraction of tasks with extremely low success probabilities collectively warp the aggregate success trend into a power law - even as each problem scales exponentially on its own. We further demonstrate that this distributional perspective explains previously observed deviations from power law scaling, and provides a simple method for forecasting the power law exponent with an order of magnitude lower relative error, or equivalently, ${\sim}2-4$ orders of magnitude less inference compute. Overall, our work contributes to a better understanding of how neural language model performance improves with scaling inference compute and the development of scaling-predictable evaluations of (multimodal) language models.

We introduce Best-of-N (BoN) Jailbreaking, a simple black-box algorithm that jailbreaks frontier AI systems across modalities. BoN Jailbreaking works by repeatedly sampling variations of a prompt with a combination of augmentations---such as random shuffling or capitalization for textual prompts---until a harmful response is elicited. We find that BoN Jailbreaking achieves high attack success rates (ASRs) on closed-source language models, such as 89% on GPT-4o and 78% on Claude 3.5 Sonnet when sampling 10,000 augmented prompts. Further, it is similarly effective at circumventing state-of-the-art open-source defenses like circuit breakers and reasoning models like o1. BoN also seamlessly extends to other modalities: it jailbreaks vision language models (VLMs) such as GPT-4o and audio language models (ALMs) like Gemini 1.5 Pro, using modality-specific augmentations. BoN reliably improves when we sample more augmented prompts. Across all modalities, ASR, as a function of the number of samples (N), empirically follows power-law-like behavior for many orders of magnitude. BoN Jailbreaking can also be composed with other black-box algorithms for even more effective attacks---combining BoN with an optimized prefix attack achieves up to a 35% increase in ASR. Overall, our work indicates that, despite their capability, language models are sensitive to seemingly innocuous changes to inputs, which attackers can exploit across modalities.

We are at a tipping point for digital communication: moving beyond 'ways of seeing' to include 'ways of feeling'.Much as optical technologies transformed sight and the visual (from the telescope and microscope to Google Glass), the rapid expansion in digital touch technologies is set to reconfigure touch and the tactile in significant ways.Advances in haptics, virtual reality, and physiological sensing provide new sensory ways of communicating, as well as new ways to capture the quality of touch.These state-of-the-art digital touch technologies promise to supplement, heighten, extend and reconfigure how people communicate.They are reshaping what and who can be touched, as well as when and how they can be touched, changing existing forms of communication and giving rise to changes in co-located and remote communication between humans, and between humans and robots.These developments sit alongside social discourses of concern and loss, with the digital being associated with the removal of touch from the material sensory landscape (Jewitt et al., 2020).Yet, in our current climate of social distancing and disengaging with touch in our everyday interactions, the promise of the digital becomes increasingly appealing and significant, in re-enabling touch possibilities.As the global population emerges from and comes to terms with their experiences of the COVID-19 pandemic, these questions become all the more significant.The breadth and interdisciplinary interest in this growing field-across designers, artists, computer scientists, engineers, psychologists and social scientists, with interests in robotics and touch, affective computing, wearables, and digital installations -brings attention to the growing need to engage with 'social' aspects of digital touch, moving beyond technologies and physiological foci to engage with touch practices (Jewitt et al., 2021).We argue for the need to think about touch beyond the physiological act of sensing and perceiving to a more rich and nuanced interpretation of touch that takes account of its emotional and psychological significance, the social, cultural and historical evolution of touch practices in human communication, and approaches to touch of the "lived, social body as a site of meaning-making, where the skin acts as both a boundary between and a point of connection with others" (Karpashevich et al.).Given this landscape, this special issue addresses timely and important questions around the need to think about touch in different ways.This provides a new direction in the field that seeks to address the complexity of human touch and interrogates the limitation of today's haptic devices.Papers in this special issue contribute to understanding this gap by engaging with socio-cultural issues around digital touch communication and identifying new ways to study touch, specifically aiming to explore the nuance and subtlety of touch and an expanded view of touch.

Despite the importance of touch in human–human relations, research in affective tactile practices is in its infancy, lacking in-depth understanding needed to inform the design of remote digital touch communication. This article reports two qualitative studies that explore tactile affective communication in specific social contexts, and the bi-directional creation, sending and interpretation of digital touch messages using a purpose-built research tool, the Tactile Emoticon. The system comprises a pair of remotely connected mitts, which enable users in different locations to communicate through tactile messages, by orchestrating duration and level of three haptic sensations: vibration, pressure and temperature. Qualitative analysis shows the nuanced ways in which 68 participants configured these elements to make meaning from touch messages they sent and received. It points to the affect and emotion of touch, its sensoriality and ambiguity, the significance of context, social norms and expectations of touch participants. Findings suggest key design considerations for digital touch communication, where the emphasis shifts from generating ‘recognizable touches’ to tools that allow people to shape their touches and establish common understanding about their meaning.

Drawing from a present need as well as a growing interest towards developing methods to engage participants in research remotely [1,2], this workshop focuses on identifying opportunities and challenges around designing with children from an embodiment perspective [3,4,5,6,10,11,12,13,15]. Themed around whole body interaction and digital technologies [3,7,8,9,10,11,14,15,16,17,18,20,21] the aim of this workshop is to gain methodological insights for the remote participation of children in design, and arrive at design insights addressing the question how we can design remote, inclusive whole body interactions for and with children while attending to child participants’ multimodal communication and bodily engagement [19,21] in a physically dislocated setting [1,2]. The workshop will involve discussions around data gathering from different angles, including relational, ethical, safety and practical perspectives, to create a comprehensive picture of what it entails to orchestrate embodied co-design workshops with children virtually. Outputs from the workshop will contribute to the IDC community by offering methodological insights from attendees’ own experience and presented case studies from researchers and practitioners from both non-academic and academic settings to support children's embodied engagement in remote workshop situations and beyond.

Haptic devices have the potential to enhance the learning experience by foregrounding embodied, sensory and multi-modal elements of learning topics. In this paper, we report on-going work investigating a game prototype with haptic feedback for seven year old children's engagement with geometrical concepts as part of an iterative design study. Our findings include a new game play mode adopted by the children, that empowers the use of haptic feedback in game play and has the potential to enable the enactment of shape properties in the game play process.

Museums increasingly use digital technology to enhance exhibition experiences for families, notably in relation to physically mediated installations for young children through natural user interfaces. Yet little is known about how families and children engage with such installations and the kinds of interactive experiences they engender in museum spaces. This paper addresses a pressing need for research to adopt an analytical focus on the body during such digitally mediated interactions in order to understand how bodily interaction contributes to meaning making in the museum context. It reports an observation study of families and children interacting with a whole-body interface (using Kinect) in the context of an installation in a museum exhibit on rare Chinese paintings. The study shows how the installation design engenders particular forms of bodily interaction, collaboration and meaning making. It also contributes design insights into whole-body interaction installations in museums and public spaces.

The methods of data collection that we choose determine the kinds of data that we have access to, and thus shape analyses. In the context of novel interfaces where different modes, available through the environment and context, mediate the interaction, understanding methodological approaches is critical. This paper examines alternative methods of data collection for exploring student's embodied interaction with novel technology in a learning context. Specifically it analyses non-facilitated interaction in a tangible learning environment, in conjunction with three different post activity interview approaches: semi-structured interviews; semi-structured interview with video prompted recall; and interviews using the technology itself. Findings suggest that the different interview approaches change the nature of information elicited, and that non-facilitated interaction offers clearer insight into interpretation, both in terms of the meaning that emerges through, and is, therefore, embodied in the interaction, and in terms of representation, directly informing design.

Tangible and multitouch technologies offer new opportunities for physically interacting with objects and digital representations, foregrounding the role of the body in interaction and learning. Developing effective methodologies for examining real time cognition and action, and the relationship between embodiment, interaction and learning is challenging. This paper draws on multimodality, with its emphasis on examining the use of multiple semiotic resources for meaning making, to examine the differential use of semiotic resources by pairs of students interacting with a tangible learning environment. Specifically the analysis details the role of body position, gaze, manipulation and speech in shaping interaction. The analysis illustrates the interaction between these modes and 'multimodal action flow', particularly in terms of pace, rhythm and interaction structure, and the implications of this for interaction and the meaning making process.

Students with intellectual disabilities tend to be reliant on other people's opinions and attitudes, and fear taking the initiative. Thus, they are reluctant to independently undertake activities required in exploratory learning contexts -- a pedagogical approach recommended by constructivist theories. This research aims to investigate how different aspects of tangibles, like physicality, multisensory and dynamic feedback, can better support more independent exploration for students with intellectual disabilities. Empirical studies were undertaken where forty-six students with intellectual disabilities used four different tangible systems for exploratory activities. Preliminary results provide indications for adequate design choices under the categories of action-effect mapping, types of representations, embedded meanings and conceptual metaphors, and actions.

Students with intellectual disabilities tend to be reliant on other people's opinions and attitudes, and fear taking initiatives. Thus, they are reluctant to independently undertake activities of exploratory learning -- a pedagogical approach recommended by constructivist theories. This research aims to investigate how different aspects of tangibles can better support more independent exploration for students with intellectual disabilities. In this paper we discuss three relevant themes that have emerged from ongoing analysis of empirical studies where children with intellectual disabilities experimented with four different tangible systems: the importance of both space and time dimensions of embodiment; the complexity of conveying concepts through audio representations; and the role of actions both as cognitive resources for thinking and expression.

The role of physical actions in learning has fostered enthusiasm for developing novel learning representations using emerging technologies such as tangibles. Indeed, as emerging devices blur the distinction between physical and graphical interfaces, it is important to build our understanding of how different forms of action and interaction will affect children's conceptual development. However, the mechanisms underpinning which actions support learning are often unclear. This paper offers a new perspective for examining the role of physical manipulation in learning by drawing a distinction between the actions generated by the hands and the resulting change in external representation. Arguments from existing research are revisited before using this distinction to identify implications for design.

Engagement is a frequently reported measure in evaluation studies of technology mediated learning environments. In the context of novel digital technologies, where the ‘novelty’ factor is inherent in interaction with emergent technologies, the concept of engagement is often reported at a general level of description of fun and enjoyment. Although this importantly indicates positive appraisal of the environment it does not provide any detail about how the participants are ‘engaged’. For example, what are they doing and thinking about, where is their attention focused – is it on a learning task, a tangential entertaining activity or even the technology itself? This paper offers a more detailed analysis of students’ foci of interaction, to provide insight into the different ways that children are both cognitively and physically engaged during a discovery-based learning experience. Three key foci of interaction within the learning space (learning concept, tangential activity, technology) were derived from video data analysis. A coding scheme for identifying these interaction foci was developed and applied to empirical data with a tangible learning environment. In depth analysis showed close relationships between the different foci of interaction and the learning process: engaging with a tangential activity in a exploratory interaction engendered cognitive engagement with domain related concepts; while engaging with technology to understand the basics of the system’s functioning could facilitate higher levels of conceptual abstraction. This article highlights the different interaction foci that students take in innovative, technology-enhanced learning environments, and provides an analytical approach, which informs and extends current evaluation approaches towards student engagement in novel digital environments.

The advent of new technologies is expanding the possibilities for a richer, multi-sensory interaction to support children with learning difficulties in schools. However, little research has yet investigated how such innovative interaction can enhance the learning experience of these children. Effectively informing research and design of innovative educational, technological resources for children with learning difficulties requires relevant field study. Findings from a qualitative classroom study indicate the potential of tangible technologies to enhance these children's experiences by providing a variety of modes of representation, and opportunities for collaboration, physical engagement and hands-on exploration. They also provide an effective foundation for investigating how tangible interaction can help to structure exploratory learning: a recommended but problematic approach for these children.

In tangible learning systems, the facility to promote physically active engagement highlights the need to understand how different designs impact on action and interaction, and the subsequent implications for learning. This paper draws on studies involving two tangible learning systems to analyse the effect of design choices on the kinds of (inter)actions engendered and how they create, shape and constrain different learning opportunities. Main findings suggest the need to promote and allow for different kinds of opportunities for conceptual reflection within the collective physical interaction; the importance of balancing collective representations and individual action-effect links; and the need to enhance appropriate awareness when dealing with several loci of attention.

Ubiquitous technologies offer new opportunities for digitally augmenting children's toys and play experiences. A key question is how augmented toy environments affect children's playful learning, and whether this differs from non-augmented play environments. This paper presents preliminary results of a user study we conducted to evaluate an augmented toy environment that we built --- the Augmented Knights Castle --- in terms of fun and storytelling, particularly when compared with an identical, non-augmented version. All sessions were observed, video-recorded and further feedback was elicited through small group interviews and questionnaires. Findings suggest ways in which digitally augmented play environments promote different kinds of activity from an equivalent non-augmented play environment.

In tangible learning environments the potential to exploit different physical-digital links increases representational power but also broadens the complexity of design. This paper presents studies that illustrate the effect of physical correspondence design choices on learners' interpretations, particularly regarding meaning making and conceptual mappings between objects and representations, and learners' ability to generalize. Preconceptions and associations with familiar real settings were found to have a significant level of interference in children's perception, interpretation and comprehension of the concepts.

Drawing on the 'representation' TUI framework [21], this paper reports a study that investigated the concept of 'representation location' and its effect on interaction and learning. A reacTIVision-based tangible interface was designed and developed to support children learning about the behaviour of light. Children aged eleven years worked with the environment in groups of three. Findings suggest that different representation locations lend themselves to different levels of abstraction and engender different forms and levels of activity, particularly with respect to speed of dynamics and differences in group awareness. Furthermore, the studies illustrated interaction effects according to different physical correspondence metaphors used, particularly with respect to combining familiar physical objects with digital--based table-top representation. The implications of these findings for learning are discussed.

Tangibles, in the form of physical artefacts embedded with sensor technologies, offer the opportunity to exploit and build on our everyday interaction and experience with the world, enabling new forms of engagement and access to tools for supporting learning. The implications for learning are considerable, potentially bringing about a radical change in the way we conceptualise learning and learning activities. However, we know little about the specific learning benefits, and currently lack an effective structure within which to establish them. Although several frameworks have been proposed for conceptualizing tangible environments, none highlight the central role that external representations have in tangible environments. This paper argues for the importance of placing primary emphasis on representation, and the role that this might play in mediating interaction and cognition in tangible environments. The representation-tangible relationship is outlined, together with their differential potentials for learning. Based on this the paper then proposes a conceptual framework for systematically investigating how different ways of linking digital information with physical artefacts influence interaction and cognition, to gain a clearer understanding of their role for learning.

Ubiquitous and mobile technologies provide opportunities for designing novel learning experiences that move out of the classroom. Information can be presented and interacted with in a variety of ways while exploring a physical environment. A key issue this raises is when, where, what and how much? Our research is concerned with the design, delivery and interaction of digital information when learning about ecology outdoors. We present a framework of the different forms of digital augmentation and the different processes by which they can be accessed. Using the framework, we designed an outdoors learning experience, aimed at encouraging students to carry out contextualized scientific enquiry and to reflect on their interactions. Pairs of 11-12 year olds explored a woodland and were presented at certain times with different forms of digital augmentation. Our study showed that this kind of exploration promoted interpretation and reflection at a number of levels of abstraction.

We present a new way of conceptualising tangibles for learning. This scheme adopts Heidegger's analysis of the ways a user can treat a tool: either as 'ready-to-hand' or 'present-at-hand'. It also proposes two types of activity a learner can engage in when using a tangible: either exploratory or expressive activity. Finally, two types of models that a user can explore are proposed: theoretical and practical models.Examples from the literature are described in terms of this framework and an example is given from our own work of an attempt to use this conceptualisation in design.

Tangibles, in the form of physical artefacts that are electronically augmented and enhanced to trigger various digital events to happen, have the potential for providing innovative ways for children to play and learn, through novel forms of interacting and discovering. They offer, in addition, the scope for bringing playfulness back into learning. To this end, we designed an adventure game, where pairs of children have to discover as much as they can about a virtual imaginary creature called the Snark, through collaboratively interacting with a suite of tangibles. Underlying the design of the tangibles is a variety of transforms, which the children have to understand and reflect upon in order to make the Snark come alive and show itself in a variety of morphological and synaesthesic forms. The paper also reports on the findings of a study of the Snark game and discusses what it means to be engrossed in playful learning.