Pen-based interaction interfaces are widely used in precision tasks such as computer-aided design (CAD) modeling. However, frequent mode switching in traditional interfaces often leads to reduced efficiency, while the unstable layout of dynamically adaptive interfaces may cause user discomfort. To address these issues, this paper proposes a novel adaptive pen-based interface, GestApt, which integrates operation recommendation and pen gesture technology to provide an efficient and intuitive interaction method. The core design of GestApt includes a gated recurrent unit (GRU)-based neural network for a predictive toolbar that dynamically forecasts the user’s next action and a convolutional neural network (CNN)-based gesture recognition module to assist with unpredicted operations. To evaluate the effectiveness of GestApt, we conducted a user study with 20 participants performing CAD modeling tasks, comparing their performance with a traditional interface. The results showed that GestApt significantly reduced task completion time while enhancing user experience. Additionally, the optimized toolbar design of GestApt notably reduced the workload on users’ non-dominant hands, helping to alleviate physical fatigue from prolonged interface use. This work demonstrates the advantages of GestApt in improving efficiency, optimizing user interaction, and reducing physical strain, offering a new solution for adaptive design in pen-based interfaces. This design approach not only promotes the development of more efficient and ergonomically friendly interfaces, but also provides a reference for the design of future intelligent CAD interfaces.

Visual storytelling is a new approach to creative expression based on verbal and figural creativity. The keys to visual storytelling are narrating and drawing over a period of time, which can be beneficial but also demanding on creativity for children. Informed by need-finding investigations, we developed StoryDrawer, a co-creative system that supports visual storytelling for children aged 6–10 years through collaborative drawing between children and artificial intelligence (AI). The system includes a context-based voice agent and two AI-driven collaborative strategies: the real-time transformation of children's telling into drawings, and the generation of abstract sketches with semantic similarity to existing story content. We conducted a 2 × 2 study with 64 children to evaluate the efficacy of StoryDrawer by varying the two strategies in four conditions. The results suggest that StoryDrawer provoked participants’ creative and elaborate ideas and contributed to their creative outcomes during an engaging visual storytelling experience.