Silent speech input offers a fast alternative to other text entry methods in VR, such as virtual keyboard. However, in Chinese, the lack of tone creates ambiguity, as similar lip movements can convey different meanings depending on the tone. We proposed a tone-assisted silent speech input method that improves accuracy. We first explored five tone-input methods in VR, demonstrating that tone input can be synchronized with silent speech. Next, we implemented the most preferred tone-input method and designed a corresponding multimodal lip-reading model, ToneLipNet. ToneLipNet showed significantly higher accuracy than the baseline model without tone. Then, we conducted a five-day study to investigate the long-term potential of our input method. Participants quickly adapted, achieving high input speed and good accuracy. Finally, we conducted a user study to investigate how users coordinate silent speech input and tone input in VR.

Monte Carlo rendering often faces a dilemma, namely, whether to choose an unbiased estimator or a biased one. Although different integrators have been developed to address various scenarios, no single method can effectively manage all situations. Thus, finding a good approach to combine different integrators has always been a topic that warrants exploration. This work proposes DSCombiner, a new shrinkage estimator that flexibly combines unbiased and biased estimators (typically generated by different integrators) in image space into a single estimating procedure, strategically utilizing the strengths of different integrators while minimizing their weaknesses. DSCombiner overcomes the limitation of single shrinkage combiners by introducing a two-step shrinkage towards a noise-free radiance prior. We derive optimal shrinkage factors for the two steps within a hierarchical Bayesian framework, and provide a deep learning-based method to improve the results. Comprehensive qualitative and quantitative validations across diverse scenes demonstrate visible improvements in image quality, as compared with previous image-space and path-space combiners.

Sketching is a common way to depict design ideas freely. Augmented reality technology extends the human–computer interaction space from the screen to the real world, providing a new application space for 3D sketching. However, due to the difficulty of depth perception, it is difficult for sketching strokes to maintain good planar and spatial consistency, making drawing 3D sketches in AR challenging. In this article, we propose FMASketch—a method of drawing mid-air sketches similar to building blocks. The auxiliary surfaces can ensure that the strokes drawn by users can always stay on the same plane as drawing in 2D space. Gestures are used to stitch together multiple drawing planes, and the strokes on the planes form a 3D sketch, which is very similar to how 3D modeling is done on the desktop by drawing different views and assembling them. Experiments have verified the effectiveness of the auxiliary surface generation method and gesture operations in this article. The mid-air sketching method is easy to learn and provides a new idea for rapid modeling in AR scenarios.

Handwritten Mathematical Expression Recognition (HMER) is an important task in pattern recognition. It is a challenging task due to symbols resembling each other in appearance("z/2", "B/β") and the complex mathematical syntax. The encoder-decoder architecture has been widely used in recent HMER methods. Several works introduce HMER-related tasks that enhance the performance of HMER. We propose the General Category Recognition Task (GCRT) and design the General Category Network(GCN) to perform HMER and GCRT in parallel. GCRT alleviates the symbol confusion problem and also helps the model generate results conforming to mathematical syntax. Compared with SOTA methods, the experimental results show that Expression Recognition Rates(ExpRate) of our method are increased by 1.12%, 2.45% and 1.84% on CROHME 2014, 2016 and 2019 test set respectively. In addition, current works explore the effect of a single auxiliary task on HMER. We investigated the effect of multiple tasks on HMER. We performed many experiments and drew multiple valuable conclusions.We will publish all codes in the future.

3D sketching in virtual reality (VR) provides an immersive drawing experience for designs. However, due to the lack of depth perception cues in VR, scaffolding surfaces that constrain strokes to 2D are usually used as visual guides to reduce the difficulty of drawing accurate strokes. When the dominant hand is occupied by the pen tool, the efficiency of scaffolding-based sketching can be improved by using gesture input to reduce the idleness of the non-dominant hand. This paper presents GestureSurface, a bi-manual interface that uses non-dominant hand performing gestures to operate scaffolding and the other hand drawing with controller. We designed a set of non-dominant gestures to create and manipulate scaffolding surfaces, which are assembled by automatic combination based on five predefined primitive surfaces. We evaluated GestureSurface through a 20-person user study and found that the method of scaffolding-based sketching using non-dominant hand has the advantages of high efficiency and low fatigue.

Neurobiologists recently found the brain can use sudden emerged channels to process information. Based on this finding, we put forward a question whether we can build a computation model that is able to integrate a sudden emerged new type of perceptual channel into itself in an online way. If such a computation model can be established, it will introduce a channel-free property to the computation model and meanwhile deepen our understanding about the extendibility of the brain. In this article, a biologically inspired neural network named artificial evolution (AE) network is proposed to handle the problem. When a new perceptual channel emerges, the neurons in the network can grow new connections to connect the emerged channel according to the Hebb rule. In this article, we design a sensory channel expansion experiment to test the AE network. The experimental results demonstrate that the AE network can handle the sudden emerged perceptual channels effectively.

Stroke grouping in sketch recognition is both difficult and time-consuming. Our preliminary experiment indicates that, when people drawing flowcharts, their gaze focused on non-arrow areas, which providing a spatial cue for stroke grouping. Therefore, we present a novel stroke grouping method aided by gaze information. Based on gaze data that is collected simultaneously during natural drawing process, we generate hotspot areas serving as the position reference of semantic symbols. Strokes are first roughly grouped by the hotspot areas, so as to efficiently decrease the searching space. Experiment on a dataset of 54 flowcharts shows that time efficiency of stroke grouping can be greatly improved in our method and there is much potential for introducing eye-gaze data in sketch recognition.

Due to the more and more intense competition in software market, usability is gaining more and more attention. For software engineering students, learning and mastering human-computer interaction knowledge can help them design software products with higher usability. In this paper, we describe a preliminary experience of teaching HCI to undergraduate software engineering students in Nanjing University (NJU), including curriculum design, course implementation and improvements based on some feedback from students. We believe that this information is useful at how to teach HCI in a practical way, and we hope it can serve as guidance for other teachers struggling with similar course design issues.