The continued advancement in computer interfaces to support 3D tasks requires a better understanding of how users will interact with 3D user interfaces in a virtual workspace. This article presents two studies that investigated the effect of visual, auditory, and haptic sensory feedback modalities presented by a virtual button in a 3D environment on task performance (time on task and task errors) and user rating. Although we expected task performance to improve for conditions that combined two or three feedback modalities over a single modality, we instead found a significant emergent behavior that decreased performance in the trimodal condition. We found a significant increase in the number of presses when a user released the button before closing the virtual switch, suggesting that the combined visual, auditory, and haptic feedback led participants to prematurely believe they actuated a button. This suggests that in the design of virtual buttons, considering the effect of each feedback modality independently is not sufficient to predict performance, and unexpected effects may emerge when feedback modalities are combined.

The simple action of pressing a button is a multimodal interaction with an interesting depth of complexity. As the development of computer interfaces supporting 3D tasks progresses, there is a need to understand how users will interact with virtual buttons that generate multimodal feedback. Using a phone number dialing task on a virtual keypad, this study examined the effects of visual, auditory, and haptic feedback combinations on task performance and on the motion of individual button presses. The results suggest that the resistance of haptic feedback alone was not enough to prevent participants from pressing the button farther than necessary. Reinforcing haptic feedback with visual or auditory feedback shortened the depth of the presses significantly. However, the shallower presses that occurred with trimodal feedback may have led participants to release some buttons too early, which may explain an unexpected increase in mistakes where the participant missed digits from the phone number.

We describe a prototype software system for investigating novel human-computer interaction techniques for 3-D geospatial data. This system, M4-Geo (Multi-Modal Mesh Manipulation of Geospatial data), aims to provide a more intuitive interface for directly manipulating 3-D surface data, such as digital terrain models (DTM). The M4-Geo system takes place within a 3-D environment and uses a Phantom haptic force feedback device to enhance 3-D computer graphics with touch-based interactions. The Phantom uses a 3-D force feedback stylus, which acts as a virtual "finger tip" that allows the user to feel the shape (morphology) of the terrain's surface in great detail. In addition, it acts as a touch sensitive tool for different GIS tasks, such as digitizing (draping) of lines and polygons directly onto a 3-D surface and directly deforming surfaces (by pushing or pulling the stylus in or out). The user may adjust the properties of the surface deformation (e.g., soft or hard) locally by painting it with a special "material color."

We describe M4, the multi-modal mesh manipulation system, which aims to provide a more intuitive desktop interface for freeform manipulation of 3D meshes. The system combines interactive 3D graphics with haptic force feedback and provide several virtual tools for the manipulation of 3D objects represented by irregular triangle meshes. The current functionality includes mesh painting with pressure dependent brush size and paint preview, mesh cutting via drawing a poly-line on the model and two types of mesh deformations. We use two phantoms, either in a co-located haptic/3D-stereo setup or as a fish tank VR setup with a 3D flat panel. In our system, the second hand assists the manipulation of the object, either by ";holding"; the mesh or by affecting the manipulation directly. While the connection of 3D artists and designers to such a direct interaction system may be obvious, we are also investigating its potential benefits for landscape architects and other users of spatial geoscience data. Feedback from an upcoming user study will evaluate the benefits of this system and its tools for these different user groups.