Camera pose estimation is a fundamental task in visual localization but often relies on image retrieval or dense rendering, leading to high memory and computational cost. We propose a real-time retrieval-free camera pose estimation framework based on sparse cross-modal 2D–3D matching. Our approach learns descriptors for a compact point cloud using dense-render guidance and performs localization through projection-guided uncertainty-aware correspondence refinement, avoiding image database search and dense matching.

Virtual Reality (VR) has emerged as a promising solution to address the pressing concern of transferring know-how in the manufacturing industry. Making an immersive training experience often involves designing an instrumented replica of a tool whose use is to be learned through virtual training. The process of making a replica can alter its mass, making it different from that of the original tool. As far as we know, the influence of this difference on learning outcomes has never been evaluated. To investigate this subject, an immersive training experience was designed with pre and post-training phases under real conditions, dedicated to learning the use of a rotary tool. 80 participants took part in this study, split into three groups: a control group performing the virtual training using a replica with the same mass as the original tool ($\mathrm{m}=100\%$), a second group that used a replica with a lighter mass than the original tool ($\mathrm{m}= 50\%$) and a third group using a replica heavier than the original tool ($\mathrm{m}=150\%$). Despite variations in the mass of the replica used for training, this study revealed that the learning outcomes remained comparable across all groups, while also demonstrating significant enhancements in certain performance measures, including task completion time. Overall, these findings provide useful insights regarding the design of tool replicas for immersive training.

Weathering phenomena are ubiquitous in urban environments, where it is easy to observe severely degraded old buildings as a result of water penetration. Despite being an important part of any realistic city, this kind of phenomenon has received little attention from the Computer Graphics community compared to stains resulting from biological or flow effects on the building exteriors. In this paper, we present physically-inspired deep weathering effects, where the penetration of humidity (i.e., water particles) and its interaction with a building’s internal structural elements result in large, visible degradation effects. Our implementation is based on a particle-based propagation model for humidity propagation, coupled with a spring-based interaction simulation that allows chemical interactions, like the formation of rust, to deform and destroy a building’s inner structure. To illustrate our methodology, we show a collection of deep degradation effects applied to urban models involving the creation of rust or of ice within walls.

In this special issue ofIEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the journal papers from the 22nd IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2023), which will be held as a hybrid conference between October 16 and 20, 2023 in Sydney, Australia. ISMAR continues the over twenty-year long tradition of IWAR, ISMR, and ISAR, and is the premier conference for Mixed and Augmented Reality in the world.

Text entry in Virtual Reality (VR) is becoming an increasingly important task as the availability of hardware increases and the range of VR applications widens. This is especially true for VR industrial applications where users need to input data frequently. Large-scale industrial adoption of VR is still hampered by the productivity gap between entering data via a physical keyboard and VR data entry methods. Data entry needs to be efficient, easy-to-use and to learn and not frustrating. In this paper, we present a new data entry method based on handwriting recognition (HWR). Users can input text by simply writing on a virtual surface. We conduct a user study to determine the best writing conditions when it comes to surface orientation and sensory feedback. This feedback consists of visual, haptic, and auditory cues. We find that using a slanted board with sensory feedback is best to maximize writing speeds and minimize physical demand. We also evaluate the performance of our method in terms of text entry speed, error rate, usability and workload. The results show that handwriting in VR has high entry speed, usability with little training compared to other controller-based virtual text entry techniques. The system could be further improved by reducing high error rates through the use of more efficient handwriting recognition tools. In fact, the total error rate is 9.28% in the best condition. After 40 phrases of training, participants reach an average of 14.5 WPM, while a group with high VR familiarity reach 16.16 WPM after the same training. The highest observed textual data entry speed is 21.11 WPM.

Technological developments provide solutions to alleviate the tremendous impact on the health and autonomy due to the impact of dementia on navigation abilities. We systematically reviewed the literature on devices tested to provide assistance to people with dementia during indoor, outdoor and virtual navigation (PROSPERO ID number: 215585). Medline and Scopus databases were searched from inception. Our aim was to summarize the results from the literature to guide future developments. Twenty-three articles were included in our study. Three types of information were extracted from these studies. First, the types of navigation advice the devices provided were assessed through: (i) the sensorial modality of presentation, e.g., visual and tactile stimuli, (ii) the navigation content, e.g., landmarks, and (iii) the timing of presentation, e.g., systematically at intersections. Second, we analyzed the technology that the devices were based on, e.g., smartphone. Third, the experimental methodology used to assess the devices and the navigation outcome was evaluated. We report and discuss the results from the literature based on these three main characteristics. Finally, based on these considerations, recommendations are drawn, challenges are identified and potential solutions are suggested. Augmented reality-based devices, intelligent tutoring systems and social support should particularly further be explored.

In this special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the journal papers from the 21st IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2022), which will be held as a hybrid conference between October 17 and 21, 2022 in Singapore. ISMAR continues the over twenty year long tradition of IWAR, ISMR, and ISAR, and is the premier conference for Mixed and Augmented Reality in the world.

The set of physical and sensitive phenomena that interacts with the urban morphology acts on the resulting perception from the users of a place. Its study and representation provides elements beyond the aesthetics aspects that can allow a better understanding of the space and future urban projects. We aim to analyze the effects of three different wind representations in terms of perception and sense of presence in virtual reality (VR). We focus on the following conditions: (R) reference scene with the audiovisual representation of the mechanical effects of the wind on the elements of the context, (V) reference scene plus the visualization of the wind flow, present (among others) in the architecture field, (T) reference scene plus tactile restitution of wind and eventually (V+T) assembling all previous conditions. For the experiment, we present to the participants (R), then (V) followed by (T), and finish with (V+T). 37 participants evaluated 12 different stop points (divided into four routes in the same simulated street), where they had to determine the perceived wind force and direction concerning the four different conditions (each one corresponding to one route). At the end of each route, participants evaluated their sense of presence in the VR scene. Our analysis showed significant effects of tactile restitution over the visual effects used in the study, both for understanding wind properties and for increasing the sense of presence in the VR scene. In terms of wind direction, (T) reduced the estimation error by 27% compared to (V). Concerning wind force, the reduction was 9.8%. As far as presence was concerned, (T) increased the sense of presence by 12.2% compared to (V).

Commonly used Head Mounted Displays (HMDs) in Augmented Reality (AR), namely Optical See-Through (OST) displays, suffer from a main drawback: their focal lenses can only provide a fixed focal distance. Such a limitation is suspected to be one of the main factors for distance misperception in AR. In this paper, we studied the use of an emerging new kind of AR display to tackle such perception issues: Retinal Projection Displays (RPDs). With RPDs, virtual images have no focal distance and the AR content is always in focus. We conducted the first reported experiment evaluating egocentric distance perception of observers using Retinal Projection Displays. We compared the precision and accuracy of the depth estimation between real and virtual targets, displayed by either OST HMDs or RPDs. Interestingly, our results show that RPDs provide depth estimates in AR closer to real ones compared to OST HMDs. Indeed, the use of an OST device was found to lead to an overestimation of the perceived distance by 16%, whereas the distance overestimation bias dropped to 4% with RPDs. Besides, the task was reported with the same level of difficulty and no difference in precision. As such, our results shed the first light on retinal projection displays' benefits in terms of user's perception in Augmented Reality, suggesting that RPD is a promising technology for AR applications in which an accurate distance perception is required.

While perceptual biases have been widely investigated in Virtual Reality (VR), very few studies have considered the challenging environment of Optical See-through Augmented Reality (OST-AR). Moreover, regarding distance perception, existing works mainly focus on the assessment of egocentric distance perception, i.e. distance between the observer and a real or a virtual object. In this paper, we study exocentric distance perception in AR, hereby considered as the distance between two objects, none of them being directly linked to the user. We report a user study (n=29) aiming at estimating distances between two objects lying in a frontoparallel plane at 2.1m from the observer (i.e. in the medium-field perceptual space). Four conditions were tested in our study: real objects on the left and on the right of the participant (called real-real), virtual objects on both sides (virtual-virtual), a real object on the left and a virtual one on the right (real-virtual) and finally a virtual object on the left and a real object on the right (virtual-real). Participants had to reproduce the distance between the objects by spreading two real identical objects presented in front of them. The main findings of this study are the overestimation (20%) of exocentric distances for all tested conditions. Surprisingly, the real-real condition was significantly more overestimated (by about 4%, p=.0166) compared to the virtual-virtual condition, i.e. participants obtained better estimates of the exocentric distance for the virtual-virtual condition. Finally, for the virtual-real/real-virtual conditions, the analysis showed a non-symmetrical behavior, which suggests that the relationship between real and virtual objects with respect to the user might be affected by other external factors. Considered together, these unexpected results illustrate the need for additional experiments to better understand the perceptual phenomena involved in exocentric distance perception with real and virtual objects.

The topic of distance perception has been widely investigated in Virtual Reality (VR). However, the vast majority of previous work mainly focused on distance perception of objects placed in front of the observer. Then, what happens when the observer looks on the side? In this paper, we study differences in distance estimation when comparing objects placed in front of the observer with objects placed on his side. Through a series of four experiments (n=85), we assessed participants' distance estimation and ruled out potential biases. In particular, we considered the placement of visual stimuli in the field of view, users' exploration behavior as well as the presence of depth cues. For all experiments a two-alternative forced choice (2AFC) standardized psychophysical protocol was employed, in which the main task was to determine the stimuli that seemed to be the farthest one. In summary, our results showed that the orientation of virtual stimuli with respect to the user introduces a distance perception bias: objects placed on the sides are systematically perceived farther away than objects in front. In addition, we could observe that this bias increases along with the angle, and appears to be independent of both the position of the object in the field of view as well as the quality of the virtual scene. This work sheds a new light on one of the specificities of VR environments regarding the wider subject of visual space theory. Our study paves the way for future experiments evaluating the anisotropy of distance perception in real and virtual environments.

In this paper we present an immersive virtual reality user study aimed at investigating how customers perceive and if they would purchase non-standard (i.e. misshaped) fruits and vegetables (FaVs) in supermarkets and hypermarkets. Indeed, food waste is a major issue for the retail sector and a recent trend is to reduce it by selling non-standard goods. An important question for retailers relates to the FaVs' “level of abnormality” that consumers would agree to buy. However, this question cannot be tackled using “classical” marketing techniques that perform user studies within real shops since fresh produce such as FaVs tend to rot rapidly preventing studies to be repeatable or to be run for a long time. In order to overcome those limitations, we created a virtual grocery store with a fresh FaVs section where 142 participants were immersed using an Oculus Rift DK2 HMD. Participants were presented either “normal”, “slightly misshaped”, “misshaped” or “severely misshaped” FaVs. Results show that participants tend to purchase a similar number of FaVs whatever their deformity. Nevertheless participants' perceptions of the quality of the FaV depend on the level of abnormality.

We present a fast and accurate method for reconstructing surfaces of revolution (SoR) on 3D data and its application to structural modeling of a cluttered scene in real-time. To estimate a SoR axis, we derive an approximately linear cost function for fast convergence. Also, we design a framework for reconstructing SoR on dense SLAM. In the experiment results, we show our method is accurate, robust to noise and runs in real-time.

Projectors are important display devices for large scale augmented reality applications. However, precisely calibrating projectors with large focus distances implies a trade-off between practicality and accuracy. People either need a huge calibration board or a precise 3D model [12]. In this paper, we present a practical projectorcamera calibration method to solve this problem. The user only needs a small calibration board to calibrate the system regardless of the focus distance of the projector. Results show that the rootmean-squared re-projection error (RMSE) for a 450cm projection distance is only about 4mm, even though it is calibrated using a small B4 (250×353mm) calibration board.

Adaptive Hypermedia Environments are a suitable means for developing personalized educational content that can respond to the needs of heterogeneous cohorts. These resources are increasingly built upon the Semantic Web, powered by the development and deployment of ontologies. After experimenting the automatic creation of domain ontologies from educational reference books and their use within semantic wikis, we have envisaged the development of applications that allow learners to harness the potential of semantic tools, providing recommended paths for content discovery and effective learning. These applications address issues observed within educational contexts, notably within our institution. The applications will provide learners with a supplementary resource to provide support to those seeking increasingly tailor-made learning experiences.

Virtual reality is a good tool to design and assess urban projects and to study perception in cities. Climate perception significantly influences the perception and use of urban spaces, however, virtual urban environments are scarcely represented with different climatic aspects. In this paper, we study the role that visual cues (sky aspect, shadows, sun location, and light effects) specifically play in climate perception (season, daytime and temperature) in virtual urban environments. We present and discuss the data we collected from a recent virtual reality experiment in which ten variations of the climatic context in the same urban space were assessed.

Virtual reality is a good tool to design and assess urban projects and to study perception in cities. Climate perception significantly influences the perception and use of urban spaces; however, virtual urban environments are scarcely represented with different climatic aspects. In this paper, we study the role that visual cues (sky aspect, shadows, sun location, and light effects) specifically play in climate perception (season, daytime and temperature) in virtual urban environments. We present and discuss the data we collected from a recent virtual reality experiment in which ten variations of the climatic context in the same urban space were assessed. The results prove the feasibility of suggesting complex climatic perceptions and thermal feelings by processing only sky, shadow and lighting effects. Furthermore, our results highlight the role of subjective interpretation, personal background and current real-world environment in interpretation and understanding of visual cues. Finally, we show that perceived climate influences global perception of virtual urban environments.

Interaction with computers using freehand gestures becomes more and more popular. However, it is hard to make precise inputs by making gestures in air without a physical support. In this paper, we present HorizontalDragger, a new selection technique aimed to improve the selection precision by converting the 2D selection problem into a 1D selection problem. After setting a region of interest in the display, all the objects inside the region are considered as potential targets. The user can drag the index finger horizontally to choose the desired target.

To make constrained manipulation of 3D objects in desktop 3D applications, 3D transformation widgets are commonly used. However, their performance degrades on touchscreens because of low accuracy of touch inputs and the fingertip occlusion problem. In this paper, we present TouchSketch, a touch-based interface which allows users to perform independently fine-grained object translation, rotation and scaling on mobile devices. Our manipulation technique permits using the non-dominant hand to specify the manipulation reference while the dominant hand is used to determine the operation mode and control the transformation. In addition to 3D manipulation, TouchSketch provides also a set of functions to edit the shape of 3D objects. Users can use this application to accomplish rapid sketching tasks. We have conducted a user study to evaluate the efficiency of our manipulation technique. The results show that our technique outperforms a Widget-based technique regarding both the efficiency and fluency.

We present a method which can quickly and robustly match 2D and 3D point patterns based on their sole spatial distribution, but it can also handle other cues if available. This method can be easily adapted to many transformations such as similarity transformations in 2D/3D, and affine and perspective transformations in 2D. It is based on local geometric consensus among several local matchings and a refinement scheme. We provide two implementations of this general scheme, one for the 2D homography case (which can be used for marker or image tracking) and one for the 3D similarity case. We demonstrate the robustness and speed performance of our proposal on both synthetic and real images and show that our method can be used to augment any (textured/textureless) planar objects but also 3D objects.

Encyclopedic knowledge bases can be powerful tools for the acquisition of fundamental knowledge for learners. However, the structure and the very nature of these documents can impede learning processes. By extracting domain ontologies from reference books and using this same material to populate an intelligent learning system, we propose a methodology for lifelong learners.

Augmented maps have many important applications. However, no mature registration method exists to associate unprepared maps with a Geographical Information System (GIS) database which would be used to superimpose simulation results or route display on a paper map. In this paper, we propose a method called Robust Random Dot Markers (RRDM) that can robustly track coplanar random dot patterns, which can be used to address this problem. RRDM is based on the same idea of the Random Dot Markers (RDM) proposed by [Uchiyama and Saito 2011] and it can serve as fiducial markers as well as texture independent "natural markers". We conduct a series of experiments and show that RRDM is more robust than RDM in terms of jitter, perspective distortion, under and over detection of dots in the pattern. As an example of "natural marker", we show that RRDM can successfully register unprepared printed maps only with pure geographic features, i.e. road intersections coordinates, which we retrieve from a GIS. Our method does not suffer from the drawbacks of traditional "feature-point" based registration methods which mainly based on textures, since textures may change according to different maps.

Climatic effects represent the physical signals (temperature, wind, humidity, light) which can have a perceptual impact for human-beings. They significantly influence perception and use of urban spaces. Nevertheless, while virtual environments have been proven as good tools to assess urban projects by non-expert people or to study perception in cities, climatic atmospheres are very seldom integrated in the virtual urban model [Drettakis et al. 2007; Tahrani and Moreau 2007]. Indeed, virtual urban environments are represented in ideal situations: the sky is blue, there are a few white clouds, trees are leafy or flowered and the sun is shining. Thus, participants can rarely experiment various seasons, times of day or weathers and and evaluate the urban design under different climatic or lighting situations.

Reference books and encyclopedic knowledge bases present learners with an important source of fundamental concepts. However, the resulting knowledge acquisition process is often hindered by the linearity that is inherent to these resources, making it difficult for learners to realize the many links that exist between these concepts. This research project aims at establishing and implementing a rich semantic model for the identification and classification of knowledge in a core-periphery structure at the service of graduate level students of engineering sciences. This model will allow learners to obtain a global view of knowledge domains and to identify roadmaps for the knowledge acquisition process.

For applications such as outdoor Augmented Reality (AR) or 3D City Model construction of updating, a real-time registration between a video sequence and a Geographic Information System (GIS) is required. In this work, we present a registration system using a GPS prior and a skyline matching method to estimate the camera pose from images. A skyline rectification method through vertical vanishing point detection is also developed to enable arbitrary camera orientation, with non null tilt. The proposed approach is robust, fully automatic and computationally inexpensive which makes it a possible solution for mobile device applications. The performance of our approach is demonstrated in several experimental evaluations. The possible refinement of camera location through a systematic method is also investigated.

Stereoscopic devices are widely used (immersion-based working environments, stereoscopically-viewed movies, auto-stereoscopic screens). In some instances, exposure to stereoscopic immersion techniques can be lengthy, and so eye strain sets in. We propose a method for reducing eye strain induced by stereoscopic vision. After reviewing sources of eye strain linked to stereoscopic vision, we focus on one of these sources: images with high frequency content associated with large disparities. We put forward an algorithm for removing the irritating high frequencies in high horizontal disparity zones (i.e., for virtual objects appearing far from the real screen level). We elaborate on our testing protocol to establish that our image processing method reduces eye strain caused by stereoscopic vision, both objectively and subjectively. We subsequently quantify the positive effects of our algorithm on the relief of eye strain and discuss further research perspectives.

To develop and manage urban territories and to analyze urbanisation impact on environment, an accurate knowledge of the city and its urban fabric is necessary. A physical description like building morphology or land cover/use allows some characterization of the urban terrain. A specific package for a GIS software is developed for urban analysis: UrbSAT (Urban Spatial Analysis Tool). It allows extracting knowledge from various data sources and gives some indicators to real applications (sustainable development, management policy, air quality improvement, etc). It raises issues such as database accuracy and quality, cell shape, size and orientation for the construction of spatial indicators. We present some tests to address those issues.

Wind turbines (WT) are socially controversial because of their visual and acoustic impacts on landscape. Virtual reality (VR) is here proposed - thanks to immersion and interaction potentialities - as an immersive and multisensory approach in order to assess WT impacts. For that, a comparison between a real park and the same virtual one is needed to evaluate VR for landscape impacts restitution. The parks are evaluated using an urban path-based method (perception in motion): the real walking is simulated by a Wiimote in vitro. The results, while very similar to the in situ study show the limits of the lack of free motion.

The urban environment is defined by the interaction between the city morphological characteristics and its physical behavior based on the human interpretation. Contemporary psychology emphasizes the relation between perception and action and shows that perception can be enhanced by the receptor's motion. Basing our work on these principles, we assume that virtual reality techniques, bringing both immersion and interaction to static computer generated images can be of use to study solar effects and therefore daylight ambience. In this paper, we show the methodology for solar effects analysis in both real and virtual world and build demonstrators to compare the emergence of solar effects

In this paper we present a low cost approach to automated reconstruction of 3D roof models which can be used for generation of a realistic 3D environment of an urban area in a virtual reality application. The key aspect of the method which differs from the other studies is that only one aerial view is used and architectural knowledge is introduced to form 3D roof models of buildings. First buildings in the aerial image are located using footprints of buildings in the digital cadastre map and roof image of each building is retrieved from the original image. The corresponding roof lines of the buildings are obtained by processing each roof image separately. Finally by merging 2D roof model and architectural knowledge about the building such as the building type and possible roof slope angles, 3D roof models are created.

In stereo research the construction of a dense disparity map is a complicated task when the scene contains a lot of occlusions. In this case in the neighborhood of occlusions, we could consider that the images have a non-stationary behavior. In this paper we propose a new method for computing a dense disparity map using the decomposition of the 2D wavelet transform in four quarters allowing us to find a corresponding pixel in the case of occlusion as well. Our algorithm constructs in each pixel of two images four estimators corresponding to each quarter. The matching of the four wavelet coefficient estimators in the right image with the other four in the left image allow us to construct a dense map disparity map in each pixel of an image.

3D reconstruction from a set of stereoscopic image pairs requires the ability to match both images: any point in an image should match an homologous point in the other image. Knowing the relative position of the cameras, it is then possible to build a dense disparity map and a Z-buffer which will be used for 3D reconstruction. We propose a new method for computing a dense disparity map thanks to the use of a special wavelet transform less sensitive to discontinuity in disparities and partial occlusions. This method is based on the quarter of wavelet transform. Some experimental results are presented at the end of the paper.