This paper presents a hand-tracking-based digital Theremin that integrates Unity3D real-time audio synthesis with Leap Motion Controller 2 optical tracking. The system maps 3D hand positions to core musical parameters, pitch and volume, while allowing real-time control of tone type, waveform blending, reverb, and octave through additional spatial dimensions. A modular architecture combines continuous hand-tracking input, visual feedback, and procedural audio processing within a responsive user interface. Visual guidance panels support intuitive learning by displaying real-time feedback for pitch and volume control. A user study with both musicians and non-musicians demonstrated that the system was learnable, expressive, and responsive, though limitations in tracking precision and sound smoothing were observed. A user evaluation indicates that the system provides responsive and expressive control, and is learnable for both musicians and non-musicians.

Music visualization is presented in different formats. The mainstream music visualization remains focused on providing animated images for visual pleasure or for enhancing the perception of sheet music with colors. Static visualization of music, on the other hand, has the potential to provide quick insights into the music’s attributes, particularly its mood. However, despite our previous work exploring several approaches to music mood visualization, a major problem still exists in how to objectively evaluate the mood of music pieces and the generated images. Thus, this paper proposes IMEMNextan improvement to the existing framework IMEMNet [1], which serves as a novel and objective metric for music mood visualization. Not only does it allow for a user study-free evaluation of music mood visualization, but the new network also enables for creation of datasets of music-image pairs with the same mood. The new IMEMNext network achieved a significant performance improvement compared to its original implementation.

The manufacturing process in the pharmaceutical industry requires for using vessels or tanks which inner surfaces can undergo fouling and/or may develop visible defects that have to be cleaned away. Surface contamination can present itself in different forms, and there is no available image database of stains. In this article, we propose to use an unsupervised anomaly detection approach, which trains models on one or just a few images of clean surfaces. First, we propose an inpainting GAN with global and local generators to make full use of every single image and to reduce the overfitting caused by a limited number of images in the training dataset. Furthermore, we propose a periodic noise injection technique to increase the number of images for training and to improve the detection performance of the network. The experimental results demonstrate that the proposed network and noise injection technique achieve significant performance gains in the challenging task of anomaly detection on the surfaces of pharmaceutical equipment and other real-world industrial datasets. We also consider possible generalizations of the proposed method.

Music visualization has become an integral part of music performance, appreciation, and study. Even before computers, people have tried to visualize different aspects of music. Today, music visualization is conducted on a wide variety of musical characteristics, of which timbre is one of the least visualized. Moreover, timbre is commonly labeled with semantic descriptors that vary from person to person, and therefore it is difficult to quantify and categorize. Through the analysis of audio spectral characteristics, we have devised method to create a timbre-to-color conversion. The proposed conversion was successfully tested on different users and hence can be used for further improvement.

Music has been visualized in different forms. Majority of the existing methods of music visualization utilized only a select few parameters of the music, of which pitch and frequency were the most visualized. Also, current musk visualizers usually produce animated visual backgrounds for the music being played. Visualization of music as static images was only addressed by some people who claimed to have a neurological condition called synesthesia or chromesthesia. In this paper, we use artificial intelligence to simulate such music visualization. Specifically, we consider how the mood of music can be visualized in static images using deep learning techniques. We consider two approaches. First, we use the deep learning network to generate abstract paintings based on the music sentiments obtained from the Spotify library. In the second approach, two different deep learning networks are used to both classify the musk and to generate the respective landscape images representing its mood. The results are analyzed and compared as well as subjected to user tests.

This paper presents an approach to procedural modeling and the interactive software tool ShapeExplorer designed to make geometric shapes and visual appearances defined by mathematical functions (explicit, implicit and parametric). It can run on Windows, MacOS and Linux. It is just one interactive window where the user can type the definitions scripts and other parameters. The purpose of ShapeExplorer is to work as a quick all-in-one multi-platform design and visualization tool providing mathematical models for virtual environments.

We performed a feasibility study on new ways of using sound in interactive computer graphics to improve visual interaction as well as to replace or augment haptic interaction. We considered using sound during surface interaction tasks performed with a desktop haptic device and an optical hand tracking device and evaluated the efficiency of such interactive geometry sonification in terms of its precision and speed. We also considered scenarios of using sound for easing navigation while moving along a path or surface.

Anomaly detection is a challenging task in data analysis, especially when it comes to unsupervised pixel-level segmentation of anomalies in images. In this paper, we present a novel multi-stage defect repaired image resynthesis framework for the detection and segmentation of anomalies in images. In contrast to the existing reconstruction-based approaches, our reconstruction is free from artifacts caused by defective regions so that the defects can be identified from the residual map between input samples and their resynthesized defect-eliminated outputs. Our method outperforms the state-of-art benchmarks in most categories using the publicly available MVTec dataset. Besides, the method also demonstrates an excellent capability of repairing defects in abnormal samples.

The hand tracking device Leap Motion Controller has been applied in many areas since it was released. Two innovative interaction mechanisms were designed by the author for virtual object manipulation, in which collision detection between the virtual hands and the objects is not required. Instead, it controls the grasping and releasing with the gestures of one hand; and controls the direction and amount of the object's displacement with the other hand. After several tests, the mechanisms are proven to have reasonably high precision. They allow the users to adjust the parameters to create customized systems, leading to an optimal outcome.

Following the rapid advancement of Artificial Intelligence and transition into the era of Big Data, researchers have started to explore the possibility of using machine learning in creative domains such as music generation. However, most research were focused on musical composition and removed expressive attributes during data pre-processing, which resulted in mechanical-sounding generated music. To address this issue, music elements, such as pitch, time and velocity, were extracted from MIDI tracks and encoded with piano-roll data representation. With the piano-roll data representation, Deep Convolutional Generative Adversarial Network (DCGAN) learned the data distribution from the given dataset and generated new data derived from the same distribution. The generated music was evaluated based on its incorporation of music dynamics and a user study. The evaluation results verified that DCGAN could generate expressive music comprising of music dynamics and syncopated rhythm.

Generative Adversarial Networks (GANs) are a very recent chapter in generative models and music visualisation. This paper explores the application of GANs in music visualisation via latent space exploration using latent directions. The output music visualisation tool is then evaluated via a user study to verify its effectiveness.

Artificial intelligence is being increasingly applied in manufacturing to maximize industrial productivity. Image clustering, as a fundamental research direction in unsupervised learning, has been used in various fields. Since no label information is required in clustering, it can perform a preliminary analysis of the data while saving lots of manpower. In this paper, we propose a novel end-to-end clustering network called Self-supervised Pairing Image Clustering (SPIC) for industrial application, which produces clustering prediction for input images in an advanced pair classification network. For training this network, a self-supervised pairing module is built to form balanced pairs accurately and efficiently without label information. Since the existence of trivial solutions cannot be avoided in most of unsupervised learning methods, two additional information theoretic-constraints regularize the training that ensures the clustering prediction to be unambiguous and close to the real data distribution during training. Experimental results indicate that the proposed SPIC outperforms the state-of-art approaches on manufacturing datasets-NEU and DAGM. It also shows the execellent generalization capability on other genral public datasets, such as MNIST, Omniglot, CIFAR10, and CIFAR100.

Recent interest and development in artificial intelligence spurred researchers to explore the role that deep learning could play in creative music. There exists a variety of deep learning music generation systems and methods that fulfil different objectives such as generation of a specific musical texture or accompaniment for a melody. However, current implementations lack content variability as they are only able to generate music with a fixed length or varied length with fixed temporal granularity. We propose a new hybrid temporal scope representation that generates music with both varied length and temporal granularity, enabling the generation of progressive meters such as irregular and mixed meters. We introduce a model made up of a Bidirectional Long Short-Term Memory (Bi-LSTM) network that uses our hybrid temporal scope representation to generate coherent music with irregular and mixed meters. Evaluation of our model using score analysis and user study demonstrates that it successfully generates irregular rhythmic properties with commendable musical value.

This paper proposes an integrated detection framework of solder joint defects in the context of Automatic Optical Inspection (AOI) of Printed Circuit Boards (PCBs). Both localization and classifications tasks were considered. For the localization part, in contrast to the existing methods that are highly specified for particular PCBs, we used a generic deep learning method which can be easily ported to different configurations of PCBs and soldering technologies and also gives real-time speed and high accuracy. For the classification part, an active learning method was proposed to reduce the labeling workload when a large labeled training database is not easily available because it requires domain-specified knowledge. The experiments show that the localization method is fast and accurate. In addition, high accuracy with only minimal user input was achieved in the classification framework on two different datasets. The results also demonstrated that our method outperforms three other active learning benchmarks.

Methods from supervised machine learning allow the classification of new data automatically and are tremendously helpful for data analysis. The quality of supervised maching learning depends not only on the type of algorithm used, but also on the quality of the labelled dataset used to train the classifier. Labelling instances in a training dataset is often done manually relying on selections and annotations by expert analysts, and is often a tedious and time-consuming process. Active learning algorithms can automatically determine a subset of data instances for which labels would provide useful input to the learning process. Interactive visual labelling techniques are a promising alternative, providing effective visual overviews from which an analyst can simultaneously explore data records and select items to a label. By putting the analyst in the loop, higher accuracy can be achieved in the resulting classifier. While initial results of interactive visual labelling techniques are promising in the sense that user labelling can improve supervised learning, many aspects of these techniques are still largely unexplored. This paper presents a study conducted using the mVis tool to compare three interactive visualisations, similarity map, scatterplot matrix (SPLOM), and parallel coordinates, with each other and with active learning for the purpose of labelling a multivariate dataset. The results show that all three interactive visual labelling techniques surpass active learning algorithms in terms of classifier accuracy, and that users subjectively prefer the similarity map over SPLOM and parallel coordinates for labelling. Users also employ different labelling strategies depending on the visualisation used.

Not many people know about the first electronic musical instrument - the theremin - and can play it. The idea of this instrument is very groundbreaking: it is played without physical contact with it and in the same way as we sing but by using hands in place of our vocal cords. In this paper we consider how to implement the theremin with a computer using very different physical principles of optical hand tracking and by adding advantages of visual interfaces. The goal of this research is to eventually fulfill the dream of the inventor to make the theremin a musical instrument for everyone and to prove that everyone can play music.

Automatic optical inspection for manufacturing traditionally was based on computer vision. However, there are emerging attempts to do it using deep learning approach. Deep convolutional neural network allows to learn semantic image features which could be used for defect detection in products. In contrast to the existing approaches where supervised or semi-supervised training is done on thousands of images of defects, we investigate whether unsupervised deep learning model for defect detection could be trained with orders of magnitude smaller amount of representative defect-free samples (tenths rather than thousands). This research is motivated by the fact that collection of large amounts of defective samples is difficult and expensive. Our model undergoes only one-class training and aims to extract distinctive semantic features from the normal samples in an unsupervised manner. We propose a variant of transfer learning, that consists of combination of unsupervised learning used upon VGG16 with pre-trained on ImageNet weight coefficients. To demonstrate a defect detection, we used a set of Printed Circuit Boards (PCBs) with different types of defects - scratch, missing washer/extra hole, abrasion, broken PCB edge. The trained model allows us to make clusters of normal internal representations of features of PCB in high-dimensional feature space, and to localize defective patches in PCB image based on distance from normal clusters. Initial results show that more than 90% of defects were detected.

Detecting defects is an integral part of any manufacturing process. Most works still utilize traditional image processing algorithms to detect defects owing to the complexity and variety of products and manufacturing environments . In this paper, we propose an approach based on deep learning which uses autoencoders for extraction of discriminative features . It can detect different defects without using any defect samples during training. This method, where samples of only one class (i.e. defect-free samples) are available for training, is called One Class Classification (OCC). This OCC method can also be used for training a neural network when only one golden sample is available by generating many copies of the reference image by data augmentation . The trained model is then able to generate a descriptor—a unique feature vector of an input image. A test image captured by an Automatic Optical Inspection (AOI) camera is sent to the trained model to generate a test descriptor, which is compared with a reference descriptor to obtain a similarity score. After comparing the results of this method with a popular traditional similarity matching method SIFT , we find that in the most cases this approach is more effective and more flexible than the traditional image processing-based methods, and it can be used to detect different types of defects with minimum customization .

Supervised machine learning techniques require labelled multivariate training datasets. Many approaches address the issue of unlabelled datasets by tightly coupling machine learning algorithms with interactive visualisations. Using appropriate techniques, analysts can play an active role in a highly interactive and iterative machine learning process to label the dataset and create meaningful partitions. While this principle has been implemented either for unsupervised, semi-supervised, or supervised machine learning tasks, the combination of all three methodologies remains challenging. In this paper, a visual analytics approach is presented, combining a variety of machine learning capabilities with four linked visualisation views, all integrated within the mVis (multivariate Visualiser) system. The available palette of techniques allows an analyst to perform exploratory data analysis on a multivariate dataset and divide it into meaningful labelled partitions, from which a classifier can be built. In the workflow, the analyst can label interesting patterns or outliers in a semi-supervised process supported by active learning. Once a dataset has been interactively labelled, the analyst can continue the workflow with supervised machine learning to assess to what degree the subsequent classifier has effectively learned the concepts expressed in the labelled training dataset. Using a novel technique called automatic dimension selection, interactions the analyst had with dimensions of the multivariate dataset are used to steer the machine learning algorithms. A real-world football dataset is used to show the utility of mVis for a series of analysis and labelling tasks, from initial labelling through iterations of data exploration, clustering, classification, and active learning to refine the named partitions, to finally producing a high-quality labelled training dataset suitable for training a classifier. The tool empowers the analyst with interactive visualisations including scatterplots, parallel coordinates, similarity maps for records, and a new similarity map for partitions.

Video-conferencing and video phone calls are common nowadays. However more immersion can be achieved if we not only see and hear but also physically feel each other. In contrast to previous attempts which were mostly setting up integral video-haptic communication systems, we propose a different way where we use any existing video communication tools while asynchronously exchanging over the Internet cloud server very little data packets containing only haptic interaction data. In case of desktop haptic devices, we only exchange between the participating computers haptic interface point coordinates, orientation angles of the device handles and computed force vectors. More options for haptic communication appear when we replace common video cameras with depth-sensing cameras and hand-tracking devices. Then, motion of the hand, arm or even the whole body of one party can be both seen and physically felt by the other party participating in such Internet tangible video communication. We validated the proposed way of communication by technical measurements of the transmission times over the LAN and Amazon Cloud server, as well as by conducting a successful user study.

This paper proposes a method for automatic image-based classification of solder joint defects in the context of Automatic Optical Inspection (AOI) of Printed Circuit Boards (PCBs). Machine learning-based approaches are frequently used for image-based inspection. However, a main challenge is to manually create sufficiently large labeled training databases to allow for high accuracy of defect detection. Creating such large training databases is time-consuming, expensive, and often unfeasible in industrial production settings. In order to address this problem, an active learning framework is proposed which starts with only a small labeled subset of training data. The labeled dataset is then enlarged step-by-step by combining K-means clustering with active user input to provide representative samples for the training of an SVM classifier. Evaluations on two databases with insufficient and shifting solder joints samples have shown that the proposed method achieved high accuracy while requiring only minimal user input. The results also demonstrated that the proposed method outperforms random and representative sampling by ~ 3.2% and ~ 2.7%, respectively, and it outperforms the uncertainty sampling method by ~ 0.5%.

Video-conferencing and video calls are common nowadays. However, without being able to physically touch or feel each other, we cannot have full immersive communication achieved. Unlike earlier attempts of joining video and haptic communication into one integrated system, we propose to setup an asynchronous method of exchanging haptic interaction data while using traditional ways of video communication. The data packets are exchanged over the Internet cloud server where only participating haptic interface point coordinates, orientation angles of the device handles are transmitted. We also explore more options like using depth-sensing cameras and handtracking devices to capture motion of hand, arm or the whole body of one user so that he becomes both visible and tangible to the other party. The proposed way of communication is validated by technical measurements and a user study. A test of remote connections over long distances was carried out to pave way for further studies.

Surface level defect detection, such as detecting missing components, misalignments and physical damages, is an important step in any manufacturing process. In this paper, similarity matching techniques for manufacturing defect detection are discussed. We are proposing an algorithm which detects surface level defects without relying on the availability of defect samples for training. Furthermore, we are also proposing a method which works when only one or a few reference images are available. It implements a deep autoencoder network and trains input reference image(s) along with various copies automatically generated by data augmentation. The trained network is then able to generate a descriptor-a unique signature of the reference image. After training, a test image of the same product is sent to the trained network to generate a test image descriptor. By matching the reference and test descriptors, a similarity score is generated which indicates if a defect is found. Our experiments show that this approach is more generic than traditional hand-engineered feature extraction methods and it can be applied to detect multiple type of defects.

Haptic feedback has always been a missing link in online shopping. In this project, we study whether a commonly-used haptic device with only one Haptic Interface Point (HIP) can be used in online shopping for compensating lack of physical touch. A user study was conducted in which data-driven haptic weight, shape and texture information was simulated and provided to the users. Despite the limitations of the device, the results have shown positive effects of providing haptic feedback in enhancing users' understanding of physical properties of a product.1

Interactive rendering of translucent materials in virtual worlds has always proved to be challenging. In our work, we develop a voxel illumination framework for translucent materials illuminated by area lights. Our voxel illumination framework consists of two voxel structures. They are the Enhanced Subsurface Light Propagation Volumes (ESLPV), which handles the local translucent material appearance and the Light Propagation Volumes (LPV), which handles indirect illumination for the entire scene. Using a set of sparsely distributed Poisson disk samples in the ESLPV and LPV, illumination can be gathered from area lights. A uniform set of Poisson disk samples on the translucent objects is resampled and chosen as Translucent Planar Lights (TPLs) and is used to distribute lighting from translucent objects into the LPV by an additional gathering process. Our technique allows for direct and indirect illuminations from highly scattering translucent materials to be rendered interactively under area lighting at good quality. We can achieve similar effects, such as scattered light illumination from translucent materials, when compared to offline renderers without precomputations.

Naturalness of the mid-air interaction interface for virtual assembly and shape modeling is important. In order to design an interface perceived as "natural" by most people, common behaviors and mental patterns for mid-air interaction of people have to be recognized, which is an area merely explored yet. This paper serves this purpose of understanding the users' mental interaction models, in order to provide standards and recommendation for devising a natural virtual interaction interface. We tested three kinds of tasks --- manipulating tasks, deforming tasks and tool-based operating tasks on 16 participants. We have found that: 1) different features of mental models were observed for different types of tasks. Interaction techniques should be designed to match these features; 2) virtual hand self-avatar helps estimate size of virtual objects, as well as helps plan and visualize the complex process and procedures of a task, which is especially helpful for tool-based tasks; 3) bimanual interaction is witnessed as a dominant interaction mode preferred by the majority; 4) natural gestures for deforming tasks always reflect forces exerted. These suggestions are useful for designing a midair interaction interface matching users' mental models.

In this paper, we propose a web-enabled shape modeling system with natural free-hand interaction, which can be easily learned by users while imposing least mental load on them. The deformation interface allows for performing various deformations, including stretching, compressing, squeezing, enlarging, twisting and tapering, on shapes interactively mimicking how they are done in real life. The manipulation interface allows an object to be directly grabbed and manipulated with either one or two hands, while also smoothly switching between them. Constrained methods are also provided for precise manipulation. An intuitive metaphor is designed to help the users to discover the interaction techniques by themselves without any manuals or instructions. A rendering pipeline, based on function-based extension of VRML/X3D, is designed with hidden complexity to support the proposed functionalities of the system. Hands motions are captured by Leap Motion controller. The user study proves the naturalness of the modeling system, and its easiness to be learned and remembered.

Creation of procedural 3D building models can significantly lessen the costs of modeling, since it allows generating a variety of similar shapes from one procedural description. The common field of application for procedural modeling is modeling of straight building facades, which are very well suited for shape grammars – a special kind of procedural modeling system. In order to generate round building geometry, we present a way to setup different coordinate systems in shape grammars. Besides Cartesian, these are primarily cylindrical and spherical coordinate systems for generation of structures like towers or domes, that can procedurally adapt to different dimensions and parameters. The users can apply common splitting idioms from shape grammars in their familiar way, for creating round instead of straight geometry.

Interactive rendering of illumination from area lights in virtual worlds has always proved to be challenging. In this paper, we extend the work of multi resolution rendering for direct illumination from area lights. We propose a deferred shading method for direct illumination which subdivides screenspace into multi resolution 2D-fragments in which higher resolution fragments are created to represent geometric and depth discontinuities as well as shadow boundaries. To detect shadow boundaries, our subdivision scheme, sub-fragment visibility test (SFVT), performs a visibility discontinuity check within each fragment and subdivides the fragment to a higher resolution level if discontinuity is found. In addition, our proposed gradient aware screenspace subdivision (GASS) algorithm accelerates the refinement by increasing the number of subdivisions based on gradient differences. Our technique utilizes the streamcompaction feature of the transform feedback shader (TFS) in the graphics shading pipeline to filter out fragments for soft shadow refinement. A single pass screenspace irradiance upsampling scheme which uses radial basis functions (RBF) is proposed for interpolating scattered fragments. This reduces artifacts caused by large fragments. OuInteractive rendering of illumination from area lights in virtual worlds has always proved to be challenging. In this paper, we extend the work of multi resolution rendering for direct illumination from area lights. We propose a deferred shading method for direct illumination which subdivides screenspace into multi resolution 2D-fragments in which higher resolution fragments are created to represent geometric and depth discontinuities as well as shadow boundaries. To detect shadow boundaries, our subdivision scheme, sub-fragment visibility test (SFVT), performs a visibility discontinuity check within each fragment and subdivides the fragment to a higher resolution level if discontinuity is found. In addition, our proposed gradient aware screenspace subdivision (GASS) algorithm accelerates the refinement by increasing the number of subdivisions based on gradient differences. Our technique utilizes the stream-compaction feature of the transform feedback shader (TFS) in the graphics shading pipeline to filter out fragments for soft shadow refinement. A single pass screenspace irradiance upsampling scheme which uses radial basis functions (RBF) is proposed for interpolating scattered fragments. This reduces artifacts caused by large fragments. Our technique does not require precomputations and is able to run at interactive rates.r technique does not require precomputations and is able to run at interactive rates.

Multi-view reconstruction methods are able to produce polygon meshes of a complex scene with millions of triangles which are well suited for visualization purposes. However, these large-scale, dense meshes normally cannot be haptically rendered directly with readily available APIs. In this paper, we present a method to extend meshes reconstructed from images to visual-haptic applications by using images for visual display while an improved hybrid collision detection method is used to meet the real-time requirements for haptic rendering. Moreover, three main imperfections (holes, outliers and degenerated facets) inherent in reconstructed models are also handled to help provide a smooth and consistent force feedback. Given a reconstructed model and the corresponding image, the proposed method provides a way to virtually explore the scene in the image, making it possible to appreciate art works and historic monuments within a touching distance.

Haptic interaction is a form of a user-computer interaction where physical forces are delivered to the user via vibrations, displacements and rotations of special haptic devices. When quality of the experience of the haptic interaction is assessed, mostly subjective tests using various questionnaires are performed. We proposed novel neurocognitive tools for assessing both overall experience of the haptic interaction, as well as particular time-stamped activities. Our assessment tools are based on recognition of emotions and stress obtained from Electroencephalograms (EEG). We used them in a feasibility study on adding haptic interaction to Skype video conversation.

Video interaction is a common way of communication in cyberspace. It can become more immersive by incorporating hap tic modality. Using commonly available depth sensing controllers like Microsoft Kinect, information about the depth of a scene can be captured in real-time together with the video. In this paper, we present a method for real-time hap tic interaction with videos containing depth data. Forces are computed based on the depth information. Spatial and temporal filtering of the depth stream is used to provide stability of force feedback delivered to the hap tic device. Fast collision detection ensures the proposed approach to be used in real-time. The usefulness of the approach is illustrated by a tangible video application example.

Abstract We add new modality to image‐based visualization by converting ordinary photos into tangible images, which can be then haptically rendered. This is performed by interactive sketching haptic models on the photos so that the models match the image parts, which will become tangible. In contrast to common geometric modelling, we define the haptic models in a three‐dimensional haptic modelling space distorted by the central projection. Analytic FRep functions (variants of implicit functions) are mostly used for defining the haptic models. The tangible images thus created can realistically simulate some actual three‐dimensional scenes by implementing the principle “What You See Is What You Touch” while in fact still be 2D images. Copyright © 2015 John Wiley & Sons, Ltd.

We augment images with additional data allowing us to visualize them with elements of animation as well as interact with the images using haptic devices as if they were actual 3D scenes. First, we apply procedural simulation to simulate image distortions, which are characteristic for various phenomena like blowing wind, fluid flow, etc. Next, we consider how haptic rendering of images can be performed. This in turn includes haptic rendering of forces augmented to the images as well as haptic interaction with haptic objects augmented to the images. The immersive images thus obtained can be exchanged across the internet to realistically simulate 3D virtual scenes while in fact still be 2D images augmented with rather little additional data files in a way how movie subtitles files are added to the video files. The proposed approach is illustrated with a few representative examples.

Common shape modelling is usually done with virtual tools controlled by interactive or hap tic devices, which have one interface point for simulating collision between the object and the modelling tool. Shape modeling with some kind of virtual hand where several fingers deform the object was not very common since the respective devices are either too expensive or not very precise. Introduction of affordable Leap Motion device opens new prospects for free hand shape modeling. This paper is a feasibility study on using Leap Motion for shape modeling in VRML/X3D environments where virtual objects are defined by mathematical functions and hence can easily be exchanged across the internet due to their small size.

While the growing demand for new building models contained in virtual worlds, games, and movies, makes the easy and fast creation of modifiable models more and more important, 3D modeling of buildings can be a tedious task due to their sometimes complex geometry. For historic buildings, especially the roofs can be challenging. We present a new method of combining simple building solids to form more complex buildings, and give an emphasis on the blending of roof faces. This can be integrated in common pipelines for procedural modeling of buildings and will bring more expressiveness than existing methods.

Minimally invasive arthroscopic surgery has become the gold standard for orthopedic surgery procedures on joints done by making small incisions on the skin through which special miniature pencil-like cameras and tools are inserted. Simulation of the arthroscopic procedures in virtual environments is based on the specifics of the arthroscopic camera which is normally not a common forward-looking but a short-focus side-looking camera. In this paper we write about two problems which arise when modeling views seen through the virtual arthroscopic camera controlled by a desktop hap tic device in a hybrid image-based virtual environment. These are automatic creation of panoramic images of the surgical field from the actual arthroscopic videos and removing fish-eye distortions from the individual video frames.

Haptic interaction significantly augments our experience with a computer and in cyber worlds in particular. However, haptic interaction requires the content creators to make physical or haptic models of the virtual objects while it is not always possible or feasible, especially when it comes to using real images or videos as elements of interaction. We, therefore, propose to use image-driven haptic rendering where a displayed image, real or simulated, is used as a source of the force-feedback calculations at any pixel touched by the haptic device. We introduce the main idea and describe how it is implemented as a core algorithm for image-driven haptic rendering, as well as for a few particular cases of haptic rendering of different dominant colors, textures and contours of the objects displayed in the images. Implementations of the proposed method to desktop tangible image application and haptic video communication on the web are presented as a proof of concept.

Minimally invasive arthroscopic surgery has replaced the common orthopaedic surgery procedures on joints. However it demands from surgeons to acquire very different motor-skills for using special miniature pencil-like instruments and cameras inserted through little incisions on the body while observing the surgical field on a video monitor. Training in virtual reality is becoming an alternative to traditional surgical training based on either real patients or increasingly difficult to procure cadavers. In this paper we propose solutions for simulation in virtual environments a few basic arthroscopic procedures including incision of the arthroscopic camera, positioning of the instrument in front of it, as well as using scissors and graspers. Our approach is based on both full 3D simulation and haptic interaction as well as image-based visualization and haptic interaction.

We propose a new educational web-enabled 3D shape modeling framework that does not restrict to a list of predefined primitive geometric objects and allows the user to interactively create objects by incremental modifications of the basic shape with the tool shapes that are defined by analytical functions. The shape definition is eventually a function script which can be rendered on any suitable graphics system. The function script includes accelerating structures that significantly improve the shape rendering performance allowing us to sustain interactivity with large number of interactive operations applied.

In this paper, we introduce a new approach to virtual palpation for medical training in cyber worlds. We analyze palpation as a medical procedure. Then, we survey the existing virtual palpation projects, which use haptic devices, and propose a new image-driven approach to haptic palpation that can be easily ported to any web-enabled and collaborative environments. We also introduce variable haptic interaction point that allows us to implement multiple-point haptic interaction while using a single-point desktop haptic device. Lastly, we prove our hypothesis by implementing the proposed approach for abdominal palpation and validating it with medical practitioners. We also discuss the advantages of our method over other existing works in terms of flexibility, simplicity and scalability.

Polygon and point based models dominate in virtual reality. These models also affect haptic rendering algorithms which are often based on collision with polygons. We use mathematical functions to define and implement geometry (curves, surfaces and solid objects), visual appearance (3D colors and geometric textures) and various tangible physical properties (elasticity, friction, viscosity, and force fields). The function definitions are given as analytical formulas (explicit, implicit and parametric), function scripts and procedures. Since the defining functions are very small we can efficiently use them in collaborative virtual environments to exchange between the participating clients. We proposed an algorithm for haptic rendering of virtual scenes including mutually penetrating objects with different sizes and arbitrary location of the observer without a prior knowledge of the scene to be rendered. The algorithm is based on casting multiple haptic rendering rays from the Haptic Interaction Point (HIP), and it builds a stack to keep track on all colliding objects with the HIP. The algorithm uses collision detection based on implicit function representation of the object surfaces. The proposed approach allows us to be flexible when choosing the actual rendering platform. The function-defined objects and parts constituting them can be used together with other common definitions of virtual objects such as polygon meshes, point sets, voxel volumes, etc. We implemented an extension of X3D and VRML which allows for defining complex geometry, appearance and haptic effects in virtual scenes by functions and common polygon-based models, with various object sizes, mutual penetrations, arbitrary location of the observer and variable precision.

Visual and haptic rendering pipelines exist concurrently and compete for computing resources while the refresh rate of haptic rendering is two orders of magnitude higher than that of visual rendering (1000 Hz vs. 30-50Hz). However, in certain cases, 3D visual rendering can be replaced by merely displaying 2D images, thus releasing the resources to image-driven haptic rendering algorithms. A number of approaches have been proposed to provide haptic interaction with 2D images but they suffer from various problems and do not provide a fully believable impression of haptic sensation of the 3D scene displayed in the image. Based on the method of haptic effect generation, the existing approaches can be classified into techniques that use information derived from the image, and techniques that use additional information along with the image to enable haptic interaction. Previously, we proposed our own method of augmenting images with haptic data for using them in shared virtual environments [Rasool and Sourin 2010]. The images were augmented with invisible haptic objects defined by implicit, explicit and parametric mathematical functions using the function-based extension of the Virtual Reality Modeling Language [Wei and Sourin, 2011].

Various Partial Differential Equations (PDE) have been used in computer graphics for approximating surfaces of geometric shapes by finding solutions to PDEs subject to suitable boundary conditions. The PDE boundary conditions are defined as 3D curves on the surface of the shapes. We propose how to automatically derive these curves as boundaries of curved patches on the surface of the original polygon mesh. The analytic solution to the PDE used throughout this work is fully determined by finding a set of coefficients associated with parametric functions according to the particular set of boundary conditions. The PDE coefficients require an order of magnitude smaller space compared to the original polygon data and can be interactively rendered with different level of detail. It allows for an efficient exchange of the PDE shapes in 3D Cyber worlds and their web visualization. In this paper we analyze and formulate the requirements for extracting suitable boundary conditions, describe the algorithm for the automatic deriving of the boundary curves, and present its implementation as a part of the function-based extension of VRML and X3D.

Haptic interaction is commonly used with 3D objects defined by their geometric and solid models. Extension of the haptic interaction to 3D Cyber worlds is a challenging task due to the Internet bandwidth constraints and often prohibitive sizes of the models. We study how to replace visual and haptic rendering of shared 3D objects with 2D image visualization and 3D haptic rendering of the forces reconstructed from the images or augmenting them, which will eventually simulate realistic haptic interaction with 3D objects. This approach allows us to redistribute the computing power so that it can concentrate mainly on the tasks of haptic interaction and rendering. We propose how to implement such interaction with small function descriptions of the haptic information augmenting images and video. We illustrate the proposed ideas with the function-based extension of VRML and X3D.

Creating cyber-instructors or virtual humans in cyberspace capable of maintaining conversation with learners as well as offering different educational services may become an excellent teaching tool implementing personal mentoring in Cyber worlds. In this paper, a framework of the cyber-instructor based on the commonly available software tools is proposed. The components of the framework are replaceable and can be easily tuned to particular educational needs and technical implementations. Several implementations of the cyber-instructor with various looks and feels of the HCI have been studied and implemented such as 3D talking avatar, web chat with multimedia components and chat engine communication.

Commonly, surface and solid haptic effects are separated for haptic rendering. We propose a method for defining surface and solid haptic effects as well as various force fields in 3D cyber worlds containing mixed geometric models, including polygon meshes, point clouds, image-based billboards and layered textures, voxel models and functions-based models of surfaces and solids. We also propose a way how to identify location of the haptic tool in such haptic scenes as well as consistently and seamlessly determine haptic effects when the haptic tool moves in the scenes with objects having different sizes, locations, and mutual penetrations.

We propose the use of partial differential equations (PDEs) for shape modelling within visual cyberworlds. PDEs, especially those that are elliptic in nature, enable surface modelling to be defined as boundary-value problems. Here we show how the PDE based on the Biharmonic equation subject to suitable boundary conditions can be used for shape modelling within visual cyberworlds. We discuss an analytic solution formulation for the Biharmonic equation which allows us to define a function based geometry whereby the resulting geometry can be visualised efficiently at arbitrary levels of shape resolutions. In particular, we discuss how function based PDE surfaces can be readily integrated within VRML and X3D environments.

We seek to further expand the shared collaborative potential of cyberworlds by using haptic forcefeedback in shared virtual scenes. We propose how to define density of the objects, together with their geometry and appearance, by using mathematical functions. We illustrate this concept by developing software which allows us to touch and feel surfaces of VRML and X3D objects, convert them to solid objects as well as create any other solid objects using the function-based extension of VRML and X3D. We define geometry, appearance and density of the solid objects by implicit, explicit and parametric functions straight in the VRML/X3D code or in dynamic-link libraries. Since the function-based models are small in size, it is possible to perform their collaborative interactive modifications with concurrent synchronous visualization at each client computer with any required level of detail. We illustrate the proposed with several application examples.

Geometric shape modeling becomes increasingly complex and resource-demanding task. In this paper we propose a method to leverage the power of grid to provide users with high-precision free-form shape modeling environment. Function-based representation allows for defining shapes with an unlimited level of detail while keeping a small size of the model. The model compactness allows us to create optimal infrastructure for distributing work between processing nodes. A novel parallel surface extraction algorithm was proposed to avoid a uniform sampling of the defining function. Only points located close to the surface of the shape are sampled. A protocol for interactive communication between the user of the modeling system and the processing modules in grid has been developed to support shape modification, visualization with arbitrary precision and dynamic load-dependent allocation of computing resources during the modeling session.

We have proposed and implemented function-based extensions of X3D and its predecessor VRML which allow for defining time-dependent geometric shapes, their appearance and transformations with analytically defined parametric, implicit and explicit functions. The function-defined shapes can be used together with the standard X3D and VRML shapes. Besides defining shapes by analytical functions, we have developed interactive function-based shape modeling tools. We have also extended these interactive shape modeling tools to work on the grid. Function-defined models can be efficiently rendered at the media grid portal

In this paper we propose FVRML - a function-based extension of Virtual Reality Modeling Language which allows for time-dependent shape modeling on the Web. Shape's geometry, 3D texture, color and transformations can be defined with analytical functions typed straight in VRML code. These analytical functions can be functions of time. FVRML allows us to greatly extend the abilities of VRML with a very intuitive approach to shape modeling based on using analytical functions. Several functions can be combined into one using Java scripts emulated in FVRML, which allows for even greater flexibility and convenience of shape modeling

In the recent years, the usage of 3D cyberworlds for educational purposes has increased. The metaphors behind the design of virtual places are quite diverse, from replication of real universities to art museums and scientific labs. Based on the results of a case study we have performed, this paper provides an initial set of requirements for a cyberworld representing an existing university. In this connection, we analyze place metaphors and associated design features of the virtual campus of Nanyang Technological University in Singapore in the context of related work. Finally, we discuss the correspondence between the identified metaphors and associated educational goals, providing directions for further development of the virtual campus

Shared virtual worlds require exchanging shape models over the Internet. Since complex shapes such as VRML objects are often defined with polygonal meshes, the size of models may become very large. It slows down their visualization and limits the model's precision. Using function-based models incorporated into VRML or other visualization frameworks allows for reducing the overall size of models as well as provides any required level of detail. In this paper, we propose a fast function-based shape modeling framework. The developed software is used for collaborative shape modeling in VRML shared virtual worlds. It allows for passive visualization of the complex function-based models in cyberworlds as well as for their interactive modeling.

In this paper we introduce a function-defined extension of Virtual Reality Modeling Language where analytical formulas are used for defining geometry and appearance of 3D shapes. By "functions" we understand analytical definitions with parametric, implicit and FRep formulas. In our model, different analytical representations can be used concurrently for defining geometry and appearance of the shapes. The proposed structure of the software allows for its extending with any proprietary function-based models. The extension is designed for collaborative shape modeling as well as for building shared virtual worlds. This is illustrated with the implementation of the collaborative shape modeling hands-on experience, which is a part of the Virtual Campus - a shared cyberworld of Nanyang Technological University.

This paper addresses interactive function-based shape modeling where relatively small formulas are used rather than thousands of polygons. Interactive modification of the function model with concurrent visualization of the respective polygonal mesh provides both interactivity and any required level of detail leading to a photorealistic appearance of the resulting shapes. We have proposed a rendering method capable of handling local shape modifications with any desired precision. We have also proposed methods which let us accelerate the final function evaluation - a common bottleneck for function-based shape modeling. Finally, we describe applications of interactive function-based shape modeling to photorealistic virtual embossing and carving.

This paper describes how the web-based visualization can be greatly improved using the function-based shape modeling technique. The improvement is possible because the proposed function-defined VRML shape node allows the content creators to describe any complex models with relatively small functions compared to the large-size polygonal mesh based VRML nodes. These function-defined shapes can be used together with the common VRML shapes. The proposed node has a few implementations capable of visualizing geometric shapes defined with HyperFun language as well as in any proprietary function-defined data formal. For fast visualization of the function-defined shapes, we have developed an improved continuation polygonization algorithm specifically designed for VRML visualization. The design, the implementation details, and the application examples of the proposed node are discussed.

This article describes how virtual embossing and wood cutting can be done using the function representation of a shape and tools. The software is implemented as an interactive shape modeler where a functional model of the shape is subsequently modified with offset and set-theoretic operations. For visualization, interactive ray tracing is used. Bounding boxes together with the spatial organization of the functional model provide the required fast function evaluation that is usually a bottleneck for functionally based shape modeling systems. The software runs on a personal computer. Categories and Subject Descriptors I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling - Curve, surface, solid, and object representations; I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism -- Raytracing, Virtual reality; I.3.8 [Computer Graphics]: Applications. Keywords Computer art, embossing, carving, virtual reality, functionally based shape modeling. 1

We present our approach to volume modelling which combines volume representations by voxel data and by continuous real functions. We discuss the main differences between direct volume visualization and modelling with voxel data, questions of conversion between two representations including volume reconstruction from contour data. We illustrate the approach by several advanced operations on a volumetric object: set-theoretic operations, sweeping, hypertexturing, feature-based sculpting, splitting, spatial and temporal transformations.

Studies a function representation of point sets swept by moving solids. The original solid-generator is defined by an inequality f(x,y,z,t)/spl ges/0 where x, y, z are Cartesian coordinates and t is treated as the time. This definition allows us to include solids which change their shapes in time. Constructive solids can be used as generators also when described by R-functions. The trajectory of the generator can be defined in parametric form as movement of its local coordinate system. In the paper, we did it with superposition of time-dependent affine transformations. To get the function representation F(x,y,z)/spl ges/0 of the swept solid, we apply the concept of an envelope, previously used basically for boundary represented objects. We have reduced the problem of swept solid description to a global extremum search by the t variable. The algorithm for procedural swept solid modeling is discussed. The benefit of our model is that it is applied not only for visualization but allows one to use the swept solid as an argument for other operations. For example, the swept solid can be intersected with other ones that are useful for the implementation of such operations as cutting and drilling. Ordinary texture mapping and hypertexturing can also be applied to it. The possibility of using a functionally defined generator with variable shape allows us to achieve a complexity of swept solids which was hardly possible before.