Dynamic object modeling is a critical challenge in 3D scene reconstruction. Previous methods typically maintain a canonical space to represent the object model, and a deformation field to express the object motion. However, this approach fails when the object undergoes large motions. The position variation caused by significant motion not only complicates the establishment of a canonical space, but also misleads the interpretation of the deformation field. To overcome the above weaknesses, we propose Motion Decoupled Dynamic 3D Gaussian Splatting (M5D-GS), the first 3D-GS model that separates motion and deformation modeling for dynamic object representation with large motion from a monocular camera. M5D-GS increases the practicality of 3D-GS, as it is common for objects to move, rotate, and deform simultaneously. Current datasets only contain object deformations with slight motions. We introduce a pipeline to reuse current benchmarks by adding large motions into the scene. We also introduce a new benchmark featuring several new scenes with complex motions, scenes augmented from previous datasets, and some real world recorded testcases, to fully demonstrate our improvements. Our M5D-GS significantly increases the accuracy under large motion scenarios while maintaining high rendering speed, which makes it suitable for dynamic object representation tasks including 4D novel view synthesis and real-time rendering.

Recent methods have made significant progress in optical flow estimation. However, the evaluation of these methods focuses mainly on improved accuracy in benchmarks and often overlooks the analysis of network robustness, which may be important in safety-critical scenarios such as autonomous driving. In this paper, we propose a novel method for robustness evaluation by modifying data from original benchmarks. Unlike previous benchmarks that focus on complex scenes, we propose to modify shape and texture of objects from the original images in order to analyze the sensitivity to these changes observed in the output. Our aim is to identify common failure cases of state-of-the-art (SOTA) methods to evaluate their robustness and understand their behaviors. We show that: Optical flow methods are more sensitive to shape changes than to texture changes; and optical flow methods tend to "remember" objects seen during training and may "ignore" the motion of unseen objects. Our experimental results and findings provide a more in-depth understanding of the behavior of recent optical flow methods. (See https://github.com/llesky/Robustness-Analysis-Optical-Flow).

Referring video object segmentation (RVOS) extracts objects from videos based on provided text narrations. Previous approaches typically work on all video frames simultaneously through offline processing, but this is not always possible. Offline processing becomes also ineffective for long videos, as directly cutting the video into short clips to fit memory limits results in the loss of temporal consistency. To make RVOS more applicable to real-world video streams or long video scenarios, we introduce a Filtering Framework for RVOS (FF-RVOS), the first model capable of operating in online, semi-online, and offline modes with just one training session. We redesign RVOS as a stochastic optimization problem and leverage filtering to optimize object states across temporal sequences. Our method enhances temporal consistency for video streams and when a long video is processed in shorter clip sequences due to memory limitations. FF-RVOS demonstrates superior performance compared to previous state-of-the-art methods on public benchmarks with clear improvements, especially when the video is cut into short clips for processing. Our framework can also be embedded into different offline methods to boost temporal consistency. The project page is https://github.com/haliphinx/FF-RVOS.

Unrealistic parser-based dialogue systems limit player agency. Large language model (LLM) characters can enhance agency but lack structure and measurable objectives. In this article, we propose a framework for structured interactions that tracks player progress through specific objectives, while also improving character LLM responses. This approach frames interactions as puzzles with states representing goal-based milestones. We employ an LLM to analyze dialogue history and enforce state transitions for state awareness and to enable specific actions like tailored LLM prompts and multimodal content changes. This results in a robust dialogue state tracking system for goal-based interactions. Using our method, a designer can craft transition rules as abstract goals that allow players to invent their own solutions rather than discovering the designer's intent. We demonstrate this with a hostage scenario game, where the player negotiates with a hostage-taker adversary. The game's effectiveness is assessed through qualitative gameplay analysis and a quantitative evaluation of our state tracking method.

Video frame interpolation based on optical flow has made great progress in recent years. Most of the previous studies have focused on improving the quality of clean videos. However, many real-world videos contain large obstructions making the video discontinuous. To address this challenge, we propose our Obstruction Robustness Framework (ORF) that enhances the robustness of existing VFI networks in the face of large obstructions. The ORF contains two components: (1) A feature repair module that first captures ambiguous pixels in the synthetic frame by a region similarity map, then repairs them with a cross-overlap attention module. (2) A data augmentation strategy that enables the network to handle dynamic obstructions without extra data. To the best of our knowledge, this is the first work that explicitly addresses the error caused by large obstructions in video frame interpolation. By using previous state-of-the-art methods as backbones, our method does not only improve the results in original benchmarks but also significantly enhances the interpolation quality for videos with obstructions.

The goal of Referring Video Object Segmentation is to extract an object from a video clip based on a given expression. While previous methods have utilized the transformer’s multi-modal learning capabilities to aggregate information from different modalities, they have mainly focused on spatial information and paid less attention to temporal information. To enhance the learning of temporal information, we propose TCE-RVOS with a novel frame token fusion (FTF) structure and a novel instance query transformer (IQT). Our technical innovations maximize the potential information gain of videos over single images. Our contributions also include a new classification of two widely used validation datasets for investigation of challenging cases. Our experimental results demonstrate that TCERVOS effectively captures temporal information and outperforms the previous state-of-the-art methods by increasing the J&F score by 4.0 and 1.9 points using ResNet-50 and VSwin-Tiny as the backbone on Ref-Youtube-VOS, respectively, and +2.0 mAP on A2D-Sentences dataset by using VSwin-Tiny backbone. The code is available at https://github.com/haliphinx/TCE-RVOS

Feature pyramids and iterative refinement have recently led to great progress in optical flow estimation. However, downsampling in feature pyramids can cause blending of foreground objects with the background, which will mislead subsequent decisions in the iterative processing. The results are missing details especially in the flow of thin and of small structures. We propose a novel Residual Feature Pyramid Module (RFPM) which retains important details in the feature map without changing the overall iterative refinement design of the optical flow estimation. RFPM incorporates a residual structure between multiple feature pyramids into a downsampling module that corrects the blending of objects across boundaries. We demonstrate how to integrate our module with two state-of-the-art iterative refinement architectures. Results show that our RFPM visibly reduces flow errors and improves state-of-art performance in the clean pass of Sintel, and is one of the top-performing methods in KITTI. According to the particular modular structure of RFPM, we introduce a special fine-tuning approach that can dramatically decrease the training time compared to a typical full optical flow training schedule on multiple datasets.

Our augmented reality online assistance platform enables an expert to specify 6DoF movements of a component and apply the geometrical and physical constraints in real-time. We track the real components on the expert’s side to monitor the operations of an expert. We leverage a remote rendering technique that we proposed previously to relieve the rendering burden of the augmented reality end devices. By conducting a user study, we show that the proposed method outperforms conventional instructional videos and sketches. The answers to the questionnaires show that the proposed method receives higher recommendation than sketching, and, compared to conventional instructional videos, is outstanding in terms of instruction clarity, preference, recommendation, and confidence of task completion. Moreover, as to the overall user experience, the proposed method has an advantage over the video method.

In this paper, we investigate effective neural network layers for optical flow estimation and in particular for omni-directional optical flow. Optical flow has many applications in computer graphics, augmented reality and in 3D modeling. We create a simple dataset that enables us to efficiently assess the effectiveness of different neural network layers for optical flow. Based on this small-sized diagnostic dataset, FlowCLEVR, we conclude that a deformable convolution layer is highly effective in reducing motion and occlusion boundary blur. Based on these results, we are able to design modifications to various existing network architectures improving their performance. We demonstrate improved performance on FlowCLEVR, on standard datasets for optical flow in planar images and on a novel omni-directional optical flow dataset. We also extend our work to omni-directional stereo.

In this paper, we propose a new feature representation for automatic classification of different tissue types in histopathological images. Our proposed approach is to combine different sets of features into a novel texture feature space. We compute all features in the Complex Shearlet domain, where we have not only examined the magnitude coefficients, but we also incorporate the relative phase (RP) coefficients to form the feature representation. Our Shearlet feature space consists of Haralick texture features, segmentation-based Fractal Texture Analysis, Local Binary patterns, and Local Oriented Statistic Information Booster. These descriptors are also used jointly to increase the classification accuracy of histopathological images when used with standard classifier. We demonstrate the effectiveness of our novel feature space with a support vector machine (SVM). The proposed features provide high classification performance competing with state-of-the-art systems equally on four publicly available datasets: Warwick-QU, Epistroma, BreaKHis, multi-class Kather.

In this paper we present an approach for 6 DoF panoramic videos from omni-directional stereo (ODS) images using convolutional neural networks (CNNs). More specifically, we use CNNs to generate panoramic depth maps from ODS images in real-time. These depth maps would then allow for re-projection of panoramic images thus providing 6 DoF to a viewer in virtual reality (VR). As the boundaries of a panoramic image must touch in order to envelope a viewer, we introduce a border weighted loss function as well as new error metrics specifically tailored for panoramic images. We show experimentally that training with our border weighted loss function improves performance by benchmarking a baseline skip-connected encoder-decoder style network as well as other state-of-the-art methods in depth map estimation from mono and stereo images. Finally, a practical application for VR using real world data is also demonstrated.

In this paper, we present a real-time approach for 3D object detection using a single, mobile and uncalibrated camera. We develop our algorithm using a feature-based method based on two novel naive Bayes classifiers for viewpoint and feature matching. Our algorithm exploits the specific structure of various binary descriptors in order to boost feature matching by conserving descriptor properties (e.g., rotational and scale invariance, robustness to illumination variations and real-time performance). Unlike state-of-the-art methods, our novel naive classifiers only require a database with a small memory footprint because we store efficiently encoded features. In addition, we also improve the indexing scheme to speed up the matching process. Because our database is built from powerful descriptors, only a few images need to be 'learned' and constructing a database for a new object is highly efficient.

The matching of oriented local image feature descriptors like SIFT, SURF or ORB often includes the refinement and filtering of matches based on the relative orientation of the features. This is important since the computational cost for subsequent tasks like camera pose estimation or object detection increases dramatically with the number of outliers. Simple 2D orientation descriptions are unsuitable for Omni directional images because of image distortions and non-monotonic mapping from camera rotations to image rotations. In this work we introduce 3D orientation descriptors which, unlike 2D descriptors, are suitable for match refinement on Omni directional images and improve matching results on images from cameras and camera rigs with a wide field of view. We evaluate different match refinement strategies based on 2D and 3D orientations and show the fundamental advantages of our approach.

Vortex methods increasingly receive attention from the computer graphics community for simple and direct modeling of complex flow phenomena such as turbulence. The coupling between free-form solids, represented by arbitrary surface meshes, and fluids simulated with vortex methods, leads to visually rich simulations. In this paper, we introduce a novel approach for simulating the interaction between solids and inviscid fluids for high-quality simulations using Lagrangian vortex particles. The key aspect of our method is simulating the creation of vorticity at a solid's surface. While previous vortex simulators only focus on modeling the solid as a boundary for the fluid, our approach allows the accurate simulation of two processes of visual interest. The first is the introduction of surface vorticity in the main flow as turbulence (vortex shedding). The second is the motion of the solid induced by fluid forces. We also introduce to computer graphics the concept of source panels to model nonturbulent flow around objects. To the best of our knowledge, this is the first work on two-way coupling of 3D solids and fluids using Lagrangian vortex methods in computer graphics.

Haptic texture represents the fine-grained attributes of an object's surface and is related to physical characteristics such as roughness and stiffness. We introduce an interactive and mobile scanning system for the acquisition and synthesis of haptic textures that consists of a visually tracked handheld touch probe. The most novel aspect of our work is an estimation method for the contact stiffness of an object based solely on the acceleration and forces measured during stroking of its surface with the handheld probe. We establish an experimental relationship between the estimated stiffness and the contact stiffness observed during compression. We also measure the height-displacement profile of an object's surface enabling us to generate haptic textures. We show an example of mapping the textures on to a coarse surface mesh obtained with an image-based technique, but the textures may also be combined with coarse surface meshes obtained by manual modeling.

Abstract Animation of viscoelastic solids in entertainment and medical applications as well as scientific simulation can be improved through observations of real world objects. This paper presents a method for simulating viscoelastic solids in real‐time for visual and haptic display along with a method for determining the parameters of the the underlying model from automated physical measurements of real world objects. The viscoelastic model is a novel extension of the discrete Green's function matrix for linear elasticity, which combines static behavior represented by Green's functions with dynamic behavior expressed by differential equations inspired by particle systems. We describe a novel estimation method of dynamic contact behavior for heterogeneous complex objects based on these measurements. For this estimation, our method relies only on measurement data previously used in the acquisition less realistic elastostatic models. In this way our method allows more physically accurate realism in animation of viscoelastic solids without large additional computational costs or any measurements besides those associated with related methods for elastostatic solids. Categories and Subject Descriptors (according to ACM CCS): I.3.5 [Computer Graphics]: Physically based modeling I.3.7 [Computer Graphics]: Virtual reality, animation

Translucent objects are characterized by diffuse light scattering beneath the object's surface. Light enters and leaves an object at possibly distinct surface locations. This paper presents the first method to acquire this transport behavior for arbitrary inhomogeneous objects. Individual surface points are illuminated in our DISCO measurement facility and the object's impulse response is recorded with a high-dynamic range video camera. The acquired data is resampled into a hierarchical model of the object's light scattering properties. Missing values are consistently interpolated resulting in measurement-based, complete and accurate representations of real translucent objects which can be rendered with various algorithms.

We present a method to estimate a positive semidefinite matrix by linear least squares, and we apply this method to the estimation of local compliance matrices during deformable object modeling. Estimation of physical quantities from measurements has to consider noise due to measurement and modeling inaccuracy. Enforcing constraints during estimation can guarantee physically plausible results even under difficult measurement conditions.

High quality, virtual 3D models are quickly emerging as a new multimedia data type with applications in such diverse areas as e-commerce, online encyclopedias, or virtual museums, to name just a few. We present new algorithms and techniques for the acquisition and real-time interaction with complex textured 3D objects and show how these results can be seamlessly integrated with previous work into a single framework for the acquisition, processing, and interactive display of high quality 3D models. In addition to pure geometry, such algorithms also have to take into account the texture of an object (which is crucial for a realistic appearance) and its reflectance behavior. The measurement of accurate material properties is an important step towards photorealistic rendering, where both the general surface properties as well as the spatially varying effects of the object are needed. Recent work on the image-based reconstruction of spatially varying BRDFs enables the generation of high quality models of real objects from a sparse set of input data. Efficient use of the capabilities of advanced PC graphics hardware allows for interactive rendering under arbitrary viewing and lighting conditions and realistically reproduces the appearance of the original object.

The estimation of the bi-directional reflectance distribution function (BRDF) of a 3D object requires reflectance measurements under numerous viewing and lighting directions. This sketch summarizes our method to select advantageous directions. Uncertainty minimization of the estimated BRDF parameters forms the theoretical underpinning of our acquisition planner. Our hardware-accelerated planner can aid manual and automatic measurement, reduces measurement effort and increases the quality of the acquired models.

Abstract This paper presents a rendering method for translucent objects, in which viewpoint and illumination can be modified at interactive rates. In a preprocessing step, the impulse response to incoming light impinging at each surface point is computed and stored in two different ways: The local effect on close‐by surface points is modeled as a per‐texel filter kernel that is applied to a texture map representing the incident illumination. The global response (i.e. light shining through the object) is stored as vertex‐to‐vertex throughput factors for the triangle mesh of the object. During rendering, the illumination map for the object is computed according to the current lighting situation and then filtered by the precomputed kernels. The illumination map is also used to derive the incident illumination on the vertices which is distributed via the vertex‐to‐vertex throughput factors to the other vertices. The final image is obtained by combining the local and global response. We demonstrate the performance of our method for several models. ACM CSS:I.3.7 Computer Graphics—Three‐Dimensional Graphics and Realism Color Radiosity

Abstract Measuring reflection properties of a 3D object involves capturing images for numerous viewing and lightingdirections. We present a method to select advantageous measurement directions based on analyzing the estimationof the bi‐directional reflectance distribution function (BRDF). The selected directions minimize the uncertaintyin the estimated parameters of the BRDF. As a result, few measurements suffice to produce models that describethe reflectance behavior well. Moreover, the uncertainty measure can be computed fast on modern graphics cardsby exploiting their capability to render into a floating‐point frame buffer. This forms the basis of an acquisitionplanner capable of guiding experts and non‐experts alike through the BRDF acquisition process. We demonstratethat spatially varying reflection properties can be captured more efficiently for real‐world applications using ouracquisition planner. Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Three‐DimensionalGraphics and Realism Virtual Reality I.4.1 [Computer Vision]: Digitization and Image Capture, Reflectance

We describe techniques for automatically acquiring observations of a deforming object and for estimating a model of the deformation from these observations. A robotic system was developed for measuring deformation, and has been previously reported (Pai et al., 2000, 2001). The present paper describes new techniques for the estimation of deformable models based on discrete Green's functions, from measurements acquired using stereo vision and a robot arm's position and force sensors. This kind of robotic measurement presents new challenges for measurement and estimation, that we address here. Our techniques for robotic acquisition of deformable, object models have potential applications in robotics, haptic interfaces, simulation, computer graphics and virtual reality.

This paper presents a rendering method for translucent objects, in which view point and illumination can be modified at interactive rates. In a preprocessing step the impulse response to incoming light impinging at each surface point is computed and stored in two different ways: The local effect on close-by surface points is modeled as a per-texel filter kernel that is applied to a texture map representing the incident illumination. The global response (i.e. light shining through the object) is stored as vertex-to-vertex throughput factors for the triangle mesh of the object. During rendering, the illumination map for the object is computed according to the current lighting situation and then filtered by the precomputed kernels. The illumination map is also used to derive the incident illumination on the vertices which is distributed via the vertex-to-vertex throughput factors to the other vertices. The final image is obtained by combining the local and global response. We demonstrate the performance of our method for several models.

We describe a system for constructing computer models of several aspects of physical interaction behavior, by scanning the response of real objects. The behaviors we can successfully scan and model include deformation response, contact textures for interaction with force-feedback, and contact sounds. The system we describe uses a highly automated robotic facility that can scan behavior models of whole objects. We provide a comprehensive view of the modeling process, including selection of model structure, measurement, estimation, and rendering at interactive rates. The results are demonstrated with two examples: a soft stuffed toy which has significant deformation behavior, and a hard clay pot which has significant contact textures and sounds. The results described here make it possible to quickly construct physical interaction models of objects for applications in games, animation, and e-commerce.