3D Gaussian splats (3DGS) have rapidly gained traction as a 3D scene representation technique that enables efficient real-time rendering and highfidelity novel view synthesis. This paper reports on ongoing MPEG GSC efforts conducted jointly by the MPEG Video Coding group (WG 4) and the Coding of 3D Graphics and Haptics group (WG 7) to define a practical and interoperable compression framework for 3DGS. MPEG GSC is planning GSC standardization with short-term and long-term timelines to address market requirements. The short-term objective is to standardize coding tools that build on proven MPEG ecosystems while introducing only the minimal set of extensions, syntax, and processing required for INRIA-3DGS format (referred to in MPEG as I-3DGS). In the long-term, MPEG is also investigating broader alternatives for 3DGS representation and compression, including approaches that integrate training during compression, collectively referred to as Alternative-3DGS (A-3DGS). More specifically, this paper focuses on the I-3DGS and explores both geometrybased and video-based coding frameworks within MPEG GSC.

3D dynamic meshes are extensively utilized to represent immersive 3D content. The need to transmit a vast quantity of these captured dynamic meshes over networks has created a significant demand for efficient dynamic mesh compression techniques. In response to this critical requirement, the Moving Picture Expert Group (MPEG), an international standard working group of ISO, initiated a new project in October 2021, called Video-based Dynamic Mesh Coding (V-DMC). This initiative seeks to leverage existing, mature video codecs and their prevalent hardware as well as direct usage of future video coding solutions. The inaugural version of the V-DMC test model was released in July 2022. Over the past two years, continuous efforts have led to the development of V-DMC test model version 8.0. This paper aims to detail the technical advancements and provide an update on the current status of V-DMC standardization activities.

In this paper, we introduce a novel 3D mesh convolution-based autoencoder for geometry compression, able to deal with irregular mesh data without requiring neither preprocessing nor manifold/watertightness conditions. The proposed approach extracts meaningful latent representations by learning features directly from the mesh faces, while preserving connectivity through dedicated pooling and unpooling operations. The encoder compresses the input mesh into a compact base mesh space, which ensures that the latent space remains comparable. The decoder reconstructs the original connectivity and restores the compressed geometry to its full resolution. Extensive experiments on multi-class datasets demonstrate that our method outperforms state-of-the-art approaches in both 3D mesh geometry reconstruction and latent space classification tasks.

Convolutional neural networks (CNNs) have been pivotal in various 2D image analysis tasks, including computer vision, image indexing and retrieval or semantic classification. Extending CNNs to 3D data such as point clouds and 3D meshes raises significant challenges since the very basic convolution and pooling operators need to be completely revisited and re-defined in an appropriate manner to tackle irregular connectivity issues. In this paper, we introduce MeshConv3D, a 3D mesh-dedicated methodology integrating specialized convolution and face collapse-based pooling operators. MeshConv3D operates directly on meshes of arbitrary topology, without any need of prior remeshing/conversion techniques. In order to validate our approach, we have considered a semantic classification task. The experimental results obtained on three distinct benchmark datasets show that the proposed approach makes it possible to achieve equivalent or superior classification results, while minimizing the related memory footprint and computational load.

Virtual reality (VR) training and teleoperation tech-nologies play a pivotal role in revolutionizing workforce de-velopment and remote machinery operation in Industry 4.0. By providing immersive, interactive, and cost-effective training solutions, they thereby enhance operational efficiency and safety standards. However, to achieve such results, they need the right tool for the right task, or in the case of VR, the right controller for the right application. This study assesses the performance of different controller types, including gaming gamepads, traditional VR controllers, and sim-ulators joysticks, in terms of user experience, task performance, and ease of use. Through a series of experiments and user feedback analyses, we identify the strengths and limitations of each controller. The results show that the joysticks, the controllers closest to the real-life interface present in excavators, are the most suited interface for our VR earthwork application despite the lack of user familiarity with this controller.

Point clouds, crucial for representing 3D objects and scenes, offer immersive and precise depictions of the real world. Despite their superiority, the substantial data volume challenges current multimedia ecosystems. To address this, the Moving Picture Expert Group (MPEG) initiated the point cloud compression project in 2017, leading to two branches: Video-based Point Cloud Compression (V-PCC) and Geometry-based Point Cloud Compression (G-PCC). The first edition of G-PCC was published in March 2023, and ongoing efforts over the past three years have advanced towards G-PCC Edition 2. This paper aims to present recent technical achievements and the current status of G-PCC standardization activities.

Recently, high-quality, humanoid-like 3D point clouds have become extensively used in various use cases related to VR/AR applications. Such high-density point clouds, represented by a huge number of points (e.g., 1 million points) carrying various photometric attributes, require efficient compression techniques for storage and transmission. However, most research works in the literature mainly focus on geometry compression, while only a few consider the spatio-temporal compression of color attributes. In this paper, we propose a novel color attribute prediction method, which exploits a skeleton-based affine motion estimation technique. The skeleton and the motion parameters are compressed in a lossless manner, to preserve accurate color prediction. The color residuals are lossy compressed using a video-based coding solution. Our proposal has been integrated into the Video-based Point Cloud Compression (V-PCC) test model of MPEG. The experimental results demonstrate that the proposed method outperforms the reference V-PCC test model, notably in low bitrate conditions.

Introducing sparsity in a convnet has been an efficient way to reduce its complexity while keeping its performance almost intact. Most of the time, sparsity is introduced using a three-stage pipeline: 1) training the model to convergence, 2) pruning the model, 3) fine-tuning the pruned model to recover performance. The last two steps are often performed iteratively, leading to reasonable results but also to a time-consuming process. In our work, we propose to remove the first step of the pipeline and to combine the two others in a single training-pruning cycle, allowing the model to jointly learn the optimal weights while being pruned. We do this by introducing a novel pruning schedule, named One-Cycle Pruning (OCP), which starts pruning from the beginning of the training, and until its very end. Experiments conducted on a variety of combinations between architectures (VGG-16, ResNet-18), datasets (CIFAR-10, CIFAR-100, Caltech-101), and sparsity values (80%, 90%, 95%) show that not only OCP consistently outperforms common pruning schedules such as One-Shot, Iterative and Automated Gradual Pruning, but also that it drastically reduces the required training budget. More-over, experiments following the Lottery Ticket Hypothesis show that OCP allows to find higher quality and more stable pruned networks.

This paper reports on a user study to comparatively evaluate two AR training systems designed for step-by-step manual operations: ATOFIS - recently proposed in the literature, and Microsoft Dynamics 365 Guides (hereinafter Guides) - one of the most relevant state-of-the-art commercial solutions. The user study (N=16) was conducted in two stages- i.e., training and authoring, on a partial replica of a real-world assembly workstation. During training, the participant learns a sequence of manual operations by performing two assembly cycles, guided by each of the two AR training systems. During authoring, the participant creates the two sets of AR work instructions used in the next training session, one set with each of the two authoring systems. We bound the authoring and training procedures during the experiment to comparatively assess the AR systems overall, and address at the same time an evaluation gap observed in the literature. The experimental results demonstrated advantages of the authoring approach proposed by ATOFIS (i.e., low-cost, formalized, in-situ, immersive and on-the-fly), proved the usability and effectiveness of the AR instructions authored with ATOFIS and validated a set of hypotheses formulated by the authors of the system. ATOFIS authoring was $1.72\times$ faster and unanimously preferred by the participants; ATOFIS training reported zero assembly errors and was 13% faster than Guides. ATOFIS reported excellent system usability (i.e., SUS) and mental workload (i.e., NASA-TLX) scores for both authoring and training, outperforming Guides on all dimensions.

Augmented Reality (AR) is an emerging technology offering a great potential in assisting humans in a wide range of industrial processes, from manufacturing to validation and maintenance. However, very few AR solutions have been adopted so far in industrial sectors, mainly because of technical and acceptability issues. This paper has three main contributions: (1) it identifies potential barriers for AR adoption in manufacturing environments; (2) it proposes an AR training methodology that overcomes these challenges and finally, (3) it evaluates the proposed AR training approach in a concrete, real-world use case by conducting a field experiment with 12 participants. Our findings indicate that low-cost assistive visual assets (i.e. text, images, videos and arrows) can be sufficient for effectively delivering complex manual assembly instructions through AR head-mounted displays (i.e. Hololens 2). We finally discuss how the proposed methodology simplifies considerably the authoring of the AR instructions and how this technique can potentially be generalized to other similar industrial scenarios.

In this article, a survey of the point cloud compression (PCC) methods by organizing them with respect to the data structure, coding representation space, and prediction strategies is presented. Two paramount families of approaches reported in the literature-the projection- and octree-based methods-are proven to be efficient for encoding dense and sparse point clouds, respectively. These approaches are the pillars on which the Moving Picture Experts Group Committee developed two PCC standards published as final international standards in 2020 and early 2021, respectively, under the names: video-based PCC and geometry-based PCC. After surveying the current approaches for PCC, the technologies underlying the two standards are described in detail from an encoder perspective, providing guidance for potential standard implementors. In addition, experiment evaluations in terms of compression performances for both solutions are provided.

The aim of this work is to present an open solution for building an Augmented Reality (AR) system without using any existing SDK. The proposed approach relies upon 2D planar object recognition for mobile real-time tracking applications. The transformation relating the world and the camera coordinate systems is determined using pose estimation. Once the projective transform relating 3D and 2D features is computed, a virtual 3D graphic is registered on the image. Many tests have been performed to show the efficiency of the proposed approach and to prove its relevance in terms of accuracy and time computation. The final application enabled real-time mobile tracking of markerless images augmented with 3D models to enrich the visual perception of the user.

This paper outlines the objectives of the working group on developing a model Augmented Reality curriculum for higher education. We motivate the need for the model curriculum by the growing Augmented Reality industry and subsequent demand for trained professionals. While the industry is growing, the educational offers that train the required skills remain limited and fragmented. The working group will address this challenge by surveying the state of the art in Augmented Reality education are reviewing available data on industry requirements. Based on the results, the group will develop a new model Augmented Reality curriculum. The working group will also develop future work recommendations for the design of teaching materials and integration of Augmented Reality in computing curricula.

With the rapid development of point cloud acquisition technologies, high-quality human-shape point clouds are more and more used in VR/AR applications and in general in 3D Graphics. To achieve near-realistic quality, such content usually contains an extremely high number of points (over 0.5 million points per 3D object per frame) and associated attributes (such as color). For this reason, disposing of efficient, dedicated 3D Point Cloud Compression (3DPCC) methods becomes mandatory. This requirement is even stronger in the case of dynamic content, where the coordinates and attributes of the 3D points are evolving over time. In this paper, we propose a novel skeleton-based 3DPCC approach, dedicated to the specific case of dynamic point clouds representing humanoid avatars. The method relies on a multi-view 2D human pose estimation of 3D dynamic point clouds. By using the DensePose neural network, we first extract the body parts from projected 2D images. The obtained 2D segmentation information is back-projected and aggregated into the 3D space. This procedure makes it possible to partition the 3D point cloud into a set of 3D body parts. For each part, a 3D affine transform is estimated between every two consecutive frames and used for 3D motion compensation. The proposed approach has been integrated into the Video-based Point Cloud Compression (V-PCC) test model of MPEG. Experimental results show that the proposed method, in the particular case of body motion with small amplitudes, outperforms the V-PCC test mode in the lossy inter-coding condition by up to 83% in terms of bitrate reduction in low bit rate conditions. Meanwhile, the proposed framework holds the potential of supporting various features such as regions of interests and level of details.

In this paper we present a novel approach for object identification and tracking in large image datasets. Objects of interest are represented by feature points and descriptors extracted and compared to a set of reference data. An optimized matching paradigm is designed to deal with scalable image databases while keeping a good recognition rate in real-life environment conditions. Experiments are conducted to evaluate the effectiveness of the method and the obtained results demonstrate a true interest of the proposed approach.

Today an experience developer must choose tools for authoring AR experiences based on many factors including ease of use, performance across a variety of platforms, reach and discoverability and cost. The commercially viable options are organized in closed technology silos (beginning with SDKs). A publisher of experiences must choose one or develop for multiple viewing applications, then promote one or more application to the largest possible audience. Developers of applications must then maintain the customized viewing application over time across multiple platforms or have the experience (and the application) expire at the end of a campaign.

This paper presents an approach for tracking natural objects in augmented reality applications. The targets are detected and identified using a markerless approach relying upon the extraction of image salient features and descriptors. The method deals with large image databases using a novel strategy for feature retrieval and pairwise matching. Further-more, the developed method integrates a real-time solution for 3D pose estimation using an analytical technique based on camera perspective transformations. The algorithm associates 2D feature samples coming from the identification part with 3D mapped points of the object space. Next, a sampling scheme for ordering correspondences is carried out to establishing the 2D/3D projective relationship. The tracker performs localization using the feature images and 3D models to enhance the scene view with overlaid graphics by computing the camera motion parameters. The modules built within this architecture are deployed on a mobile platform to provide an intuitive interface for interacting with the surrounding real world. The system is experimented and evaluated on challenging scalable image dataset and the obtained results demonstrate the effectiveness of the approach towards versatile augmented reality applications.

In this work we present a novel interaction approach based on a gesture recognition system using a Microsoft Kinect sensor. Gestures are defined and interpreted in order to activate controls on a media device. This natural interface enables intuitive interaction with the multimedia content. The depth sensor observes the scene to detect a request for control in the form of a gesture. Then, the application assigns a control for the media system. The application is tested under various scenarios and proved the reliability and the effectiveness of the proposed approach.

Online Engineering allows users to perform tele-operation over the Internet. It is used in collaboratories, distance learning curriculum, and remote maintenance processes. Tele-operation over the Internet is however restrained by the development time and cost of ad hoc solutions. These solutions a) are hardly reusable, b) present a low fidelity of the human computer interface, and c) barely enable pervasive accesses (standalone clients or Web forms). We present in this paper a novel approach for building tele operation Human Computer Interfaces based on interactive multimedia. The end-user commands the remote device using interactive elements in the multimedia interface. The feedback information is seen as a combination of natural video content, produced by an IP camera capturing the instrument and synthetic graphics corresponding to elements of the device input interface (control panel). The proposed architecture fosters the reuses and gives a high level of interoperability between command terminals, since the interface can be displayed in any terminal able to lay out multimedia content.

In this paper, we propose an authoring tool for creating mobile mixed reality games (MMRGs), referred to as M2RGC (Mobile Mixed Reality Game Creator). Goal is to transform the game creation process in a task that is as interesting as playing the game itself. Therefore we offer a user-friendly interface through which different MMRGs can be created by defining the objects used in the game, their behavior and interaction with sensors (e.g. GPS, compass). The tool itself uses the power of MPEG-4 BIFS[1] capabilities, such as the possibility of grouping multimedia assets in a scene. We build a wrapper over these MPEG-4 BIFS functionalities allowing anyone to discover the experience of MMRGs creation. Concerning the client runtime for playing the game on mobile phones, we deploy a MPEG-4 player on the Windows Mobile and Android platforms.

This work aims to realize a recognition system for a software engine that will automatically generate a quiz starting from a video content and reinsert it into the video, turning thus any available foreign-language video (such as news or TV series) into a remarkable learning tool. Our system includes a face tracking application which integrates the eigen face method with a temporal tracking approach. The main part of our work is to detect and identify faces from movies and to associate specific quizzes for each recognized character. The proposed approach allows to label the detected faces and maintains face tracking along the video stream. This task is challenging since characters present significant variation in their appearance. Therefore, we employed eigen faces to reconstruct the original image from training models and we developed a new technique based on frames buffering for continuous tracking in unfavorable environment conditions. Many tests were conducted and proved that our system is able to identify multiple characters. The obtained results showed the performance and the effectiveness of the proposed method.

This paper proposes a low computational method to perform ROI (Region Of Interest) based video encoding and adaptive streaming for remote rendering applications. The main objective of the proposed solution is to minimize the latency in the interactive loop even when facing poor transmission conditions. In order to do that, the knowledge of the depth map information provided by the rendering engine is exploited by the real-time video encoder to adapt the bitrate of the transmitted stream. Especially, thanks to an efficient coupling between the rendering and the video encoding stages, the macroblocks of each video frame are encoded with different quantization steps that follow an ROI partitioning. The details of this partitioning algorithm are provided as well with some implementation considerations. The simulation results demonstrate the benefit of our adaptive approach from the user experience point of view.

This paper presents a survey on current trends in several technologies related to avatars and introduces a new solution for interoperability of avatars in Virtual Worlds. A short examination of several standards, recommendations, markup languages and VWs presentation definitions addressing different aspects of avatars are provided. As a result of this analysis, a new standard, called MPEG-V, aiming to provide an interchange format for virtual worlds is presented. Providing the container of avatar properties/capabilities is the vision of MPEG-V. By its descriptive format it aims to facilitate the deployment of VW recognizing that their success depends on maintaining acceptable development cost and one mechanism in doing so consists in ensuring interoperability between them at least at the level of avatars.

In this paper we present an on-line platform, called MyMultiMediaWorld.com, designed to help standardization bodies and communities to benchmark compression algorithms for still 3D objects. Based on MPEG standards, this platform provides an easy-to-use API to integrate the algorithms to be evaluated. The platform automatically benchmarks them from the platform extensible 3D database. When uploading new 3D contents, benchmark results for the various available compression schemes are automatically updated. Filtering tools are provided to help performance assessment at scales ranging from individual 3D content to database subsets.

Most current multi-player 3D games can only be played on dedicated platforms, requiring specifically designed content and communication over a prede-fined network. To overcome these limitations, the OLGA (On-Line GAming) consortium has devised a framework to develop distributive, multi-player 3D games. Scalability at the level of content, platforms and networks is exploited to achieve the best trade-offs between complexity and quality. Besides, stan-dardized content compression formats (MPEG-4, JPEG 2000) are used in OLGA's framework, enabling easy deployment over existing infrastructure, while keeping hooks to well-established practices in the game industry.

The paper gives a short overview on the new feature of the recent MPEG-4 standard, namely interactive facilitation of multimedia contents. Some experiments on dealing with BIFS and MPEG-J - the two main motors enabling the interactivity in MPEG-4 - are presented. These experiments serve for two targets. First, they demonstrate a very perspective capability of MPEG-4 based multimedia contents, which are not only flexible in composition but also make interaction with viewer possible. Next, they expose a gap occurring in merging the computing and broadcasting technology in MPEG-4, namely temporal constraint versus random access. The work also outlines some solution to overcome this problem. The goal is to improve the performance of the interactivity enabling feature so that MPEG-4 standard actually integrates the computing capability into compression technique, enabling the generation of sophisticated multimedia scene.

Enriched multimedia applications and services aim at combining images, sounds, videos, and synthetic objects into hybrid and interactive scenes. The core technologies discussed here deal with the representation and integration within such complex scenes of a specific kind of synthetic data, namely virtual character animation. This paper analyzes how an integrated and standardized framework is currently emerging in order to ensure application interoperability, universal content access, and user interactivity. We first compare how virtual character animation has been addressed within virtual reality modeling language (VRML) and MPEG-4 synthetic and natural hybrid coding standardization processes. A comparative synthesis between the objectives and the capabilities of each framework is exposed, specifically MPEG-4 Face and Body Animation (FBA) versus H-Anim (Humanoid Animation Working Group, WEB3D Consortium, h-anim.org.) 1.1, and MPEG-4 Bone-Based Animation (BBA) versus H-Anim 2001. The Animation Framework eXtension (AFX) specifications that are a part of MPEG-4 Systems Part 16 include the BBA framework. The BBA animation concepts, based on generic skeleton representation and curve-based deformations, are introduced. The definition of the related nodes and how they successfully address the BBA concepts are as well discussed. Some comments with respect to the rotation representation, the interpolation methods, the animation mask, and value parameters of the animation stream are made. This set of specifications provides an efficient framework, which is appropriate to real-time and animation realistic applications within networked environments.

This paper explores the new features of the recent MPEG- 4 standard, namely the description and the composition of an interactive multimedia scene. Providing with such advanced facilities, the MPEG-4 standard is no longer considered only as an efficient compression tool for conventional video / audio application. It is now granted the power of computing mechanism as any other programming languages to facilitate the creation of high interactive multimedia contents. Game-applications therefore can be also considered as a new output-feature of the MPEG-4 standard. The work describes in detail how game can be created within the MPEG-4 framework. The process of making game with MPEG-4 is studied comprehensively to demonstrate the compatibility as well as the advantages of the MPEG-4 standard regarding to Flash technology, a well-known tool for making games over the Internet.

The MPEG-4 face and body animation (FBA) specifications aims at standardizing an interchange format for specifying virtual face and body modeling and related animation parameters. The scene graph concept, the FBA nodes, semantics and representation of the animation parameters as well as compression and streaming capabilities are successively addressed. The compatibility between virtual character animation techniques and MPEG-4 FBA is analyzed in terms of modeling, deformation behavior and animation capabilities.

The paper presents an advanced animation framework for virtual characters within the MPEG-4 standard. The bone-based animation (BBA) specifications are described and it is shown how they extend the MPEG-4 face and body animation (FBA) specifications. Specifically, the scene graph concept, the BBA nodes related to bone and muscle modeling, the animation parameters and the compression capabilities are successively addressed and discussed within a specific application related to a sign language communication system.