The ACM Multimedia 2025 Industry Program presents a comprehensive overview of how multimodal AI is revolutionizing real-world applications. The program features contributions from over twenty industry leaders, covering a spectrum of domains from content creation and distribution to healthcare, manufacturing, and foundational technology. Keynotes by leaders from NEC and Google DeepMind address critical challenges in business transformation and media integrity, respectively. A dedicated seminar from Google explores advancements in video codecs like AV2 and the development of next-generation quality metrics using Large Language Models (LLMs). The program also includes twelve expert talks and seven demonstrations that showcase practical innovations, such as LLM-driven recommendation systems at Meta, generative AI for industrial optimization by Mitsubishi Electric, and a Siemens-developed protocol for semantic interoperability in manufacturing. These components collectively highlight the profound societal and industrial impact of multimedia research and bridge the gap between academic theory and real-world deployment.

The social metaverse is a growing digital ecosystem that blends virtual and physical worlds. It allows users to interact socially, work, shop, and enjoy entertainment. However, privacy remains a major challenge, as immersive interactions require continuous collection of biometric and behavioral data. At the same time, ensuring high-quality, low-latency streaming is difficult due to the demands of real-time interaction, immersive rendering, and bandwidth optimization. To address these issues, we propose adaptive social metaverse streaming (ASMS), a novel streaming system based on federated multiagent proximal policy optimization (F-MAPPO). ASMS leverages F-MAPPO, which integrates federated learning (FL) and deep reinforcement learning (DRL) to dynamically adjust streaming bit rates while preserving user privacy. Experimental results show that ASMS improves user experience by at least 14% compared to existing streaming methods across various network conditions. Therefore, ASMS enhances the social metaverse experience by providing seamless and immersive streaming, even in dynamic and resource-constrained networks, while ensuring that sensitive user data remain on local devices.

Extended reality (XR) is rapidly advancing and poised to revolutionize content creation and consumption. In XR, users integrate various sensory inputs to form a cohesive perception of the virtual environment. This survey reviews the state-of-the-art in XR streaming, focusing on multiple paradigms. To begin, we define XR and introduce various XR headsets along with their multimodal interaction methods to provide a foundational understanding. We then analyze XR traffic characteristics to highlight the unique data transmission requirements. We also explore factors that influence the quality of experience in XR systems, aiming to identify key elements for enhancing user satisfaction. Following this, we present visual attention-based optimization methods for XR streaming to improve efficiency and performance. Finally, we examine current applications and highlight challenges to provide insights into ongoing and future developments of XR.

The ACM Multimedia 2024 industry program offers a unique platform for fostering collaboration between academia and industry. This year's program features a diverse range of industry keynotes, expert talks, seminars, and demonstrations, showcasing the latest advancements in multimedia technology. Renowned experts from industry and academia will share their insights on topics such as generative AI, automotive design, computer vision, spatial experience, healthcare, and more. Attendees will have the opportunity to network with industry leaders, learn about cutting-edge technologies, and explore potential collaborations. The industry program highlights the growing importance of multimedia technology in various domains and demonstrates the innovative ways in which AI and other emerging technologies are transforming industries. By participating in this program, attendees can gain valuable knowledge, expand their professional networks, and contribute to the advancement of the field.

Over the last few years, 360video traffic on the network has grown significantly. A key challenge of 360video playback is ensuring a high quality of experience (QoE) with limited network bandwidth. Currently, most studies focus on tile-based adaptive bitrate (ABR) streaming based on single viewport prediction to reduce bandwidth consumption. However, the performance of models for single-viewpoint prediction is severely limited by the inherent uncertainty in head movement, which can not cope with the sudden movement of users very well. This paper first presents a multimodal spatial-temporal attention transformer to generate multiple viewpoint trajectories with their probabilities given a historical trajectory. The proposed method models viewpoint prediction as a classification problem and uses attention mechanisms to capture the spatial and temporal characteristics of input video frames and viewpoint trajectories for multi-viewpoint prediction. After that, a multi-agent deep reinforcement learning (MADRL)-based ABR algorithm utilizing multi-viewpoint prediction for 360video streaming is proposed for maximizing different QoE objectives under various network conditions. We formulate the ABR problem as a decentralized partially observable Markov decision process (Dec-POMDP) problem and present a MAPPO algorithm based on centralized training and decentralized execution (CTDE) framework to solve the problem. The experimental results show that our proposed method improves the defined QoE metric by up to 85.5% compared to existing ABR methods.

With the continuous evolution of networking technologies, multi-modal services that involve video, audio, and haptic contents are expected to become the dominant multimedia service in the near future. Edge caching is a key technology that can significantly reduce network load and content transmission latency, which is critical for the delivery of multi-modal contents. However, existing caching approaches only rely on a limited number of factors, e.g., popularity, to evaluate their importance for caching, which is inefficient for caching multi-modal contents, especially in dynamic network environments. To overcome this issue, we propose a content importance-based caching scheme which consists of a content importance evaluation model and a caching model. By leveraging dueling double deep Q networks (D3QN) model, the content importance evaluation model can adaptively evaluate contents' importance in dynamic networks. Based on the evaluated contents' importance, the caching model can easily cache and evict proper contents to improve caching efficiency. The simulation results show that the proposed content importance-based caching scheme outperforms existing caching schemes in terms of caching hit ratio (at least 15% higher), reduced network load (up to 22% reduction), average number of hops (up to 27% lower), and unsatisfied requests ratio (more than 47% reduction).

Various congestion control protocols have been designed to achieve high performance in different network environments. Modern online learning solutions that delegate the congestion control actions to a machine cannot properly converge in the stringent time scales of data centers. We leverage multi-agent reinforcement learning to design a system for dynamic tuning of congestion control parameters at end-hosts in a data center. The system includes agents at the end-hosts to monitor and report the network and traffic states, and agents to run the reinforcement learning algorithm given the states. Based on the state of the environment, the system generates congestion control parameters that optimize network performance metrics such as throughput and latency. As a case study, we examine BBR, an example of a prominent recently-developed congestion control protocol. Our experiments demonstrate that the proposed system has the potential to mitigate the problems of static parameters.

Multiaccess edge computing (MEC) is a promising solution to the computation-intensive, low-latency rendering tasks of the metaverse. However, how to optimally allocate limited communication and computation resources at the edge to a large number of users in the metaverse is quite challenging. In this article, we propose an adaptive edge resource allocation method based on multiagent soft actor–critic with graph convolutional networks (SAC-GCN). Specifically, SAC-GCN models the multiuser metaverse environment as a graph where each agent is denoted by a node. Each agent learns the interplay between agents by graph convolutional networks with a self-attention mechanism to further determine the resource usage for one user in the metaverse. The effectiveness of SAC-GCN is demonstrated through the analysis of user experience, balance of resource allocation, and resource utilization rate by taking a virtual city park metaverse as an example. Experimental results indicate that SAC-GCN outperforms other resource allocation methods in improving overall user experience, balancing resource allocation, and increasing resource utilization rate by at least 27%, 11%, and 8%, respectively.

With the increase in health consciousness, noninvasive body monitoring has aroused interest among researchers. As one of the most important pieces of physiological information, researchers have remotely estimated the heart rate (HR) from facial videos in recent years. Although progress has been made over the past few years, there are still some limitations, like the processing time increasing with accuracy and the lack of comprehensive and challenging datasets for use and comparison. Recently, it was shown that HR information can be extracted from facial videos by spatial decomposition and temporal filtering. Inspired by this, a new framework is introduced in this paper to remotely estimate the HR under realistic conditions by combining spatial and temporal filtering and a convolutional neural network. Our proposed approach shows better performance compared with the benchmark on the MMSE-HR dataset in terms of both the average HR estimation and short-time HR estimation. High consistency in short-time HR estimation is observed between our method and the ground truth.

This article presents an image-based real-time facial expression recognition system that is able to recognize the facial expressions of several subjects on a webcam at the same time. Our proposed methodology combines a supervised transfer learning strategy and a joint supervision method with center loss, which is crucial for facial tasks. A newly proposed Convolutional Neural Network (CNN) model, MobileNet, which has both accuracy and speed, is deployed in both offline and in a real-time framework that enables fast and accurate real-time output. Evaluations towards two publicly available datasets, JAFFE and CK+, are carried out respectively. The JAFFE dataset reaches an accuracy of 95.24%, while an accuracy of 96.92% is achieved on the 6-class CK+ dataset, which contains only the last frames of image sequences. At last, the average run-time cost for the recognition of the real-time implementation is around 3.57ms/frame on a NVIDIA Quadro K4200 GPU.

Context-aware recommendations offer the potential of exploiting social contents and utilize related tags and rating information to personalize the search for content considering a given context. Recommendation systems tackle the problem of trying to identify relevant resources from the vast number of choices available online. In this study, we propose a new recommendation model that personalizes recommendations and improves the user experience by analyzing the context when a user wishes to access multimedia content. We conducted empirical analysis on a dataset from last.fm to demonstrate the use of latent preferences for ranking items under a given context. Additionally, we use an optimization function to maximize the mean average precision measure of the resulted recommendation. Experimental results show a potential improvement to the quality of the recommendation in terms of accuracy when compared with state-of-the-art algorithms.

Three-dimensional (3D) sensing and printing technologies have reshaped our world in recent years. In this article, a comprehensive overview of techniques related to the pipeline from 3D sensing to printing is provided. We compare the latest 3D sensors and 3D printers and introduce several sensing, postprocessing, and printing techniques available from both commercial deployments and published research. In addition, we demonstrate several devices, software, and experimental results of our related projects to further elaborate details of this process. A case study is conducted to further illustrate the possible tradeoffs during the process of this pipeline. Current progress, future research trends, and potential risks of 3D technologies are also discussed.

In this paper, we introduce an MPEG-V based haptic authoring tool intended for simplifying the development process of haptics-enabled multimedia applications. The developed tool provides a web-based interface for users to create haptic environments by importing 3D models and adding haptic properties to them. The user can then export the resulting environment to a standard MPEG-V format. The latter can be imported to a haptic player that renders the described haptics-enabled 3D scene. The proposed tool can support many haptic devices, including Geomagic Devices, Force Dimension Devices, Novint Falcon Devices, and Moog FCS HapticMaster Devices. We conduct a proof of concept HTML5 haptic game project and user studies on haptic effects, development process and user interface, which shows our tool's effectiveness in simplifying the development process of haptics-enabled multimedia applications.

With the introduction of new depth sensing technologies, interactive hand-gesture devices are rapidly emerging. However, the hand-gestures used in these devices do not follow a common vocabulary, making certain control command device-specific. In this paper we present an initial effort to create a standardized interactive hand-gesture vocabulary for the next generation of television applications. We conduct a user-elicitation study using a survey in order to define a common vocabulary for specific control commands, such as Volume up/down, Menu open/close, etc. This survey is entirely user-oriented and thus it has two phases. In the first phase, we ask open questions about specific commands. In the second phase, we use the answers suggested from the first phase to create a multiple choice questionnaire. Based on the results from the survey, we study the gesture attitudes and preferences between gender groups, and between age groups with a quantitative and qualitative statistical analysis. Finally, the hand-gesture vocabulary is derived after applying an agreement analysis on the user-elicited gestures. The proposed methodology for gesture set design is comparable with existing methodologies and yields higher agreement levels than relevant user-elicited studies in the field.

We propose a 6D RGB-D odometry approach that finds the relative camera pose between consecutive RGB-D frames by keypoint extraction and feature matching both on the RGB and depth image planes. Furthermore, we feed the estimated pose to the highly accurate KinectFusion algorithm, which uses a fast ICP (Iterative Closest Point) to fine-tune the frame-to-frame relative pose and fuse the depth data into a global implicit surface. We evaluate our method on a publicly available RGB-D SLAM benchmark dataset by Sturm et al. The experimental results show that our proposed reconstruction method solely based on visual odometry and KinectFusion outperforms the state-of-the-art RGB-D SLAM system accuracy. Moreover, our algorithm outputs a ready-to-use polygon mesh (highly suitable for creating 3D virtual worlds) without any postprocessing steps.

We focus on generating consistent reconstructions of indoor spaces from a freely moving handheld RGB-D sensor, with the aim of creating virtual models that can be used for measuring and remodeling. We propose a novel 6D RGBD odometry approach that finds the relative camera pose between consecutive RGB-D frames by keypoint extraction and feature matching both on the RGB and depth image planes. Furthermore, we feed the estimated pose to the highly accurate KinectFusion algorithm, which uses a fast ICP (Iterative-Closest-Point) to fine-tune the frame-to-frame relative pose and fuse the Depth data into a global implicit surface. We evaluate our method on a publicly available RGB-D SLAM benchmark dataset by Sturm et al. The experimental results show that our proposed reconstruction method solely based on visual odometry and KinectFusion outperforms the state-of-the-art RGB-D SLAM system accuracy. Our algorithm outputs a ready-to-use polygon mesh (highly suitable for creating 3D virtual worlds) without any post-processing steps.

In this paper, we propose an “anti-fatigue” control method for bionic actuated systems. Specifically, the proposed method is illustrated on an over-actuated bionic arm. Our control method consists of two steps. In the first step, a set of linear equations is derived by connecting the acceleration description in both joint and muscle space. The pseudo inverse solution to these equations provides an initial optimal muscle force distribution. As a second step, we derive a gradient direction for muscle force redistribution. This allows the muscles to satisfy force constraints and generate an even distribution of forces throughout all the muscles (i.e. towards "anti-fatigue"). The overall proposed method is tested for a bending-stretching movement. We used two models (bionic arm with 6 and 10 muscles) to verify the method. The force distribution analysis verifies the “anti-fatigue” property of the computed muscle force. The efficiency comparison shows that the computational time does not increase significantly with the increase of muscle number. The tracking error statistics of the two models show the validity of the method.

Redundant muscle-driven arms have numerous advantages, such as increased robustness, ability for load distribution, impedance change etc. However, controlling such a muscle-driven arm is a difficult task. This is mainly due to its redundancy, specially when the muscle force is required to follow certain output constraints and fulfill optimization objectives. In this paper, a new method for controlling such muscle-like systems is proposed. By considering both joint and muscle acceleration contributions, a set of linear equations was constructed. Driving muscle activation is thus framed as the only unknown vector. To solve this linear equation set, a pseudo-inverse solution was used. The null space within this solution represents the internal force, which was used to evenly distribute the muscle forces, which is considered as "anti-fatigue" way. Moreover, to make the proposed method adaptive to modeling errors, an estimated system model was added to represent the real model. By updating the parameters of the estimated model based on prediction error, the estimated model approaches the real model gradually in real time. The overall method was tested for the case of a bending-stretching movement. The presented results verify the validity of the method, and illustrate its useful features and advantages.

In this paper, we introduce the From Sense to Print system. It is a system where a 3D sensing device connected to the cloud is used to reconstruct an object or a human and generate 3D CAD models which are sent automatically to a 3D printer. In other words, we generate ready-to-print 3D models of objects without manual intervention in the processing pipeline. Our proposed system is validated with an experimental prototype using the Kinect sensor as the 3D sensing device, the KinectFusion algorithm as our reconstruction algorithm and a fused deposition modeling (FDM) 3D printer. In order for the pipeline to be automatic, we propose a semantic segmentation algorithm applied to the 3D reconstructed object, based on the tracked camera poses obtained from the reconstruction phase. The segmentation algorithm works with both inanimate objects lying on a table/floor or with humans. Furthermore, we automatically scale the model to fit in the maximum volume of the 3D printer at hand. Finally, we present initial results from our experimental prototype and discuss the current limitations.

As the computer vision and virtual reality (VR) technology become mature nowadays, there is a huge potential to connect VR with the real world. In this paper, a 3D interactive VR system with adaptive treadmill control is developed. The subject walks on the treadmill at will. Interaction force between feet and treadmill are measured to estimate the subject's intended walking speed. The treadmill belt velocity and 3D display adjust to the walking pace according to the subject's walking intention, which makes the subject feel that he (or her) is walking in the real market. The experiment shows that the control results are smooth, which verifies the validity of the whole system.

Mobile robot could navigate in unknown environment autonomously with the help of simultaneous localization and mapping (SLAM). Recently, SLAM based on information matrix enjoys much popularity since it is naturally sparse. However, the computational burden related to information matrix balloons with respect to the increase of the mapped landmarks. In this paper, by considering the features of information matrix, we present a novel method which wipes off nearly half of the elements in information matrix. The errors that come from sparsification decrease apparently by loop-closure. Furthermore, the relationship between sparsification and SLAM accuracy is analyzed theoretically. A large scale simulation and experiment conducted on a real robot suggest that the technique is effective for a robot's SLAM in real-world applications.