Obtaining the clear contours of ship targets via Synthetic aperture radar (SAR) is extremely valuable for monitoring the sea. Now there are many deep learning imaging methods for ground scenes with good results, but they will face three main challenges when imaging ship targets: 1) The translational and rotational motions of ship targets during travel and due to waves respectively bring spatial invariant and variant errors that are tough to be estimated and compensated, resulting in defocused SAR imaging results; 2) The varying motion of ships demands high generalization ability of the imaging reconstruction network to adapt to the ship targets with sample-wise variant motion parameters; 3) Since ships are noncooperative targets, the accuracy of motion parameter estimation should be evaluated based on image quality, whereas the available SAR image quality assessment functions exhibit limited robustness. To address these issues, this article proposes a deep learning-based SAR imaging framework for ship targets via deep unfolding. Firstly, the motion model and characteristics of ship targets are analyzed, and the SAR imaging model for ship target with complex translational and rotational motion is established. Secondly, an imaging network with high generalization ability is proposed to adapt echoes for imaging under different motion parameters. On this basis, a SAR ship image quality assessment network is proposed to assess the imaging results of SAR ship targets with different focusing qualities. Then, the high-resolution imaging problem of ship targets is regarded as a motion parameter optimization problem, with the image quality assessment results as the objective function. Finally, this problem is optimized to search for the most accurate translational and rotational motion parameter variables of the ship target, achieving error compensation and imaging. The validity of the method is verified though the simulation of point targets and real SAR scenario data.

The separated platforms of a bistatic synthetic aperture radar (SAR) introduce synchronization error in the echo, resulting in offset and defocused target position and other characteristics. To deal with the problem, this paper firstly analyzes the synchronization error characteristics, then proposes a synchronization error separated method based on variational mode decomposition techniques, and ultimately uses the echo after the error separation to get the high precision SAR imaging results. Simulation experiment results verify the effectiveness of the proposed method.

Multi-core digital signal processor (DSP) is widely used in SAR real-time imaging system for its high-speed operation capacity and parallel working ability. The matching of the algorithm and the parallel architecture has great impact on imaging speed of SAR processing. This paper proposed an efficient imaging architecture based on multi-core DSP TMS320C6678. The system implements bistatic polar format algorithm (PFA), using two-dimensional Chirp-Z Transform and one-dimensional interpolation to realize two-dimensional resampling. The experimental results show that the proposed architecture can complete 1024×1024 points echo processing and output a 1024×1024 pixels image within 1.7 seconds.

Synthetic Aperture Radar (SAR), an all-weather microwave remote sensing technology, is widely used in environmental monitoring, earth resource surveys and other fields, especially in the military field to achieve monitoring purposes. In the monitoring process, it is crucial to achieve real-time imaging of the target scene. Therefore, in this paper, an efficient SAR real-time imaging system is designed to achieve timely parallel processing of the scene when the radar platform scans the target scene, so as to reduce the consecutive frame cycles of real-time imaging. In which, the signal processing board structure consists of two FPGAs and six DSPs is used to quickly implement high precision SAR real-time imaging for airborne radar, so that the radar system can display the target image in time with the platform moving and check whether the current echo is valid. In addition, the architecture proposed in this paper can be extended in case the continuous frame period for real-time imaging is short.

As a new addition, this year the ACM Multimedia conference is introducing Thematic Workshops. Inspired by the NIPS model, Thematic Workshops allow papers that could not be accommodated in the main conference to be presented to the research community.

Near-eye multi-focal (light field) displays are known to be more advantageous than conventional stereoscopic 3D displays, because they can alleviate the vergence-accommodation conflict. While stereoscopic 3D displays are restricted to projecting images on a single focal plane, near-eye light field displays can form multiple focal planes and volumetrically render 3D data. Most existing near-eye light field displays use simplistic, uniformly spaced focal plane configuration (in dioptric space). In this paper, we present a novel technique that optimizes the focal plane configuration based on characteristics of the content to be rendered. We show that this technique can significantly improve the perceived visual quality of content visualized on such displays.

We present a study of depth-disparity calibration for augmented reality applications using binocular optical see-through displays. Two techniques were proposed and compared. The "paired-eyes" technique leverages the Panum's fusional area to help viewer find alignment between the virtual and physical objects. The "separate-eyes" technique eliminates the need of binocular fusion and involves using both eyes sequentially to check the virtual-physical object alignment on retinal images. We conducted a user study to measure the calibration results and assess the subjective experience of users with the proposed techniques.

We describe a study that aims to understand physical activity and sedentary behavior in free-living settings. We employed a wearable camera to record 3 to 5 days of imaging data with 40 participants, resulting in over 360,000 images. These images were then fully annotated by experienced staff with a rigorous coding protocol. We designed a deep learning based classifier in which we adapted a model that was originally trained for ImageNet [1]. We then added a spatio-temporal pyramid to our deep learning based classifier. Our results show our proposed method performs better than the state-of-the-art visual classification methods on our dataset. For most of the labels our system achieves more than 90% average accuracy across different individuals for frequent labels and more than 80% average accuracy for rare labels.

The last few decades have witnessed tremendous advances in telecommunication, with the invention of technologies such as radio, telephone, voice-over-IP, and video conferencing. While all these communication tools are useful and valuable, the ultimate goal of telecommunication is to enable fully immersive remote interaction in a way that simulates or even surpasses the face-to-face experience. Immersive 3D communication technologies are developed aiming at that goal. The objective of this tutorial is to present an overview of the recent advances in immersive 3D communication. Topics include the basics of human 3D perception, new systems and algorithms in real-time 3D scene capture and reconstruction, 3D data compression and dissemination, 3D displays, etc. We intend to provide insights into the latest immersive 3D communication technologies, and highlight some open research challenges for the future.

ITU-T G.1070 [7] is widely cited for evaluating the subjective quality of the video conferencing, but its findings cannot be applied to the tele-immersion. The reasons are two fold. First, a tele-immersive system offers end users an unmatched realistic and immersive experience by allowing them to collaborate in the joint virtual space. Second, the human activities in the shared space are not limited to the conferencing conversation. In this paper, we conduct a user study with 19 participants to investigate the human perceptions of two tele-immersive shared activities, where media samples of different qualities are evaluated using the comparative category rating method [9] in case of each activity. We compare our subjective results to those presented in G.1070, and demonstrate heterogeneous human perceptual impacts in different activities.

This article presents a psychophysical study that measures the perceptual thresholds of a new factor called Color-plus-Depth Level-of-Details (CZLoD) peculiar to polygon-based 3D tele-immersive video. The results demonstrate the existence of Just Noticeable Degradation and Just Unacceptable Degradation thresholds on the factor. In light of the results, we design and implement a real-time perception-based quality adaptor for 3D tele-immersive video. Our experimental results show that the adaptation scheme can reduce resource usage (e.g., CPU cycles) while considerably enhancing the overall perceived visual quality. Our analysis confirms the potential temporal and spatial performance benefits achievable with CZLoD adaptation.

In this demonstration, we present two 3D tele-immersive games: light-saber dual and block fencing that merge 3D video representations of participants in real-time to enable remote interactions in a virtual world. The light-saber dual arranges participants in a symmetric setup where both participants interact with each other in a virtual world with similar goals. On the other hand, the block fencing creates an asymmetric setup where participants interact with virtual objects having different goals. Using these two setups, we address the challenges and novelty of our solutions in portable environment setup, data acquisition, multi-stream synchronization, multi-stream session management, mobile device rendering, and overlay communication in the design and implementation of advanced 3D tele-immersive systems.

The voice communication industry is undergoing a rapid phase change with continuously evolving technologies such as cellular, mobile and Internet telephony. Effective evaluation of system performance is becoming critical, which will serve as an important instrument for service providers to monitor and manage quality. Recently, Quality of Experience (QoE) metrics are found to be more valuable quality assessment mechanism since they are more closely related to human perception as compared to traditional QoS-based techniques. In this paper, we investigate a new affect-based approach for QoE evaluation. Our hypothesis states that QoE in voice communication is correlated to user's affective behavior, which will vary across networking conditions. We conduct extensive user study based on natural human-human voice communication. The experimental results indicate very promising prospect of this approach.

This paper presents a psychophysical study that measures the perceptual thresholds of a new factor called Color-plus-Depth Level-of-Detail peculiar to polygon-based 3D tele-immersive video. The results demonstrate the existence of Just Noticeable Degradation and Just Unacceptable Degradation thresholds on the factor. In light of the results, we describe the design and implementation of a real-time perception-based quality adaptor for 3D tele-immersive video. Our experimental results show that the adaptation scheme can reduce resource usage while considerably enhancing the overall perceived visual quality.

This paper presents a psychophysical approach to control a new factor called Color-plus-Depth Level-of-Detail in polygon-based 3D tele-immersive video. Based on our psychophysical study that demonstrates the existence of perceptual thresholds on the factor, we present a real-time perception-based quality adaptor for 3D tele-immersive video. Our experimental results show that the adaptation scheme can reduce resource usage while considerably enhancing the overall perceived visual quality.

3D tele-immersion (3DTI) has the potential of enabling virtual reality interaction among remote people with real-time 3D video. However, today's 3DTI systems still suffer from various performance issues, limiting their broader deployment, due to the enormous demand on temporal (computing) and spatial (networking) resources. Past research focused on system-centric approaches for technical optimization, without taking human users into the loop. We argue that human factors (including user preferences, semantics, limitations, etc.) are an important and integral part of the cyber-physical 3DTI systems, and should not be neglected. This thesis proposes a comprehensive, human-centric framework for managing video data and functions across the 3DTI pipeline. Our approach is comprehensive because it involves all components of the video function pipeline. It is also comprehensive in the sense that both temporal and spatial resource challenges are considered and tackled.

An ideal interactive 3D tele-immersion (3DTI) system is expected to disseminate and synchronize multi-streams with a shortest-possible latency among participating sites, achieve inter-stream synchronization, and bound both inter-sender and inter-receiver skews. This is, however, a key challenge because of (1) the coexistence of multi-modal, correlated, bandwidth-savvy streams from multiple source media, (2) the bounded bandwidth resources at each site, (3) the heterogeneous transmission end-to-end delays (EED) between sites and (4) the diversity of 3D views requested by multiple users. Our study of the existing content dissemination topologies reveals their inadequacy of handlings the complication and dynamics present in 3DTI systems.

This paper presents a novel paradigm of Streaming as a Service (SAS) to model correlated multi-streaming in Distributed Interactive Multimedia Environments. We propose SAS Kernel, a generic, distributed, and modular service kernel realizing SAS paradigm. SAS Kernel features high exibility by employing a configurable interface to allow for input of correlated multi-streams (bundle of streams) from diverse types of sensory devices. It is also highly extensible by allowing user-controlled functions to be applied to bundle of streams in runtime. Experiments with real-world applications demonstrate that the SAS Kernel incurs low overhead in delay, CPU, and bandwidth demands.

Distributed Interactive Multimedia Environments (DIMEs) enable geographically distributed people to interact with each other in a joint media-rich virtual environment for a wide range of activities, such as art performance, medical consultation, sport training, etc. The real-time collaboration is made possible by exchanging a set of multi-modal sensory streams over the network in real time. The characterization and evaluation of such multi-stream interactive environments is challenging because the traditional Quality of Service metrics (e.g., delay, jitter) are limited to a per stream basis. In this work, we present a novel cyber-physical, spatio-temporal QoS model that includes new metrics to measure and evaluate QoS for a bundle of streams and across bundles of streams. The new model allows better assessment of performance of any DIME application which employs multiple sensors. We validate our model by comparing the QoS performance of bundle of streams over three transport protocols - UDP, DCCP, and TCP in a 3D tele-immersive test bed.

In this paper, we present the results from a quantitative and qualitative study of distributed gaming in 3D tele-immersive (3DTI) environments. We explore the Quality of Experience (QoE) of users in the new cyber-physical gaming environment. Guided by a theoretical QoE model, we conducted a case study and evaluated the impact of various Quality of Service (QoS) metrics (e.g., end-to-end delay, visual quality, etc.) on 3DTI gaming experience. We also identified a number of non-technical factors that are not captured by the original theoretical model, such as age, social interaction, and physical setup. Our analysis highlights new implications for the next-generation gaming system design, as well as a more comprehensive conceptual framework that captures non-technical influences for user experience in such environments.

Synchronization is a challenge in the multi-site 3D tele-immersion (3DTI) because it is complicated by the coexistence of highly correlated heterogenous streams from multiple sources, and the need for multi-stream resynchronization when user views change. To address the problems, we present TSync: a new multi-tier synchronization framework for 3DTI which can effectively reduce the multi-stream sync skews. Our contributions are focusing on (1) the use of timed synchronization points for multi-stream synchronization; (2) Internet bandwidth estimation based on machine learning; (3) the cooperative frame rate allocation for correlated multi-streams and (4) the resynchronization protocol used when user views change. Experimental results show that TSync can successfully achieve the synchronization of multi-source heterogeneous streams in 3DTI under Internet dynamics.

Three-dimensional tele-immersive (3DTI) environments have great potential to promote collaborative work among geographically distributed users. However, most existing 3DTI systems only work with two sites due to the huge demand of resources and the lack of a simple yet powerful networking model to handle connectivity, scalability, and quality-of-service (QoS) guarantees. In this article, we explore the design space from the angle of multi-stream management to enable multi-party 3DTI communication. Multiple correlated 3D video streams are employed to provide a comprehensive representation of the physical scene in each 3DTI environment, and are rendered together to establish a common cyberspace among all participating 3DTI environments. The existence of multi-stream correlation provides the unique opportunity for new approaches in QoS provisioning. Previous work mostly concentrated on compression and adaptation techniques on the per-stream basis while ignoring the application layer semantics and the coordination required among streams. We propose an innovative and generalized ViewCast model to coordinate the multi-stream content dissemination over an overlay network. ViewCast leverages view semantics in 3D free-viewpoint video systems to fill the gap between high-level user interest and low-level stream management. In ViewCast, only the view information is specified by the user/application, while the underlying control dynamically performs stream differentiation, selection, coordination, and dissemination. We present the details of ViewCast and evaluate it through both simulation and 3DTI sessions among tele-immersive environments residing in different institutes across the Internet2. Our experimental results demonstrate the implementation feasibility and performance enhancement of ViewCast in supporting multi-party 3DTI collaboration.

Three-dimensional tele-immersive (3DTI) environments have great potential to promote collaborative work among geographically distributed users. However, most existing 3DTI systems only work with two sites due to the huge demand of resources and the lack of a simple yet powerful networking model to handle connectivity, scalability, and quality-of-service (QoS) guarantees.

The next-generation social communication medium, tele-immersion, is receiving increasing attention from both the research and industrial communities. It enables interaction between geographically distributed sites through realistic reconstruction of physical scenes in immersive spaces. While the previous research focused on real-world measurements of implemented tele-immersive systems, we take a model-based approach to study the system more generally and thoroughly. With a model validated with actual measurements, we evaluate the system performance in a larger scale, and understand the quantitative effect of the algorithmic enhancements that were proposed in the literature. We also predict futuristic performance characteristic and show that high interactivity can be foreseen with hardware advances in the near future.

The past decades have witnessed a rapid growth of Distributed Interactive Multimedia Environments (DIMEs). Despite their intensity of user-involved interaction, the existing evaluation frameworks remain very much system-centric. As a step toward the human-centric paradigm, we present a conceptual framework of Quality of Experience (QoE) in DIMEs, to model, measure, and understand user experience and its relationship with the traditional Quality of Service (QoS) metrics. A multi-displinary approach is taken to build up the framework based on the theoretical results from various fields including psychology, cognitive sciences, sociology, and information technology. We introduce a mapping methodology to quantify the correlations between QoS and QoE, and describe our controlled and uncontrolled studies as illustrating examples. The results present the first deep study to model the multi-facet QoE construct, map the QoS-QoE relationship, and capture the human-centric quality modalities in the context of DIMEs.

We present MobileTI, a portable tele-immersive system that merges 3D video representations of users in real time to enable remote collaboration across geographical distances. With portability as a main goal, we address the challenges in the camera setup, time synchronization, video acquisition, and networking in the design and implementation of the system. Having been deployed in public performances, MobileTI proves to be effective, efficient, and user-friendly. Our experimental findings in terms of technical performance and user feedback are presented.

Multi-stream/multi-site 3D video collaborative systems are promising as they enable remote users to interact in a 3D virtual space with a sense of co-presence. However, the decentralized content dissemination remains a challenge. In this work, we explore approaches to construct adaptive overlay based on the users' visual interest in the collaborative space. Particularly, we consider the practical challenge that the user's interest might change dynamically. We propose, compare, and evaluate three algorithms to handle the view dynamics. With extensive experiments, we demonstrate that an algorithm that exploits view locality can achieve efficient bandwidth utilization, high topology stability, and great scalability.

3D Teleimmersive (TI) environments have gained significant attention as they show promising results for a wide number of applications. These environments are usually distributed and multi platform which makes performance analysis and monitoring difficult. We present a Unified Time Service (UTS) and its measurement framework for evaluating components in terms of their reflected QoS (e.g. delay). Then we utilize UTS to analyze the delay introduced by the virtualized rendezvous points (RP) in the H-MEDIA TI system. We find that the delay overhead heavily depends on frame sizes of the 3D video streams.

3D tele-immersion (3DTI) has recently emerged as a new way of video-mediated collaboration across the Internet. Unlike conventional 2D video-conferencing systems, it can immerse remote users into a shared 3D virtual space so that they can interact or collaborate "virtually". However, most existing 3DTI systems can support only two sites of collaboration, due to the huge demand of networking resources and the lack of a simple yet efficient data dissemination model. In this paper, we propose to use a general publish-subscribe model for multi-site 3DTI systems, which efficiently utilizes limited network resources by leveraging user interest. We focus on the overlay construction problem in the publish-subscribe model by exploring a spectrum of heuristic algorithms for data dissemination. With extensive simulation, we identify the advantages of a simple randomized algorithm. We present optimization to further improve the randomized algorithm by exploiting semantic correlation. Experimental results demonstrate that we can achieve an improvement by a factor of five.

The 3D tele-immersive (3DTI) environments are emerging as the next generation technique for tele-communication. In this paper, we present a novel and generic interface for view control in 3D environments. The interface uses Wii Remote, a wireless controller for a video game console. It allows the user to manipulate the virtual camera and 3D objects intuitively with buttons and through motions (pitch and roll). We conducted user studies to evaluate the interface with a professional dancer and average audiences. The results demonstrate that the Wii Remote interface is much better suited to view control in 3DTI environments since it is much easier to use, more effective, and more accurate than the conventional mouse-based interfaces.

In this demonstration, we present WiiView, a view control interface for 3D tele-immersive (3DTI) environments. This interface allows the user to intuitively manipulate the virtual camera and graphical objects in a 3D tele-immersive virtual space, thus significantly enhancing the viewing experience for users in 3DTI collaboration. We have used WiiView in a dance class for creative choreography in Spring 2008 at University of Illinois at Urbana-Champaign. It allows for great flexibility, usability, and portability, as the dancers reported. Furthermore, the interface has become an interesting creativity tool for digital choreographers, as it creates a new element in the design space when dancers control the virtual view change and perform physical body movement simultaneously.

We present our implementation and evaluation of TEEVE, a distributed 3D tele-immersive system. TEEVE is among the first to support multi-stream/multi-site 3D tele-immersive environments with commercial off-the-shelf hardware and software infrastructures. It promotes collaborative physical activities among geographically dispersed sites by immersing the 3D representations of remote participants into a joint 3D virtual space. In this paper, we describe our implementation of TEEVE and introduce the recent advances in its different system components. In particular, we present an implemented protocol for semantic-aware data dissemination to support multi-site remote collaboration. We evaluate the TEEVE system by deploying it on the Internet. The experimental results demonstrate that it achieves stable visual quality, soft real-time delay, and efficient resource usage.

The Tele-immersive Dance Environment (TED) is a geographically distributed, real-time 3-D virtual room where multiple participants interact independent of physical distance. TED, a highly interactive collaborative environment, offers digital options with multiple viewpoints, enhancing the creative movement composition involved with dance choreography. A symbiotic relationship for creativity and design exists between dance artists and computer scientists as the tele-immersive environment is analyzed as a creativity and learning tool. We introduce the advancements of the interactive digital options, new interface developments, user study results, and the possibility of a computational model for human creativity through Laban Movement Analysis.

We demonstrate the Tele-immersive Dance Environment (TED), a geographically distributed, real-time 3-D virtual room where multiple participants interact independent of physical distance. TED, a highly interactive collaborative environment, offers digital options with multiple viewpoints, enhancing the creative movement composition involved with dance choreography. We present the advancements of the interactive digital options, new interface developments, and user study results.

The study of 3D Tele-immersion impact on remote collaborative work represents a very interesting and challenging research topic. In this paper, we introduce the latest accomplishments of TEEVE research which merges computer science with dance choreography. This collaborative research model is ideal for creative, interdisciplinary problem solving. TEEVE offers an entirely new interface for dance choreography as a creative tool and alternative performance venue.

Tele-immersion is emerging as a new medium that creates 3D photorealistic, immersive, and interactive experience between geographically dispersed users. However, most existing tele-immersive systems can only support two-way collaboration. In this paper we propose a multi-layer framework and a new data dissemination protocol to support multi-site collaboration. The problem context is unique as multiple remote sites participate in an interactive tele-immersive session, where each site has multiple correlated 3D video streams to send (later referred as multi-stream/multi-site environments). The key challenge is to disseminate such large number of 3D live video streams among these sites subject to the bandwidth and latency constraints while satisfying QoS guarantees in visual quality. Among our findings is that the simple randomized algorithm outperforms many other static algorithms in the unique context. Moreover, the streams generated from one site have high semantic correlation, because often cameras at one site are shooting the same scene, only from different angles. We exploit the stream correlation in the multicast protocol to minimize the level of loss.

Real-time distributed multi-party/multi-stream systems are becoming more popular in many areas such as 3D tele-immersion, multi-camera conferencing and security surveillance. However, the construction of such systems in large scale is impeded by the huge demand of computing and networking resources and the lack of a simple yet powerful networking model to handle interconnection, scalability and quality of service (QoS) guarantees. We make two main contributions in the paper: (1) we propose a novel generalized ViewCast model for multi-party/multi-stream video-mediated systems that fills the gap between high-level user interest and low level per-stream management, and (2) we demonstrate the ViewCast model by applying it to the multi-party 3D Tele-Immersive (3DTI) collaboration among geographically dispersed users. More specifically, we show how the ViewCast model is used in supporting stream data dissemination, coordination and QoS management among multiple 3D tele-immersive environments. We present our experimental results in both real implementation and simulation to show that our ViewCast-based solution achieves high efficiency, scalability, and quality in supporting multi-party 3DTI collaboration.

We first present the tele-immersive environments developed jointly by University of Illinois at Urbana-Champaign and University of California at Berkeley. The environment features 3D full and real body capturing, wide field of view, multi-display 3D rendering, and attachment free participant. We then describe a study of collaborative dancing between remotely located dancers in the shared virtual space. Two professional dancers are invited to the tele-immersive site of each university. As a preliminary experiment, we let the dancers perform elementary body movements and coordinate their dancing. The coordination requires one dancer to take the lead while the other follows her by appropriate movements. During the experiment, the dancers are dancing at various motion rates to evaluate how well the collaborative dancing is supported with the current technical boundary. Our important findings indicate that 1) tele-immersive environments have strong potential impact on the concept of choreography and communication of live dance performance, 2) the presence of multi-display system, real body 3D rendering, audio channel, and less intrusive-ness greatly enhances the immersive and dancing experience, and 3) the level of synchronization achieved by the dancers is higher than that expected from the video rate

We present a study of collaborative dancing between remote dancers in a tele-immersive environment which features 3D full and real body capturing, wide field of view, multi-display 3D rendering, and attachment free participant. We invite two professional dancers to perform collaborative dancing in the environment. The coordination requires one dancer to take the lead while the other follows by appropriate movement. Throughout the experiment, the dancers are dancing at various motion rates to evaluate how well the collaborative dancing is supported with the current technical boundary. Our important findings indicate that 1) tele-immersive environments have strong potential impact on the concept of choreography and communication of live dance performance, 2) the presence of multi-view display, real body 3D rendering, audio channel, and less intrusiveness greatly enhances the immersive and dancing experience, and 3) the level of synchronization achieved by the dancers is higher than that expected from the video rate.