Recently, transformer-based methods have been introduced to estimate 3D human pose from multiple views by aggregating the spatial-temporal information of human joints to achieve the lifting of 2D to 3D. However, previous approaches cannot model the inter-frame correspondence of each view's joint individually, nor can they directly consider all view interactions at each time, leading to insufficient learning of multi-view associations. To address this issue, we propose a Spatial-View-Temporal transformer (SVTformer) to decouple spatial-view-temporal information in sequential order for correlation learning and model dependencies between them in a local-to-global manner. SVTformer includes an attended Spatial-View-Temporal (SVT) patch embedding to attentively capture the local features of the input poses and stacked SVT encoders to extract global spatial-view-temporal dependencies progressively. Specifically, SVT encoders perform three reconstructions sequentially to attended features with the learning through view decoupling for temporal-enhanced spatial correlation, temporal decoupling for spatial-enhanced view correlation, and another view decoupling for spatial-enhanced temporal relationship. This decoupling-coupling-decoupling multi-view scheme enables us to alternatively model the inter-joint spatial relationships, cross-view dependencies, and temporal motion associations. We evaluate the proposed SVTformer on three popular 3D HPE datasets, and it yields state-of-the-art performance. It effectively deals with ill-posed problems and enhances the accuracy of 3D human pose estimation.

In Visual Indoor Navigation (VIN), Deep Reinforcement Learning (DRL) is commonly used by agents to achieve end-to-end mapping from vision to action when navigating toward a target based on observation. However, current DRL-based work suffers from two challenges: partial observability resulting from using solely a single first-person view and poor generalization in the case of unknown scenes and unknown objects. In light of these issues, this paper introduces the integration of multi-view as an expansion of observability and meta-learning as a primary generalization technique into the DRL framework and presents the meta-reinforcement learning method that leverages Multi-View Semantic Spatial Context (MVSSC). Specifically, aiming to explore the informative multi-view context for better efficiency in searching for and navigating to the target, we model the objects’ relationship from two aspects, multi-view semantic context (MVSEC) and multi-view spatial context (MVSPC). MVSEC enables agents to encode prior semantic relationships adaptively via a multi-view modulated graph. Meanwhile, MVSPC enhances the spatial representation of target-related objects’ correlation through similarity grids of multi-view. After adaptively fusing the multi-view and context information under the meta-reinforcement learning framework, our method can encourage efficient target search and robust navigation with stronger generalization performance to unknown scenes and unknown objects. Extensive experimental results on the AI2-THOR simulator demonstrate that our method outperforms the current state-of-the-art approaches.

In robot-assisted cardiac surgery, predicting heart motion can help improve the operation accuracy and safety of surgical robots. Different from the conventional prediction schemes which model the point of interest (POI) with only temporal correlation of past observations, this paper proposes an LSTM-based method by exploiting the spatio-temporal correlation of the 3D movements of POI and auxiliary points (APs) on the same surface of the heart. Three different LSTM models are investigated. The first two models define the POI prediction as a pure time-series forecasting problem based on past POI trajectory, and the third model combines the past observations of POI and new observations of APs to take into consideration the extra spatial correlations for prediction. Experimental comparison studies based on 3D coordinates obtained from real stereo-endoscopic videos demonstrate the superior performance of the proposed spatio-temporal LSTM model.

This paper investigates the backscattering properties of vegetation fire based on ground-based scatterometer measurement in the combustion period. According to the different states of combustion during and after vegetation fire, respectively, X-band and C-band scattering coefficient of full polarization (HH, HV, VH, and VV ) are measured to exploiting the spatial and temporal responses of fire effect on vegetation. Meanwhile, the relevant influencing factors, which include the parameters of vegetation, ground and environment, are obtained in observation area. Besides the process before and after fire, it is discussed in detail for analysis of microwave scattering coefficients changing with different burning degrees in the fire development process. Based on the experimental result, the distinction through combustion is shown by the microwave method effectively, which proves its potential in information extraction from vegetation fire.