DNNs are rapidly evolving from streamlined single-modality single-task (SMST) to multi-modality multi-task (MMMT) with large variations for different layers and complex data dependencies among layers. To support such models, hardware systems also evolved to be heterogeneous. The heterogeneous system comes from the prevailing trend to integrate diverse accelerators into the system for lower latency. FPGAs have high computation density and communication bandwidth and are configurable to be deployed with different designs of accelerators, which are widely used for various machine-learning applications. However, scaling from SMST to MMMT on heterogeneous FPGAs is challenging since MMMT has much larger layer variations, a massive number of layers, and complex data dependency among different backbones. Previous mapping algorithms are either inefficient or over-simplified which makes them impractical in general scenarios. In this work, we propose CHEF to enable efficient implementation of MMMT models in realistic heterogeneous FPGA clusters, i.e. deploying heterogeneous accelerators on heterogeneous FPGAs (A2F) and mapping the heterogeneous DNNs on the deployed heterogeneous accelerators (M2A). We propose CHEF-A2F, a two-stage accelerators-to-FPGAs deployment approach to co-optimize hardware deployment and accelerator mapping. In addition, we propose CHEF-M2A, which can support general and practical cases compared to previous mapping algorithms. To the best of our knowledge, this is the first attempt to implement MMMT models in real heterogeneous FPGA clusters. Experimental results show that the latency obtained with CHEF is near-optimal while the search time is 10000X less than exhaustively searching the optimal solution.

Panoramic video is considered to be an attractive video format since it provides the viewers with an immersive experience. However, only the viewers’ focused region of a panoramic video, viewport, is shown on the screen. Numerous applications may benefit from the future viewport prediction, such as the adaptive viewport delivery. Two aspects may be considered for predicting the viewports. Firstly, the viewport trajectory within a time interval is highly correlated, hence the historical viewport trajectory should be utilized. Secondly, the panoramic video content within the viewport highly correlates with human visual attention. In this paper, we propose the Emotional Attention map aided panoramic Viewport Estimation (EAVE) model for predicting the future viewport trajectory. In this model, the historical viewport trajectory and the emotional attention map are jointly exploited by a Long Short-Term Memory (LSTM) network. Our experiments show that the proposed method outperforms several baseline methods in terms of prediction accuracy on a panoramic dataset containing viewing trajectory data from 50 viewers for watching 75 panoramic videos.

In this paper, we propose a portrait relighting method based on deep reinforcement learning (called PR-RL). Our PR-RL model could conduct portrait relighting by sequentially predicting local light editing strokes, and use strokes to conduct dodge and burn operations on the image lightness, simulating image editing by artists using brush strokes. Reinforcement learning with Deep Deterministic Policy Gradient is introduced to design our PR-RL model, defining the action (stroke parameters) in a continuous space, through which a reward can be designed to guide the agent to learn and relight a portrait image like an artist. To optimize the relighting effect, we further enable the reward to be location relevant and hence a coarse-to-fine strategy can be applied to select corresponding actions and maximize the performance of the proposed method. In comparison with the existing efforts, our proposed PR-RL method is locally effective, scale-invariant and interpretable. We apply the proposed method to tasks of portrait relighting based on both SH-lighting and reference images. The experiments show that our PR-RL method outperforms state-of-the-art methods in generating locally effective and interpretable high resolution relighting results for wild portrait images.

In order to help orthopedists evaluate the morphological characteristics of the acetabulum of patients with osteoarthritis of the hip more efficiently and accurately to identify the type of acetabular deformities which benefits the personalized preoperative planning, a 3-dimensional (3D) acetabular morphologic parameters measurements software dedicated to the hip was developed. The system includes four modules: 1) Identify the anterior pelvic plane (APP) of the pelvis model; 2) Identify the circular rim of the acetabular wall; 3) Automatically measure the 3D morphological parameters of the dysplastic acetabulum; 4) Interactively measure the 3D morphological parameters of the dysplastic acetabulum. The automatic parameter measurement function of this software could fast and accurately measure the 3D morphological parameters of the dysplastic acetabulum. These automatically measured parameters were close to those measured manually with error generally less than 2mm, and their average measurement time was nearly 10 times faster than that using the Mimics 17.0 system. For patients with large osteophyte, the 3D morphological parameters of the dysplastic acetabulum could be efficiently measured using the interactive measurement function of this software with simple operations. This software was used to measure acetabular morphological parameters in 61 patients. Two types of dysplastic acetabula were identified by the thickness of the medial wall on the lower margin of the acetabulum Tb: type I was a thin acetabulum (35 cases,$Tb \leqslant 10.0$mm) and type II was a thick acetabulum (26 cases,$Tb > 10.0$mm). According to the result of the acetabular morphological characteristic analysis, it can be found that the thickness of the medial wall is an important morphological characteristic for the THA preoperative surgical planning, and the thickened medial wall could be a misleading factor for the suboptimal placement of the cup