Vehicle driving simulators have been widely used in fields including road design, automotive development, and driver training. As a core component of the simulators, motion cueing algorithms (MCAs) aim to reproduce realistic vehicle motion sensations while respecting workspace limitations for a high-fidelity driving experience. Recent advancements have identified model predictive control (MPC) as a promising approach for MCA. However, conventional MPC-based MCA faces computational bottlenecks that limit its performance. Although recent studies have employed swarm intelligence optimization algorithms, such as the genetic algorithm and grey wolf optimizer, to enhance the performance of MPC-based MCA through horizon adjustments, the improvement path from the computational cost perspective is still overlooked. Therefore, this paper proposes a costs-optimization framework for MPC-based MCA (COMPC-based MCA), which integrates two key components: an Operator Splitting Quadratic Program (OSQP) solver and a Snow Ablation Optimizer (SAO) that considers computational costs and motion sensation errors for parameter selection. Experiment results validated the performance of the proposed framework. COMPC-based MCA can achieve fast quadratic programming (QP) solving and select appropriate parameters for performance improvement while considering computational costs.

Elucidating the reasoning process with structured explanations from question to answer is crucial, as it significantly enhances the interpretability, traceability, and trustworthiness of question-answering (QA) systems.However, structured explanations demand models to perform intricately structured reasoning, which poses great challenges.Most existing methods focus on single-step reasoning through supervised learning, ignoring logical dependencies between steps.Moreover, existing reinforcement learning (RL) based methods overlook the structured relationships, underutilizing the potential of RL in structured reasoning.In this paper, we propose SEER, a novel method that maximizes a structure-based return to facilitate structured reasoning and explanation.Our proposed structure-based return precisely describes the hierarchical and branching structure inherent in structured reasoning, effectively capturing the intricate relationships between different reasoning steps.In addition, we introduce a fine-grained reward function to meticulously delineate diverse reasoning steps.Extensive experiments show that SEER significantly outperforms state-of-theart methods, achieving an absolute improvement of 6.9% over RL-based methods on En-tailmentBank, a 4.4% average improvement on STREET benchmark, and exhibiting outstanding efficiency and cross-dataset generalization performance.Our code is available at https://github.com/Chen-GX/SEER.

In adaptive video streaming, the design of an adaptive bitrate (ABR) strategy is critical for the quality-of-experience (QoE) perceived by users. Though current learning-based ABR algorithms achieve state-of-the-art performance for users with a given QoE metric setting for training, they may unfortunately suffer the poor generalization issue for other users with different QoE preferences. Besides, how to quantitatively characterize the distinct QoE preference for a user has also not been extensively studied yet. In this paper, we propose STEER, a successor feature-based transfer reinforcement learning framework for fast learning the ABR strategies on heterogeneous QoE preferences. Specifically, we first develop a QoE preference analysis scheme to infer the personal QoE preference of a single user based on the user's actual viewing history. We then formulate the personalized QoE maximization problem as a reinforcement learning (RL) task, which optimizes the ABR strategy to maximize the overall QoE perceived by the user. Further, we model the QoE maximization problem for multiple users with heterogeneous QoE preferences as a multi-task RL problem, with each task distinguished by the user-distinct QoE preference. To efficiently address this problem, the proposed STEER solves for each RL-based ABR task by learning its optimal successor feature (SF) function, which can be exploited as shared knowledge across tasks to facilitate the transfer between tasks. With SF functions, STEER can quickly evaluate the optimal policies of previously learned tasks on a new task, and further use the generalized policy improvement operation to obtain a jumpstart policy. Both theoretically and empirically, we show that this jumpstart policy is a good initial policy with a performance guarantee for better generalization in the new task, and can also lead to a faster convergence to the optimal policy of the new task.

In this paper, we investigate the adaptive video delivery for multiple users over time-varying and mutually interfering multi-cell wireless networks. The key research challenge is to jointly design the physical-layer resource allocation scheme and application-layer rate adaptation logic, such that the users' long-term fair quality of experience (QoE) can be maximized. Due to the timescale mismatch between these two layers and the asynchrony of user requests, however, it is difficult to directly model the cross-layer stochastic control problem by using a reinforcement learning framework. To address this difficulty, we propose a novel two-level decision framework where an optimization-based beamforming scheme (performed at the base stations) and a deep reinforcement learning (DRL)-based rate adaptation scheme (performed at the user terminals) are, respectively, developed, such that a highly complex long-term multi-user QoE fairness problem is decomposed into some relatively simple problems and solved effectively. Our strategy represents a significant departure from the existing schemes with consideration of either a short-term multi-user QoE maximization or a long-term single-user point-to-point QoE maximization. Extensive simulations demonstrate that the proposed cross-layer design is effective and promising.

In this paper, we propose a deep reinforcement learning (DRL)-based rate adaptation algorithm for adaptive 360-degree video streaming, which is able to maximize the quality of experience of viewers by adapting the transmitted video quality to the time-varying network conditions. Specifically, to reduce the possible switching latency of the field of view (FoV), we design a new QoE metric by introducing a penalty term for the large buffer occupancy. A scalable FoV method is further proposed to alleviate the combinatorial explosion of the action space in the DRL formulation. Then, we model the rate adaptation logic as a Markov decision process and employ the DRL-based algorithm to dynamically learn the optimal video transmission rate. Simulation results show the superior performance of the proposed algorithm compared to the existing algorithms.

Sending only the viewport of interest provides a solution for 360-degree video streaming under the current bandwidth-constrained infrastructure. To this end, the user viewport requires to be prefetched in advance by conducting viewport prediction. To more accurately capture the nonlinear and long-term dependent relation between the future and past viewports, we develop a single viewport prediction model using convolutional neural network (CNN), in which the pooling layers are dropped and more convolutional layers are added for stronger nonlinear fitting ability. Further, we design a viewport trajectory prediction model based on recurrent neural network (RNN) which learns long-term dependency in sequential viewports. Specially, it is capable to estimate future viewport trajectory and support variable-size prediction window with low complexity. Finally, a correlation filter-based viewport tracker (CFVT) is proposed to perform content-aware viewport prediction. The combination of the RNN and the CFVT through a fusion model enables them to complement each other which is validated by significant improvement in prediction accuracy.

Policy evaluation with smooth and nonlinear function approximation has shown great potential for reinforcement learning. Compared to linear function approxi- mation, it allows for using a richer class of approximation functions such as the neural networks. Traditional algorithms are based on two timescales stochastic approximation whose convergence rate is often slow. This paper focuses on an offline setting where a trajectory of $m$ state-action pairs are observed. We formulate the policy evaluation problem as a non-convex primal-dual, finite-sum optimization problem, whose primal sub-problem is non-convex and dual sub-problem is strongly concave. We suggest a single-timescale primal-dual gradient algorithm with variance reduction, and show that it converges to an $\epsilon$-stationary point using $O(m/\epsilon)$ calls (in expectation) to a gradient oracle.

The high-definition and panorama characteristics of 360-degree video make its transmission in existing networks more challenging. To alleviate this problem, this paper leverages transcoding-enabled edge caching in tile-granularity and makes joint caching, transcoding and delivery decisions among multiple edge servers for adaptive 360-degree video streaming. We formulate the joint caching, transcoding and delivery decision-making problem as a multivariate nonlinear integer programming problem, aiming at minimizing the aggregate network operational cost and balancing the server load, subject to storage and computation capacity constraints of the edge servers. Through decomposition and relaxation approach, the original intractable optimization problem is solved by an effective iterative algorithm. Finally, simulation results show that the proposed iterative algorithm outperforms the existing schemes in terms of resource utilization, cost reduction and server coordination.

In this paper, we propose a cross-layer decision framework for multiuser adaptive video delivery over time-varying and mutually interfering wireless cellular network. The key idea is to synthetically design the physical-layer optimization-based beamforming scheme (performed at the base stations) and the application-layer deep reinforcement learning (DRL)-based rate adaptation scheme (performed at the user terminals), so that a very complex multi-user overall fair long-term quality of experience (QoE) maximization problem can be decomposed to two layers and solved effectively. Extensive simulations show that the proposed cross-layer design is effective and promising.

Real-time video services are usually delay sensitive and have strict constraints on the transmission reliability, which poses challenges to live video streaming over multi-hop wireless networks, since the unpredictable packet losses and network congestions caused by time-varying wireless channels greatly degrade the received video quality. To address this, in this paper, we propose a reinforcement learning (RL)-based opportunistic routing (OR) scheme for wireless video streaming with high-reliability and low-delay requirements. It can exploit the broadcast nature of the wireless shared medium and path diversity through OR to improve the transmission reliability, and find the low-delay paths between the source-destination pair dynamically for video packets through the RL module embedded in each relay node. Specifically, we design for the OR a new path-cost metric called the expected anypath delay (EAD), to estimate the end-to-end delay of a packet between the current relay node and the destination. The EAD is dynamically measured and updated over time, thereby reflecting the changes of link quality and the congestion level at the relay node. Moreover, we utilize the ACK message to piggyback the EAD of each relay node to its previous-hop node. Based on the local communication of the EADs from the neighbors, each node in the network can iteratively and independently run the RL module to update its own EAD value. Then, the next-hop forwarder node on a low delay route can be determined by assigning higher relay priority to the candidate forwarder nodes with lower EADs in OR. Simulation results show that the proposed RLOR algorithm can achieve a proper tradeoff between the transmission reliability and latency, so as to support the low-delay transmission of wireless video streams with high received video quality.