In this paper we are introducing a new reinforcement learning method for control problems in environments with delayed feedback. Specifically, our method employs stochastic planning, versus previous methods that used deterministic planning. This allows us to embed risk preference in the policy optimization problem. We show that this formulation can recover the optimal policy for problems with deterministic transitions. We contrast our policy with two prior methods from literature. We apply the methodology to simple tasks to understand its features. Then, we compare the performance of the methods in controlling multiple Atari games.

Investors and regulators can greatly benefit from a realistic market simulator that enables them to anticipate the consequences of their decisions in real markets. However, traditional rule-based market simulators often fall short in accurately capturing the dynamic behavior of market participants, particularly in response to external market impact events or changes in the behavior of other participants. In this study, we explore an agent-based simulation framework employing reinforcement learning (RL) agents. We present the implementation details of these RL agents and demonstrate that the simulated market exhibits realistic stylized facts observed in real-world markets. Furthermore, we investigate the behavior of RL agents when confronted with external market impacts, such as a flash crash. Our findings shed light on the effectiveness and adaptability of RL-based agents within the simulation, offering insights into their response to significant market events.

In 2012, Securities and Exchange Commission (SEC) mandated all corporate filings for any company doing business in the U.S. be entered into the electronic data gathering, analysis, and retrieval (EDGAR) system. In this work, we are investigating ways to analyze the data available through the EDGAR database. This may serve portfolio managers (pension funds, mutual funds, insurance, and hedge funds) to get automated insights into companies they invest in, to better manage their portfolios. The analysis is based on artificial neural networks applied to the data. In particular, one of the most popular machine learning methods, the convolutional neural network (CNN) architecture, originally developed to interpret and classify images, is now being used to interpret financial data. This work investigates the best way to input data collected from the SEC filings into a CNN architecture. We incorporate accounting principles and mathematical methods into the design of three image encoding methods. Specifically, two methods are derived from accounting principles (sequential arrangement, category chunk arrangement) and one is using a purely mathematical technique [the Hilbert vector arrangement (HVA)]. In this work, we analyze fundamental financial data as well as financial ratio data and study companies from the financial, healthcare, and information technology sectors in the United States. We find that using imaging techniques to input data for CNN works better for financial ratio data but is not significantly better than simply using the 1-D input directly for fundamental data. We do not find the HVA technique to be significantly better than other imaging techniques.

Visual tracking algorithms have important robotic applications such as mobile robot guidance and servoed wide area surveillance systems. These applications ideally require vision algorithms which are robust to camera motion and scene change but are cheap and fast enough to run on small, low power embedded systems. Unfortunately most robust visual tracking algorithms are either computationally expensive or are restricted to a stationary camera. This paper describes a new color based tracking algorithm, the Adaptive Background CAMSHIFT (ABCshift) tracker and an associated technique, mean shift servoing, for efficient pan-tilt servoing of a motorized camera platform. ABCshift achieves robustness against camera motion and other scene changes by continuously relearning its background model at every frame. This also enables robustness in difficult scenes where the tracked object moves past backgrounds with which it shares significant colors. Despite this continuous machine learning, ABCshift needs minimal training and is remarkably computationally cheap. We first demonstrate how ABCshift tracks robustly in situations where related algorithms fail, and then show how it can be used for real time tracking with pan-tilt servo control using only a small embedded microcontroller.