Reinforcement learning tasks that involve visual input continue to pose a challenge when it comes to partial observability problems. Although prior research has introduced methods such as LSTM, GTrXL, and DNC, each of these approaches possesses its own limitations. To address the problem of partial observability in a more universal context, this paper proposed the Observation-Time-Action deep stacking algorithm. First, observations, actions, and time data were combined into a tuple and then stacked into a longer sequence. Then, convolution and fully connected layers were utilized to extract relevant features from the sequence, which were then fed into the algorithm for processing. We designed a number of experiments with partial observability, corresponding to different typical scenarios in reinforcement learning. The experiment results demonstrated that the proposed method obtained a higher success rate. Moreover, we also investigated the effect of stacking frame length and different reinforcement learning elements on the algorithm. Finally, we conducted a HITL (Hardware-in-the-Loop) experiment to further verify the effectiveness of our algorithm.

Memory corruption diagnosis, especially at the binary level where all high-level program abstractions are missing, is a tedious and time-consuming task. Given a crash, memory corruption diagnosis is expected to not only locate the root cause of the vulnerability, but also deliver rich semantics to understand the vulnerability. However, existing techniques can barely satisfy the above requirements. In this article, we present${{\sf MemRay}}$, a dynamic memory corruption diagnosis technique. The insight behind our approach is that most memory corruption is caused by malformed inputs, which further leads the vulnerable program to manipulate inputs by referencing invalid data structures. We design the “data structure reference sequence” to characterize how a program references various data structures to manipulate program inputs. Then, we identify memory corruptions by detecting violations in the input manipulations via data structures. We demonstrate the effectiveness of${{\sf MemRay}}$on a wide range of memory-corruption vulnerabilities. The result shows that${{\sf MemRay}}$precisely locates the root cause of vulnerabilities. Moreover, the “data structure reference” enables${{\sf MemRay}}$to deliver rich semantics and context information to assist vulnerability diagnosis on binary code.