In this paper, we study tracking by language that localizes the target box sequence in a video based on a language query. We propose a framework called GTI that decomposes the problem into three sub-tasks: Grounding, Tracking, and Integration. The three sub-task modules operate simultaneously and predict the box sequence frame-by-frame. “Grounding” predicts the referred region directly from the language query. “Tracking” localizes the target based on the history of the grounded regions in previous frames. “Integration” generates final predictions by synergistically combining grounding and tracking. With the “integration” task as the key, we explore how to indicate the quality of the grounded regions in each frame and achieve the desired mutually beneficial combination. To this end, we propose an “RT-integration” method that defines and predicts two scores to guide the integration: 1) R-score represents the Region correctness whether the grounding prediction accurately covers the target, and 2) T-score represents the Template quality whether the region provides informative visual cues to improve tracking in future frames. We present our real-time GTI implementation with the proposed RT-integration, and benchmark the framework on LaSOT and Lingual OTB99 with highly promising results. Moreover, we produce a disambiguated version of LaSOT queries to facilitate future tracking by language studies.

Increasing the scale of reinforcement learning experiments has allowed researchers to achieve unprecedented results in both training sophisticated agents for video games, and in sim-to-real transfer for robotics. Typically such experiments rely on large distributed systems and require expensive hardware setups, limiting wider access to this exciting area of research. In this work we aim to solve this problem by optimizing the efficiency and resource utilization of reinforcement learning algorithms instead of relying on distributed computation. We present the "Sample Factory", a high-throughput training system optimized for a single-machine setting. Our architecture combines a highly efficient, asynchronous, GPU-based sampler with off-policy correction techniques, allowing us to achieve throughput higher than $10^5$ environment frames/second on non-trivial control problems in 3D without sacrificing sample efficiency. We extend Sample Factory to support self-play and population-based training and apply these techniques to train highly capable agents for a multiplayer first-person shooter game. Github: https://github.com/alex-petrenko/sample-factory

Traditional profiling techniques typically identify performance hot-spots. Other specialized techniques such as WCET analysis cater to safety critical real-time systems with hard constraints and thus make conservative assumptions. However, several domains motivate the need to systematically characterize and represent differential timing properties such as the variance in execution time of application artifacts such as the functions. Such domains include, vulnerability analysis of applications with regard to differential timing attacks, and the optimization of soft-real-time applications to reduce frame-rate fluctuations. In this paper, we motivate the need for execution variance as a performance measure and propose a variance-based analysis scheme. We introduce a new program representation called Variance Characterization Graph (VCG) that is used both as the intermediate representation for the variance-based analysis, and as the final representation that provides concise actionable information to programmers and optimization frameworks. We develop a methodology based on statistical pattern matching to summarize the dominant patterns of application behavior into a very compact VCG representation useful for tuning application behavior such as the soft-real-time properties.

The introduction of multicore processors on desktops and other personal computing platforms has given rise to multiple interesting end-user application possibilities. One important trend is the increased presence of resource hungry applications like gaming and multimedia applications. One of the key distinguishing factors of these applications is that they are amenable to variable semantics (ie, multiple possibilities of results) unlike traditional applications wherein a fixed, unique answer is expected. For example, varying degrees of image processing improves picture quality, different model complexities used in game physics allow different degrees of realism during game play, and so on. The goal of this paper is to demonstrate that scalable semantics in applications such as video games can be enriched with optional tasks that can be launched and thus adapt to the amount of available resources at runtime. We propose a C/C++ API that allows the programmer to define how the current semantics of a program can be opportunistically enriched, as well as the underlying runtime system that orchestrates the different computations We show how this infrastructure can be used to enrich a well known game called Quake 3. Our results show that it is possible to perform significant enrichment without degrading the application's performance by utilizing additional cores.

With core counts on the rise, the sequential components of applications are becoming the major bottleneck in performance scaling as predicted by Amdahl's law. We are therefore faced with the simultaneous problems of occupying an increasing number of cores and speeding up sequential sections. In this work, we reconcile these two seemingly incompatible problems with a novel programming model called N-way. The core idea behind N-way is to benefit from the algorithmic diversity available to express certain key computational steps. By simultaneously launching in parallel multiple ways to solve a given computation, a runtime can just-in-time pick the best (for example the fastest) way and therefore achieve speedup.

For important classes of interactive consumer applications, such as gaming and video, the Quality-of-Service requirement is to create a maximally immersive experience for the interactive user. This necessitates a trade-off between maximizing the computational complexity of application features versus the need to maintain a smooth and sufficiently high frame-rate. The implementation of these applications using conventional C/C++/Java development flows, their highly data-dependent time-varying nature, and the lack of analytical models for their execution time behavior pose unique challenges in obtaining significant QoS improvements.

Soft real-time applications lack a formal methodology for their design optimization. Well-established techniques from hard real-time systems cannot be directly applied to soft real-time applications, without losing key benefits of the soft real-time paradigm. We introduce a statistical analysis framework that is well-suited for discovering opportunities for optimization of soft real-time applications. We demonstrate how programmers can use the analysis provided by our framework to perform aggressive soft real-time design optimizations on their applications. The paper introduces the Context Execution Tree (CET) representation for capturing the statistical properties of function calls in the context of their execution in the program. The CET is constructed from an offline-profile of the application. Statistical measures are coupled with techniques that extract runtime distinguishable call-chains. This combination of techniques is applied to the CET to find statistically significant patterns of activity that i) expose slack in the execution of the application with respect to its soft real-time requirements, and ii) can be predicted with low overhead and high reliability during the normal execution of the application.