Image processing workflows typically consist of a series of different functions, each working with “image” inputs and outputs in an abstract sense. However, the specific in-memory representation of images differs between and sometimes within libraries. Conversion is therefore necessary when integrating functions from several sources into a single program. The conversion process forces users to consider low-level implementation details, including data types, color channels, channel order, minibatch layout, memory locations, and pixel intensity ranges. Specifically in the case of visual programming languages (VPLs), this distracts from high- level operations. We introduce im2im, a Python library that automates the conversion of in-memory image representations. The central concept of this library is a knowledge graph that describes image representations and how to convert between them. The system queries this knowledge graph to generate the conversion code and execute it, converting an image to the desired representation. The effectiveness of the approach is evaluated through two case studies in VPLs. In each case, we compared a workflow created in a basic blockbased VPL with the same workflow enhanced using im2im. These evaluations show that im2im automates type conversions and eliminates the need for manual intervention. Additionally, we compared the overhead of using explicit intermediate representations versus im2im, both of which avoid manual type conversions in VPLs. The results indicate that im2im generates only the necessary conversions, avoiding the runtime overhead associated with converting to and from intermediate formats. A performance comparison between the step-by-step approach used by im2im and a single-function approach demonstrates that the overhead introduced by im2im does not impact practical usability. While focused on block-based VPLs, im2im can be generalized to other VPLs and textual programming environments. Its principles are also applicable to domains other than images. The source code and analyses are available via GitHub.

Memory efficiency is crucial in training deep learning networks on resource-restricted devices. During backpropagation, forward tensors are used to calculate gradients. Despite the option of keeping those dependencies in memory until they are reused in backpropagation, some forward tensors can be discarded and recomputed later from saved tensors, so-called checkpoints . This allows, in particular, for resource-constrained heterogeneous environments to make use of all available compute devices. Unfortunately, the definition of these checkpoints is a non-trivial problem and poses a challenge to the programmer—improper or excessive recomputations negate the benefit of checkpointing. In this article, we present XEngine, an approach that schedules network operators to heterogeneous devices in low memory environments by determining checkpoints and recomputations of tensors. Our approach selects suitable resources per timestep and operator and optimizes the end-to-end time for neural networks taking the memory limitation of each device into account. For this, we formulate a mixed-integer quadratic program (MIQP) to schedule operators of deep learning networks on heterogeneous systems. We compare our MIQP solver XEngine against Checkmate [ 12 ], a mixed-integer linear programming (MILP) approach that solves recomputation on a single device. Our solver finds solutions that are up to 22.5% faster than the fastest Checkmate schedule in which the network is computed exclusively on a single device. We also find valid schedules for networks making use of both central processing units and graphics processing units if memory limitations do not allow scheduling exclusively to the graphics processing unit.

In this paper we present a novel method to detect guard rails in highway scenarios using a stereo camera setup. In contrast to previous methods, we combine geometry information with appearance cues using a state-of-the-art feature encoding method. In our system pipeline, we follow a hough-based approach to localize potential guard rails in the image and require each detected line to fulfill linearity in depth as well as certain height expectations. To leverage the appearance information, we exploit an efficient bag-of-features representation that relies on randomized clustering forests. The effectiveness of our approach is demonstrated on a large novel dataset with pixel-level annotations of guard rails in real-world highway scenarios.