Recent advances in machine learning have leveraged dramatic increases in computational power, a trend expected to continue in the future. This paper introduces the first Hyperscale Hardware Optimized Neural Architecture Search (H2O-NAS) to automatically design accurate and performant machine learning models tailored to the underlying hardware architecture. H2O-NAS consists of three key components: a new massively parallel “one-shot” search algorithm with intelligent weight sharing, which can scale to search spaces of O(10280) and handle large volumes of production traffic; hardware-optimized search spaces for diverse ML models on heterogeneous hardware; and a novel two-phase hybrid performance model and a multi-objective reward function optimized for large scale deployments.

We present a novel approach for unsupervised learning of depth and ego-motion from monocular video. Unsupervised learning removes the need for separate supervisory signals (depth or ego-motion ground truth, or multi-view video). Prior work in unsupervised depth learning uses pixel-wise or gradient-based losses, which only consider pixels in small local neighborhoods. Our main contribution is to explicitly consider the inferred 3D geometry of the whole scene, and enforce consistency of the estimated 3D point clouds and ego-motion across consecutive frames. This is a challenging task and is solved by a novel (approximate) backpropagation algorithm for aligning 3D structures. We combine this novel 3D-based loss with 2D losses based on photometric quality of frame reconstructions using estimated depth and ego-motion from adjacent frames. We also incorporate validity masks to avoid penalizing areas in which no useful information exists. We test our algorithm on the KITTI dataset and on a video dataset captured on an uncalibrated mobile phone camera. Our proposed approach consistently improves depth estimates on both datasets, and outperforms the state-of-the-art for both depth and ego-motion. Because we only require a simple video, learning depth and ego-motion on large and varied datasets becomes possible. We demonstrate this by training on the low quality uncalibrated video dataset and evaluating on KITTI, ranking among top performing prior methods which are trained on KITTI itself.

We consider the problem of next frame prediction from video input. A recurrent convolutional neural network is trained to predict depth from monocular video input, which, along with the current video image and the camera trajectory, can then be used to compute the next frame. Unlike prior next-frame prediction approaches, we take advantage of the scene geometry and use the predicted depth for generating the next frame prediction. Our approach can produce rich next frame predictions which include depth information attached to each pixel. Another novel aspect of our approach is that it predicts depth from a sequence of images (e.g. in a video), rather than from a single still image. We evaluate the proposed approach on the KITTI dataset, a standard dataset for benchmarking tasks relevant to autonomous driving. The proposed method produces results which are visually and numerically superior to existing methods that directly predict the next frame. We show that the accuracy of depth prediction improves as more prior frames are considered.

Creating and maintaining a platform for reliably producing and deploying machine learning models requires careful orchestration of many components---a learner for generating models based on training data, modules for analyzing and validating both data as well as models, and finally infrastructure for serving models in production. This becomes particularly challenging when data changes over time and fresh models need to be produced continuously. Unfortunately, such orchestration is often done ad hoc using glue code and custom scripts developed by individual teams for specific use cases, leading to duplicated effort and fragile systems with high technical debt.

We present a framework for specifying, training, evaluating, and deploying machine learning models. Our focus is on simplifying cutting edge machine learning for practitioners in order to bring such technologies into production. Recognizing the fast evolution of the field of deep learning, we make no attempt to capture the design space of all possible model architectures in a domain-specific language (DSL) or similar configuration language. We allow users to write code to define their models, but provide abstractions that guide developers to write models in ways conducive to productionization. We also provide a unifying Estimator interface, making it possible to write downstream infrastructure (e.g. distributed training, hyperparameter tuning) independent of the model implementation.

This article describes a Lagrangian finite element method that simulates the behavior of liquids and solids in a unified framework. Local mesh improvement operations maintain a high-quality tetrahedral discretization even as the mesh is advected by fluid flow. We conserve volume and momentum, locally and globally, by assigning to each element an independent rest volume and adjusting it to correct for deviations during remeshing and collisions. Incompressibility is enforced with per-node pressure values, and extra degrees of freedom are selectively inserted to prevent pressure locking. Topological changes in the domain are explicitly treated with local mesh splitting and merging. Our method models surface tension with an implicit formulation based on surface energies computed on the boundary of the volume mesh. With this method we can model elastic, plastic, and liquid materials in a single mesh, with no need for explicit coupling. We also model heat diffusion and thermoelastic effects, which allow us to simulate phase changes. We demonstrate these capabilities in several fluid simulations at scales from millimeters to meters, including simulations of melting caused by external or thermoelastic heating.

This paper extends Galerkin projection to a large class of non-polynomial functions typically encountered in graphics. We demonstrate the broad applicability of our approach by applying it to two strikingly different problems: fluid simulation and radiosity rendering, both using deforming meshes. Standard Galerkin projection cannot efficiently approximate these phenomena. Our approach, by contrast, enables the compact representation and approximation of these complex non-polynomial systems, including quotients and roots of polynomials. We rely on representing each function to be model-reduced as a composition of tensor products, matrix inversions, and matrix roots. Once a function has been represented in this form, it can be easily model-reduced, and its reduced form can be evaluated with time and memory costs dependent only on the dimension of the reduced space.

Understanding human mobility in an environment can be approached in many forms, one of which is to recover the underlying structure of user movement. In our work, we show that we can use a network of binary proximity sensors to detect paths between nodes and also extract highly popular trajectories users take. We show that with sufficient amount of these binary data, even with no prior knowledge of the location of these sensors, we can capture a correlation between the detection timestamps in the case where a physical path exists between any two nodes. Our algorithm also generates characteristics of the path, such as the distribution of transition times and volume. We further show that with sampling techniques we can estimate the underlying trajectories that generated the time stamps. We have tested our algorithm on a simulator and two sensor network deployments. We found that, despite the lack of position information about the sensor nodes, with timestamps alone our algorithm can accurately detect the trajectories and is robust enough to use in a real-world office building.

In this paper, we present a routing scheme that exploits knowledge about the behavior of mobile sinks within a network of data sources to minimize energy consumption and network congestion. For delay-tolerant network applications, we propose to route data not to the sink directly, but to send it instead to a relay node along an announced or predicted path of the mobile node that is close to the data source. The relay node will stash the information until the mobile node passes by and picks up the data. We use linear programming to find optimal relay nodes that minimize the number of necessary transmissions while guaranteeing robustness against link and node failures, as well as trajectory uncertainty.

Abstract We present physically based algorithms for interactive deformable shape and motion modelling. We coarsely sample the objects with simulation nodes, and apply a meshless finite element method to obtain realistic deformations at interactive frame rates. This shape deformation algorithm is then used to specify keyframe poses and a smooth interpolating motion is obtained by solving for an energy‐minimizing trajectory. We show how to handle collisions between different deformable objects as well as with static or moving scene objects. Secondary motion is added as a post‐process by running a meshless elastic solid simulation. We enforce precomputed trajectories using control forces computed using shape matching. Key to the efficiency of our method is a sparse deformation representation and an adaptive optimization algorithm that automatically introduces new degrees of freedom in problematic regions. An accurate temporal interpolation scheme that exactly recovers rigid motions keeps the number of unknowns low and achieves realistic deformations with very few keyframes. We also show how the algorithm allows combining purely physical simulation with keyframe‐based scripted animation. The presented results illustrate that our framework can handle complex shapes at interactive rates, making it a valuable tool for animators to realistically model deformable 3D shapes and their motion.

We propose a finite element simulation method that addresses the full range of material behavior, from purely elastic to highly plastic, for physical domains that are substantially reshaped by plastic flow, fracture, or large elastic deformations. To mitigate artificial plasticity, we maintain a simulation mesh in both the current state and the rest shape, and store plastic offsets only to represent the non-embeddable portion of the plastic deformation. To maintain high element quality in a tetrahedral mesh undergoing gross changes, we use a dynamic meshing algorithm that attempts to replace as few tetrahedra as possible, and thereby limits the visual artifacts and artificial diffusion that would otherwise be introduced by repeatedly remeshing the domain from scratch. Our dynamic mesher also locally refines and coarsens a mesh, and even creates anisotropic tetrahedra, wherever a simulation requests it. We illustrate these features with animations of elastic and plastic behavior, extreme deformations, and fracture.

We present a method to extract topology information from detection events of mobile entities moving through a network of binary sensors. We extract the topological structure of possible paths in the network by analyzing the time correlation of events at different sensors. The histograms of time delays between any two sensors contain the necessary information to reconstruct the network topology. This data is heavily corrupted by noise due to multiple agents in the network. We therefore use a mixture model of multiple Gaussian and a uniform distribution to explicitly isolate the noise. Our algorithm yields a graph representing the topology of our sensor network along with average travel time between nodes.

Abstract We introduce a novel algorithm that decomposes a deformable shape into meaningful parts requiring only a single input pose. Using modal analysis, we are able to identify parts of the shape that tend to move rigidly. We define a deformation energy on the shape, enabling modal analysis to find the typical deformations of the shape. We then find a decomposition of the shape such that the typical deformations can be well approximated with deformation fields that are rigid in each part of the decomposition. We optimize for the best decomposition, which captures how the shape deforms. A hierarchical refinement scheme makes it possible to compute more detailed decompositions for some parts of the shape. Although our algorithm does not require user intervention, it is possible to control the process by directly changing the deformation energy, or interactively refining the decomposition as necessary. Due to the construction of the energy function and the properties of modal analysis, the computed decompositions are robust to changes in pose as well as meshing, noise, and even imperfections such as small holes in the surface.

We present a new approach to fluid simulation that balances the speed of model reduction with the flexibility of grid-based methods. We construct a set of composable reduced models, or tiles , which capture spatially localized fluid behavior. We then precompute coupling terms so that these models can be rearranged at runtime. To enforce consistency between tiles, we introduce constraint reduction . This technique modifies a reduced model so that a given set of linear constraints can be fulfilled. Because dynamics and constraints can be solved entirely in the reduced space, our method is extremely fast and scales to large domains.

Abstract We present a robust and efficient algorithm for the pairwise non‐rigid registration of partially overlapping 3D surfaces. Our approach treats non‐rigid registration as an optimization problem and solves it by alternating between correspondence and deformation optimization. Assuming approximately isometric deformations, robust correspondences are generated using a pruning mechanism based on geodesic consistency. We iteratively learn an appropriate deformation discretization from the current set of correspondences and use it to update the correspondences in the next iteration. Our algorithm is able to register partially similar point clouds that undergo large deformations, in just a few seconds. We demonstrate the potential of our algorithm in various applications such as example based articulated segmentation, and shape interpolation.

Abstract Finite element simulations in computer graphics are typically based on tetrahedral or hexahedral elements, which enables simple and efficient implementations, but in turn requires complicated remeshing in case of topological changes or adaptive refinement. We propose a flexible finite element method for arbitrary polyhedral elements, thereby effectively avoiding the need for remeshing. Our polyhedral finite elements are based on harmonic basis functions, which satisfy all necessary conditions for FEM simulations and seamlessly generalize both linear tetrahedral and trilinear hexahedral elements. We discretize harmonic basis functions using the method of fundamental solutions, which enables their flexible computation and efficient evaluation. The versatility of our approach is demonstrated on cutting and adaptive refinement within a simulation framework for corotated linear elasticity.

Abstract We propose a new adaptive space deformation method for interactive shape modeling. A novel energy formulation based on elastically coupled volumetric cells yields intuitive detail preservation even under large deformations. By enforcing rigidity of the cells, we obtain an extremely robust numerical solver for the resulting nonlinear optimization problem. Scalability is achieved using an adaptive spatial discretization that is decoupled from the resolution of the embedded object. Our approach is versatile and easy to implement, supports thin‐shell and solid deformations of 2D and 3D objects, and is applicable to arbitrary sample‐based representations, such as meshes, triangle soups, or point clouds.

Abstract We present a method for animating deformable objects using a novel finite element discretization on convex polyhedra. Our finite element approach draws upon recently introduced 3D mean value coordinates to define smooth interpolants within the elements. The mathematical properties of our basis functions guarantee convergence. Our method is a natural extension to linear interpolants on tetrahedra: for tetrahedral elements, the methods are identical. For fast and robust computations, we use an elasticity model based on Cauchy strain and stiffness warping. This more flexible discretization is particularly useful for simulations that involve topological changes, such as cutting or fracture. Since splitting convex elements along a plane produces convex elements, remeshing or subdivision schemes used in simulations based on tetrahedra are not necessary, leading to less elements after such operations. We propose various operators for cutting the polyhedral discretization. Our method can handle arbitrary cut trajectories, and there is no limit on how often elements can be split.

We present a novel framework for the efficient simulation and animation of discrete thin shells.Our method takes a point sampled surface as input and performs all necessary computations without intermediate triangulation.We discretize the thin shell functional using so-called fibers.Such fibers are locally embedded parametric curves crisscrossing individual point samples.In combination, they create a dense mesh representing the surface structure and connectivity for the shell computations.In particular, we utilize the fibers to approximate the differential surface operators of the thin shell functional.The polynomials underlying the fiber representation allow for a robust and fast simulation of thin shell behavior.Our method supports both elastic and plastic deformations as well as fracturing and tearing of the material.To compute surfaces with rich surface detail, we designed a multiresolution representation which maps a high-resolution surface onto a fiber network of lower resolution.This makes it possible to animate densely sampled models of very high surface complexity.While being tuned for point sampled objects, the presented framework is versatile and can also take triangle meshes or triangle soups as input.

This paper presents an algorithm for rendering of point-sampled CSG models. The approach works with arbitrary CSG trees of surfel models with arbitrary sampling densities. Edges and corners are rendered by reconstructing the involved surfaces separately. The reconstructed surfaces are clipped at intersections. This way, blurring at any magnification is avoided. As opposed to existing methods, which resample surfaces close to object intersections, the proposed approach preserves the original object representation. Since no resampling is needed, dynamic scenes can be handled very flexible. Complex intersections involving any number of objects can be rendered.