Deep Learning Recommendation Models (DLRM) are widespread, account for a considerable data center footprint, and grow by more than 1.5x per year. With model size soon to be in terabytes range, leveraging Storage Class Memory (SCM) for inference enables lower power consumption. This paper evaluates the major challenges in extending the memory hierarchy to SCM for DLRM, and presents different techniques to improve performance through a Software Defined Memory. We show how underlying technologies such as Nand Flash and 3DXP differentiate, and relate to real world scenarios, enabling from 5% to 29% power savings.

The needs of a realtime vision aid require that it functions immediately, not merely for production of a picture or video record to be viewed later. Therefore the vision system must offer a high dynamic range (often hundreds of millions to one) that functions in real time. In compliment with the existing efficient and real-time HDR compositing algorithms, we propose a novel method for compressing High Dynamic Range (HDR) images by Per-Pixel Exposure Mapping (PPEM). Unlike any existing methods, PPEM only varies exposure to achieve tone mapping. It takes advantage of the camera response to enable exposure synthesis which we call Wyckoff set expansion. The method evaluates the synthetic exposures in a recursive pairwise process to generate a tone mapped HDR image. The results can be approximated using a look-up table, which can be used for real-time HDR applications.

Three-Dimensional (3D) range cameras have recently appeared in the marketplace for use in surveillance (e.g. cameras affixed to inanimate objects) applications. We present FreeGlass™ as a wearable hands-free 3D gesture-sensing Digital Eye Glass system. FreeGlass comprises a head-mounted display with an infrared range camera, both connected to a wearable computer. It is based on the MannGlas™ computerized welding glass, which embodies HDR (High Dynamic Range) and AR (Augmented/Augmediated Reality). FreeGlass recontextualizes the 3D range camera as a sousveillance (e.g. cameras attached to people) camera. In this sousveillance context, the range camera is worn by the user and shares the same point-of-view as the user. Computer vision algorithms therefore benefit from the use of the range camera to allow image segmentation by using both the infrared and depth information from the device for 3D hand gesture recognition system. The gesture recognition is then accomplished by using a neural network on the segmented hand. Recognized gestures are used to provide the user with interactions in an augmediated reality environment. Additionally, we present applications of FreeGlass for serendipitous gesture recognition in everyday life, as well as for interaction with real-world objects (with and without gesture recognition). A plurality of FreeGlass units can be used together, each sensor having a different spreading sequence, or the like, so that a number of people can collaborate and share the same or similar Augmediated Reality space(s).

This paper presents the invention and implementation of 3D (Three Dimensional) HDR (High Dynamic Range) sensing, along with examples. We propose a method of 3D HDR veillance (sensing, computer vision, video capture, or the like) by integrating tonal and spatial information obtained from multiple HDR exposures for use in conjunction with one or more 3D cameras. In one embodiment, we construct a 3D HDR camera from multiple 3D cameras such as Kinect sensors. In this embodiment the 3D cameras are arranged in a fixed array, such that the geometric relationships between them remain constant over time. Only a single camera calibration step is required at the initial time of assembling and fixing the cameras into the array. Preferably the cameras either view from the same position through beam splitters, or are fixed close to one another, so that they capture approximately the same subject matter. The system is designed so the cameras each capture a differently exposed image or video of approximately the same subject matter. In one embodiment, two Kinect cameras are attached together facing in the same direction, with an ND (Neutral Density) filter over one of them, so as to obtain a darker exposure. The dark and light exposures are combined to obtain more accurate 3D sensing in high contrast scenes. In another embodiment, a single 3D camera is exposure-sequenced (alternating light and dark exposures). 3D HDR might, more generally, be incorporated into existing 3D cameras, resulting in a new kind of 3D sensor that can work in nearly any environment, including high contrast scenes such as outdoor scenes, or scenes where a bright light is shining directly into the sensor.

We present highly parallelizable and computationally efficient High Dynamic Range (HDR) image compositing, reconstruction, and spatotonal mapping algorithms for processing HDR video. We implemented our algorithms in the EyeTap Digital Glass electric seeing aid, for use in everyday life. We also tested the algorithms in extreme dynamic range situations, such as, electric arc welding. Our system runs in real-time, and requires no user intervention, and no fine-tuning of parameters after a one-time calibration, even under a wide variety of very difficult lighting conditions (e.g. electric arc welding, including detailed inspection of the arc, weld puddle, and shielding gas in TIG welding). Our approach can render video at 1920x1080 pixel resolution at interactive frame rates that vary from 24 to 60 frames per second with GPU acceleration. We also implemented our system on FPGAs (Field Programmable Gate Arrays) for being miniaturized and built into eyeglass frames.

We present a way finding system that uses a range camera and an array of vibrotactile elements which we built into a helmet.