Tesla’s in-house supercomputer for Machine Learning

This article consists of a collection of slides from the author's conference presentation.

The special section in this issue focuses on the topic of computer arithmetic.

An IEEE compliant, 1 GHz Sparc64-V Floating-Point Unit (FPU) with reliability-accessibility-serviceability (RAS) features and partial support for denormal operands and results is presented. The FPU has two functional units, each with an adder (FADD) and a multiplier (FMUL). Additionally, one of the functional units contains a graphics unit (VIS). Two floating-point instructions can be scheduled out of order each cycle, one to each functional unit. A peak performance of 4 GFLOP is achieved by scheduling two floating-point multiply add (FMA) instructions each cycle. The FADD unit is fully pipelined and can execute an addition, subtraction, conversion, or compare instruction every cycle. The FMUL unit executes pipelined multiply instructions. Divide and square-root instructions are executed with multiple iterations through the multiplier pipeline. The VIS unit is also pipelined and executes SIMD fixed-point graphics instructions. The adder and multiplier have latencies of 3 and 4 cycles respectively. Novel techniques are presented in the adder and multiplier implementations to reduce area and cycle time. The FPU provides RAS support for enhanced server reliability by using selective parity error detection. The FPU has been implemented in 0.15 u, 6-layer metal CMOS technology.

Describes a table compression method employing finite-precision linear interpolation in reciprocal tables. The interpolation method employs a compressed look-up table and a small-sized multiplier to yield an output reciprocal as a simple direct operation. The leading bits of the arbitrarily precise input are used to index the table, and a limited number of succeeding fractional bits are used to interpolate on the table employing a multiply/add operation. The low-order bits of the product are rounded off so that the output reciprocals are guaranteed to be correct to a unit in the last place and provide a round-to-nearest reciprocal for over 90% of arbitrarily precise input arguments. The interpolation method generates 2k-bit faithful reciprocals employing a k-bits-in, (2k+2)-bits-out table and a (k+3)/spl times/(k+3)-bit multiplier. A single-precision faithful reciprocal can be generated employing a table of size 13 kBytes and a 15/spl times/15-bit multiplier, compared to a table size of 46 MBytes for conventional reciprocal tables. The table and dedicated small multiplier efficiently characterize a functional reciprocator unit with, at most, a 2-cycle latency.

We describe bipartite reciprocal tables that employ separate table lookup of the positive and negative portions of a borrow-save reciprocal value. The fusion of the parts includes a rounding so the output reciprocals are guaranteed correct to a unit in the last place, and typically provide a round-to-nearest reciprocal for over 90% of input arguments. The output rounding can be accomplished in conjunction with multiplier recoding yielding practically no cost in logic complexity or time in employing bipartite tables. We demonstrate these tables to be 2 to 4 times smaller than conventional 4-bit reciprocal tables. For 10-16 bit reciprocal table lookup the compression grows from a factor of 4 to over 16, making possible the use of larger seed reciprocals than previously considered cost effective.>

We prove that a convenient ROM reciprocal table construction algorithm generates tables that minimize the relative error. The worst case relative errors realized for such optimally computed k-bits-in, m-bits-out ROM reciprocal tables are then determined for all table sizes 3/spl les/k,m/spl les/12. We next prove the table construction algorithm always generates a k-bits-in, k-bits-out table with relative errors never any greater than 3/4 2/sup -k/, and more generally with g guard bits that for (k+g)-bits-out the relative error is never any greater than 2-(k+1)(1+1/(2g+1)). To provide for determining extreme case test data and to compute the precision of a reciprocal table without prior construction of the full ROM reciprocal table, we describe a procedure that requires generation and inspection of only a small portion of such a table to identify input values guaranteed to include the worst case relative errors in the table.>

It is proved that a conventional ROM reciprocal table construction algorithm generates tables that minimize the relative error. The worst case relative errors realized for such optimally computed k-bits-in, m-bits-out ROM reciprocal tables are then determined for all table sizes 3 /spl les/ k, m /spl les/ 12. It is then proved that the table construction algorithm always generates a k-bits-in, k-bits-out table with relative errors never any greater than 3(2/sup -k/)/4 for any k, and, more generally with g guard bits, that for (k + g)-bits-out the relative error is never any greater than 2/sup -(k+1)/(1 + 1/(2/sup g+1/)). To provide for determining test data without prior construction of a full ROM reciprocal table, a procedure that requires generation and searching of only a small portion of such a table to determine regions containing input data yielding the worst case relative errors is described.>