We present an algorithm to compute the geodesic $L_1$ farthest-point Voronoi diagram of $m$ point sites in the presence of $n$ rectangular obstacles in the plane. It takes $O(nm+n \log n + m\log m)$ construction time using $O(nm)$ space. This is the first optimal algorithm for constructing the farthest-point Voronoi diagram in the presence of obstacles. We can construct a data structure in the same construction time and space that answers a farthest-neighbor query in $O(\log(n+m))$ time.