One of the long-standing questions in search systems is the role of diversity in results. From a product perspective, showing diverse results provides the user with more choice and should lead to an improved experience. However, this intuition is at odds with common machine learning approaches to ranking which directly optimize the relevance of each individual item without a holistic view of the result set. In this paper, we describe our journey in tackling the problem of diversity for Airbnb search, starting from heuristic based approaches and concluding with a novel deep learning solution that produces an embedding of the entire query context by leveraging Recurrent Neural Networks (RNNs). We hope our lessons learned will prove useful to others and motivate further research in this area.

The application of deep learning to search ranking was one of the most impactful product improvements at Airbnb. But what comes next after you launch a deep learning model? In this paper we describe the journey beyond, discussing what we refer to as the ABCs of improving search: A for architecture, ℬ for bias and ℂ for cold start. For architecture, we describe a new ranking neural network, focusing on the process that evolved our existing DNN beyond a fully connected two layer network. On handling positional bias in ranking, we describe a novel approach that led to one of the most significant improvements in tackling inventory that the DNN historically found challenging. To solve cold start, we describe our perspective on the problem and changes we made to improve the treatment of new listings on the platform. We hope ranking teams transitioning to deep learning will find this a practical case study of how to iterate on DNNs.

The application to search ranking is one of the biggest machine learning success stories at Airbnb. Much of the initial gains were driven by a gradient boosted decision tree model. The gains, however, plateaued over time. This paper discusses the work done in applying neural networks in an attempt to break out of that plateau. We present our perspective not with the intention of pushing the frontier of new modeling techniques. Instead, ours is a story of the elements we found useful in applying neural networks to a real life product. Deep learning was steep learning for us. To other teams embarking on similar journeys, we hope an account of our struggles and triumphs will provide some useful pointers. Bon voyage!

High-quality, photorealistic image-based rendering datasets are typically too large to download entirely before viewing, even when compressed. It is more suitable to instead stream the required image data to a remote user who can start interacting with the dataset immediately. This paper presents an interactive light field streaming system and proposes packet scheduling for transmitting the encoded image data in a rate-distortion optimized manner. An interactive light field streaming system must have low user latency. The system presented in this paper predicts the future user viewing trajectory to mitigate the effects of the low-latency constraints. Experimental results show that view prediction can improve performance, and that this improvement is limited by the prediction accuracy. The proposed packet scheduling algorithm considers network conditions and rate-distortion cost, knowledge of sent and received images, and the distortion for a set of images, to optimize the rendered image quality for the remote user. Rate-distortion optimized scheduling can be implemented either at the receiver or the sender. It is shown that this rate-distortion optimized packet scheduling can significantly improve performance over a heuristic scheduling approach. Experimental results also show that the encoding prediction dependency structure affects streaming performance both through the compression efficiency of the encoding and also through any decoding dependencies that may be introduced

We propose disparity-compensated lifting for wavelet compression of light fields. With this approach, we obtain the benefits of wavelet coding, such as scalability in all dimensions, as well as superior compression performance. Additionally, the proposed approach solves the irreversibility limitations of previous light field wavelet coding approaches, using the lifting structure. Our scheme incorporates disparity compensation into the lifting structure for the transform across the views in the light field data set. Another transform is performed to exploit the coherence among neighboring pixels, followed by a modified SPIHT coder and rate-distortion optimized bitstream assembly. A view-sequencing algorithm is developed to organize the views for encoding. For light fields of an object, we propose to use shape adaptation to improve the compression efficiency and visual quality of the images. The necessary shape information is efficiently coded based on prediction from the existing geometry model. Experimental results show that the proposed scheme exhibits superior compression performance over existing light field compression techniques.

Previous work on rate-distortion optimized streaming of light fields assumes that the scheduling task is performed by a central server. To scale to large numbers of simultaneous users, however, it is necessary to shift the computational burden of image scheduling to the clients. In this paper, we present a receiver-driven streaming framework where the packet scheduling is performed at the client. One key challenge of receiver-based streaming is in efficiently transmitting the rate-distortion preamble necessary for the scheduling algorithm. We present a method to efficiently represent and estimate the view-dependent distortions, and show that the performance is nearly as good as if the actual distortion values were used. We also show that the sender-driven and receiver-driven systems have very similar rate-distortion streaming performance.

Image-based rendering data sets, such as light fields, require efficient compression due to their large data size, but also easy random access when rendering from the data set. Efficient compression usually depends upon prediction between images, which creates dependencies between them, conflicting with the requirement of having easy random access. Existing light field coders concentrate either on compression efficiency, or use ad hoc methods to design prediction that balances random access and compression efficiency requirements. In this paper, we study this joint problem of compression efficiency and random access. We propose a model for light field image generation, light field image coding and rendering novel views from these light field images. We present a view-dependent rate-distortion measure that allows us to consider random access and compression efficiency simultaneously. We compare the theoretical results from the model with the experimental results from our DCT-based coder and show that they qualitatively give similar results. Finally, we suggest how. with this model, we can better optimize the prediction dependency structure in our coder for random access and compression efficiency performance.

Image-based rendering data sets, such as light fields, require efficient compression due to their large data size, but also easy random access when rendering from the data set. Efficient compression usually depends upon prediction between images, which creates dependencies between them, conflicting with the requirement of having easy random access. In this paper we propose to eliminate prediction at the encoder by using Wyner-Ziv coding for compressing the light field images. The images are independently encoded by a Wyner-Ziv encoder. At the receiver, previously reconstructed images are used by the Wyner-Ziv decoder as side information to exploit similarities among images. Simulation results show significant compression performance gains compared to conventional independent image coding while maintaining random access capabilities.

Image-based rendering data sets, such as light fields, require efficient compression due to their large data size, but also easy random access when rendering from or streaming the data set. Efficient compression usually depends upon prediction between images, which creates dependencies between them, making random access more difficult. Independent encoding of the images is less efficient but provides random access, and in many cases is superior to predictive coding when considering the rate-distortion performance for certain viewing trajectories of the light field. In this paper, we describe a method which can obtain the best of both worlds: random access to any image; and compression efficiency, through predictive coding. We store multiple representations of each light field image, and decode using the appropriate representation depending on the available images that are already decoded. To eliminate prediction mismatch between the representations, we employ the idea of "SP-frames" from video coding. This allows us to generate identical reconstruction images from all representations. Our experimental results show a significant improvement in terms of view-trajectory-dependent rate-distortion performance over independent coding of the images.

We propose a novel approach for light field compression that incorporates disparity compensation into 4D wavelet coding using disparity-compensated lifting. With this approach, we obtain the benefits of wavelet coding, including compression efficiency and scalability in all dimensions. Additionally, our proposed approach solves the irreversibility limitations of previous wavelet coding approaches. Experimental results show that the compression efficiency of the proposed technique outperforms current state-of-the-art wavelet coding techniques by a wide margin.

We propose a method of using shape adaptation for compression of light fields of 3D objects. Shape adaptation is incorporated into two light field coders, both applying disparity compensation with an explicit geometry model, to improve the compression efficiency. The shape information can be derived at the decoder from an accurate geometry model. If the available geometry is inaccurate, we propose to code the exact 2D shapes with the aid of the approximate geometry. Experiments show that shape adaptation greatly improves the compression performance of both coders.

We extend a recently-proposed framework for the rate-distortion optimized transmission of packetized media. The original framework assumed that each packet has a single arrival deadline and that a packet is useless if it arrives after its deadline. In practice, however, packets may be associated with multiple deadlines. Examples include the case of compressed video that uses bi-directional prediction and the case of decoders that can recover from late packet arrivals through the accelerated retroactive decoding of the dependency chain. We extend the original framework to consider multiple deadlines. In our experimental results for the case of the accelerated retroactive decoding of late packets, the multiple-deadline formulation yields up to a 3.15 dB improvement in rate-distortion performance compared to the original, single-deadline formulation. The results indicate, furthermore, that accelerated retroactive decoding offers significant benefit only when coupled with a scheduler that considers multiple deadlines.

We propose a framework for the streaming of light fields over a lossy error-prone packet network. This system is optimized for the end-user according to a rate-distortion criterion. We build upon recent rate-distortion optimized streaming work for audio and video data. We extend this work to light field image data by introducing view-dependent distortion, multiple playout deadlines, and state-based distortion. In our experimental results, the rate-distortion optimized framework has a bit-rate reduction of up to 75% over a heuristic rule-based system.

We propose a novel approach for light field compression that incorporates disparity compensation into 4-D wavelet coding using disparity-compensated lifting. With this approach, we obtain the benefits of wavelet coding, including compression efficiency and scalability in all dimensions. Additionally, our proposed approach solves the irreversibility limitations of previous wavelet coding approaches. Experimental results show that the compression efficiency of the proposed technique outperforms current state-of-the-art wavelet coding techniques by a wide margin.

To store and transmit the large amount of image data necessary for Image-based Rendering (IBR), efficient coding schemes are required. This paper presents two different approaches which exploit three-dimensional scene geometry for multi-view compression. In texture-based coding, images are converted to view-dependent texture maps for compression. In model-aided predictive coding, scene geometry is used for disparity compensation and occlusion detection between images. While both coding strategies are able to attain compression ratios exceeding 2000:1, individual coding performance is found to depend on the accuracy of the available geometry model. Experiments with real-world as well as synthetic image sets show that texture-based coding is more sensitive to geometry inaccuracies than predictive coding. A rate-distortion theoretical analysis of both schemes supports these findings. For reconstructed approximate geometry models, model-aided predictive coding performs best, while texture-based coding yields superior coding results if scene geometry is exactly known.

In geometry-aided light field compression, a geometry model is used for disparity-compensated prediction of light field images from already encoded light field images. This geometry model, however, may have limited accuracy. We present an algorithm that refines a geometry model to improve the overall light field compression efficiency. This algorithm uses an optical-flow technique to explicitly minimize the disparity-compensated prediction error. Results from experiments performed on both real and synthetic data sets show bit-rate reductions of approximately 10% using the improved geometry model over a silhouette-reconstructed geometry model.

In current light field compression algorithms, geometry information is critical for efficient compression of light field image data. Inaccuracies in the geometry can greatly affect the compression efficiency. In this paper, we present a theoretical framework with which we can analyze and understand the effects of geometry inaccuracy on the compression efficiency of a light field coder. This framework combines ideas from statistical signal processing and rate-distortion theory. We demonstrate using simulations that the theory qualitatively matches the actual results of light field coding algorithms.

In this paper, we describe a multi-hypothesis prediction technique for disparity-compensated light field compression. Multi-hypothesis prediction has been used extensively in video compression. In this work, we apply multi-hypothesis prediction to the problem of light field compression. Most current techniques for light field compression utilize some form of disparity compensation with one hypothesis. We demonstrate that a multi-hypothesis approach, where two hypotheses are used, improves the overall efficiency of a light field coder. Our experimental results show an image quality gain of up to 1 dB in PSNR on our test data sets.