The rapid growth in both computational density and scale in AI models in recent years motivates the construction of an efficient and reliable dedicated network infrastructure. This paper presents the design, implementation, and operation of Meta's Remote Direct Memory Access over Converged Ethernet (RoCE) networks for distributed AI training.

Video scene detection is the task of temporally segmenting a video into its basic story units called scenes. We propose a temporal context aware scene detection method. For each shot in a video, we store the time-indexed features of its surrounding shots as its context memory. A context-reading operation is performed to read the most relevant information from the memory which is used to update the feature of the query shot. To adaptively determine the temporal scale of context memory for different queries, we apply a bank of context memories of different temporal scales to generate multiple context reads, and adaptively aggregate them according to their confidence scores. The adaptive context-reading is guided by a structure learning objective which encourages each shot to read the most appropriate context such that the global structure of scene can be revealed in the feature space. With the context-aware shot features learned by our method, we perform clustering to find the scene boundaries. Our experiments demonstrate that adaptively modeling temporal context yields the state-of-the-art results on the existing video scene detection datasets. We also construct a large-scale dataset for the task and our ablation studies on it show that the performance gains owe to the proposed adaptive context reading.

Automatic image aesthetics assessment is important for a wide variety of applications such as on-line photo suggestion, photo album management and image retrieval. Previous methods have focused on mapping the holistic image content to a high or low aesthetics rating. However, the composition information of an image characterizes the harmony of its visual elements according to the principles of art, and provides richer information for learning aesthetics. In this work, we propose to model the image composition information as the mutual dependency of its local regions, and design a novel architecture to leverage such information to boost the performance of aesthetics assessment. To achieve this, we densely partition an image into local regions and compute aesthetics-preserving features over the regions to characterize the aesthetics properties of image content. With the feature representation of local regions, we build a region composition graph in which each node denotes one region and any two nodes are connected by an edge weighted by the similarity of the region features. We perform reasoning on this graph via graph convolution, in which the activation of each node is determined by its highly correlated neighbors. Our method naturally uncovers the mutual dependency of local regions in the network training procedure, and achieves the state-of-the-art performance on the benchmark visual aesthetics datasets.

The problem of broadcasting a parallel Gaussian source over an additive white Gaussian noise broadcast channel under the mean-squared error distortion criterion is studied. A hybrid digital-analog coding strategy which combines source coding with side information, channel coding with side information, layered source coding, and superposition broadcast channel coding is presented. When specialized to the open problem of broadcasting a white Gaussian source over an additive white Gaussian noise broadcast channel with bandwidth mismatch, which has been the subject of several previous investigations, this coding scheme strictly improves on the state of the art.

The problem of transmitting a Gaussian source with memory to a digital and a linear-analog receiver, over an arbitrarily colored, nondegraded Gaussian broadcast channel is studied. The main result of this work is a complete characterization of the set of achievable distortion pairs at the two receivers given a power constraint at the transmitter. Further, a constructive hybrid uncoded-coded scheme consisting of the cascade of source coding with side information and channel coding with side information systems is shown to achieve the entire power-mean squared error (MSE) distortion region associated with the problem. An interesting operating point in this region is one where the digital receiver obtains the classical point-to-point optimal quality and the analog receiver attains the best possible simultaneously achievable distortion. This problem is motivated by the practical application of the seamless ldquoin-bandrdquo digital upgrade of legacy analog transmission systems.

We describe PRISM, a video coding paradigm based on the principles of lossy distributed compression (also called source coding with side information or Wyner-Ziv coding) from multiuser information theory. PRISM represents a major departure from conventional video coding architectures (e.g., the MPEGx, H.26x families) that are based on motion-compensated predictive coding, with the goal of addressing some of their architectural limitations. PRISM allows for two key architectural enhancements: (1) inbuilt robustness to "drift" between encoder and decoder and (2) the feasibility of a flexible distribution of computational complexity between encoder and decoder. Specifically, PRISM enables transfer of the computationally expensive video encoder motion-search module to the video decoder. Based on this capability, we consider an instance of PRISM corresponding to a near reversal in codec complexities with respect to today's codecs (leading to a novel light encoder and heavy decoder paradigm), in this paper. We present encouraging preliminary results on real-world video sequences, particularly in the realm of transmission losses, where PRISM exhibits the characteristic of rapid recovery, in contrast to contemporary codecs. This renders PRISM as an attractive candidate for wireless video applications.

We consider the problem of spatio-temporal resolution enhancement of a dynamic video scene by synergistically combining information from multiple video sequences corresponding to the different views of the scene. While prior work has focused on spatio-temporal superresolution by primarily exploiting the blurring correlation induced by the scene imaging process, in this work we exploit another and perhaps a stronger form of correlation the intrinsic correlation present in the dynamic scene. We use the knowledge base from the areas of multi-view computer vision, video coding and signal/image processing to efficiently model the dynamic video scene by exploiting inter-camera (space) as well as intra-camera correlations (time). We present simulation results corresponding to a lowframe rate two-camera stereo video configuration where the sequences are combined to generate higher frame-rate sequences. Our proposed framework has direct application to problems such as multi-view video surveillance and relates closely to problems such as multi-view video compression.

In this work, we analytically study the complexity-performance trade-offs associated with video codecs based on the principle of source coding with side information at the decoder. We address three important aspects. First, we quantify the theoretical performance gains attained with side-information based codecs over prediction-based coders like MPEG under a lossy transmission scenario when there is drift. Secondly, we show that it is possible to closely approach MPEG's compression performance using side-information video coding principles with accurate DFD (displaced frame difference) modeling even without sophisticated channel codes. Thirdly, we analytically show that the value of accurately estimating DFD statistics diminishes as the channel gets noisier.

We study the problem of estimating a physical process at a central processing unit (CPU) based on noisy measurements collected from a distributed, bandwidth-constrained, unreliable, network of sensors, modeled as an erasure network of unreliable "bit-pipes" between each sensor and the CPU. The CPU is guaranteed to receive data from a minimum fraction of the sensors and is tasked with optimally estimating the physical process under a specified distortion criterion. We study the noncollaborative (i.e., fully distributed) sensor network regime, and derive an information-theoretic achievable rate-distortion region for this network based on distributed source-coding insights. Specializing these results to the Gaussian setting and the mean-squared-error (MSE) distortion criterion reveals interesting robust-optimality properties of the solution. We also study the regime of clusters of collaborative sensors, where we address the important question: given a communication rate constraint between the sensor clusters and the CPU, should these clusters transmit their "raw data" or some low-dimensional "local estimates"? For a broad set of distortion criteria and sensor correlation statistics, we derive conditions under which rate-distortion-optimal compression of correlated cluster-observations separates into the tasks of dimension-reducing local estimation followed by optimal distributed compression of the local estimates.

In this Part II of a two-part paper, we present a new achievable rate-distortion region for the symmetric n-channel multiple-descriptions coding problem (n>2) where the rate of every description is the same, and the reconstruction distortion depends only on the number of descriptions received. Using a new approach for the random coding constructions, along with a generalization of the technique used in the two-channel El Gamal and Cover region to any n, the rate region presented here achieves points that have not been known in the literature previously. This rate region is derived from a concatenation of source-channel erasure codes developed in Part I of this work by deploying the framework of source coding with side information ("random binning"). The key idea is that by using the framework of source coding with side information, multiple statistically identical realizations representing the coarse version of a source can be simultaneously refined by a single encoding. We point out that there is an important conceptual difference in random coding construction for the multiple-descriptions coding problem between the case of n=2 and n>2. To illustrate the framework, we also present the important case of the Gaussian source in detail.

In this two-part paper, we present a new achievable rate region for the general n-channel symmetric multiple descriptions problem. In part I, inspired by the concept of maximum-distance separable (MDS) erasure channel codes, we consider a special case of this rate region, where the source is encoded into n descriptions each with rate R. These descriptions are transmitted over n bandwidth constrained and errorless channels. During transmission, a subset of these channels can break down, thus erasing the corresponding descriptions. The decoder is interested in recovering the source with the reception of at least k descriptions. Thus, the encoder is allowed to sample only one realization of this breakdown process during the entire transmission. For Gaussian sources, we have the following interesting result: when any k descriptions arrive, the achievable distortion exactly matches the optimal distortion-rate performance corresponding to a source rate of kR bits; with the reception of any m > k descriptions, the source reconstruction quality is strictly better, the improvement being nearly linear in the number of descriptions received.

In this work, we present PRISM (Power-efficient Robust high-compression Syndrome-based Multimedia coding), a new video coding paradigm based on the principles of distributed source coding. Incurring the low encoding complexity of intra-frame (still-image) video coding, PRISM approaches the high compression efficiency of full-motion inter-frame video coding, while simultaneously offering natural robustness to the drift problem and channel loss in direct contrast to conventional video coding architectures (MPEG and H.26L). These traits make it well-matched to uplink-rich applications such as multimedia over wireless networks, wireless video and sensor cameras, etc.

In this work, we present PRISM (power-efficient, robust, high-compression, syndrome-based multimedia coding), a video coding paradigm based on the principles of coding with side information (which, unlike the classical Wyner-Ziv coding scenario Wyner, A et al. (1976), is characterized by an ambiguous state of nature characterizing the side-information Ishwar, P et al. (2003)). PRISM's architectural goals are to inherit the low encoding complexity and robustness of motion-JPEG style intra-frame video codecs while approaching the high compression efficiency of full-motion interframe video codecs. The PRISM paradigm roughly swaps the encoder-decoder complexity with respect to conventional video coding architectures through the novel concept of moving the motion compensation task from the encoder to the decoder. These traits make PRISM well-matched to uplink-rich media applications involving wireless video and security cameras, multimedia-equipped phones and PDA's etc.

We present new achievable rate regions and code constructions for the symmetric n-channel multiple descriptions (MD) coding problem (Puri et al. (2002)) for n>2. Our approach is inspired by unexplored connections between MD and the problem of distributed source coding (Slepian et al. (1973); Wyner et al. (1976)). For illustrative clarity, we restrict our focus to the important special case relating to (n, k) source-channel erasure codes (Pradhan et al. (2001)). This involves n encodings of a source with the goal of maximizing its reconstruction fidelity with the availability of any k of them, while strictly improving this reconstruction fidelity with the availability of more than k descriptions. We describe the underlying information-theoretic framework, and then formulate practical constructions based on scalar quantizers and linear channel codes for the n=3 case to illustrate our concepts.

We address the problem of streaming video efficiently from multiple servers in a distributed manner. Invoking a scalable video encoding format, we describe a network-friendly, rate-distortion efficient distributed streaming algorithm based on a robust distributed encoding paradigm. The algorithm is efficiently matched dynamically to both the video content and the network channel conditions. Due to the diversity effect of multiple servers, our distributed algorithm is immune to single points of failure, provides natural load-balancing of the multiple servers and a graceful degradation in quality with an increase in multi-server load or a decrease in network throughput.

We address the problem of streaming video efficiently from mUl­ tiple servers in a distributed manner. Invoking a scalable video encoding format, we describe a network-friendly , rate-distortion efficient distributed streaming algorithm based on a robust dis­ tributed encoding paradigm. The algorithm is efficiently matched dynamically to both the video content and the network channel conditions. Due to t he d iversity effect of multiple servers, our dis­ tributed algorithm is immune to single points of failure, provides natural load-balancing of the multiple servers and a graceful degra­ dation in quality with an Increase in multi-server load or a decrease in network throughput. We present the results of our Internet ex­ periments that validate our hypotheses.

We consider the problem of distributing high-quality video signals over IP multicast in large local area networks (LANs), under realtime delay constraints, and with software-only processing. In a large LAN (such as the network of a university campus, or the network of a large company), the source of channel heterogeneity that the video communications system must cope with is not that of different bandwidth constraints available to each receiver, but it is essentially variations in the available CPU power that each receiver has in order to decode the incoming signal. We propose a new architecture for a video multicast system, present the design of the different components of this system, and show results obtained in a real implementation. Our feeds consist of video encoded at about 3 Mbits/sec and 16 frames/sec, capable of tolerating the loss of about 300 Kbits/sec worth of data. Decoding is performed on Linux PCs, and the quality of our reconstructed signals degrades gracefully with the speed of the CPU on which the receiver runs.

We propose multiple description (MD) video coders which use motion-compensated predictions. Our MD video coders utilize MD transform coding and three separate prediction paths at the encoder to mimic the three possible scenarios at the decoder: both descriptions received or either of the single descriptions received. We provide three different algorithms to control the mismatch between the prediction loops at the encoder and decoder. We present simulation results comparing the three approaches to two standards-based approaches to MD video coding. We show that when the main prediction loop at the encoder uses a two-channel reconstruction, it is important to have side prediction loops and transmit some redundancy information to control mismatch. We also examine the performance of our MD video coder with partial mismatch control in the presence of random packet loss, and demonstrate a significant improvement compared to more traditional approaches.

In this work we present an end-to-end optimized video streaming system comprising of synergistic interaction between a source packetization strategy and an efficient and responsive, TCP-friendly congestion control protocol [Linear Increase Multiplicative Decrease with History (LIMD/H)]. The proposed source packetization scheme transforms a scalable/layered video bitstream so as to provide graceful resilience to network packet drops. The congestion control mechanism provides low variation in transmission rate in steady state and at the same time is reactive and provably TCP-friendly. While the two constituent algorithms identified above are novel in their own right, a key aspect of this work is the integration of these algorithms in a simple yet effective framework. This "application-transport" layer interaction approach is used to maximize the expected delivered video quality at the receiver. The integrated framework allows our system to gracefully tolerate and quickly react to sudden changes in the available connection capacity due to the onset of congestion, as verified in our simulations.

We describe a framework for video transmission over the Internet that features the coordinated operation of an application-layer video source coding algorithm and a transport-layer rate control mechanism. The proposed video coding scheme operates on a progressively encoded video stream and provides graceful resilience to network packet drops. The robustness is enabled through a generalized multiple description (MD) coding strategy, architectured as an adaptive array of packet-erasure correction codes. The video coding algorithm is matched to an efficient and reactive rate control mechanism that minimizes the fluctuation of rate and uses the profile of past losses to adjust the rate in a TCP-friendly manner. While the two constituent algorithms identified above are interesting in their own right, a key feature of this work is the integration of these algorithms in a simple framework that seeks to maximize the expected delivered video quality at the receiver through coordinated adaptation of the two components. We present simple simulation results to illustrate the utility of our approach.

We propose multiple description (MD) video coders which use motion compensated predictions. Our MD video coders utilize MD transform coding and three separate prediction paths at the encoder, to mimic the three possible scenarios at the decoder: both descriptions received or either of the single descriptions received. We provide three different algorithms to control the mismatch between the prediction loops at the encoder and decoder. The results show that when the main prediction loop is the central loop, it is important to have side prediction loops and transmit some redundancy information to control mismatch.

We propose an automatic repeat request (ARQ)/forward error correction (FEC) scheme for synchronous transmission of video over a binary symmetric constant rate channel. The approach consists of jointly allocating source and channel rates to video blocks from a given admissible set subject to the buffer or equivalently end-end delay constraints. The channel codes used are the popular class of powerful FEC codes known as rate-compatible punctured convolutional (RCPC) codes. The method used involves independent coding of the video units and optimization of the end-to-end expected delivered video quality. The existence of a return channel is assumed through which the decoder informs the encoder about the success/failure of the transmission. In the event of a failure, incremental parity information is sent to the decoder for correcting errors and a reallocation performed at the encoder. The simulations done point out the efficacy of the proposed scheme.