Recent advances in Generative Adversarial Networks (GANs) have made it possible to conduct multi-domain image-to-image translation using a single generative network. While recent methods such as Ganimation and SaGAN are able to conduct translations on attribute-relevant regions using attention, they do not perform well when the number of attributes increases as the training of attention masks mostly rely on classification losses. To address this and other limitations, we introduce Attribute Manipulation Generative Adversarial Networks (AMGAN) for fashion images. While AMGAN's generator network uses class activation maps (CAMs) to empower its attention mechanism, it also exploits perceptual losses by assigning reference (target) images based on attribute similarities. AMGAN incorporates an additional discriminator network that focuses on attribute-relevant regions to detect unrealistic translations. Additionally, AMGAN can be controlled to perform attribute manipulations on specific regions such as the sleeve or torso regions. Experiments show that AMGAN outperforms state-of-the-art methods using traditional evaluation metrics as well as an alternative one that is based on image retrieval.

In this paper, we investigate ways of conducting a detailed fashion search using query images and attributes. A credible fashion search platform should be able to (1) find images that share the same attributes as the query image, (2) allow users to manipulate certain attributes, e.g. replace collar attribute from round to v-neck, and (3) handle region-specific attribute manipulations, e.g. replacing the color attribute of the sleeve region without changing the color attribute of other regions. A key challenge to be addressed is that fashion products have multiple attributes and it is important for each of these attributes to have representative features. To address these challenges, we propose the FashionSearchNet which uses a weakly supervised localization method to extract regions of attributes. By doing so, unrelated features can be ignored thus improving the similarity learning. Also, FashionSearchNet incorporates a new procedure that enables region awareness to be able to handle region-specific requests. FashionSearchNet outperforms the most recent fashion search techniques and is shown to be able to carry out different search scenarios using the dynamic queries.

In this paper, we propose an attribute-based query & retrieval system designed for fashion products. Our system addresses the problem of carrying out fashion searches by the query image and attribute manipulation, e.g. replacing long sleeve attribute of a dress to sleeveless. We present the attributes in two groups: (1) general attributes (category, gender etc.) and (2) special attributes (sleeve length, collar etc.). The special attributes are more suitable for the attribute manipulation and thus conducting searches. In order to solve the mentioned fashion search problem, it is crucial for the deep neural networks to understand attribute similarities. To facilitate more specific similarity learning, clothing items are represented by their structural subcomponents or "parts". The parts are estimated using an unsupervised segmentation method and used inside the proposed Convolutional Neural Network (CNN) as an attention mechanism. Meaning, different parts are connected to the special attributes, e.g. sleeve part is connected with sleeve length attribute. With this mechanism, part-based triplet ranking constraint is applied to learn similarity of each special attribute independently from one another in a single network. In the end, the well-defined features are used to conduct the fashion search. Additionally, an adaptive relevance feedback module is used to personalize the fashion search process with the feature descriptions. For our experiments, a new dataset is constructed containing 101,021 images which consist of pure clothing items. Besides achieving decent retrieval results in our dataset, the experiments show that proposed technique outperforms different baselines and is able to adapt towards user's requests.

A type of application that occupies most of the Internet is online multimedia, which is usually referred to online video or video streaming with providers such as Youtube, Netflix and Hulu. With the advancement in the Internet technology, people can watch an entire 2-hours long movie. However, people sometimes want to skip to different sections of a video to see whether the whole video is of their interest. Doing so usually requires the video to restart buffering and causes delay. To alleviate this issue, this research proposes a parallel streaming method based on the modification of the Dynamic Adaptive Streaming over HTTP (DASH) technique. By splitting video into smaller segments and modifying the segment scheduling, we can stream a video at many different positions in parallel so that users can jump to the different sections with minimal delay. The performance evaluation of the proposed technique shows that such streaming is possible through video segment scheduling with higher throughput in exchange for higher bandwidth requirement and utilization.

In this paper, we explore the use of spatial and frequency information of features in the biologically inspired model of HMAX. We discuss and refine previous models which use a similar framework and build specialized features which are better tuned to image structures by using unsupervised methods of clustering and picking the most frequent features using the statistics of the occurrence of the features in different spatial zones. Our classification results on the Caltech 101 dataset show significant improvements of up to 6% compared to previous improvements of the biologically inspired model of HMAX.

Consecutive corrupted MBs or slice errors are commonly seen in modern video transmission systems. Temporal error concealment is an effective approach to reduce the impact of errors. Conventional temporal error concealment techniques recover slice errors on a MB basis. We propose a new novel temporal scheme for slice error concealment based on a size-adaptive region basis. Size-adaptive region boundary matching can significantly reduce the misalignments or side mismatches that are commonly seen in MB-based approaches. Instead of using MSE or MAE as the matching criteria, We propose a matching criterion based on structural similarity (SSIM). Our proposed matching criterion can achieve better structural alignments and better visual perceptual quality. We also propose a method to detect motion vector outliers at the last stage of the error concealment scheme, which can further improve the visual quality of the concealed videos. Our proposed scheme conceals slice errors with good accuracy as well as low computation complexity, and it can be applied for real-time applications.

Video transmission over wireless networks suffers from packet loss due to either temporary packet drop or fading-induced bit errors. To ensure that the quality of the decoded video is not overly affected by the channel unreliability, the video applications have to provide sufficient robustness. In this paper, a novel slice interleaving algorithm is proposed for compressed video packetization. Each packet is constructed by interleaving independently decodable slices of the coded video bit-stream in a group of frames. We group slices into packets according to the maximum minimal distance of these slices. The scattered pattern guarantees that each lost slice has, as many as possible, spatial and temporal neighbors that belong to other correctly received packets. This scheme greatly strengthens the error concealment performance compared with traditional methods. Experimental results demonstrate the superiority of the proposed algorithm.

Video communication is often afflicted by packet loss over unreliable networks. To alleviate the quality degradation in the situation of packet loss, this paper proposes a novel error resilient slice interleaving method for compressed video packetization. Each packet is constructed by interleaving independently decodable slices of the coded video bit-stream in consecutive frames. The optimal interleaving pattern for minimizing the overall distortion of the decoded video, subject to a delay constraint, is determined based on the error concealment technique employed at the user end. Compared with traditional error-resilient methods, this scheme greatly improves the overall performance without requiring an increase in bit-rate. Experimental results on different video sequences demonstrate the effectiveness of the proposed approach.

With an empirically-derived complexity-rate-distortion model of a complexity scalable video encoder, we study the distributed optimization of several encoders sharing a wireless network, using the PSNR of the decoded video as the utility function. We will demonstrate how universal frequency reuse and source rate control can help the system degrade gracefully with increasing users, show the value of a simple pricing mechanism on transmission power in improving performance and study the effect of power allocation between transmitter and source encoder.

The H.264/AVC video coding standard encapsulates the most advanced video coding tools. Since the various techniques that lead to better coding efficiency of the coding standard also inevitably increase the complexity of the video encoder, real-time H.264 encoding of video streams is a challenging task. If available computational resource does not allow the entire encoding process to be carried out in time, a complexity scalable technique that ensures a graceful degradation of coding performance will be a valuable tool. We designed a video encoding scheme that allows the rate distortion (R-D) process to be carried out in a complexity scalable fashion. Our proposed singularly parameterized complexity scalable scheme allows the control of complexity-coding performance tradeoff when available resources are limited and the optimal R-D performance is unattainable.

In this paper, we introduce a singularly-parameterized complexity scalable H.264-compliant encoder. Through modeling its complexity-rate-distortion relationships, we derive optimized operating mode of the encoder (rate and complexity) and show through experiment that such optimization can help a video encoder operate within rate and time constraints. The design of the complexity scalable encoding scheme enables the encoder to perform optimization while taking into consideration the availability of computational resource. This extension of traditional rate-distortion optimization is necessary when time or power constraints do not allow a video encoder to achieve rate-distortion optimized coding performance. Our optimization scheme outputs parameters that allow the encoder to be as close to being rate-distortion optimized as possible, within rate and complexity constraints.

The H.264/AVC video coding standard encapsulates the most advanced video coding tools. The various techniques that lead to better coding efficiency of the coding standard also inevitably increase the complexity of the video encoder. Thus, real-time encoding of video streams with H.264 coding standard is a challenging task. If available computational resource does not allow the entire encoding process to be carried out in time, a complexity scalable technique that ensures a graceful degradation of coding performance will be a useful tool. This work proposes a singularly-parameterized complexity scalable rate-distortion framework for H.264/AVC encoders.

This paper presents a real-time H.264/AVC baseline profile video encoder. The encoder hardware is implemented using a cost-effective, low-power ADI Blackin-561 DSP and related peripherals for real-time video capturing, coding and streaming. The encoder software is developed using a two-stage pipelining framework for efficient parallel video data encoding. A synchronization mechanism with shared memory semaphores is used to schedule the hardware processes and software procedures to acquire the real-time encoding performance. Techniques for reducing the execution time in both stages are also described in this paper. Performance evaluation results verified that the encoder is capable of performing real-time encoding of CIF-resolution and medium-motion VGA-resolution videos, while maintaining good video quality.

In this paper, a singularly parameterized complexity-scalable scheme is proposed for designing power-aware and power-adaptive H.264 video encoder. The proposed scheme enables adaptive control of the trade-off between the encoder's complexity and coding performance by dynamically updating a control parameter during encoding. This enables the encoder to perform optimally on a variety of computing platforms and runtime environments by adaption to instantaneous available computing resources and varying source video characteristics. The proposed scheme also allows computational resource allocation across layers in SVC encoders.

Temporal error concealment (TEC) algorithms do not produce good results in the presence of cut scenes, camera panning or object occlusion in videos due to the lack of a suitable reference frame. Under such circumstances, spatial EC (SEC) may perform better. Thus, EC algorithms should decide adaptively between TEC or SEC. In this paper, we propose an integrated spatio-temporal EC framework which extends our previous work in TEC and SEC. The proposed algorithm decides adaptively on the best EC mode and produces average PSNR improvements of up to 4.59 dB over a coding-mode based EC algorithm. Most importantly, our algorithm produces results of much better perceptual quality.

We consider the on-demand delivery of layered video over peer-to-peer (P2P) networks. A parallel server architecture enables the service of requests by more than one server, thus reducing load at individual servers and dispersing network load. In a P2P network, peer upload bandwidth is limited and heterogenous, so a P2P streaming session may involve multiple supplying peers of different uplink. Partitioning the video data is a crucial first step in the collaborative delivery of data by multiple servers. If done right, the client's viewing experience can be significantly improved. In this paper, we describe a retrieval strategy that significantly reduces access delay necessary for continous playback when retrieving layers of heterogenous bitrate over multiple connections of heterogenous bandwidth.

The paper describes a fast MPEG-2 to H.264 transcoder by leveraging the MPEG-2 metadata set to reduce the computational complexity of the H.264 encoding process. With the proposed fast transcoding algorithms, our software-based transcoder can perform real-time transcoding a MPEG-2 movie from a DVD disc (at D1 resolution and full frame-rate) and simultaneously performing RTP/UDP streaming of the H.264 video with AC-3 audio over IP networks using only a 1.6 GHz PC notebook. We can achieve about 20-30% faster than the full re-encoding method even at the same target bit-rate and without visible quality degradation. Benchmarking with existing software systems has shown similar results.

Due to the advance of technologies in multimedia compression and network communications, scalable media streaming services have been availed to provide QoS differentiated services for heterogeneous users. However, it is still a big challenge to support consistent end-to-end Quality of Services (QoS) for the users due to the dynamic feature of the Internet, and abrupt variability of the network resources may severally affect the client perceived QoS. In this paper, we address the issues of smooth QoS adaptation for scalable streaming services. We propose an Optimal Smooth QoS Adaptation (OS-QA) strategy which allocates the server resource adaptively to cope with the variability of network bandwidth and protects the service quality of different quality classes under dynamic resource constraints. We analyze the quality variation caused by resource fluctuation and proposed OS-QA to minimize the average QoS variance under the resource constraints. Simulations are conducted to compare our proposed method with other QoS adaptation methods, and performance is analyzed in terms of QoS variance and PSNR. Results show that our proposed method is able to gracefully adapt the QoS and protect the client perceived QoS by minimizing the QoS variance under dynamic network resource constrains.

Temporal scalability in H.264/SVC video compression standard can be achieved with the hierarchical B-pictures (HB) structure. When performing motion estimation (ME) in the HB structure, the temporal distance between the current frame and reference frame(s) can be large (up to 32 frames apart). This limits the performance of fast search algorithm as larger search window is often necessary. Extensive experiments showed that popular fast suboptimal block ME algorithms are ineffective at tracking large motions across several frames. In this paper, we propose a new framework called Lacing which integrates well with any fast block ME techniques to significantly improve the motion prediction accuracy in quality of the motion-compensated frame and also result in smoother motion vector fields with lower entropy.

In temporal error concealment (EC), the sum of absolute difference (SAD) is commonly used to identify the best replacement macroblock. Even though the use of SAD ensures spatial continuity and produces visually good results, it is insufficient to ensure edge alignment. Other distortion criteria based solely on structural alignment may also perform poorly in the absence of strong edges. In this paper, we propose a spatio-temporal EC search algorithm using an edge weighted SAD distortion criterion. This distortion criterion ensures both edge alignment and spatial continuity. We assume the loss of motion information and use zero motion vector as the starting search point. We show that the proposed algorithm outperforms the use of unweighted SAD in general. Most importantly, the perceptual quality of EC is improved due to edge alignment while ensuring spatial continuity.

In H.264 video coding, there are a substantial number of 4x4 blocks becoming all-zeros after transformation and quantization. This is a waste of computational resources because these skipped blocks do not require forward transform and quantization. We proposed a very effective early detection of fast skipped block detections based on the theoretical derivation of H.264 integer transform and quantization. The experimental results show that the algorithm can detect 9.71%-43.35% more zero blocks than Yong's method.

It is well known that motion estimation is the most computationally intensive processing unit of the H.264 video encoder. Various fast motion estimation algorithms have been proposed to reduce its complexity. Generally, these approaches achieve speedup by reducing the number of candidate search points within the search window. In this paper, we propose a new method, which uses the Sum-of-Absolute-Differences mapping (SAD map) to dynamically cache the SAD values and then reuse them for different block sizes. Experimental results on standard video sequences verified that the proposed method is capable of increasing the encoder speed by up to 15% without any loss in PSNR value or increase in bit rate. Due to its generic nature, this method can be applied in any fast motion estimation methods although it is especially effective in the full search motion estimation method.

This paper introduces a novel class of length-4N orthonormal scalar wavelets, and presents the theory, implementational issues, and their applications to image compression. We first give the necessary and sufficient conditions for the existence of this class. The parameterized representation of filters with different lengths are then given. Next, we derive new and efficient decomposition and reconstruction algorithms specifically tailored to this class of wavelets. We show that the proposed discrete wavelet transformations are orthogonal and have lower computational complexity than conventional octave-bandwidth transforms using Daubechies' (1989) orthogonal filters of equal length. In addition, we also verify that symmetric boundary extensions can be applied. Finally, our image compression results further confirm that improved performance can be achieved with lower computational cost.

Approximation order, linear phase symmetry, time-frequency localization, regularity, and stopband attenuation are some criteria that are widely used in wavelet filter design. We propose a new criterion called good multifilter properties (GMPs) for the design and construction of multiwavelet filters targeting image compression applications. We first provide the definition of GMPs, followed by a necessary and sufficient condition for an orthonormal multiwavelet system to have a GMP order of at least 1. We then present an algorithm to construct orthogonal multiwavelets possessing GMPs, starting from any length-2N scalar CQPs. Image compression experiments are performed to evaluate the importance of GMPs for image compression, as compared to other common filter design criteria. Our results confirmed that multiwavelets that possess GMPs not only yield superior PSNR performances, but also require much lower computations in their transforms.

We introduce a highly scalable video compression system for very low bit-rate videoconferencing and telephony applications around 10-30 kbits/s. The video codec first performs a motion-compensated three-dimensional (3-D) wavelet (packet) decomposition of a group of video frames, and then encodes the important wavelet coefficients using a new data structure called tri-zerotrees (TRI-ZTR). Together, the proposed video coding framework forms an extension of the original zero tree idea of Shapiro (1992) for still image compression. In addition, we also incorporate a high degree of video scalability into the codec by combining the layered/progressive coding strategy with the concept of embedded resolution block coding. With scalable algorithms, only one original compressed video bit stream is generated. Different subsets of the bit stream can then be selected at the decoder to support a multitude of display specifications such as bit rate, quality level, spatial resolution, frame rate, decoding hardware complexity, and end-to-end coding delay. The proposed video codec also allows precise bit rate control at both the encoder and decoder, and this can be achieved independently of the other video scaling parameters. Such a scheme is very useful for both constant and variable bit rate transmission over mobile communication channels, as well as video distribution over heterogeneous multicast networks. Finally, our simulations demonstrated comparable objective and subjective performance when compared to the ITU-T H.263 video coding standard, while providing both multirate and multiresolution video scalability.

The widespread use of block-based interframe motion estimation for video sequence compression in both MPEG and H.263 standards is due to its effectiveness and simplicity of implementation. Nevertheless, the high computational complexity of the full-search algorithm has motivated a host of suboptimal but faster search strategies. A popular example is the three-step search (TSS) algorithm. However, its uniformly spaced search pattern is not well matched to most real-world video sequences in which the motion vector distribution is nonuniformly biased toward the zero vector. Such an observation inspired the new three-step search (NTSS) which has a center-biased search pattern and supports a halfway-stop technique. It is faster on average, and gives better motion estimation as compared to the well-known TSS. Later, the four-step search (4SS) algorithm was introduced to reduce the average case from 21 to 19 search points, while maintaining a performance similar to NTSS in terms of motion compensation errors. We propose a novel unrestricted center-biased diamond search (UCBDS) algorithm which is more efficient, effective, and robust than the previous techniques. It has a best case scenario of only 13 search points and an average of 15.5 block matches. This makes UCBDS consistently faster than the other suboptimal block-matching techniques. This paper also compares the above methods in which both the processing speed and the accuracy of motion compensation are tested over a wide range of test video sequences.