We analyze neuron-level representations of generalizing and memorizing networks, using a synthetic dataset designed through aggregating hidden patterns into supervision classes.We observe that the average pre-activation of the most activated patterns of a class (and inversely) in each neuron increases during training: a winner-take-most phenomenon.The network applies a divide-and-conquer strategy, where each neuron specializes in classifying different patterns of a class.Through an ablation study, describe three necessary conditions for this phenomenon.Finally, we provide intuition for why it occurs, drawing links with existing work on sample difficulty, gradient coherence, and implicit clustering.

Estimating 3D human poses from 2D images remains challenging due to occlusions and projective ambiguity. Multi-view learning-based approaches mitigate these issues but often fail to generalize to real-world scenarios, as large-scale multi-view datasets with 3D ground truth are scarce and captured under constrained conditions. To overcome this limitation, recent methods rely on 2D pose estimation combined with 2D-to-3D pose lifting trained on synthetic data. Building on our previous MPL framework, we propose RUMPL, a transformer-based 3D pose lifter that introduces a 3D ray-based representation of 2D keypoints. This formulation enables a model agnostic to camera parameters that can be universally deployed across arbitrary camera configurations in a given area without retraining or fine-tuning. A new View Fusion Transformer leverages learned fused-ray tokens to aggregate information along rays, further improving multi-view consistency. Evaluation on standard benchmarks shows that RUMPL significantly outperforms existing methods, yielding a 56.6% MPJPE (All KP) reduction on Human3.6M over triangulation-based methods and exceeding 70% improvement on the CMU Panoptic dataset when compared to transformer-based image-representation approaches. Results on new benchmarks, including in-the-wild multi-view and multi-person datasets, confirm its robustness and scalability.

The zero-shot capabilities of Vision-Language Models (VLMs) have been widely leveraged to improve predictive performance. However, previous works on transductive or test-time adaptation (TTA) often make strong assumptions about the data distribution, such as the presence of all classes. Our work challenges these favorable deployment scenarios and introduces a more realistic evaluation framework, including (i) a variable number of effective classes for adaptation within a single batch, and (ii) non-i.i.d. batches of test samples in online adaptation settings. We provide comprehensive evaluations, comparisons, and ablation studies that demonstrate how current transductive or TTA methods for VLMs systematically compromise the models’ initial zero-shot robustness across various realistic scenarios, favoring performance gains under advantageous assumptions about the test sample distributions. Furthermore, we introduce StatA, a versatile method that can handle a wide range of deployment scenarios, including those with a variable number of effective classes at test time. Our approach incorporates a novel regularization term designed specifically for VLMs, which acts as a statistical anchor preserving the initial text-encoder knowledge, particularly in low-data regimes. Code available at https://github.com/MaxZanella/StatA.

peer reviewed

We present a scalable framework designed to craft efficient lightweight models for video object detection utilizing self-training and knowledge distillation techniques. We scrutinize methodologies for the ideal selection of training images from video streams and the efficacy of model sharing across numerous cameras. By advocating for a camera clustering methodology, we aim to diminish the requisite number of models for training while augmenting the distillation dataset. The findings affirm that proper camera clustering notably amplifies the accuracy of distilled models, outperforming the methodologies that employ distinct models for each camera or a universal model trained on the aggregate camera data.

We present a novel algorithm for pulmonary embolism segmentation, designed to alleviate the need for expert annotation.Our approach integrates deep learning with a conventional image segmentation techniques, operating in two distinct stages.Specifically, graph cut is used for initial segmentation, followed by manual refinement, to define the labels required to train a CNN.This CNN is then employed to generate pseudolabels on a large dataset, enabling the training of an improved CNN*.Our findings demonstrate enhanced performance of CNN* over CNN.Overall, the CNN* builds on a very limited amount of manual intervention.Moreover, the injection of expert knowledge in the graph-cut avoids the need for expert knowledge in this manual intervention.

This paper considers the deployment of flexible and high-performance surveillance systems. These systems must continuously integrate new sensors and sensing algorithms, which are autonomous (e.g., capable of making decisions independently of a central system) and possess interaction skills (e.g., capable of exchanging observations). For this purpose, our work proposes adopting an agent-based architecture derived from an organizational and holonic (i.e., system of systems) multi-agent model. It leverages autonomous processing methods, resulting in a scalable and modular multisensor and multimethod surveillance systems. A vehicle tracking case study demonstrates the relevance of our approach in terms of effectiveness and runtime.

Reconstruction-based anomaly detection typically relies on the reconstruction of a defect-free output from an input image.Such reconstruction can be obtained by training an autoencoder to reconstruct clean images from inputs corrupted with a synthetic defect.Previous works have shown that adopting an autoencoder with skip connections improves reconstruction sharpness.However, it remains unclear how skip connections aect the latent representations learned during training.Here, we compare internal representations of autoencoders with and without skip connections.Experiments over the MVTec AD dataset reveal that skip connections enable the autoencoder latent representations to intrinsically discriminate between clean and defective images.

We introduce a simple but robust method to restore the visibility of hazy images. Our non deep-learning strategy refines a simplistic approximation of the airlight by taking advantage of the O-HAZE dataset that contains also the corresponding haze-free images. Knowing that the transmission is generally characterized by small values in hazy scenes, based on the optical model, we first assumes zero transmission, and approximate the airlight by the input hazy image. Then, using hazy and corresponding haze-free images available from the O-HAZE dataset, a second airlight estimate can be computed by solving the optical model assuming a simplified transmission map derived from dark channel prior. Observing that the difference between these two airlight estimates, primarily contains the low frequencies of the hazy image, we refine the airlight approximation derived from a zero transmission by reinforcing its low frequency component. We extensively tested our approach on real world hazy images. The qualitative and quantitative evaluations demonstrate that our approach yields better results than previous physically-based image dehazing techniques, and favorably compares with the deep learning dehazing approaches.

Image noise is ubiquitous in photography. However, image noise is not compressible nor desirable, thus attempting to convey the noise in compressed image bitstreams yields sub-par results in both rate and distortion. We propose to explicitly learn the image denoising task when training a codec. Therefore, we leverage the Natural Image Noise Dataset, which offers a wide variety of scenes captured with various ISO numbers, leading to different noise levels, including insignificant ones. Given this training set, we supervise the codec with noisy-clean image pairs, and show that a single model trained based on a mixture of images with variable noise levels appears to yield best-in-class results with both noisy and clean images, achieving better rate-distortion than a compression-only model or even than a pair of denoising-then-compression models with almost one order of magnitude fewer GMac operations.

Occluded person re-identification (ReID) is a person retrieval task which aims at matching occluded person images with holistic ones. For addressing occluded ReID, part-based methods have been shown beneficial as they offer fine-grained information and are well suited to represent partially visible human bodies. However, training a part-based model is a challenging task for two reasons. Firstly, individual body part appearance is not as discriminative as global appearance (two distinct IDs might have the same local appearance), this means standard ReID training objectives using identity labels are not adapted to local feature learning. Secondly, ReID datasets are not provided with human topographical annotations. In this work, we propose BPBreID, a body part-based ReID model for solving the above issues. We first design two modules for predicting body part attention maps and producing body part-based features of the ReID target. We then propose GiLt, a novel training scheme for learning part-based representations that is robust to occlusions and non-discriminative local appearance. Extensive experiments on popular holistic and occluded datasets show the effectiveness of our proposed method, which outperforms state-of-the-art methods by 0.7% mAP and 5.6% rank-1 accuracy on the challenging Occluded-Duke dataset. Our code is available at https://github.com/VlSomers/bpbreid.

Turning the weights to zero when training a neural network helps in reducing the computational complexity at inference. To progressively increase the sparsity ratio in the network without causing sharp weight discontinuities during training, our work combines soft-thresholding and straight-through gradient estimation to update the raw, i.e. non-thresholded, version of zeroed weights. Our method, named ST-3 for straight-through/soft-thresholding/sparse-training2, obtains SoA results, both in terms of accuracy/sparsity and accuracy/FLOPS trade-offs, when progressively increasing the sparsity ratio in a single training cycle. In particular, despite its simplicity, ST-3 favorably compares to the most recent methods, adopting differentiable formulations [42] or bio-inspired neuroregeneration principles [25]. This suggests that the key ingredients for effective sparsification primarily lie in the ability to give the weights the freedom to evolve smoothly across the zero state while progressively increasing the sparsity ratio.

JPEG-XS offers low complexity image compression for applications with constrained but reasonable bit-rate, and low latency. Our paper explores the deployment of JPEG-XS on lossy packet networks. To preserve low latency, Forward Error Correction (FEC) is envisioned as the protection mechanism of interest. Although the JPEG-XS codestream is not scalable in essence, we observe that the loss of a codestream fraction impacts the decoded image quality differently, depending on whether this codestream fraction corresponds to codestream headers, to coefficient significance information, or to low/high frequency data. Hence, we propose a rate-distortion optimal unequal error protection scheme that adapts the redundancy level of Reed-Solomon codes according to the rate of channel losses and the type of information protected by the code. Our experiments demonstrate that, at 5% loss rates, it reduces the Mean Squared Error by up to 92% and 65%, compared to a transmission without and with optimal but equal protection, respectively.

The need for Explainable AI is increasing with the development of deep learning. The saliency maps derived from convolutional neural networks generally fail in localizing with accuracy the image features justifying the network prediction. This is because those maps are either low-resolution as for CAM, or smooth as for perturbation-based methods, or do correspond to a large number of widespread peaky spots as for gradient-based approaches. In contrast, our work proposes to combine the information from earlier network layers with the one from later layers to produce a high resolution Class Activation Map that is competitive with the previous art in term of insertion-deletion faithfulness metrics, while outperforming it in term of precision of class-specific features localization.

Collecting image annotations remains a significant burden when deploying CNN in a specific applicative context. This is especially the case when the annotation consists in binary masks covering object instances. Our work proposes to delineate instances in three steps, based on a semi-automatic approach: (1) the extreme points of an object (left-most, right-most, top, bottom pixels) are manually defined, thereby providing the object bounding-box, (2) a universal automatic segmentation tool like Deep Extreme Cut is used to turn the bounded object into a segmentation mask that matches the extreme points; and (3) the predicted mask is manually corrected. Various strategies are then investigated to balance the human manual annotation resources between bounding-box definition and mask correction, including when the correction of instance masks is prioritized based on their overlap with other instance bounding-boxes, or the outcome of an instance segmentation model trained on a partially annotated dataset. Our experimental study considers a teamsport player segmentation task, and measures how the accuracy of the Panoptic-Deeplab instance segmentation model depends on the human annotation resources allocation strategy. It reveals that the sole definition of extreme points results in a model accuracy that would require up to 10 times more resources if the masks were defined through fully manual delineation of instances. When targeting higher accuracies, prioritizing the mask correction among the training set instances is also shown to save up to 80% of correction annotation resources compared to a systematic frame by frame correction of instances, for a same trained instance segmentation model accuracy.

In industrial vision, the anomaly detection problem can be addressed with an autoencoder trained to map an arbitrary image, i.e. with or without any defect, to a clean image, i.e. without any defect. In this approach, anomaly detection relies conventionally on the reconstruction residual or, alternatively, on the reconstruction uncertainty. To improve the sharpness of the reconstruction, we consider an autoencoder architecture with skip connections. In the common scenario where only clean images are available for training, we propose to corrupt them with a synthetic noise model to prevent the convergence of the network towards the identity mapping, and introduce an original Stain noise model for that purpose. We show that this model favors the reconstruction of clean images from arbitrary real-world images, regardless of the actual defects appearance. In addition to demonstrating the relevance of our approach, our validation provides the first consistent assessment of reconstruction-based methods, by comparing their performance over the MVTec AD dataset [1], both for pixel- and image-wise anomaly detection. Our implementation is available at https://github.com/anncollin/AnomalyDetection-Keras.

We introduce an effective fusion-based technique to enhance both day-time and night-time hazy scenes. When inverting the Koschmieder light transmission model, and by contrast with the common implementation of the popular dark-channel DehazeHeCVPR2009, we estimate the airlight on image patches and not on the entire image. Local airlight estimation is adopted because, under night-time conditions, the lighting generally arises from multiple localized artificial sources, and is thus intrinsically non-uniform. Selecting the sizes of the patches is, however, non-trivial. Small patches are desirable to achieve fine spatial adaptation to the atmospheric light, but large patches help improve the airlight estimation accuracy by increasing the possibility of capturing pixels with airlight appearance (due to severe haze). For this reason, multiple patch sizes are considered to generate several images, that are then merged together. The discrete Laplacian of the original image is provided as an additional input to the fusion process to reduce the glowing effect and to emphasize the finest image details. Similarly, for day-time scenes we apply the same principle but use a larger patch size. For each input, a set of weight maps are derived so as to assign higher weights to regions of high contrast, high saliency and small saturation. Finally the derived inputs and the normalized weight maps are blended in a multi-scale fashion using a Laplacian pyramid decomposition. Extensive experimental results demonstrate the effectiveness of our approach as compared with recent techniques, both in terms of computational efficiency and the quality of the outputs.

This article introduces a novel solution to improve image enhancement in terms of color appearance. Our approach, called Color Channel Compensation (3C), overcomes artifacts resulting from the severely non-uniform color spectrum distribution encountered in images captured under hazy night-time conditions, underwater, or under non-uniform artificial illumination. Our solution is founded on the observation that, under such adverse conditions, the information contained in at least one color channel is close to completely lost, making the traditional enhancing techniques subject to noise and color shifting. In those cases, our pre-processing method proposes to reconstruct the lost channel based on the opponent color channel. Our algorithm subtracts a local mean from each opponent color pixel. Thereby, it partly recovers the lost color from the two colors (red-green or blue-yellow) involved in the opponent color channel. The proposed approach, whilst simple, is shown to consistently improve the outcome of conventional restoration methods. To prove the utility of our 3C operator, we provide an extensive qualitative and quantitative evaluation for white balancing, image dehazing, and underwater enhancement applications.

This paper introduces a novel tone mapping operator, designed to offer a good rendering of the local structures. The new operator fusions the multiple versions of a single HDR input obtained by clipping and normalizing its intensity based on a complete set of disjoint intervals. Defining the weight map associated to each version to be its clipping interval indicator function promotes contrast enhancement, but induces artifacts when neighboring pixels belong to distinct intervals. We thus propose to smooth out the indicators across neighboring pixels with similar intensity, using a standard cross-bilateral filter. With such weight maps, the fusion operator becomes equivalent to applying histogram equalization on the image regions on which the cross-bilateral filter diffuses the indicators, and is therefore referred to as Bilateral Histogram Equalization (BHE) operator. It compares favorably to previous tone mapping algorithms.

In this letter we introduce a simple but effective concept, Color Channel Transfer (CCT), that is able to substantially improve the performance of various dehazing techniques. CCT is motivated by a key observation: in scattering media the information from at least one color channel presents high attenuation. To compensate for the loss of information in one color channel, CCT employs a color-transfer strategy and operates in a color opponent space that helps to compensate automatically the chromatic loss. The reference is computed by combining the details and saliency of the initial image with uniform gray image that assures a balanced chromatic distribution. The extensive qualitative and quantitative experiments demonstrate the utility of CCT as a preprocessing step for various dehazing problems such as day-time dehazing, night-time dehazing, and underwater image dehazing.

This paper introduces an effective strategy to enhance the visibility of hazy images, especially those obtained in night-time conditions. Compared to day-time, in night-time scenes, the lighting generally arises from multiple artificial sources and therefore may be considered intrinsically as being non-uniform. As a result, conventional global atmospheric light (airlight) estimation strategies become irrelevant. In this work, we propose a simple yet effective patch-based atmospheric light estimation. To circumvent the problem of selecting an appropriate patch size, we propose to estimate the atmospheric light on several patch sizes, and to define the local airlight as the average of those estimates. An extensive experimental validation demonstrates that the proposed strategy is able to recover the scene radiance without unwanted color-shifting, and proves that our approach is competitive compared to recent techniques in terms of restored image quality.

We introduce an effective technique to enhance the images captured underwater and degraded due to the medium scattering and absorption. Our method is a single image approach that does not require specialized hardware or knowledge about the underwater conditions or scene structure. It builds on the blending of two images that are directly derived from a color-compensated and white-balanced version of the original degraded image. The two images to fusion, as well as their associated weight maps, are defined to promote the transfer of edges and color contrast to the output image. To avoid that the sharp weight map transitions create artifacts in the low frequency components of the reconstructed image, we also adapt a multiscale fusion strategy. Our extensive qualitative and quantitative evaluation reveals that our enhanced images and videos are characterized by better exposedness of the dark regions, improved global contrast, and edges sharpness. Our validation also proves that our algorithm is reasonably independent of the camera settings, and improves the accuracy of several image processing applications, such as image segmentation and keypoint matching.

Imaging in the underwater environment suffers from color degradation and poor visibility since the light spectrum is selectively absorbed and scattered by water and floating particles. In this paper we introduce a simple but effective underwater dehazing approach that builds on an original color transfer strategy to align the color statistics of a hazy input to the ones of a reference image, also captured underwater, but with neglectable water attenuation. As an original specificity, our proposed color transfer approach is designed to promote the preservation of salient regions, as well as of the details obtained by subtracting from the input an edge-preserving smoothed version of itself. The color-transferred input is then restored by inverting a simplified version of the McGlamery underwater image formation model, using the conventional Dark Channel Prior to estimate the transmission map and the back-scattered light parameter involved in the model. We demonstrate that our color transfer step is crucial for a good transmission estimation but mostly for underwater dehazing where other specialized techniques fail. Extensive qualitative and quantitative results demonstrate the effectiveness of the proposed approach to estimate the transmission, including for cases where traditional specialized techniques fail, and to improve the image quality.

Given a set of detections, detected at each time instant independently, we investigate how to associate them across time. This is done by propagating labels on a set of graphs, each graph capturing how either the spatio-temporal or the appearance cues promote the assignment of identical or distinct labels to a pair of detections. The graph construction is motivated by a locally linear embedding of the detection features. Interestingly, the neighborhood of a node in appearance graph is defined to include all the nodes for which the appearance feature is available (even if they are temporally distant). This gives our framework the uncommon ability to exploit the appearance features that are available only sporadically. Once the graphs have been defined, multi-object tracking is formulated as the problem of finding a label assignment that is consistent with the constraints captured each graph, which results into a difference of convex (DC) program. We propose to decompose the global objective function into node-wise sub-problems. This not only allows a computationally efficient solution, but also supports an incremental and scalable construction of the graph, thereby making the framework applicable to large graphs and practical tracking scenarios. Moreover, it opens the possibility of parallel implementation.

Due to its robustness and effectiveness, multi-scale fusion (MSF) based on the Laplacian pyramid decomposition has emerged as a popular technique that has shown utility in many applications. Guided by several intuitive measures (weight maps) the MSF process is versatile and straightforward to be implemented. However, the number of pyramid levels increases with the image size, which implies sophisticated data management and memory accesses, as well as additional computations. Here, we introduce a simplified formulation that reduces MSF to only a single level process. Starting from the MSF decomposition, we explain both mathematically and intuitively (visually) a way to simplify the classical MSF approach with minimal loss of information. The resulting single-scale fusion (SSF) solution is a close approximation of the MSF process that eliminates important redundant computations. It also provides insights regarding why MSF is so effective. While our simplified expression is derived in the context of high dynamic range imaging, we show its generality on several well-known fusion-based applications, such as image compositing, extended depth of field, medical imaging, and blending thermal (infrared) images with visible light. Besides visual validation, quantitative evaluations demonstrate that our SSF strategy is able to yield results that are highly competitive with traditional MSF approaches.

We consider the synthesis of intermediate views of an object captured by two widely spaced and calibrated cameras. This problem is challenging because foreshortening effects and occlusions induce significant differences between the reference images when the cameras are far apart. That makes the association or disappearance/appearance of their pixels difficult to estimate. Our main contribution lies in disambiguating this ill-posed problem by making the interpolated views consistent with a plausible transformation of the object silhouette between the reference views. This plausible transformation is derived from an object-specific prior that consists of a nonlinear shape manifold learned from multiple previous observations of this object by the two reference cameras. The prior is used to estimate the evolution of the epipolar silhouette segments between the reference views. This information directly supports the definition of epipolar silhouette segments in the intermediate views, as well as the synthesis of textures in those segments. It permits to reconstruct the epipolar plane images (EPIs) and the continuum of views associated with the EPI volume, obtained by aggregating the EPIs. Experiments on synthetic and natural images show that our method preserves the object topology in intermediate views and deals effectively with the self-occluded regions and the severe foreshortening effect associated with wide-baseline camera configurations.

This paper approximates the 3D geometry of a scene by a small number of 3D planes. The method is especially suited to man-made scenes, and only requires two calibrated wide-baseline views as inputs. It relies on the computation of a dense but noisy 3D point cloud, as for example obtained by matching DAISY descriptors [35] between the views. It then segments one of the two reference images, and adopts a multi-model fitting process to assign a 3D plane to each region, when the region is not detected as occluded. A pool of 3D plane hypotheses is first derived from the 3D point cloud, to include planes that reasonably approximate the part of the 3D point cloud observed from each reference view between randomly selected triplets of 3D points. The hypothesis-to-region assignment problem is then formulated as an energy-minimization problem, which simultaneously optimizes an original data-fidelity term, the assignment smoothness over neighboring regions, and the number of assigned planar proxies. The synthesis of intermediate viewpoints demonstrates the effectiveness of our 3D reconstruction, and thereby the relevance of our proposed data fidelity-metric.

Dehazing is an image enhancing technique that emerged in the recent years. Despite of its importance there is no dataset to quantitatively evaluate such techniques. In this paper we introduce a dataset that contains 1400+ pairs of images with ground truth reference images and hazy images of the same scene. Since due to the variation of illumination conditions recording such images is not feasible, we built a dataset by synthesizing haze in real images of complex scenes. Our dataset, called D-HAZY, is built on the Middelbury [1] and NYU Depth [2] datasets that provide images of various scenes and their corresponding depth maps. Due to the fact that in a hazy medium the scene radiance is attenuated with the distance, based on the depth information and using the physical model of a hazy medium we are able to create a corresponding hazy scene with high fidelity. Finally, using D-HAZY dataset, we perform a comprehensive quantitative evaluation of several state of the art single-image dehazing techniques.

We introduce an effective technique to enhance night-time hazy scenes. Our technique builds on multi-scale fusion approach that use several inputs derived from the original image. Inspired by the dark-channel [1] we estimate night-time haze computing the airlight component on image patch and not on the entire image. We do this since under night-time conditions, the lighting generally arises from multiple artificial sources, and is thus intrinsically non-uniform. Selecting the size of the patches is non-trivial, since small patches are desirable to achieve fine spatial adaptation to the atmospheric light, this might also induce poor light estimates and reduced chance of capturing hazy pixels. For this reason, we deploy multiple patch sizes, each generating one input to a multiscale fusion process. Moreover, to reduce the glowing effect and emphasize the finest details, we derive a third input. For each input, a set of weight maps are derived so as to assign higher weights to regions of high contrast, high saliency and small saturation. Finally the derived inputs and the normalized weight maps are blended in a multi-scale fashion using a Laplacian pyramid decomposition. The experimental results demonstrate the effectiveness of our approach compared with recent techniques both in terms of computational efficiency and quality of the outputs.

The progress in imaging techniques have allowed the study of various aspect of cellular mechanisms. To isolate individual cells in live imaging data, we introduce an elegant image segmentation framework that effectively extracts cell boundaries, even in the presence of poor edge details. Our approach works in two stages. First, we estimate pixel interior/border/exterior class probabilities using random ferns. Then, we use an energy minimization framework to compute boundaries whose localization is compliant with the pixel class probabilities. We validate our approach on a manually annotated dataset.

Underwater images suffer from severe perceptual/visual degradation, due to the dense and non-uniform medium, causing scattering and attenuation of the propagated light that is sensed. Typical restoration methods rely on the popular Dark Channel Prior to estimate the light attenuation factor, and subtract the back-scattered light influence to invert the underwater imaging model. However, as a consequence of using approximate and global estimates of the back-scattered light, most existing single-image underwater descattering techniques perform poorly when restoring non-uniformly illuminated scenes. To mitigate this problem, we introduce a novel approach that estimates the back-scattered light locally, based on the observation of a neighborhood around the pixel of interest. To circumvent issue related to selection of the neighborhood size, we propose to fuse the images obtained over both small and large neighborhoods, each capturing distinct features from the input image. In addition, the Laplacian of the original image is provided as a third input to the fusion process, to enhance texture details in the reconstructed image. These three derived inputs are seamlessly blended via a multi-scale fusion approach, using saliency, contrast, and saturation metrics to weight each input. We perform an extensive qualitative and quantitative evaluation against several specialized techniques. In addition to its simplicity, our method outperforms the previous art on extreme underwater cases of artificial ambient illumination and high water turbidity.

This letter focuses on the estimation of low-complexity signals when they are observed through$M$uniformly quantized compressive observations. Among such signals, we consider 1-D sparse vectors, low-rank matrices, or compressible signals that are well approximated by one of these two models. In this context, we prove the estimation efficiency of a variant of Basis Pursuit Denoise, called Consistent Basis Pursuit (CoBP), enforcing consistency between the observations and the re-observed estimate, while promoting its low-complexity nature. We show that the reconstruction error of CoBP decays like${M^{ - 1/4}}$when all parameters but$M$are fixed. Our proof is connected to recent bounds on the proximity of vectors or matrices when (i) those belong to a set of small intrinsic “dimension”, as measured by the Gaussian mean width, and (ii) they share the same quantized (dithered) random projections. By solving CoBP with a proximal algorithm, we provide some extensive numerical observations that confirm the theoretical bound as$M$is increased, displaying even faster error decay than predicted. The same phenomenon is observed in the special, yet important case of 1-bit CS.

Randomized sets of binary tests have appeared to be quite effective in solving a variety of image processing and vision problems. The exponential growth of their memory usage with the size of the sets however hampers their implementation on the memory-constrained hardware generally available on low-power embedded systems. Our paper addresses this limitation by formulating the conventional semi-naive Bayesian ensemble decision rule in terms of posterior class probabilities, instead of class conditional distributions of binary tests realizations. Subsequent clustering of the posterior class distributions computed at training allows for sharp reduction of large binary tests sets memory footprint, while preserving their high accuracy. Our validation considers a smart metering applicative scenario, and demonstrates that up to 80% of the memory usage can be saved, at constant accuracy.

We propose a hybrid personalized summarization framework that combines adaptive fast-forwarding and content truncation to generate comfortable and compact video summaries. We formulate video summarization as a discrete optimization problem, where the optimal summary is determined by adopting Lagrangian relaxation and convex-hull approximation to solve a resource allocation problem. To trade-off playback speed and perceptual comfort we consider information associated to the still content of the scene, which is essential to evaluate the relevance of a video, and information associated to the scene activity, which is more relevant for visual comfort. We perform clip-level fast-forwarding by selecting the playback speeds from discrete options, which naturally include content truncation as special case with infinite playback speed. We demonstrate the proposed summarization framework in two use cases, namely summarization of broadcasted soccer videos and surveillance videos. Objective and subjective experiments are performed to demonstrate the relevance and efficiency of the proposed method.

Clipping or saturation in audio signals is a very common problem in signal processing, for which, in the severe case, there is still no satisfactory solution. In such case, there is a tremendous loss of information, and traditional methods fail to appropriately recover the signal. We propose a novel approach for this signal restoration problem based on the framework of Iterative Hard Thresholding. This approach, which enforces the consistency of the reconstructed signal with the clipped observations, shows superior performance in comparison to the state-of-the-art declipping algorithms. This is confirmed on synthetic and on actual high-dimensional audio data processing, both on SNR and on subjective user listening evaluations.

Given a set of plausible detections, detected at each time instant independently, we investigate how to associate them across time. This is done by propagating labels on a set of graphs that capture how the spatio-temporal and the appearance cues promote the assignment of identical or distinct labels to a pair of nodes. The graph construction is driven by the locally linear embedding (LLE) of either the spatio-temporal or the appearance features associated to the detections. Interestingly, the neighborhood of a node in each appearance graph is defined to include all nodes for which the appearance feature is available (except the ones that coexist at the same time). This allows to connect the nodes that share the same appearance even if they are temporally distant, which gives our framework the uncommon ability to exploit the appearance features that are available only sporadically along the sequence of detections. Once the graphs have been defined, the multi-object tracking is formulated as the problem of finding a label assignment that is consistent with the constraints captured by each of the graphs. This results into a difference of convex program that can be efficiently solved. Experiments are performed on a basketball and several well-known pedestrian datasets in order to validate the effectiveness of the proposed solution.

Multi-object tracking requires locating the targets as well as labeling their identities. Inferring identities of the targets from their appearances is a challenge when the avail- ability and the reliability of the observation process do vary along the time and space. The purpose of this paper is to assign identities to those appearance measurements using a graph-based formalism. Each node of the graph corresponds to a tracklet, which is defined to be a sequence of positions that very likely correspond to the same physical target. Tracklets are pre-computed and our work investigates how to assign them identities, knowing the reference appearance of each target. Initially, each node is assigned a probability distribution over the set of possible identities, based on the observed appearance features. Afterwards, belief propagation is considered to infer the identities of more ambiguous nodes from those of less ambiguous nodes, by exploiting the graph constraints and the measures of similarities between the nodes. In contrast to the standard belief propagation, which treats the nodes in an arbitrary order, the pro- posed method uses a priority-based belief propagation, in which less ambiguous nodes are scheduled to transmit their messages first. Validation is performed on a real-life basketball dataset. The proposed method achieves 89% identification rate, which is an improvement of 21% and 16% compared to individ- ual identity assignment, and to standard belief propagation, respectively.

A novel method is presented for trackingmultiple deformable objects in a multiview scenario. We avoid applying fixed constraints of object sizes for object isolation, but guide the splitting/merging of objects by grouping partial motion trajectories extracted from the occupancy status of objects in the homography ground-plane. We treat this clustering task as solving a rooted arborescence problem, and investigate the performance of the proposed method by experimental results.

When summarizing a video consisting of temporally continuous actions, e.g., surveillance videos or team-sport videos, fast-forwarding usually provides a better solution than content truncation to organize a semantically more complete and thus easily understandable story. Based on the resource allocation framework initially introduced in [1], we propose a fully automatic summarization system for broadcasted soccer videos, which supports adaptive fast-forwarding as well as content truncation. The proposed system involves two major contributions (i) an improved algorithm for refining the boundaries of far-view clips based on the extracted camera movement; (ii) a process to organize a summary by determining both the selected clips and their optimal fast-forwarding strategies, considering the scene-changing tempos of those clips and the given duration constraint. Experiments on real-life broad-casted soccer videos demonstrate the relevance and the efficiency of our proposed method.

In the context of team sport events monitoring, the various phases of the game must be delimited and interpreted. In the case of a basketball game, the detection and the tracking of the ball are mandatory. This paper deals with the detection of the ballistic trajectory of a ball thrown between two players or toward the basket. Ballistic trajectories build on the 3D ball candidates previously detected at each timestamp. The proposed method is based on a graph, for which the nodes are the observed velocity between two 3D candidates, the edges link two nodes that have a common 3D candidate and the cost function is proportional to the distance between the observed acceleration and the gravity one. Hence, looking for the shortest path in this graph is equivalent to searching for 3D candidates that follow a ballistic trajectory. It appears to be both quite efficient and effective.

This paper describes an adaptive streaming technique that exploits temporal concatenation of H.264/AVC video bit-streams and uses only standard RTCP reports as feedback mechanism. As a result, common media players like VLC, QuickTime or GStreamer based, can be used in a streaming session with improved viewing experience when accessing video content through bandwidth constrained connections. Further, an original probing technique that uses video packets as probing data has been developed in order to assess whether the available bandwidth allows streaming at a higher bitrate, maximizing thus video quality and user experience. The proposed solution has been tested in real wireless scenarios, showing that video quality can indeed be improved even for standard media players.

We propose a flexible framework to summarize team-sport videos that have been originally produced for broadcast purposes. The framework is able to integrate both the knowledge about displayed content (e.g., level of interest, type of view, and so on), and the individual (narrative) preferences of the user. It builds on the partition of the original video sequence into independent segments, and creates local stories by considering multiple ways to render each segment. We discuss how to segment videos automatically based on production principles, and design parametric functions to evaluate the benefit of various local stories from a segment. Summarization by selection of local stories is then regarded as a resource allocation problem, and Lagrangian relaxation is performed to find the optimum. We investigate the efficiency of our framework by summarizing soccer, basketball and volleyball videos in our experiments.

In this paper, we address the issues of analyzing and classifying JPEG 2000 code-streams. An original representation, called integral volume, is first proposed to compute local image features progressively from the compressed code-stream, on any spatial image area, regardless of the code-blocks borders. Then, a JPEG 2000 classifier is presented that uses integral volumes to learn an ensemble of randomized trees. Several classification tasks are performed on various JPEG 2000 image databases and results are in the same range as the ones obtained in the literature with noncompressed versions of these databases. Finally, a cascade of such classifiers is considered, in order to specifically address the image retrieval issue, i.e., bi-class problems characterized by a highly skewed distribution. An efficient way to learn and optimize such cascade is proposed. We show that staying in a JPEG 2000 framework, initially seen as a constraint to avoid heavy decoding operations, is actually an advantage as it can benefit from the multiresolution and multilayer paradigms inherently present in this compression standard. In particular, unlike other existing cascaded retrieval systems, the features used along our cascade are increasingly discriminant and lead therefore to a better tradeoff of complexity versus performance.

Democratic and personalized production of multimedia content is a challenge that content providers will have to face in the near future. In this paper, we address this challenge by building on computer vision tools to automate the collection and distribution of audiovisual content. Especially, we proposed a complete production process of personalized video summaries in a typical application scenario, where the sensor network for media acquisition is composed of multiple cameras, which, for example, cover a basketball field. Distributed analysis and interpretation of the scene are exploited to decide what to show or not to show about the event, so as to produce a video composed of a valuable subset of the streams provided by each individual camera. Interestingly, the selection of the streams subsets to forward to each user depends on his/her individual preferences, making the process adaptive and personalized. The process involves numerous integrated technologies and methodologies, including but not limited to automatic scene analysis, camera viewpoint selection, adaptive streaming, and generation of summaries through automatic organization of stories. The proposed technology provides practical solutions to a wide range of applications, such as personalized access to local sport events through a web portal, cost-effective and fully automated production of content dedicated to small-audience, or even automatic log in of annotations.

We propose a flexible framework for producing highly personalized basketball video summaries, by integrating contextural information, narrative user preferences on story pattern, and general production principles. Starting from the multiple streams captured by a distributed set of fixed cameras, we study the implementation of autonomous viewpoint determination and automatic temporal segment selection, and also discuss the production of visually comfortable output, by applying smoothing process to viewpoint selection and by defining efficient benefit functions to evaluate various summary organization. The efficiency of our framework is demonstrated by experimental results.

This paper presents a fully automatic system for soccer game summarization. The system takes audio-visual content as an input, and builds on the integration of two independent but complementary contributions (i) to identify crucial periods of the soccer game in a fully automatic way, and (ii) to summarize the soccer game as a function of individual narrative preferences of the user. The process involves both audio and video analysis, and handles the personalized summarization challenge as a resource allocation problem. Experiments on real-life broadcasted content demonstrate the relevance and the computational efficiency of our integrated approach.

This paper builds on an interactive streaming architecture that supports both user feedback interpretation, and temporal juxtaposition of multiple video bitstreams in a single streaming session. As an original contribution, we explain how these functionalities can be exploited to offer improved viewing experience, when accessing highresolution or multi-views video content through individual and potentially bandwidth-constrained connections. This is done by giving the client the opportunity to select interactively a preferred version among the multiple streams that are offered to render the scene. An instance of this architecture has been implemented extending the liveMedia streaming library and using the x264 video encoder. Automatic methods have been designed and implemented to generate the multiple versions of the streamed content. In a surveillance scenario, the versions are constructed by sub-sampling the original high resolution image, or by cropping the image sequence to focus on regions of interest, in a temporally consistent way. In a soccer game context, zoomed-in versions of far view shots are computed and offered as alternatives to the sub-sampled sequence. We demonstrate the feasibility and relevance of the approach through subjective experiments.

We propose a flexible summarization framework for team-sport videos, which is able to integrate both the knowledge about displayed content (e.g. level of interest, type of view, etc.), and the individual (narrative) preferences of the user. Our framework builds on the partition of the original video sequence into independent segments, and create local stories by considering multiple ways to render each segment. We discuss how to segment videos based on production principles, and design the benefit function to evaluate various local stories from a segment. Summarization by selection of local stories is regarded as a resource allocation problem, and Lagrangian relaxation is performed to find the optimum. We use a soccer video to validate our framework in our experiments.

In this paper, we describe an interactive streaming architecture. The content streamed by this architecture is encoded with the H.264/AVC standard for video compression. The architecture has three key functionalities: temporal juxtaposition of multiple video bitstreams in a unique streaming session, on the fly image bitstreams composition and user feedback interpretation. These functionalities rely mainly on H.264/AVC features, thus allowing any basic H.264/AVC compliant player to use them.

In video surveillance applications, pre-stored images are likely to be accessed remotely and interactively upon user request. In such a context, the JPEG 2000 still image compression format is attractive because it supports flexible and progressive access to each individual image of the pre-stored content, in terms of spatial location, quality level, as well as resolution. However, when the client wants to play consecutive frames of the video sequence, the purely INTRA nature of JPEG 2000 dramatically penalizes the transmission efficiency. To mitigate this drawback, conditional replenishment mechanisms are envisioned. They convey arbitrary spatio-temporal segments of the initial video sequence directly through sporadic and rate-distortion (RD) optimized refresh of JPEG 2000 packets. Hence, they preserve JPEG 2000 compliance, while saving transmission resources. The replenishment algorithms proposed in this paper are original in two main aspects. First, they exploit the specificities of the JPEG 2000 codestream structure to balance the accuracy (in terms of bit-planes) of the replenishment across image subbands in a (RD)-optimal way. Second, they take into account the still background nature of video surveillance content by maintaining two reference images at the receiver. One reference is the last reconstructed frame, as proposed in the original replenishment framework. The other is a dynamically computed estimate of the scene background, which helps to recover the background after a moving object has left the scene. As an additional contribution, we demonstrate that the embedded nature of the JPEG 2000 codestream easily supports prioritization of semantically relevant regions of interest while browsing video content. An interesting aspect of this JPEG 2000-based prioritization is that it can be regulatedaposteriori, after the codestream generation, based on the interest expressed by the user at browsing time. Simulation results demonstrate the efficiency and flexibility of the approach compared to INTER-based solutions.

In order to achieve both fast tracking and accurate object extraction, we present in this paper an original real-time active contour method, incorporating different feature maps into a common and homogeneous framework, defined by the multi-feature vector flow (MFVF). The MFVF active contour approach does not require any target prior model, and enables precise tracking of mobile deformable objects. The use of the MFVF, resulting from multiple selected features, brings robustness into the system towards complex situations, while our computationally efficient implementation of the MFVF scheme reaches the required speed range for tracking process. The proposed method has been successfully tested on real-world video sequences.

Selective encryption is a new trend in content protection. It aims at reducing the amount of data to encrypt while achieving a sufficient and inexpensive security. This approach is particularly desirable in constrained communication (real time networking with delay constraints, mobile communication with limited computational power...). In this paper we introduce selective encryption from information theory point of view. We define a set of evaluation criteria for selective encryption algorithms and propose a novel selective encryption algorithm for JPEG2000 compressed images satisfying all these criteria. The main contribution of this proposal consists of reaching the minimum amount of data to encrypt regarding a given level of security and target application requirements. For this purpose, we exploit the R-D optimization performed by JPEG2000 EBCOT algorithm.

This paper tackles the challenge of interactively retrieving visual scenes within surveillance sequences acquired with fixed camera. Contrarily to today's solutions, we assume that no a-priori knowledge is available so that the system must progressively learn the target scenes thanks to interactive labelling of a few frames by the user. The proposed method is based on very low-cost features extraction and integrates relevance feedback, multiple-instance SVM classification and active learning. Each of these 3 steps runs iteratively over the session, and takes advantage of the progressively increasing training set. Repeatable experiments on both simulated and real data demonstrate the efficiency of the approach and show how it allows reaching high retrieval performances.

The image compression standard JPEG 2000 offers a high compression efficiency as well as a great flexibility in the way it accesses the content in terms of spatial location, quality level, and resolution. This paper explores how transmission systems conveying video surveillance sequences can benefit from this flexibility. Rather than transmitting each frame independently as it is generally done in the literature for JPEG 2000 based systems, we adopt a conditional replenishment scheme to exploit the temporal correlation of the video sequence. As a first contribution, we propose a rate-distortion optimal strategy to select the most profitable packets to transmit. As a second contribution, we provide the client with two references, the previous reconstructed frame and an estimation of the current scene background, which improves the transmission system performances.

Segmentation and tracking methods have been widely explore. However, they are often computationally heavy or require constraining assumptions. We present in this paper a new system for real-time simultaneous segmentation and tracking, without any hypothesis on target appearance, image background or camera properties. The proposed approach (SUGVPB) is an active contour modeled with B-splines and which evolution process is using a speeded up gradient vector flow, characterized by a faster computation of the edge diffusion process. The synergy of these two powerful components enables precise, robust and real-time tracking of complete non-rigid mobile objects. Our method has been validated on synthetic as well as natural video sequences.

Our study considers the development of a reliable tracker for non-rigid objects evolving on cluttered background in crowded scenes captured by moving cameras. For this purpose, we propose an original method that combines two approaches, respectively based on parametric active contours (PAC) and on point distribution model (PDM). The PAC tracker relies on an effective and effcient implementation of contour convergence mechanism to bring a smooth contour to the edges of the target in real-time. The PDM approach collects feature points in the region delineated by the PAC tracker to build and update a model of the target in term of a feature point distribution. Formally, when a novel frame is considered, its feature points are matched with the PDM model. The matching information is used to initialize the novel PAC, whose convergence identify the points that are relevant to update the PDM for the next frame. Hence, the two approaches complement each others. The a priori information provided by the PDM makes the system robust towards occlusions, while the deformation of the PAC increases its robustness towards target appearance changes. Simulations on real-word video sequences demonstrate the performance of our approach.

This paper considers the issues of scheduling and caching JPEG2000 data in client/server interactive browsing applications, under memory and channel bandwidth constraints. It analyzes how the conveyed data have to be selected at the server and managed within the client cache so as to maximize the reactivity of the browsing application. Formally, to render the dynamic nature of the browsing session, we assume the existence of a reaction model that defines when the user launches a novel command as a function of the image quality displayed at the client. As a main outcome, our work demonstrates that, due to the latency inherent to client/server exchanges, a priori expectation about future navigation commands may help to improve the overall reactivity of the system. In our study, the browsing session is defined by the evolution of a rectangular window of interest (WoI) along the time. At any given time, the WoI defines the position and the resolution of the image data to display at the client. The expectation about future navigation commands is then formalized based on a stochastic navigation model, which defines the probability that a given WoI is requested next, knowing previous WoI requests. Based on that knowledge, several scheduling scenarios are considered. The first scenario is conventional and transmits all the data corresponding to the current WoI before prefetching the most promising data outside the current WoI. Alternative scenarios are then proposed to anticipate prefetching, by scheduling data expected to be requested in the future before all the current WoI data have been sent out. Our results demonstrate that, for predictable navigation commands, anticipated prefetching improves the overall reactivity of the system by up to 30% compared to the conventional scheduling approach. They also reveal that an accurate knowledge of the reaction model is not required to get these significant improvements.

We consider streaming pre-encoded and packetized media over best-effort networks in the presence of acknowledgment feedbacks. We first review a rate-distortion (RD) optimization framework that can be employed in such scenarios. As part of the framework, a scheduling algorithm selects the data to send over the network at any given time, so as to minimize the end-to-end distortion, given an estimate of channel resources and a history of previous transmissions and received acknowledgements. In practice, a greedy scheduling strategy is often considered to limit the solution search space, and reduce the computational complexity associated to the RD optimization framework. Our work observes that popular greedy schedulers are strongly penalized by early retransmissions. Therefore, we propose a scheduling algorithm that avoids premature retransmissions, while preserving the low computational complexity aspect of the greedy paradigm. Such a scheduling strategy maintains close to optimal RD performance when adapting to network bandwidth fluctuations. Our experimental results demonstrate that the proposed patient greedy scheduler provides a reduction of up to 50% in transmission rate relative to conventional greedy approaches, and that it brings up to 2 dB of quality improvement in scheduling classical MPEG-based packet video streams.

This paper addresses the problem of streaming packetized media over a lossy packet network, with sender-driven (re)transmissions and receiver acknowledgements. It extends the Markovian formulation of the rate-distortion optimized (RaDiO) streaming framework by allowing the transmission schedule of a media data unit to become contingent on the acknowledgements relative to other data units. Media decoding dependencies are generally considered in state-of-the-art rate-distortion optimized scheduling algorithms. However, the set of eligible packet schedules are restricted to independent streaming policies, where the transmission strategy envisioned for a data unit at future transmission opportunities only depends on its own acknowledgment, and not on the acknowledgments received for other data units. This paper questions the validity of this assumption in the design of rate-distortion optimal streaming solutions, and provides a first attempt in the formal derivation of the benefit offered by dependent policies. One of the main contributions of our paper is to propose a methodology that limits the search space of dependent policies to relevant dependencies that are likely to bring a rate-distortion benefit, in order to solve an optimization problem that is a priori computationally intractable. Extensive simulations validate the proposed approach that focuses on relevant dependencies between streaming policies. We further show that the benefit of dependent streaming policies is actually marginal in practical scenarios where the gain in distortion per unit of rate decreases along the media decoding dependency path. It represents the first demonstration that the common assumption of independent streaming policies is valid in many common streaming scenarios. However, experimental results also demonstrate significant benefits and encourage a careful investigation of dependent policies when the content is characterized by an increase of the benefit per transmission unit brought along the data unit dependency path.

This paper considers the remote interactive browsing of large JPEG2000 images. In contrast with previous contributions, we focus on the dynamic nature of the system. Practically, we study the conditions under which a priori knowledge about the user behavior may help to improve the browsing system reactivity, when combined with appropriate pre-fetching mechanisms. In particular, our simulations show that, due to the latency inherent to client/server exchanges, a benefit can be drawn by scheduling future window of interest (WoI) data before all current WoI data have been sent to the client. They also reveal that an accurate knowledge of the user behavior is not necessary to get important improvements over conventional scheduling approaches.

Robust image hashing defines a feature vector that characterizes the image, independently of non-significant distortions of its content. As a consequence, the comparison between robust image hash vectors is able to indicate whether the corresponding images are equivalent or not, independently of visually non-significant distortions due for example to compression or re-sampling. We define a robust image hash based on radial projections of the image pixels. Specifically, our proposed radial hASH (RASH) considers moments of different orders to describe the luminance pdf of the pixels encountered on a set of lines articulated around the center of the image. In short, each RASH component is defined based on the moment of the pixels belonging to a specific line. Our paper provides a careful analysis of the robustness and discriminating capabilities of the RASH vectors computed based on different moment orders. As a first contribution, it demonstrates that the second order moment, i.e. the variance of the pixels on a line, allows for optimal trade-offs between the robustness and the discriminating capabilities of the resulting RASH vector. As a second contribution, extensive simulations prove that a decision engine based on RASH vectors cross-correlations is able to successfully identify pairs of equivalent or distinct images. Bottom line, the RASH assets are a low computational complexity, a strong robustness to both filtering and geometrical distortions, and a risk of collision that is estimated to less than 10 per million of images.

We consider streaming of pre-encoded and packetized media over best-effort networks in presence of acknowledgment feedback. Given an estimation of future transmission resources and knowing about past transmissions and received acknowledgments, a scheduling algorithm is defined as a mechanism that selects the data to send over the network at any given time, so as to minimize the end-to-end distortion. Our work first reveals the sub-optimality of popular greedy schedulers, which might be strongly penalized by anticipated retransmissions. It then proposes an original scheduling algorithm that avoids premature retransmissions, while preserving the simplicity of the greedy paradigm. The proposed patient greedy (PG) scheduler appears to save up to 50% of rate in comparison with the conventional greedy approach.

The cost-efficient implementation of video codecs requires a set of methodologies and decision taking at different levels in the design flow. We combine upfront algorithmic tuning with memory centric optimizations to transform the video application into a system consisting of functional blocks with localized data processing and a tailored memory hierarchy. This memory optimized functional description is the leverage for the cost-efficient mapping of the system on integrated multimedia platforms. It closely reflects the real implementation constraints and consequently allows for steering the architecture selection in a correct way. The proposed approach is demonstrated on a MPEG-4 video encoder and leads to its implementation as a pipelined system. Hardware development of the motion estimation validates that the high-level memory centric concepts are applicable and realizable at the lowest level. The motion estimation kernel supports up to 30 CIF f/s with minimized processing element requirements and data input rates.

Applications using Transmission Control Protocol (TCP), such as web-browsers, ftp, and various peer-to-peer (P2P) programs, dominate most of the Internet traffic today. In many cases, users have bandwidth-limited last mile connections to the Internet which act as network bottlenecks. Users generally run multiple concurrent networking applications that compete for the scarce bandwidth resource. Standard TCP shares bottleneck link capacity according to connection round-trip time (RTT), and consequently may result in a bandwidth partition which does not necessarily coincide with the user's desires. In this work, we present a receiver-based bandwidth sharing system (BWSS) for allocating the capacity of last-hop access links according to user preferences. Our system does not require modifications to the TCP protocol, network infrastructure or sending hosts, making it easy to deploy. By breaking fairness between flows on the access link, the BWSS can limit the throughput fluctuations of high-priority applications. We utilize the BWSS to perform efficient video streaming over TCP to receivers with bandwidth-limited last mile connections. We demonstrate the effectiveness of our proposed system through Internet experiments.

Applications using TCP, such as Web-browsers, ftp, and various P2P programs, dominate most of the Internet traffic today. In many cases the last-hop access links are bottlenecks due to their limited bandwidth capability with users running many simultaneous network applications. Standard TCP shares bottleneck link capacity according to connection round-trip time (RTT), and may result in a bandwidth partition which does not necessarily coincide with the user's desires. We present a receiver-based control system for allocating bandwidth among TCP flows according to user preferences. Our system does not require any changes to network infrastructure, and works with standard TCP senders. NS-2 simulations, as well as actual Internet experiments, show that our system achieves desired bandwidth allocation in a wide variety of scenarios including interfering cross-traffic. We also demonstrate the viability of our system in multimedia streaming applications over TCP.

This paper focuses on a major requirement of rate control (RC) algorithms when implementing a low-power video coding system: RC must be compatible with a one-pass, sequential and local processing of the frame data. This requirement prevents direct use of the most efficient RC algorithms, i.e., the ones that rely on rate-distortion (R-D) models whose parameters are computed from a pre-analysis of the input frame. The paper proposes to circumvent the problem by predicting the R-D model parameter(s) without accessing the current input frame data. It avoids the pre-analysis stage while keeping the benefit from R-D based rate control. The method is designed and illustrated for standardized and conventional hybrid coding schemes (H.26x, MPEG-x). Specifically, the mean absolute difference (MAD) of the motion prediction error, which is the key R-D model parameter, is predicted before the sequential processing of the input blocks. In order to validate the prediction, the behaviors of rate control systems using either the actual (computed) or the estimated (predicted) MAD parameter are compared.

This paper provides a precise analytical study of the selection and modulus quantization of matching pursuit (MP) coefficients. We demonstrate that an optimal rate-distortion trade-off is achieved by selecting the atoms up to a quality-dependent threshold, and by defining the modulus quantizer in terms of that threshold. In doing so, we take into account quantization error re-injection resulting from inserting the modulus quantizer inside the MP atom computation loop. In-loop quantization not only improves coding performance, but also affects the optimal quantizer design for both uniform and nonuniform quantization. We measure the impact of our work in the context of video coding. For both uniform and nonuniform quantization, the precise understanding of the relation between atom selection and quantization results in significant improvements in terms of coding efficiency. At high bitrates, the proposed nonuniform quantization scheme results in 0.5 to 2 dB improvement over the previous method.

The need for reversible or lossless watermarking methods has been highlighted in the literature to associate subliminal management information with losslessly processed media and to enable their authentication. The paper first analyzes the specificity and the application scope of lossless watermarking methods. It explains why early attempts to achieve reversibility are not satisfactory. They are restricted to well-chosen images, strictly lossless context and/or suffer from annoying visual artifacts. Circular interpretation of bijective transformations is proposed to implement a method that fulfills all quality and functionality requirements of lossless watermarking. Results of several bench tests demonstrate the validity of the approach.

We provide an analytical study of the selection and modulus quantization of matching pursuits (MP) coefficients. We demonstrate that an optimal rate-distortion trade-off is achieved by selecting the atoms up to a dead-zone threshold, and by defining the modulus quantizer in terms of that threshold. In doing so, we take into account quantization error re-injection resulting from inserting the modulus quantizer inside the MP atom computation loop. In-loop quantization affects the stepsize of the uniform quantizer, and results in a non-uniform optimal entropy constrained quantizer. Improvements larger than one dB are obtained for video coding.

Digital watermarking consists of hiding subliminal information into digital media content, also called host data. It can be the basis of many applications, including security and media asset management. In this paper, we focus on the imperceptibility requirement for image watermarking. We present the main features of the human visual system (HVS) to be translated into watermarking technology. This paper highlights the need for dedicated inputs from the human vision community. The human visual system (HVS) is very complex and able to deal with a huge amount of information. Roughly speaking, it is composed of a receiver with a pre-processing stage, the eye and the retina, a transmission channel, the optic nerve, and a processing engine, the visual cortex. Mainly because of our lack of knowledge about brain behavior, i.e. about the way a stimulus is processed through its huge neural network, the large effort to understand and model the HVS behavior has partly remained fruitless. The aim of this paper is not to provide a thorough description of the HVS. For complete HVS models and more specific details, the reader is referred to existing literature. Here, we only try to understand, in a synthetic way and from an engineering perspective, the HVS features on which the designer of a watermarking algorithm can rely, i.e. its sensitivity and masking capabilities.

Digital watermarking consists of hiding subliminal information into digital media content, also called host data. It can be the basis of many applications, including security and media asset management. In this paper we focus on the imperceptibility requirement for image watermarking. We first provide a functional inventory of image watermarking applications and emphasize the dependency between the application purpose and its need for invisibility. Then, we present a global framework common to most existing watermarking systems. It illustrates the methodology followed to translate human vision research into watermarking technology. It suggests future prospects and highlights the need for dedicated inputs from the human vision community.

Due to the large amount of data transfers it involves, the motion estimation (ME) engine is one of the most power-consuming components of any predictive video codec. As a consequence, power-optimized video coding primarily relies on a carefully designed motion estimator. This paper first presents a block ME algorithm that meets high-quality inter-frame prediction and low computational complexity requirements. It relies on a set of rules common to all recent fast and adaptive ME algorithms, but is designed in order to allow for easy and prolific data reuse. The adjacent order of the candidate positions during the search increases the locality and maintains a near-regular data flow, which results in a decrease of the data transfers and a low control complexity. Together with the computational complexity reduction, it enables cost-efficient very large scale integration realizations. A pipelined parallel architecture is then proposed and discussed. It is generic in the sense that it is suited both to the full-pel and half-pel ME. It is efficient because it allows for close to 100% hardware utilization and a sharp decrease of the peak memory bandwidth. It is suited to low-power implementation, as it enables larger data reuse factors for the most probable stages of the adaptive algorithm, which reduces the average memory bandwidth and power consumption.

We propose a block motion estimation (ME) algorithm that meets high quality requirements and allows for cost efficient VLSI realizations. It relies on a set of rules common to all fast BMA algorithms and has been designed in order to allow for easy and prolific data reuse.

The need for reversible or lossless watermarking methods has been highlighted to associate information with losslessly processed media or to enable their authentication. The paper first analyzes the specificity and the application scope of lossless watermarking methods. An original circular interpretation of a bijective transformation is then proposed to implement a method that fulfill all quality and functionality requirements.

In this paper, SNR scalable representations of video signals are studied. The investigated codecs are well suited for communications applications because they are all based on backward motion-compensated predictive coding, which provides the necessary low-delay property. In a very-low bit rate context (VLBR), the matching pursuits (MP) signal representation algorithm is used to represent the displaced frame difference (DFD) of each layer of a multilevel decomposition of the video signal. A number of conventional prediction schemes that can be generalized to any DFD representation technique are considered. They are compared with an original and MP specific DFD prediction method. Two scenari have been considered. In the first scenario, an enhancement layer is built on a base layer that has been encoded using a classical, i.e., nonscalable scheme. In that case, all methods appear to be comparable, In the second scenario, the fact that the base layer is used as a reference for an enhancement layer is taken into account to build it. In that case, the proposed MP prediction method clearly outperforms all other conventional approaches, Additional lessons can be drawn from this work. The same motion vectors can be used in both SNR layers, and the DFD prediction between layers improves coding efficiency. Moreover, the MP representation of the signal enable us to measure the predictability of the high SNR layer DFD from the low SNR layer DFD, i.e., to quantify the part of the low SNR layer information that also belongs to the high SNR layer.

"Matching pursuits" is a signal expansion technique whose efficiency for video coding has already been largely demonstrated in the MPEG-4 framework. In this paper, our attention focuses on complexity issues. First, the most expensive step of the signal expansion process is significantly speeded up by exploiting results achieved in the wavelet and multiresolution theory. A subband dictionary is proposed as an alternative to the Gabor dictionary that has been used up to now. Equivalent levels of quality are achieved with both dictionaries, but the computational cost is significantly reduced when using the subband one. Then, we explain how, with any dictionary, the linearity of the inner product could be exploited to further speed up the process in return for an increased amount of memory.

"Matching pursuits" is a signal expansion technique whose efficiency for video coding has already been demonstrated in the MPEG-4 community. In this paper, we focus our attention on some aspects of the algorithm. Firstly, the relevance of the orthogonalization of the signal expansion is investigated. Secondly, a subband dictionary is proposed as an alternative to the Gabor dictionary, used up to now. Equivalent levels of quality are achieved with both dictionaries but the computational cost is significantly reduced when using the subband one. Eventually, the entropy coding stage of the algorithm is studied.

The tool introduced in this paper allows to automatically decide in an image or in a video sequence which regions are important and which ones are not. For this purpose, fuzzy logic has been used to modelize human subjective knowledge about the way to allocate priorities to regions. The resulting classification can be used in a wide range of applications going from image coding to image understanding.