The strong temporal consistency of surveillance video enables compelling compression performance with traditional methods, but downstream vision applications operate on decoded image frames with a high data rate. Since it is not straightforward for applications to extract information on temporal redundancy from the compressed video representations, we propose a novel system which conveys temporal redundancy within a sparse decompressed representation. We leverage a video representation framework called ADΔER to transcode framed videos to sparse, asynchronous intensity samples. We introduce mechanisms for content adaptation, lossy compression, and asynchronous forms of classical vision algorithms. We evaluate our system on the VIRAT surveillance video dataset, and we show a median 43.7% speed improvement in FAST feature detection compared to OpenCV. We run the same algorithm as OpenCV, but only process pixels that receive new asynchronous events, rather than process every pixel in an image frame. Our work paves the way for upcoming neuromorphic sensors and is amenable to future applications with spiking neural networks.

Neuromorphic "event" cameras, designed to mimic the human vision system with asynchronous sensing, unlock a new realm of high-speed and high-dynamic-range applications. However, researchers often either revert to a framed representation of event data for applications, or build bespoke applications for a particular camera's event data type. To usher in the next era of video systems, accommodate new event camera designs, and explore the benefits of asynchronous video in classical applications, we argue that there is a need for an asynchronous, source-agnostic video representation. In this paper, we introduce a novel, asynchronous intensity representation for both framed and non-framed data sources. We show that our representation can increase intensity precision and greatly reduce the number of samples per pixel compared to grid-based representations. With framed sources, we demonstrate that by permitting a small amount of loss through the temporal averaging of stable pixel values, we can reduce our representational sample rate by more than half, while incurring a drop in VMAF quality score of only 4.5. We also demonstrate lower latency than the state-of-the-art method for fusing and transcoding framed and event camera data to an intensity representation, while maintaining 2000X the temporal resolution. We argue that our method provides the computational efficiency and temporal granularity necessary to build real-time intensity-based applications for event video.

Enrollments in computer science courses and majors are at or exceeding capacity at the college level. This context drives local innovations that may benefit individuals across the SIGCSE community. The panelists will share how, in the context of booming enrollments and COVID, they strive to protect faculty time, engage students in larger classes, take advantage of scale, improve student-TA interactions, motivate faculty to teach larger classes, and better monitor students in large classes. During the panel Q&A, attendees will be invited to share additional strategies live on Course.Care, which will then be disseminated through CSTeachingTips.org.

Event cameras are biologically-inspired sensors that upend the framed, synchronous nature of traditional cameras. Singh et al. proposed a novel sensor design wherein incident light values may be measured directly through continuous integration, with individual pixels' light sensitivity being adjustable in real time, allowing for extremely high frame rate and high dynamic range video capture. Up to this point, there has been little research into compressing this event data, and even less that has investigated a robust method of lossy compression. This paper makes an inroad in this area, proposing a lossy model-based compression scheme with user-controlled quality levels and examine its compression efficiency and reconstructed image quality for several synthetic scenes. In our experiments, we observe space savings over the original event stream upwards of 95% with only minor losses in reconstructed image quality.

Integrating event cameras are asynchronous sensors wherein incident light values may be measured directly through continuous integration, with individual pixels' light sensitivity being adjustable in real time, allowing for extremely high frame rate and high dynamic range video capture. This paper builds on lessons learned with previous attempts to compress event data and presents a new scheme for event compression that has many analogues to traditional framed video compression techniques. We show how traditional video can be transcoded to an event-based representation, and describe the direct encoding of motion data in our event-based representation. Finally, we present experimental results proving how our simple scheme already approaches the state-of-the-art compression performance for slow-motion object tracking. This system introduces an application "in the loop" framework, where the application dynamically informs the camera how sensitive each pixel should be, based on the efficacy of the most recent data received.

Event cameras are biologically-inspired sensors that upend the framed, synchronous nature of traditional cameras. Singh et al. proposed a novel sensor design wherein incident light values may be measured directly through continuous integration, with individual pixels' light sensitivity being adjustable in real time, allowing for extremely high frame rate and high dynamic range video capture. Arguing the potential usefulness of this sensor, this paper introduces a system for simulating the sensor's event outputs and pixel firing rate control from 3D-rendered input images.

In this paper, we present - ZenCam, which is an always-on body camera that exploits readily available information in the encoded video stream from the on-chip firmware to classify the dynamics of the scene. This scene-context is further combined with simple inertial measurement unit (IMU)-based activity level-context of the wearer to optimally control the camera configuration at run-time to keep the device under the desired energy budget. We describe the design and implementation of ZenCam and thoroughly evaluate its performance in real-world scenarios. Our evaluation shows a 29.8-35% reduction in energy consumption and 48.1-49.5% reduction in storage usage when compared to a standard baseline setting of 1920×1080 at 30fps while maintaining a competitive or better video quality at the minimal computational overhead.

Modern motion camera sensors capture light in discrete frames, a residual artifact from the earliest days of video capture that imposes an unnecessary synchronizing constraint on all pixels (all pixels must start and stop measuring at the same time). While fixed time frames do offer some simplicity -- simpler sensor design, simpler encoding, and simpler transmission -- the rigidity of frames imposes significant costs in terms of dynamic range, precision and sensitivity, and we claim that with advances in hardware and software the time is ripe for a paradigm shift. We present the design of a novel sensor architecture capable of producing a frameless, asynchronous stream of high dynamic range pixel data. The key design insight is to use a high resolution timer to measure the period of time required to gather a per-pixel configurable amount of light rather than the amount of light gathered in a fixed period of time. A system model detailing how such a sensor can be integrated into existing multimedia systems and applications is proposed, exposing a number of interesting challenges with respect to representing and compressing asynchronous pixel data. Results of a preliminary investigation of a baseline scheme that repurposes existing frame synchronous video encoding methods (i.e., H.264) are presented.

Increased enrollments in computer science programs presents a new challenge of quickly accommodating higher enrollment in computer science introductory courses. Because peer teaching scales with enrollment size, it is a promising solution for supporting computer science students in this setting. However, pedagogical and logistical challenges can arise when implementing a large peer teaching program. To study these challenges, we developed a transparent online tool, My Digital Hand, for tracking one-to-one peer teaching interactions. We deployed the tool across three universities in large CS2 computer science courses. The data gathered confirms the pedagogical and logistical challenges that exist at scale and gives insight into ways we might address them. Using this information, we developed the second iteration of My Digital Hand to better support peer teaching. This paper presents the modified tool for use by the computer science education community.

Online discussion forums are widely used and hold great promise for supporting students in learning computer science. Understanding how we can best support students in learning computer science through online discussion forums is an important open question for the CS Ed community. This paper analyzes discussion forum posts from 395 students enrolled in CS2 across two different universities. The results demonstrate that students use the discussion forums often for logistical and relatively shallow questions. However, the largest portion of questions reflect some level of constructive problem-solving activity, and are positively correlated with course grades. Questions that neither describe students' reasoning nor their attempts to solve the problem constitute the smallest percentage of questions, but these questions may be particularly important to attend to because of their relationship to students' prior experience.

Prior research introduced the n-Dimensional Display Interface (NDDI) as a new "narrow waist" for the display pipeline. In this paper, we extend the NDDI architecture to provide a blending feature. We then utilize that new feature for full screen video playback, leveraging application-level framing to realize a significant data transmission reduction. We then explore new NDDI configurations for effective rate control under highly constrained transmission budgets.

The framebuffer is a simple yet powerful abstraction for representing display data. It ably serves as a "narrow waist" for the display pipeline, allowing graphics software and display technology to evolve in parallel. Converging trends such as mobile computing, very large displays, and 3DTV are challenging the inherent inefficiency of practically serializing and encoding the framebuffer tens or hundreds of times per second. In this paper, we present a flexible new software abstraction that can deliver tremendous channel efficiency when application level semantics are exploited. The goal is to develop a versatile interface from this elastic narrow waist that scales from very large displays to small, low-power displays connected over wireless links.

This paper explores a new abstraction to replace the framebuffer as the metaphor for a new display interface. Our novel approach aims to provide backward compatibility with applications that require a simple framebuffer, while also providing tremendous channel capacity savings to applications that exploit application level semantics to use the display interface in a more sophisticated way. The goal is to develop a versatile interface that scales from very large displays to small, low-power displays connected over wireless links.

Capacity planning, as well as protocol design and storage system design for video distribution systems are dependent on an understanding of the anticipated workload. Previous studies have focused on aggregate statistics of video popularity over time, and only recently has work been done which analyzes user behaviour variability over time. We are interested in a video storage/distribution system that uses peer resources to help content providers distribute video content to nodes in a network over long periods of time. Such a system is intended to operate on local principles of utility associated with each peer. More than ever, a workload model based on long-term user behaviour is required. In particular, it is unclear how request patterns driven by local utility can be shaped to match expected large-scale aggregate request characteristics such as a Zipf-distribution for overall popularity. In this paper, we describe early work in developing a workload generator for a Peer-based PVR (Personal Video Recorder) system to demonstrate some of these challenges.

We present a framework for multidimensional utility-driven adaptation for multi-stream video applications. A notable driving application is 3D tele-immersion. Our framework directly models the utility of video frames as well as representation dependencies that arise from differential encoding. The problem of evaluating past and future data utility in the presence of packet loss is specifically addressed and two possible approaches are described. One relies on reliability semantics of the underlying transport-level protocol in order to optimize encoding relationships. The other folds the decision to retransmit packets known to be lost into the utility framework. We quantitatively demonstrate the ability of both approaches to increase performance of a prototype application and show that the second approach is generally superior.

With the rise of social networking as an on-line paradigm, we are also witnessing the birth of unstructured, unmanaged, "media collectives" in which hundreds, thousands, or even millions of users create, share, tag, link, and reuse media objects (i.e., pictures, audio, and video). The level of redundancy and repurposing within these collectives is reasonably high. For example, the same video clip (or portions of it) may appear in a variety of different contexts and formats. This redundancy and diversity serve to preserve popular media objects in a completely decentralized and unmanaged manner as a result of individuals optimizing for their own interests and goals. On the other end of the spectrum, however, is a long heavy-tail of media objects which may in fact be the only copy of its kind in the entire world. This paper addresses some of the systems issues which may be at play in dealing with such an extremely bimodal distribution and speculates that there may be opportunities to leverage these media collectives while preserving the autonomy of individual users in order to increase efficient use of networking and storage resources as well as providing richer capabilities to those who participate. The main idea behind our approach is to view adaptation through the lens of multiple description coding in order to expose coding structure and compactly specify how different versions of the same media object relate to each other.

We consider the problem of flow coordination in distributed multimedia applications. Most transport-level protocols are designed to operate independently and lack mechanisms for sharing information with other flows and coordinating data transport in various ways. This limitation becomes problematic in distributed applications that employ numerous flows between two computing clusters sharing the same intermediary forwarding path across the Internet. In this article, we propose an open architecture that supports the sharing of network state information, peer flow information, and application-specific information. Called simply the coordination protocol (CP) , the scheme facilitates coordination of network resource usage across flows belonging to the same application, as well as aiding other types of coordination. The effectiveness of our approach is illustrated in the context of multistreaming in 3D tele-immersion where consistency of network information across flows both greatly improves frame transport synchrony and minimizes buffering delay.

Using multiple cameras, a dynamic real world scene can be captured, represented, and streamed for realistic interaction in a 3D immersive system. One method to represent the dynamic environment is to use multiple depth streams - streams with color and depth information. However the captured data is too large to be streamed uncompressed. Fortunately, these data exhibit spatial and temporal coherence that can be used for compression.

The goal of tele-immersion has long been to enable people at remote locations to share a sense of presence. A tele-immersion system acquires the 3D representation of a collaborator's environment remotely and sends it over the network where it is rendered in the user's environment. Acquisition, reconstruction, transmission, and rendering all have to be done in real-time to create a sense of presence. With added commodity hardware resources, parallelism can increase the acquisition volume and reconstruction data quality while maintaining real-time performance. However, this is not as easy for rendering since all of the data need to be combined into a single display.In this article, we present an algorithm to compress data from such 3D environments in real-time to solve this imbalance. We present a compression algorithm which scales comparably to the acquisition and reconstruction, reduces network transmission bandwidth, and reduces the rendering requirement for real-time performance. This is achieved by exploiting the coherence in the 3D environment data and removing them in real-time. We have tested the algorithm using a static office data set as well as a dynamic scene, the results of which are presented in the article.

This article reprises the description of the Berkeley software-only MPEG-1 video decoder originally published in the proceedings of the 1st International ACM Conference on Multimedia in 1993. The software subsequently became widely used in a variety of research systems and commercial products. Its main impact was to provide a platform for experimenting with streaming compressed video and to expose the strengths and weaknesses of software-only video decoding using general purpose computing architectures. This article compares the original performance results with experiments run on a modern processor to demonstrate the gains of processing power in the past ten years relative to this specific application and discusses the history of MPEG-1 video software decoding and the Berkeley MPEG research group.

We consider the problem of applying aggregate congestion control to a class of distributed multimedia applications known as Cluster-to-Cluster (C-to-C) applications. Flows in such an application share a common intermediary path that is the primary source of network delay and packet loss. Using the Coordination Protocol (CP) architecture, we show how aggregate congestion control can be achieved with the following properties: Almost any rate-based, single-flow congestion control algorithm may be applied to make aggregate C-to-C traffic congestion responsive. C-to-C applications may use multiple flow bandwidth shares and still exhibit correct aggregate congestion responsiveness. C-to-C applications may implement complex, application-specific adaptation schemes in which the behavior of individual flows is decoupled from the behavior of the congestion responsive aggregate flow. Bandwidth filtered loss detection (BFLD) is presented as a technique for making single-flow loss detection algorithms work when aggregate traffic uses multiple flowshares. The approach is evaluated using both ns2 simulation and an experimental implementation in Free BSD and Linux. Results demonstrate its success for a wide range of network conditions.

Article Share on A general framework for multidimensional adaptation Authors: David Gotz University of North Carolina at Chapel Hill, Chapel Hill, NC University of North Carolina at Chapel Hill, Chapel Hill, NCView Profile , Ketan Mayer-Patel University of North Carolina at Chapel Hill, Chapel Hill, NC University of North Carolina at Chapel Hill, Chapel Hill, NCView Profile Authors Info & Claims MULTIMEDIA '04: Proceedings of the 12th annual ACM international conference on MultimediaOctober 2004 Pages 612–619https://doi.org/10.1145/1027527.1027671Online:10 October 2004Publication History 14citation423DownloadsMetricsTotal Citations14Total Downloads423Last 12 Months4Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

This paper looks at the problem of multi-streaming in 3D tele-immersion and describes how a protocol architecture called CP (for Coordination Protocol) can be used to coordinate video frame transport between application clusters. CP provides application endpoints with information about current network conditions, and an open architecture for implementing application-specific coordination schemes. The scheme described in this paper apportions available bandwidth among flows such that frame transport synchrony, important for 3D reconstruction performance, is significantly enhanced.

The goal of tele-immersion has long been to enable people at remote locations to share a sense of presence. A tele-immersion system acquires the 3D representation of a collaborator's environment remotely and sends it over the network where it is rendered in the user's environment. Acquisition, reconstruction, transmission, and rendering all have to be done in real-time to create a sense of presence. With added commodity hardware resources, parallelism can increase the acquisition volume and reconstruction data quality while maintaining real-time performance. However this is not as easy for rendering since all of the data need to be combined into a single display.In this paper we present an algorithm to compress data from such 3D environments in real-time to solve this imbalance. We expect the compression algorithm to scale comparably to the acquisition and reconstruction, reduce network transmission bandwidth, and reduce the rendering requirement for real-time performance. We have tested the algorithm using a synthetic office data set and have achieved a 5 to 1 compression for 22 depth streams.

No abstract available.

We present a new representation for image-based interactive walk-throughs. The target applications reconstruct a scene from novel viewpoints using samples from a spatial image dataset collected from a plane at eye-level. These datasets consist of pose augmented 2D images and often have a very large number of samples. Our representation exploits spatial coherence and rearranges the input samples as epipolar images. The base unit corresponds to a column of the original image that can be individually addressed and accessed. The overall representation, IRW, supports incremental updates, efficient encoding, scalable performance, and selective inclusion used by different reconstruction algorithms. We demonstrate the performance of our representation on a synthetic as well as a real-world environment.

We present a general analytical model for predicting the reconstructed frame rate of an MPEG stream. Our model captures the temporal relationships between I-, P, and B-frames but is independent of the channel and media characteristics. We derive an adaptive FEC scheme from the general model and verify it by comparing it to the results of a simulation. The prediction error of the model compared to the simulation for a wide array of parameter values is less than 5%. We then use the derived adaptive FEC scheme to study the optimal rate allocation (i.e., between generating a higher frame rate or increasing the protection for a lower frame rate) when equation-based TCP rate control is used to couple packet rates to channel characteristics such as round trip time and packet loss probabilities. Surprisingly, we find that optimal protection levels for I- and P-frames are relatively static as loss rates increase from 1% to 4% while changes in the frame type pattern are used to ameliorate the effects of the increased loss. The study demonstrates how our model can be used to reveal joint source/channel coding tradeoffs and how they relate to encoding and transmission parameters.

We introduce the notion of spatially encoded video and use it for efficiently representing image-based impostors for interactive walkthroughs. As part of a pre-process, we automatically decompose the model and compute the far-fields. The resulting texture images are organized along multiple dimensions and can be accessed in a user-steered order at interactive rates. Our encoding algorithm can compress the impostors size by two orders of magnitude. Furthermore, the storage cost for additional impostors or samples grows sub-linearly. The resulting system has been applied to a complex CAD environment composed of 13 million triangles. We are able to render it at interactive rates on a PC with little loss in image quality.

We have developed a parallel software-only processing system for creating real-time video effects such as titling and compositing (e.g., picture-in-picture) using compressed Internet video sources. The system organizes processors into a hierarchy of levels. Processes at each level of the hierarchy can exploit different types of parallelism and coordinate the actions of lower levels. To control the effect, control messages must be distributed to processors in the hierarchy while preserving the independence of each level. This requires a control mechanism that supports efficient delivery of messages to groups of processors, tunable reliability semantics, and recoverable state information. We describe a mechanism that meets these requirements that uses IP-Multicast, the Scalable Reliable Multicast protocol, and the Scalable Naming and Announcement Protocol. We also describe an optimization that provides a flexible framework for linking the control of different aspects of one or more related video effects.

hternet video is emerging as an important mtitimedia ap pHcation area.Video effects are important dernents in conventional video, but they are tiequently used in hternet ~ideo.Current hard~~based video effects proc~sing SG lutions ~e not w~matched for the ~+abti@ of kternet video (ie., bit rates, packet loss, jitter, etc.).A SO*+ ody solution, in contrast, provides the fl~ubtity to match this &[email protected] key to a sohe solution is to ~\Tloit pardeh~m.This paper presents the design of a pardel soti+ordy video Meets processing system and pretinary e~~erirnentd r~ts that mTlore the use of temporal pardl&~m.hternet packet video is emerging ~an important mtitim~ dia appEcation area The hltdticast Backbone (h~one) conferencing tool vic and NetJfeeting from hficrosoft are CSamplfis of bternet packet video conferencing took.Real Video tiom Re&Ne&orks.NetShow from hficrosoft, and IP/TV tiom Precept are axaples of vide~on-demand streaming hternet packet video applications.Th~e appEcations ~e used for a variety of purposw and audienc~s.using a vide~on-demand system, a single r~ cipient views stored material.Video conferencing is used by smti groups for meetings and by larger groups to attend re mote evmts (~g., classes, conference, concerts, etc.).Large broadcasts involve thousands of people %uned" into an ongokg program originating born eithw Eve sourc~or stored archives.For tfis paper, we group ~of these applications under the general term Internet Video (N).bternet video is &aractded by ~%abtity.Frame rates, bit ratw, and jitter are d variable in W applications.h contra~t, a tdevision system, whether adog or digitd, is &aracterized by constwt frame rates, bounded bit rates, and s.mchronous transmission.~appfimtions must *O ded n 036-S~9E!OOOS S5.

The design and implementation of a software decoder for MPEG video bitstreams is described. The software has been ported to numerous platforms including PC's, workstations, and mainframe computers. Performance comparisons are given for several different bitstreams and platforms including a unique metric devised to compare price/performance across different platforms (percentage of required bit rate per dollar). We also show that memory bandwidth is the primary limitation in performance of the decoder, not the computational complexity of the inverse discrete cosine transform as is commonly thought