Semantic segmentation results in pixel-wise perception accompanied with GPU computation and expensive memory, which makes trained models hard to apply to small devices in testing. Assuming the availability of hardware in training CNN backbones, this work converts them to a linear architecture enabling the inference on edge devices. Keeping the same accuracy as patch-mode testing, we segment images using a scanning line with the minimum memory. Exploring periods of pyramid network shifting on image, we perform such sequential semantic segmentation (SE3) with a circular memory to avoid redundant computation and preserve the same receptive field as patches for spatial dependency. In the experiments on large drone images and panoramas, we examine this approach in terms of accuracy, parameter memory, and testing speed. Benchmark evaluations demonstrate that, with only one-line computation in linear time, our designed SE3 network consumes a small fraction of memory to maintain an equivalent accuracy as the image segmentation in patches. Considering semantic segmentation for high-resolution images, particularly for data streamed from sensors, this method is significant to the real-time applications of CNN based networks on light-weighted edge devices.

Driving video is available from in-car camera for road detection and collision avoidance. However, consecutive video frames in a large volume have redundant scene coverage during vehicle motion, which hampers real-time perception in autonomous driving. This work utilizes compact road profiles (RP) and motion profiles (MP) to identify path regions and dynamic objects, which drastically reduces video data to a lower dimension and increases sensing rate. To avoid collision in a close range and navigate a vehicle in middle and far ranges, several RP/MPs are scanned continuously from different depths for vehicle path planning. We train deep network to implement semantic segmentation of RP in the spatial-temporal domain, in which we further propose a temporally shifting memory for online testing. It sequentially segments every incoming line without latency by referring to a temporal window. In streaming-mode, our method generates real-time output of road, roadsides, vehicles, pedestrians, etc. at discrete depths for path planning and speed control. We have experimented our method on naturalistic driving videos under various weather and illumination conditions. It reached the highest efficiency with the least amount of data.

In autonomous driving, state-of-the-art methods detect pedestrians from their appearances in 2-D spatial images. However, these approaches are typically time-consuming in terms of the algorithm complexity to cope with the large variations in shape, pose, action, and illumination. Capturing motion needs even more efforts. In a completely different approach, this work recognizes pedestrians along with their motion directions in a temporal way. By projecting a driving video to a 2-D temporal image called Motion Profile (MP), we can robustly distinguish pedestrian in motion against smooth background motion. To ensure non-redundant data processing of deep network on a compact motion profile further, a novel temporal-shift memory (TSM) model is developed to perform deep learning of sequential input in linear processing time. In experiments containing pedestrian motion from various sensors such as video and LiDAR, we demonstrate that, with the reduced data size around 3/720th of a video volume, this motion-based method can reach the detecting rate of pedestrians at 90% in near and mid-range on the road. With the super-fast speed and good accuracy, this method is promising for intelligent vehicles.

In vision-based autonomous driving, understanding spatial layout of road and traffic is required at each moment. This involves the detection of road, vehicle, pedestrian, etc. in images. In driving video, the spatial positions of various patterns are further tracked for their motion. This spatial-to-temporal approach inherently demands a large computational resource. In this work, however, we take a temporal-to-spatial approach to cope with fast moving vehicles in autonomous navigation. We sample one-pixel line at each frame in driving video, and the temporal congregation of lines from consecutive frames forms a road profile image. The temporal connection of lines also provides layout information of road and surrounding environment. This method reduces the processing data to a fraction of video in order to catch up vehicle moving speed. The key issue now is to know different regions in the road profile; the road profile is divided in real time to road, roadside, lane mark, vehicle, etc. as well as motion events such as stopping and turning of ego-vehicle. We show in this paper that the road profile can be learned through Semantic Segmentation. We use RGB-F images of the road profile to implement Semantic Segmentation to grasp both individual regions and their spatial relations on road effectively. We have tested our method on naturalistic driving video and the results are promising.

The objective of this paper is the instantaneous computation of time-to-collision (TTC) for potential collision only from the motion information captured with a vehicle borne camera. The contribution is the detection of dangerous events and degree directly from motion divergence in the driving video, which is also a clue used by human drivers. Both horizontal and vertical motion divergence are analyzed simultaneously in several collision sensitive zones. The video data are condensed to the motion profiles both horizontally and vertically in the lower half of the video to show motion trajectories directly as edge traces. Stable motion traces of linear feature components are obtained through filtering in the motion profiles. As a result, this avoids object recognition and sophisticated depth sensing in prior. The fine velocity computation yields reasonable TTC accuracy so that a video camera can achieve collision avoidance alone from the size changes of visual patterns. We have tested the algorithm for various roads, environments, and traffic, and shown results by visualization in the motion profiles for overall evaluation.

Weather and illumination are critical factors in vision tasks such as road detection, vehicle recognition, and active lighting for autonomous vehicles and ADAS. Understanding the weather and illumination type in a vehicle driving view can guide visual sensing, control vehicle headlight and speed, etc. This paper uses sparse coding technique to identify weather types in driving video, given a set of bases from video samples covering a full spectrum of weather and illumination conditions. We sample traffic and architecture insensitive regions in each video frame for features and obtain clusters of weather and illuminations via unsupervised learning. Then, a set of keys are selected carefully according to the visual appearance of road and sky. For video input, sparse coding of each frame is calculated for representing the vehicle view robustly under a specific illumination. The linear combination of the basis from keys results in weather types for road recognition, active lighting, intelligent vehicle control, etc.

The objective of this work is the instantaneous computation of Time-to-Collision (TTC) for potential collision only from motion information captured with a vehicle borne camera. The contribution is the detection of dangerous events and degree directly from motion divergence in the driving video, which is also a clue used by human drivers, without applying vehicle recognition and depth measuring in prior. Both horizontal and vertical motion divergence are analyzed simultaneously in several collision sensitive zones. Stable motion traces of linear feature components are obtained through filtering in the motion profiles. As a result, this avoids object recognition, and sophisticated depth sensing. The fine velocity computation yields reasonable TTC accuracy so that the video camera can achieve collision avoidance alone from size changes of visual patterns.

Pre-collision systems (PCS) are designed to warn a driver or brake automatically to help avoid or mitigate accidents, protecting vulnerable road users such as pedestrians and bicyclists. A PCS system needs to understand both normal and abnormal behavior of the vulnerable users in order to have high true-positive and low false-positive detection of PCS events. In this paper, the normal bicyclist behavior is analyzed based on naturalistic driving data. These data can inform the PCS design and lead to proper testing and evaluation of PCS systems. This paper focuses on the following attributes of normal bicyclist behavior: time to collision, lateral position, and bicycle speed. The behavior was analyzed using the Transportation Active Safety Institute naturalistic driving data from 110 cars. This paper presents new analysis gleaned from these data, plotted in 2-D graphical form, showing heat maps of common bicycle positions and trajectories of the dynamic vehicle-bicycle interactions.

the objective of this work is to compute the Time-to-Collision (TTC) of surrounding vehicles of a vehicle using motion information in driving video. The key advantage in this work is the extraction of potential danger without vehicle detection and recognition in prior, but directly from the motion divergence in the video. We analyze the trace expansion both horizontally and vertically condensed in the collision sensitive zones in the driving video. Long term motion is stably obtained through filtering in the spatial-temporal video profiles at collision sensitive parts in the video. This overcomes the accuracy problem in object recognition and saved the computation cost tremendously in the real time sensing. The fine velocity computation yields reasonable TTC accuracy so that the video camera can achieve collision avoidance alone from size changes of visual patterns.

Pedestrian detection is a challenging problem studied over decades. Most algorithms are based on human appearance. Only few works consider motion as a feature component. In this paper, however, we tackle this problem only considering short periods of pedestrian walking. This motion does not depend on the variations of pedestrian pose, body shape, illumination, and background. We model pedestrian motion that has unique properties compare to background and rigid objects motion in spatial-temporal motion profiles. This observation helps us to identify pedestrian leg motion along with body motion over a short time period. Our method also works for a vehicle borne camera where background also moves. We achieved more robust results by dealing with crowds, and other degenerating cases of human motion against background and dynamic scenes. The method has a low computational cost on a motion profile and it can be combined with a shape-based method as pre-screening for reducing the false positives. It also provides a feasible way to find human behaviors.

Pedestrian detection has been intensively studied based on appearances for driving safety. Only a few works have explored between-frame optical flow as one of features for human classification. In this paper, however, a new point of view is taken to watch a longer period for non-smooth movement. We explore the pedestrian detection purely based on motion, which is common and intrinsic for all pedestrians regardless of their shape, color, background, etc. We found unique motion characteristics of humans different from rigid objects in motion profiles. Based on the explicit analysis of spatial-temporal behaviors of pedestrians, non-smooth motion points are detected at the motion trajectories of limbs and body. This method works for driving video where both pedestrians and background are moving, and it yields good results as it is less influenced from pedestrian variations in shape and environment. The method also has low computational cost and it can be combined with a shape-based method as pre-screening tool for accuracy and speed.

Surveillance videos are recorded pervasively and their retrieval currently still relies on human operators. As an intermediate representation, this work develops a new temporal profile of video to convey accurate temporal information in the video while keeping certain spatial characteristics of targets of interest for recognition. The profile is obtained at critical positions where major target flow appears. We set a sampling line crossing the motion direction to profile passing targets in the temporal domain. In order to add spatial information to the temporal profile to certain extent, we integrate multiple profiles from a set of lines with blending method to reflect the target motion direction and position in the temporal profile. Different from mosaicing/montage methods for video synopsis in spatial domain, our temporal profile has no limit on the time length, and the created profile significantly reduces the data size for brief indexing and fast search of video.

This paper visualizes surveillance video contents effectively in a 2D temporal image for indexing and retrieving a large video database. As an intermediate representation, this work extracts a temporal profile from video to convey accurate temporal information while keeps certain spatial characteristics for recognition. Different from spatial video indexing such as video synopsis that montages video frames for video summarization, our temporal indexing is achieved by designing a scheme to observe video from a side of video volume. A series of sampling lines are located over the field of view along the principal directions of targets to probe target motion, and the position varied temporal slices are obtained from the video volume. These slices are blended into the temporal profile with transparencies related to their spatial locations in the video. Static background and targets with less motion are further embedded into the temporal profile. Our goal is to provide less deformed shapes, more spatial information, and precise event occurrence in the temporal profile for video index and browsing.

Nowadays, many vehicles are equipped with a vehicle borne camera system for monitoring drivers' behavior, accident investigation, road environment assessment, and vehicle safety design. Huge amount of video data is being recorded daily. Analyzing and interpreting these data in an efficient way has become a non-trivial task. As an index of video for quick browsing, this work maps the video into a temporal image of reduced dimension with as much intrinsic information as possible observed on the road. The perspective projection video is converted to a top-view temporal profile that has precise time, motion, and event information during the vehicle driving. Then, we attempt to interpret dynamic events and environment around the vehicle in such a continuous and compact temporal profile. The reduced dimension of the temporal profile allows us to browse the video intuitively and efficiently.

The high demand from railway safety has promoted the sensing technology to avoid manual inspection that is error-prone, low efficiency, labor-intensive and high costs. However, few works have emphasized railway environmental monitoring so far which is equally important in railway safety maintenance. In this paper, we propose a railway scene rendering system for virtual maneuver and inspection. Based on the route panoramas extracted from a real train-borne video, we generate the compact and high fidelity scenes over a long distance route in cyberspace. We analyze the relation between train motion and the video sampling in order to obtain the route panoramas of the railways at the just-sampling rate. Then, we render scene tunnels based on the route panorama in a virtual space for interactive navigation, inspection, and scene indexing. The experimental results show remarkable advantages in the data storage and automatic inspection. The compact data format with sufficient visual information can be transmitted and released in cyberspace for fast access, maneuver, and automatic retrieval in railway environment monitoring.

With the rapid development of high-speed railway, the safety running of trains becomes extremely urgent. Video is a direct and effective manner for railway environment monitoring. In this paper, we propose a route panoramic representation to produce a virtual environment of the railway scene from a train-borne video, which provides an all-round display for the entire length of railway. We generate the route panorama from the forward motion video and then render 3D scene. The experimental results show many advantages of the representation in the large data storage, browsing, and examination. It will be used for railway safety checking, railway facility inspection and virtual sightseeing from the train in the future.

This paper presents a method with which 3D images of tire track and footprint impressions at crime scenes can be captured with high fidelity, while capturing high resolution 2D color texture images simultaneously. The resulting device is portable, easy to use, is non-destructive of the evidence, and saves time at crime scenes. The same technique can also be used in the laboratory to create 3D depth images of suspect tires or shoe soles. Computer-based pattern matching technology can be used to assist in matching and comparison tasks. The device produces better quality data at a close range obtained in a larger field (or span in the case of tire impressions) compared to existing devices. It avoids problems related to occlusions by using two lasers and can digitize long spans of impressions in one scan. The method includes a calibration method which is integrated into the scanning process on site, thus avoiding problems with pre-calibrated configurations becoming stale during transportation and setup.

Video indexing is important to video browsing, editing, retrieval, and summarization. This work takes a new aspect view to profile a video volume to a visual track and create a digest of video for preview. Our projected profile of video contains spatial and temporal information inclusively in a 2D image scroll that is continuous, scalable, and indexing to frames. We analyze the camera kinematics including zoom, translation and rotation, and categorize camera works for profiling various types of video. The key idea is to use a sampling line to sweep the video volume across the major optical flow so as to obtain an intrinsic scene space that is less influenced by the camera motion. We also use motion blur technique to render dynamic targets in the profile. The resulting video track can provide a video preview for guiding the access to the frames. It will facilitate video surveillance, visual archiving of environment, video retrieval, and video editing.

This work takes a new aspect view to profile a video volume to a video track as a digest for video preview. Our projected video profile contains both spatial and temporal information inclusively in a 2D image scroll that is continuous, compact, scalable, and indexing to each frame. To profile various types of video clips, we investigate the global flow field under all camera actions, and propose a uniformed scheme that uses a sampling line to cut the video volume across the major optical flow field. The resulting profile obtains an intrinsic scene space less influenced by the camera actions, and can be displayed in a video track to guide the access to video frames, and facilitate video browsing, editing, and retrieval.

This paper aims at real-time in-car video analysis to detect and track vehicles ahead for safety, autodriving, and target tracing. This paper describes a comprehensive approach to localizing target vehicles in video under various environmental conditions. The extracted geometry features from the video are continuously projected onto a 1-D profile and are constantly tracked. We rely on temporal information of features and their motion behaviors for vehicle identification, which compensates for the complexity in recognizing vehicle shapes, colors, and types. We probabilistically model the motion in the field of view according to the scene characteristic and the vehicle motion model. The hidden Markov model (HMM) is used to separate target vehicles from the background and track them probabilistically. We have investigated videos of day and night on different types of roads, showing that our approach is robust and effective in dealing with changes in environment and illumination and that real-time processing becomes possible for vehicle-borne cameras.

This work obtains shaking-free route panoramas from a vehicle borne camera. We detect the shaking and waving profile of the camera and rectify a long route panorama automatically for an ideal parallel-perspective projection. Different from the traditional flow based approach that works on entire frame by point tracking, we create a stationary blurred temporal image called condensed slice during the scene scanning. This image yields long and visible traces from distinct and distant features and horizontal features as the evidences of camera motion, and it diminishes other features irrelevant to the motion as well. We track traces in the condensed slice and rectify the route panorama at the sub-pixel level without shape analysis in video frames. The significance of the approach is in its effectiveness visually confirmable in the results, global rectification as the vehicle moves forward, and efficiency for real-time processing using the minimum data set.

This paper introduces an interface platform to visualize location-specific media on lightweight displays that are placed and moved over a two dimensional map. The position of each display in the 2D space is dynamically sensed. The media data are transmitted to the detected display accordingly via a newly invented communication sheet underneath the map. Users can move the display on the sheet to magnify and explore interesting spots or items, while the contents received are image and video at different levels of detail. The platform is multi-user oriented, power efficient, and capable of performing multi-channel wide bandwidth transmission. As a physical medium, the communication sheet provides a large 2D space for interaction that a flat screen and touch panel cannot achieve. We have done experiments on the designed platform for verifying its effectiveness in education and exposition. These experiments have demonstrated its promising features in multimedia presentation.

This work aims at detecting and tracking vehicles in in-car video. Rather than enhancing shape analysis of various vehicle types and road situations, this work focuses on vehicle and background motions because they are more general than shapes and colors of cars in various road environments. Basic features are tracked stably using corners, intensity peaks, and horizontal line segments. We use the HMM in the temporal domain to separate background and moving vehicles in the video. To realize this, we model the image motion of vehicles and background probabilistically according to the scene characteristic and vehicle driving mechanism, as well as the joint distribution of horizontal position and velocity of scenes. The identification and tracking are robust to various illumination and environments and the processing is performed in real time. The identification results based on motion only is good and a better result can be achieved further by fusing the motion result with the results from shape analysis.

Omni-directional video records complete visual information along a route. Though replaying an omni-video presents reality, it requires significant amount of memory and communication bandwidth. This work extracts distinct views from an omni-video to form a visual digest named route sheet for navigation. We sort scenes at the motion and visibility level and investigate the similarity/redundancy of scenes in the context of a route. We use source data from 3D elevation map or omni-videos for the view selection. By condensing the flow in the video, our algorithm can generate distinct omni-view sequences with visual information as rich as the omni-video for further scene indexing and navigation with GIS data.

The use of video cameras in police cars has been found to have significant value and the number of such installed systems has been increasing. In addition to recording the events in routine traffic stops for later use in legal settings, in-car video cameras can be used to analyze in real-time or near real-time to detect critical events and notify police headquarters for help. This paper presents methods for detecting critical events in such police car videos. The specific critical events are person running out of a stopped car and officer falling down while approaching a stopped car. In the above situations, the aim is to alert the control center immediately for backup forces, especially in the last example when the officer is incapacitated. In order to implement real-time video processing so that a quick response can be generated without employing complex, slow, and brittle video processing algorithms, we use the reduced spatiotemporal representation (1D projection profile) and Hidden Markov Model to detect these events. The methods are tested on many video shots under various environmental and illumination conditions.

This paper will demonstrate a media infrastructure that can distribute multimedia signals and power via a two dimensional communication sheet. Small and low power devices placed on top of the sheet will be able to receive audio and video signals transmitted from a computer. The 2D sheet is printed in three layers and the electromagnetic waves are distributed in the middle layer. Through a grid of slits printed on the top layer, the electromagnetic waves leak and are thus picked up by small multimedia devices with antennas. Compared to the wired and wireless communication, this 2D sheet has its unique properties such as free to put and move devices on it, power efficient, wide transmission bandwidth, secure communication, etc.

On a moving vehicle, speedy motion extraction from video is demanded. Different from the traditional motion estimation methods that track or match 2D features in consecutive motion-blurred images, this work explores a novel approach to find motion without explicit shape analysis. We take spatial integration of intensities in each frame and the obtained consecutive profiles provide distinct motion traces in a visual form. The camera motion can be followed in the integrated images briefly, rather than from complex feature extraction and iterative computation. The resulting motion parameters are used for normalizing length and removing shaking of route panoramas. The results are robust and the method is particularly efficient for real time processing.

Route panorama (RP) has become an image medium to archive and visualize cityscapes along streets. Collecting data from a sampling line of a video camera mounted on a moving vehicle, the route panorama may suffer from the vehicle shaking and varying speeds due to uneven roads and busy traffic respectively. These may jitter the route panorama and change object widths. To produce a high quality route panorama, we extract spatial-temporal information in the video volume and rectify the route panorama. In this paper, we propose two condensed image slices to record traces of horizontal and vertical scenes during the vehicle motion. Instead of matching video frames that requires cost computation and large storage, we track the feature traces in the condensed image slices to remove jitter and adjust the local length of route panoramas. The method is effective and efficient in acquiring and rectifying long distance route panoramas.

A linear CCD sensor reads temporal data from a CCD array continuously and forms a 2D image profile. Compared to most of the sensors in the current sensor networks that output temporal signals, it delivers more information such as color, shape, and event of a flowing scene. On the other hand, it abstracts passing objects in the profile without heavy computation and transmits much less data than a video. This paper revisits the capabilities of the sensors in data processing, compression, and streaming in the framework of wireless sensor network. We focus on several unsolved issues such as sensor setting, shape analysis, robust object extraction, and real time background adapting to ensure long-term sensing and visual data collection via networks. All the developed algorithms are executed in constant complexity for reducing the sensor and network burden. A sustainable visual sensor network can thus be established in a large area to monitor passing objects and people for surveillance, traffic assessment, invasion alarming, etc.

This paper proposes a visual representation named scene tunnel for capturing urban scenes along routes and visualizing them on the Internet. We scan scenes with multiple cameras or a fish-eye camera on a moving vehicle, which generates a real scene archive along streets that is more complete than previously proposed route panoramas. Using a translating spherical eye, properly set planes of scanning, and unique parallel-central projection, we explore the image acquisition of the scene tunnel from camera selection and alignment, slit calculation, scene scanning, to image integration. The scene tunnels cover high buildings, ground, and various viewing directions and have uniformed resolutions along the street. The sequentially organized scene tunnel benefits texture mapping onto the urban models. We analyze the shape characteristics in the scene tunnels for designing visualization algorithms. After combining this with a global panorama and forward image caps, the capped scene tunnels can provide continuous views directly for virtual or real navigation in a city. We render scene tunnel dynamically by view warping, fast transmission, and flexible interaction. The compact and continuous scene tunnel facilitates model construction, data streaming, and seamless route traversing on the Internet and mobile devices.

This work achieves an efficient acquisition of scenes and their depths along streets. During the movement of a vehicle, a slit in the camera frame is set properly to sample scenes continuously for a route panorama. This paper proposes a novel method of depth estimation by analyzing a new phenomenon named stationary blur in the route panorama. We find its relation with the depth and evaluate its degree at local and global levels. The depth estimation through filtering avoids feature matching and tracking that are error-prone in the scanning of real and complex street scenes. Our method provides reliable results but requires much less data than that of the structure from motion. This keeps the elegance of the route panorama in data representation, and is suitable for real time sensor development. Utilizing the completeness of the route panorama in the scene archiving, we can generate planar models of streets, which can be used in city visualization.

This work achieves full registration of scenes in a large area and creates visual indexes for visualization in a digital city. We explore effective mapping, indexing, and display of scenes so that an area becomes visible. Users can virtual navigate city on the Internet and achieve real guidance with a PDA. Extended images such as route panoramas, scene tunnels, panoramic views and spherical views are acquired in an urban area and associated with geospatial locations. A 3D LIDAR elevation map is used to generate a scanning plan based on visibility, image properties, and importance of scenes. Scanning scenes along streets and at spots of interest allows for compact and complete visual data collection. To access city information, visual indexes from scenes to spaces are created pervasively for flexible space exploration and transition. To visualize a space seamlessly in a large view frame and synchronize scenes with the virtual movement in the map, we stream image data on the Internet. An engine is developed for continuous space traversing, accessing spatial information, and transiting between spaces through visual links. A real urban area has been modeled to verify the effectiveness of such a system.

Abstract This work establishes a cyberspace of a real urban area for visiting on the Internet. By registering entire scenes along every street and at many locations, viewers can visually travel around and find their destinations in cyberspace. The issues we discuss here are mapping of a large‐scale area to image domains in a small amount of data, and effective display of the captured scenes for various applications. Route Panoramas captured along streets and panoramic views captured at widely opening sites are associated to a city map to provide navigation functions. This paper focuses on the properties of our extended images—route panorama, addressing the archiving process applied to an urban area, an environment developed to transmit image data as streaming media, and display for scene traversing on the WWW in real time. The created cyberspaces of urban areas have broad applications such as city tour, real estate searching, e‐commerce, heritage preservation, urban planning and construction, and vehicle navigation. Copyright © 2005 John Wiley & Sons, Ltd.

This work creates a complete view index of an area for seamless visual guidance in the cyber world. The goal is to generate continuous route views that direct people to their destination. In addition to a route map, continuous scenes will be displayed to effectively guide virtual and real trips. We employ compact and seamless image media such as route panoramas and panoramic views in the scene registration. A uniformed framework of space indexing is developed to visualize information in various spaces. View-to-space indexes allow in-depth transitions from a global space to local spaces, and transitions between adjacent spaces. The information of the same format can be displayed on different platforms such as WWW and wireless PDA. An urban area has been modeled to implement the proposed scheme of view indexing.

This paper introduces an effort to build a cyber town from geographically separated routes. It brings streets distributed in different cities and countries together for people to traverse the space, learn history, understand living style, etc. The town is constructed by using our developed technology - route panorama and data acquired at several places. We design a flexible and comprehensive indexing scheme for connecting routes at crossings, and linking Web contents from routes. User can wander around from one route to another, and enter buildings to visit museums and sites with specific topics. This design gives a unique style for virtual exhibition of real scenes and provides cross-culture environment for visit on the Internet.

This paper proposes a visual representation named scene tunnel to archive and visualize urban scenes for Internet based virtual tour. We scan cityscapes using multiple cameras on a vehicle that moves along streets, and generate scene archive more complete than a route panorama. The scene tunnel can cover high architectures and various object aspects. It contains much less data than video, which is suitable for image transmission and rendering over the Internet. It has a uniformed resolution along the camera path and provides continuous views for virtual traversing of a real city. We have developed image acquisition methods from slit setting, view scanning, to image integration. We have also achieved city visualization with scene tunnels on the Internet by transforming view, streaming data, and providing interactions.

This paper proposes a visual representation named scene tunnel to capture and visualize urban scenes for Internet based virtual city traversing. We scan cityscapes by using multiple cameras on a vehicle that moves along a street, and generate a real scene archive more complete than a route panorama. The scene tunnel can cover high architectures and various object aspects, and its data size is much less than video. It is suitable for image transmission and rendering over the Internet. The scene tunnel has a uniformed resolution along the camera path and can provide continuous views for visual navigation in a virtual or real city. This paper explores the image acquisition methods from slit calibration, view scanning, to image integration. A plane of scanning is determined for flexible camera setting and image integration. The paper further addresses the city visualization on the Internet that includes view transformation, data streaming, and interactive functions. The high-resolution scenes are mapped onto a wide window dynamically. The compact and continuous scene tunnel facilitates the data streaming and allows virtual traversing to be extended to a large area.

This work establishes a cyber space of an urban area for visiting on the Internet. By registering entire scenes along every street and scenes at many locations, people can visually travel from street to street, and reach their destinations in cyber city. The issues we discuss here are how to map a large scale area to image domain in a small amount of data, how to cover visual information as complete as possible in the mapping, and how to effectively display the captured scenes for various activities in cyber city. Route Panoramas captured along streets and Panoramic Views captured at widely open sites are associated to city maps to provide a navigation function. This paper focuses on the properties of our extended images - route panorama, the archiving process applied to an urban area, and a real environment we developed to transmit and display scenes on the WWW and portable devices. The created cyber spaces of real cities have broad applications such as city tour, real estate, E-commerce, heritage preservation, urban planning and construction.

Route panorama is a new image format extracted from a video sequence taken by a camera mounted on a moving vehicle. We select a virtual slit in the image frame and continuously capture the slit view on the frame. The consecutive slit views are connected to form a 2D spatial image. It is a continuous, compact and complete visual representation of scenes along a route, which will play an important role in digital city, virtual tour, and navigation application. Route panorama has unique properties as compared with normal perspective images. In our experiment, we found that distant objects from the camera path are blurred horizontally in the route panorama. In order to understand resolution of route panorama images, we study theoretical definition and real physical formation of route panoramas in this paper. We have examined that scenes at different depths are sampled differently by the route panorama. This work investigates various factors that determine the spatial resolution of route panoramas. The results are particularly important to the capturing of route panoramas and depth estimation based on blurring.

This paper presents a new medium called route panorama (RP) for visualizing a large-scale environment such as a town or a city. An RP is captured by a slit camera mounted on a vehicle. It is a continuous, compact and complete visual representation of scenes along a route. It can be transmitted on the Internet in real time as a streaming media and displayed in various styles for virtual city traversing. The application of RPs includes virtual tour, navigation, heritage archiving, urban planning, city indexing, etc.

Human motion modeling for animation and VR has reached a level of capturing real motion data from people using various visual and non-visual sensors. However, most of the available systems require special devices and environments, or even attachments of extra things on an actor. To realize a personal system, we developed a general type software to acquire arbitrary motion from a video sequence. A 3D articulated human model with changeable size, color and surface shape is constructed and personalized to fit with a focused figure in the video. The personal model is then driven either automatically or manually to match with the moving body in consecutive image frames. This matching starts from key frames that contain key poses. A smooth motion is then interpolated in between key frames. The evaluation of the generated motion is enhanced by image correlation. We provide various methods to make the matching feasible in order to reduce the modeling time. This approach is suitable for personal use to meet wide needs of human motion acquisition.

This paper introduces a smart laser range sensor with many advantages over the current products. The device is portable for its lightness and simple processing components. The image data is recorded in a digital video tape for processing, similar to picture shooting by a camera. The measuring scope can be freely adjusted to cover various distances, which greatly saves the manpower in measuring stationary objects or a site. The system employs a post-calibration so that it can be used easily on site. Furthermore, we obtain the color of measured points, which has been realized so far only by complicated systems in constrained environments. All these functions are realized by spatial temporal analysis of the video images. This sensor system has been used in measuring unearthed relics at a world heritage site.

We recover 3D models of objects with specular surfaces. An object is rotated and its continuous images are taken. Circular-shaped light sources that generate conic rays are used to illuminate the rotating object in such a way that highlighted stripes can be observed on most of the specular surfaces. Surface shapes can be computed from the motions of highlights in the continuous images; either specular motion stereo or single specular trace mode can be used. When the lights are properly set, each point on the object can be highlighted during the rotation. The shape for each rotation plane is measured independently using its corresponding epipolar plane image. A 3D shape model is subsequently reconstructed by combining shapes at different rotation planes. Computing a shape is simple and requires only the motion of highlight on each rotation plane. The novelty of this paper is the complete modeling of a general type of specular objects that has not been accomplished before.

This work recovers 3D graphics models of objects with specular surfaces. An object is rotated and continuous images of it are taken. Circular lights that generate cones of rays are used to illuminate the rotating object. When the lights are properly set each point on the object can be highlighted during the rotation. The shape for each rotational plane is measured independently using its corresponding epipolar plane image. A 3D graphics model is subsequently reconstructed by combining shapes at different rotation planes. Computing a shape is simple and requires only the motion of the highlight on each rotation plane. Results not obtained before are given in the 3D shape recovery experiments on real objects.

Introduces a model based approach in acquiring motion of a human body from a video image sequence. A model can not only help recognition of a human in the images, but also save the cost in 3D sensing. A 3D human model which has the function of articulate motion and changeable size, color and surface shape is provided. The model can be driven either automatically or manually through a graphics interface. We first fit the model to some particular frames so as to yield a personal model. The matching of the personal model with the image sequence for extracting human motion starts from matching key frames. The motion between key frames is interpolated. We are building a system for generating motion for multimedia, virtual world, and animation.

Archiving of archaeological excavation sites with rich data, pictures and graphs has been required for long time. This paper introduces an effort to register 3D graphics data of an excavation site at the world famous Terra-Cotta Warriors and Horse Museum. The measured data including shapes and colors are processed for archaeology research, unearthed finds preservation, multimedia textbook and virtual museum. We developed a laser range finder to imaging local distinguished parts in the excavation site. By using image transformation and virtual reality interfaces, we combine local measured data to produce a global digital 3D graphics scene.

This work reconstructs a 3D graphics model of an object with specular surfaces by its rotation. Continuous images are taken to measure highlights on the smooth surfaces and their motion. Coplanar extended lights determine a plane of rays to produce a highlight stripe on the object. 3D surfaces are then recovered from the moving stripe. We investigate global motion characteristics of the highlights in the epipolar-plane images so as to qualitatively identify surface types and control the modeling process. Under single and multiple plane-of-rays illuminations, we give two quantitative approaches for surface and normal recovery which use highlight orientation in the image and highlight motion in the epipolar-plane images. The computations employ a first-order differential equation and linear equations, respectively.

This work reconstructs 3D graphics model of an object from an image sequence taken during the object rotation. Shading and its motion information are used in estimating shapes of smooth surfaces between fixed edges from corners and patterns. Being different from conventional shape from shadings that use one or several images, we use an image sequence that provides not only shading but also its motion. Maximum diffused shading moving on surfaces is followed at each rotation plane and surface normals are obtained. Positions of points are estimated from moving shading using motion stereo principle. It is linear equations directly achieving the results rather than iterative computation in shape from shading using complex approximation.

This paper generalizes a shape recovery method for specular surfaces. We have use linear lights that generate planes of ray to highlight a rotating object so that its specular surfaces are computed from continuous images. By setting lights properly, shape at each rotation plane can be measured and a 3D graphics model can be subsequently reconstructed. In this paper, we extend the lights used to circular ones to produce cones of ray. Under such illuminations, all surface points with different normals are highlighted in a rotation period. The computation of shape is simpler than compared with the linear light case. We give results of shape recovery on real objects.

This work explores hand-eye coordination of robots by using a camera with functions of zooming and focusing. Under the guidance of the camera, a robot hand can arrive at an object to manipulate it in the 3D space. The hand movement is decomposed into the move parallel to the image plane and the move in depth, which are controlled from image position and focus value of the hand. The focal value is put on the destination object and the hand move in depth tries to increase the image sharpness of the hand. The zooming, focusing, and 2D image position control cooperate with each other and work in feedback fashion. The fovea control of eye realized by zooming from wide-angle to tele-angle brings both flexibility and accuracy in locating hand position.

This work investigates visual characteristics of specular surfaces during rotation, and gives approaches to qualitatively identify and quantitatively recover shapes of these surfaces. Continuous images are taken when an object rotates. We look at specularly reflected patterns on the surfaces and their motion in the EPIs parallel to the rotation plane, from which estimation of each surface point and construction of the object model are carried out. We find very simple and direct methods to fulfill this objective; linear equations for multiple lights illumination, and a 1st-order differential equation for single light illumination. The motion of specular reflection has nice global characteristics in EPI. The surface types range from very shiny metal surfaces to surfaces with only week specular reflectance. We give both simulation and experiments on real objects.>

This paper proposes an approach to acquire 3D models of objects with specular reflectance for graphics use. Highlight and rotation informations are employed in the model recovery. The authors control object rotation and extract motion of a highlight stripe, from which the object shape can be qualitatively inferred and quantitatively reconstructed by solving a first-order linear differential equation. The authors have experimented on simulated and real objects to obtain their models.

This paper explores shape from contour for acquiring 3-D graphics models. In this method, a continuous sequence of images is taken as an object rotates. A smooth convex shape can be estimated instantaneously from its contour and by the first derivative of contour movement (trace of contour, or contour distribution with time). We also analyze shapes that do not satisfy the conditions of smoothness and visibility, which are indispensable for modeling an object. A region that does not expose as contour yields a nonsmoothness in the tracked contour movement. We can thus detect such a region by contour distribution filtering and extract its accurate location by computing the left and right derivatives of the distribution. This has not been studied previously. These unknown regions are obtained for further investigation using other visual cues. A general approach for building a geometrical object model using contours is then described. The entire process from silhouettes to a 3-D model is based local computation; this is promising for producing shapes in real time. Our direct goal is to establish 3-D graphics models of human faces for the growing needs of visual communications. We have obtained some good results.>

The authors attempt to combine results from different vision cues such as contour, motion and shading in an image sequence to obtain as complete a 3-D model as possible. Two problems must be solved. First, different features may appear to be the same image data. They should be classified so that different methods can be applied precisely. Second, output from different approaches may supplement, or overlap, each other. They should be qualified and then combined correctly and effectively. These issues are addressed by analyzing an image sequence taken when objects rotate around an axis. The features are verified according to their behaviour in the continuous images. Unknown areas and areas with single or multiple measurements are located. A strategy for combining these data is given.>

This paper explores a shape-from-contour method for acquiring a 3D model by using a continuous sequence of images taken as an object rotates. The authors analyze the areas that are unexposed to contours and detect them for further investigation. They describe a general approach to use the shape from contours. The goal is to establish a 3D model of the human face for the growing needs of visual communications. Some good results have been obtained.>

The authors introduce an approach to building a qualitative description of scenes along a route, which is used in route recognition by a mobile robot. The description consists of a series of landmarks autonomously selected by the robot from a generalized panoramic view generated as a visual memory of scenes along routes. The basic idea to bridge the quantitative panoramic view to qualitative landmarks is to examine the distinctiveness of patterns in the image and select landmarks from unique patterns that are remarkable by which to navigate.>

A scheme that uses an active camera to guide object manipulation in an environment that has not been modeled is introduced. By actively moving camera to desired position with proper distance and viewing angle from objects of interest, it is possible to acquire good data for robot visual feedback control when locating objects. The environment is described in an object-centered coordinates system and is measured relatively in consecutive 2-D image spaces. This approach is flexible and efficient in manipulation, since it avoids complicated 3-D modeling and image processing is driven by tasks. It is shown that robot operation and camera motion are guided by visual feedback. The strategies to control this active guidance are studied.>

This paper introduces an approach to build a qualitative description of scenes along a route, which is used in route recognition by a mobile robot. The description consists of a series of landmarks autonomously selected by the robot from a panoramic view, which has been generated as a visual memory of scenes along routes. The basic idea to bridge the quantitative panoramic view to qualitative landmarks is to examine the 'distinctiveness' of patterns in the image and select landmarks from unique patterns that are remarkably by which to navigate.>

A dynamically generated panoramic representation for route recognition by a mobile robot is presented. The strategies employed are route description from experience and route recognition by visual information. In the description phase, panoramic representation, a representation of scenes along a route, is proposed. It is obtained by scanning scenes sideways along the route, which provides rich information, such as a 2-D projection of scenes called panoramic view, a path-oriented 2-1/2-D sketch, and a path description. The continuous panoramic view is more efficient in processing than integrating discrete views into a complete route model. In the recognition phase, the robot matches the panoramic representation from incoming images with that memorized in the previous scan so that it can locate and orient itself in autonomous navigation. Since the panoramic view covers a wide field of view, one can achieve reliable matching using a coarse-to-fine method, starting from a very coarse level.>

The authors explore a new intrinsic property of dynamic vision, shape from motion, and they argue its superiority over motion from shape change for a class of motion. Based on the concept of anorthoscopic perception, the authors show how a two-dimensional dynamic projection, which contains the major portion of spatial and temporal information of dynamic 3-D scenes, is obtained from consecutive 1-D images through a slit, when the relative motion between the viewer and scenes is known. The slit should be nonparallel to the optical flow in an image. Analyzing the projection reduces the redundant processing of shape. It can facilitate dynamic object recognition or 3-D structure acquisition for stable motion of objects or camera.>

Stereo images taken by a mobile robot are analyzed and integrated into a world model. The knowledge on the properties of the environment, such as the flatness of floor and the richness in vertical surfaces, is arranged into constraints on matching and interpreting images. First, the stereo matcher detects edge points on the floor by predicting disparities from a camera model and testing them in actual images. Next, the correspondences of vertical edges starting at the floor edge points are established. By iterating similar procedures, vertical edges starting from edge points at different levels in scene are also detected. The hypothesis that a region between two vertical edges in scene is a vertical planar surface is proposed and tested by examining whether the disparities predicted from the 3-D geometry of the hypothesized surface contradict with those of patterns appeared in the region.