In the evolving field of machine learning, ensuring group fairness has become a critical concern, prompting the development of algorithms designed to mitigate bias in decision-making processes. Group fairness refers to the principle that a model's decisions should be equitable across different groups defined by sensitive attributes such as gender or race, ensuring that individuals from privileged groups and unprivileged groups are treated fairly and receive similar outcomes. However, achieving fairness in the presence of group-specific concept drift remains an unexplored frontier, and our research represents pioneering efforts in this regard. Group-specific concept drift refers to situations where one group experiences concept drift over time, while another does not, leading to a decrease in fairness even if accuracy (ACC) remains fairly stable. Within the framework of federated learning (FL), where clients collaboratively train models, its distributed nature further amplifies these challenges since each client can experience group-specific concept drift independently while still sharing the same underlying concept, creating a complex and dynamic environment for maintaining fairness. The most significant contribution of our research is the formalization and introduction of the problem of group-specific concept drift and its distributed counterpart, shedding light on its critical importance in the field of fairness. In addition, leveraging insights from prior research, we adapt an existing distributed concept drift adaptation algorithm to tackle group-specific distributed concept drift, which uses a multimodel approach, a local group-specific drift detection mechanism, and continuous clustering of models over time. The findings from our experiments highlight the importance of addressing group-specific concept drift and its distributed counterpart to advance fairness in machine learning.

A high-precision, high-efficiency, and lightweight panoptic driving perception system is an essential part of autonomous driving for optimal maneuver planning of the autonomous vehicle. We propose a simple, lightweight, and efficient SCAM-P multi-task learning network that accomplishes three crucial tasks simultaneously for panoptic driving: vehicle detection, drivable area segmentation, and lane segmentation. To increase the representation power of the shared backbone of our multi-task network, we designed a novel SCAM module with spatially localized channel attention and channel localized spatial attention blocks. SCAM is a lightweight module that can be plugged into any CNN architecture to enhance the semantic features with negligible computational overhead. We integrate our SCAM module and design the SCAM-P network, which has a shared backbone for feature extraction and three independent heads to handle three tasks at the same time. We also designed a nano variant of our SCAM-P network to make it deployment-friendly on edge devices. Our SCAM-P network obtains competitive results on the BDD100K dataset with 81.1 % mAP50 for object detection, 91.6 % mIoU for drivable area segmentation, and 28.8 % IoU for lane segmentation. Our model is robust in various adverse weather conditions, such as rainy, snowy, and at night. Our SCAM-P network not only achieves improved performance but also runs efficiently in real-time at 230.5 FPS on the RTX 4090 GPU and 112.1 FPS on the Jetson Orin edge device.

LiDAR semantic segmentation is one of the crucial tasks for scene understanding in autonomous driving. Recent trends suggest that voxel- or fusion-based methods obtain improved performance. However, the fusion-based methods are computationally expensive. On the other hand, the voxel-based methods uniformly employ local operators (e.g., 3D SparseConv) without considering the varying-density property of LiDAR point clouds, which result in inferior performance, specifically on far away sparse points due to limited receptive field. To tackle this issue, we propose novel retention block to capture long-range dependencies, maintain the receptive field of far away sparse points and design RetSeg3D , a retention-based 3D semantic segmentation model for autonomous driving. Instead of vanilla attention mechanism to model long-range dependencies, inspired by RetNet, we design cubic window multi-scale retentive self-attention (CW-MSRetSA) module with bidirectional and 3D explicit decay mechanism to introduce 3D spatial distance related prior information into the model to improve not only the receptive field but also the model capacity. Our novel retention block maintains the receptive field which significantly improve the performance of far away sparse points. We conduct extensive experiments and analysis on three large-scale datasets: SemanticKITTI, nuScenes and Waymo. Our method not only outperforms existing methods on far away sparse points but also on close and medium distance points and efficiently runs in real time at 52.1 FPS on a RTX 4090 GPU. • RetSeg3D maintains the receptive field of far away sparse points. • Employs explicit decay to introduce spatial distance related prior information. • Captures long range dependencies & considers the varying-density property of LiDAR. • Experiments on three large-scale datasets: SemanticKITTI, nuScenes and Waymo. • Improved performance and efficiently runs in real time at 52.1 FPS on RTX 4090 GPU.

Unlike the earlier 3D object detection approaches that formulate hand-crafted dense (in thousands) object proposals by leveraging anchors on dense feature maps, we formulate np (in hundreds) number of learnable sparse object proposals to predict 3D bounding box parameters. The sparse proposals in our approach are not only learnt during training but also are input-dependent, so they represent better object candidates during inference. Leveraging the sparse proposals, we fuse only the sparse regions of multi-modal features and we propose Sparse Region Fusion based 3D object Detection (SRFDet3D) network with mainly three components: an encoder for feature extraction, a region proposal generation module for sparse input-dependent proposals and a decoder for multi-modal feature fusion and iterative refinement of object proposals. Additionally for optimal training, we formulate our sparse detector with many-to-one label assignment based on Optimal Transport Algorithm (OTA). We conduct extensive experiments and analysis on publicly available large-scale autonomous driving datasets: nuScenes, KITTI, and Waymo. Our LiDAR-only SRFDet3D-L network achieves 63.1 mAP and outperforms the state-of-the-art networks on the nuScenes dataset, surpassing the dense detectors on KITTI and Waymo datasets. Our LiDAR-Camera model SRFDet3D achieves 64.7 mAP with improvements over existing fusion methods.

The massive adoption of IoT devices, the recent developments in the efficiency of AI systems, and the increase of edge computational power, accelerated the deployment of edge AI systems. The implementation of these systems through the use of low-power embedded devices scattered across the edges of a network allows for reduced latency and cost, compared to traditional cloud-based AI computing systems. As a result of the low-complexity AI models and the available low-power embedded systems on the market, this paper provides a comparative study on the inference performance of convolutional neural networks for different edge devices, by exploiting low-power GPUs and dedicated AI hardware. The benchmark results were able to achieve 864 inferences/s for the Jetson AGX Xavier board on a pre-trained SqueezeNet, while reaching a high power efficiency of 52.6 inferences/s per Watt. For the dedicated Movidius neural stick, the system requires only 1.5 W for processing 24.2 inferences/s.

Inspired by recent advances in vision transformers for object detection, we propose Li3DeTr, an end-to-end LiDAR based 3D Detection Transformer for autonomous driving, that inputs LiDAR point clouds and regresses 3D bounding boxes. The LiDAR local and global features are encoded using sparse convolution and multi-scale deformable attention respectively. In the decoder head, firstly, in the novel Li3DeTr cross-attention block, we link the LiDAR global features to 3D predictions leveraging the sparse set of object queries learnt from the data. Secondly, the object query interactions are formulated using multi-head self-attention. Finally, the decoder layer is repeated Ldecnumber of times to refine the object queries. Inspired by DETR, we employ set-to-set loss to train the Li3DeTr network. Without bells and whistles, the Li3DeTr network achieves 61.3% mAP and 67.6% NDS surpassing the state-of-the-art methods with non-maximum suppression (NMS) on the nuScenes dataset and it also achieves competitive performance on the KITTI dataset. We also employ knowledge distillation (KD) using a teacher and student model that slightly improves the performance of our network.

Shape control involves bringing a deformable object to a desired shape. In the shape control literature, the positioning of the grippers on the object is usually predefined (user-defined) and therefore considered as input information. In this paper we address the gripper positioning problem for shape control. We propose a deformation process within a simulated fully-actuated scenario and introduce multi-scale centroid paths as geometry describing points for which we prove individual control feasibility. Analysis on the evolution of multi-scale centroid paths through the fully-actuated deformation process allows us to define an importance metric for gripper candidates. Final gripper positions, based on the importance metric, are obtained through optimisation. We present simulation results for global and local shape control problems.

A reliable, real time, multi-sensor fusion functionality is crucial for localization of actively controlled capsule endoscopy robots, which are an emerging, minimally invasive diagnostic and therapeutic technology for the gastrointestinal (GI) tract. In this study, we propose a novel multi-sensor fusion approach based on a particle filter that incorporates an online estimation of sensor reliability and a non-linear kinematic model learned by a recurrent neural network. Our method sequentially estimates the true robot pose from noisy pose observations delivered by multiple sensors. We experimentally test the method using 5 degree-of-freedom (5-DoF) absolute pose measurement by a magnetic localization system and a 6-DoF relative pose measurement by visual odometry. In addition, the proposed method is capable of detecting and handling sensor failures by ignoring corrupted data, providing the robustness expected of a medical device. Detailed analyses and evaluations are presented using ex vivo experiments on a porcine stomach model, proving that our system achieves high translational and rotational accuracies for different types of endoscopic capsule robot trajectories.

Reliable and real-time 3D reconstruction and localization functionality is a crucial prerequisite for the navigation of actively controlled capsule endoscopic robots as an emerging, minimally invasive diagnostic and therapeutic technology for use in the gastrointestinal (GI) tract. In this study, we propose a fully dense, non-rigidly deformable, strictly real-time, intraoperative map fusion approach for actively controlled endoscopic capsule robot applications which combines magnetic and vision-based localization, with non-rigid deformations based frame-to-model map fusion. The performance of the proposed method is evaluated using four different ex-vivo porcine stomach models. Across different trajectories of varying speed and complexity, and four different endoscopic cameras, the root mean square surface reconstruction errors vary from 1.58 to 2.17 cm.

Ingestible wireless capsule endoscopy is an emerging minimally invasive diagnostic technology for inspection of the GI tract and diagnosis of a wide range of diseases and pathologies. Medical device companies and many research groups have recently made substantial progresses in converting passive capsule endoscopes to active capsule robots, enabling more accurate, precise, and intuitive detection of the location and size of the diseased areas. Since a reliable real time pose estimation functionality is crucial for actively controlled endoscopic capsule robots, in this study, we propose a monocular visual odometry (VO) method for endoscopic capsule robot operations. Our method lies on the application of the deep recurrent convolutional neural networks (RCNNs) for the visual odometry task, where convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are used for the feature extraction and inference of dynamics across the frames, respectively. Detailed analyses and evaluations made on a real pig stomach dataset proves that our system achieves high translational and rotational accuracies for different types of endoscopic capsule robot trajectories.

Despite significant progress achieved in the last decade to convert passive capsule endoscopes to actively controllable robots, robotic capsule endoscopy still has some challenges. In particular, a fully dense three-dimensional (3D) map reconstruction of the explored organ remains an unsolved problem. Such a dense map would help doctors detect the locations and sizes of the diseased areas more reliably, resulting in more accurate diagnoses. In this study, we propose a comprehensive medical 3D reconstruction method for endoscopic capsule robots, which is built in a modular fashion including preprocessing, keyframe selection, sparse-then-dense alignment-based pose estimation, bundle fusion, and shading-based 3D reconstruction. A detailed quantitative analysis is performed using a non-rigid esophagus gastroduodenoscopy simulator, four different endoscopic cameras, a magnetically activated soft capsule robot, a sub-millimeter precise optical motion tracker, and a fine-scale 3D optical scanner, whereas qualitative ex-vivo experiments are performed on a porcine pig stomach. To the best of our knowledge, this study is the first complete endoscopic 3D map reconstruction approach containing all of the necessary functionalities for a therapeutically relevant 3D map reconstruction.

This study addresses the problem of monocular reconstruction of surfaces that deform isometrically, using points tracked in a single image. To tackle this problem, a flat three‐dimensional (3D) shape of the surface and its image are used as template. Such deformations are characterised by certain geometric constraints and to reconstruct the surfaces, these constraints have to be properly exploited. Therefore, the authors propose an algebraic formula that aims at the joint expression of the geometric constraints, namely those based on the differential properties and also those based on the upper‐bound model. This expression is, in fact, a unique formulation that results from integrating these two types of constraints, and leads to the intended reconstructions, even when the surface is not strictly isometric. The template shape is used to set the parameters of the expression, which is then optimised (along with the projection equations) by means of a semi‐definite programming (SDP) problem. This optimisation enables the estimation of 3D positions of the points on the surface. However, and for implementation purposes, this optimisation is applied separately to patches which, together, make up the whole surface. The experimental results show that the proposed approach improves the results from other methods in terms of accuracy.

A given six dimensional vector represents a 3D straight line in Plücker coordinates if its coordinates satisfy the Klein quadric constraint. In many problems aiming to find the Plücker coordinates of lines, noise in the data and other type of errors contribute for obtaining 6D vectors that do not correspond to lines, because of that constraint. A common procedure to overcome this drawback is to find the Plücker coordinates of the lines that are closest to those vectors. This is known as the Plücker correction problem. In this article we propose a simple, closed-form, and global solution for this problem. When compared with the state-of-the-art method, one can conclude that our algorithm is easier and requires much less operations than previous techniques (it does not require Singular Value Decomposition techniques).

Pose estimation is a relevant problem for imaging systems whose applications range from augmented reality to robotics. In this paper we propose a novel solution for the minimal pose problem, within the framework of generalized camera models and using a planar homography. Within this framework and considering only the geometric elements of the generalized camera models, an imaging system can be modeled by a set of mappings associating image pixels to 3-D straight lines. This mapping is defined in a 3-D world coordinate system. Pose estimation performs the computation of the rigid transformation between the original 3-D world coordinate system and the one in which the camera was calibrated. Using synthetic data, we compare the proposed minimal-based method with the state-of-the-art methods in terms of numerical errors, number of solutions and processing time. From the experiments, we conclude that the proposed method performs better, especially because there is a smaller variation in numerical errors, while results are similar in terms of number of solutions and computation time. To further evaluate the proposed approach we tested our method with real data. One of the relevant contributions of this paper is theoretical. When compared to the state-of-the-art approaches, we propose a completely new parametrization of the problem that can be solved in four simple steps. In addition, our approach does not require any predefined transformation of the dataset, which yields a simpler solution for the problem.

In this paper, we address the problem of pose estimation under the framework of generalized camera models. We propose a solution based on the knowledge of the coordinates of 3-D straight lines (expressed in the world coordinate frame) and their corresponding image pixels. Previous approaches used the knowledge of the coordinates of 3-D points (zero dimensional elements) and their corresponding images (zero dimensional elements). In this paper, pixels belonging to the image of 3-D lines are used. There is no need to establish correspondences between pixels and 3-D points. Correspondences are established between 3-D lines and their images. There is no need to identify individual pixels. The use of correspondences between pixels (that belong to the images of the 3-D lines) and 3-D lines facilitates the correspondence problem when compared to the use of world and image points. This is one of the contributions of this paper. The approach is both evaluated and validated using synthetic data and also real images.

This paper deals with structure-from-motion (SfM) for non-rigid surfaces that undergo isometric motion. Our SfM framework aims at the joint estimation of the 3D surface and the camera motion by combining a ToF range sensor and a monocular RGB camera using a template-based approach. Our goal is to use the 2D low- resolution depth estimates provided by the TOF camera and use them to facilitate the estimation of non-rigid structure using the high-resolution images obtained by means of a RGB camera. In this paper we describe the approach used to model isometric motion by means of a surface model based on a triangular mesh. The ToF sensor is used to obtain the depth of a sparse set of 3D points, from which the depth of the mesh can be recovered using a multivariate linear system. Subsequently, having the depth of the mesh vertices, we form a non-linear constraint based on the projected length of each edge. A second non-linear constraint is then used for minimizing re-projection errors. Experimental results show that the proposed approach has good performance even if only a low-resolution depth image is used.

In this article we address the problem of minimal pose under the framework of the generalized camera models. Previous approaches were based on geometric properties, such as the preservation of distance between points. In this paper we propose a novel formulation of the problem using an algebraic-based approach. We represent the pose by a 3×3 matrix. Using both the algebraic relationship between three incident 3D points and straight lines and the underlying constraints of the pose matrix, pose can be computed. In terms of experimental results, the main contribution of the proposed method is the robustness to critical configurations. In addition, a full comparison and analysis between state-of-the-art methods is made (so far, there is no published comparison between state-of-the-art methods published).

In this article, we address the pose estimation for planar motion under the framework of generalized camera models. We assume the knowledge of the coordinates of 3D straight lines in the world coordinate system. Pose is estimated using the images of the 3D lines. This approach does not require the determination of correspondences between pixels and 3D world points. Instead, and for each pixel, it is only required that we determine to which 3D line it is associated with. Instead of identifying individual pixels, it is only necessary to establish correspondences between the pixels that belong to the images of the 3D lines, and the 3D lines. Moreover and using the assumption that the motion is planar, this paper presents a novel method for the computation of the pose using general imaging devices and assuming the knowledge of the coordinates of 3D straight lines. The approach is evaluated and validated using both synthetic data and real images. The experiments are performed using a mobile robot equipped with a non-central camera.

This paper deals with pose estimation using an iterative scheme. We show that using adequate visual information, pose estimation can be performed iteratively with only three independent unknowns, which are the translation parameters. Specifically, an invariant to rotational motion is used to estimate the camera position. In addition, an adequate transformation is applied to the proposed invariant to decrease the nonlinearities between the variations in image space and 3-D space. Once the camera position is estimated, we show that the rotation can be estimated efficiently using two different direct methods. The proposed approach is compared against two other methods from the literature. The results show that using our method, pose tracking in image sequences and the convergence rate for randomly generated poses are improved.

This paper deals with pose estimation using an iterative scheme. We show that using adequate visual information, pose estimation can be performed in a decoupling the estimation of translation and rotation. More precisely, we show that pose estimation can be achieved iteratively as a function of only three independent unknowns, which are the translation parameters. An invariant to rotational motion is used to estimate the camera position. Once the camera position is estimated, we show that the rotation can be estimated efficiently using a direct method. The proposed approach is compared against two classical methods from the literature. The results show that using our method, pose tracking in image sequences and the convergence rate for randomly generated poses are improved.

Generic imaging models can be used to represent any camera. Current generic models are discrete and define a mapping between each pixel in the image and a straight line in 3D space. This paper presents a modification of the generic camera model that allows the simplification of the calibration procedure. The only requirement is that the coordinates of the 3D projecting lines are related by functions that vary smoothly across space. Such a model is obtained by modifying the general imaging model using radial basis functions (RBFs) to interpolate image coordinates and 3D lines, thereby allowing both an increase in resolution (due to their continuous nature) and a more compact representation. Using this variation of the general imaging model, we also develop a calibration procedure. This procedure only requires that a 3D point be matched to each pixel. In addition, not all the pixels need to be calibrated. As a result, the complexity of the procedure is significantly decreased. Normalization is applied to the coordinates of both image and 3D points, which increases the accuracy of the calibration. Results with both synthetic and real datasets show that the model and calibration procedure are easily applicable and provide accurate calibration results.

Catadioptric cameras combine conventional cameras and mirrors to create omnidirectional sensors providing 360° panoramic views of a scene. Modeling such cameras has been subject of significant research interest in the computer vision community leading to a deeper understanding of the image properties and also to different models for different types of configurations. Visual servoing applications using catadioptric cameras have essentially been using central cameras and the corresponding unified projection model. So far only in very few cases more general models have been used. In this paper we address the problem of visual servoing using the so-called the radial model. The radial model can be applied to many camera configurations and in particular to non-central catadioptric systems with mirror shapes that are symmetric around the optical axis. In this case we show that the radial model can be used with a non-central catadioptric camera to allow effective image-based visual servoing (IBVS) of a mobile robot. Using this model, which is valid for a large set of catadioptric cameras, new visual features are proposed to control the degrees of freedom of a mobile robot moving on a plane. Several simulation results are provided to validate the effectiveness of such features.

Generic imaging models can be used to represent any camera. These models are specially suited for non-central cameras for which closed-form models do not exist. Current models are discrete and define a mapping between each pixel in the image and a straight line in 3D space. Due to difficulties in the calibration procedure and model complexity these methods have not been used in practice. The focus of our work was to relax these drawbacks. In this paper we modify the general imaging model using radial basis functions to interpolate image coordinates and 3D lines allowing both an increase in resolution (due to their continuous nature) and a more compact representation. Using this new variation of the general imaging model we also develop a new linear calibration procedure. In this process it is only required to match one 3D point to each image pixel. Also it is not required the calibration of every image pixel. As a result the complexity of the procedure is significantly decreased.

This paper presents a general approach for the simultaneous tracking of multiple moving targets using a generic active stereo setup. The problem is formulated on the plane, where cameras are modeled as “line scan cameras,” and targets are described as points with unconstrained motion. We propose to control the active system parameters in such a manner that the images of the targets in the two views are related by a homography. This homography is specified during the design stage and, thus, can be used to implicitly encode the desired tracking behavior. Such formulation leads to an elegant geometric framework that enables a systematic and thorough analysis of the problem at hand. The benefits of the approach are illustrated by applying the framework to two distinct stereo configurations. In the first case, we assume two pan-tilt-zoom cameras, with rotation and zoom control, which are arbitrarily placed in the working environment. It is proved that such a stereo setup can track up toN= 3 free-moving targets, while assuring that the image location of each target is the same for both views. The second example considers a robot head with neck pan motion and independent eye rotation. For this case, it is shown that it is not possible to track more thanN= 2 targets because of the lack of zoom. The theoretical framework is used to derive the control equations, and the implementation of the tracking behavior is described in detail. The correctness of the results is confirmed through simulations and real tracking experiments.

This article presents a general approach for the active stereo tracking of multiple moving targets. The problem is formulated on the plane, where cameras are modeled as “line scan cameras” and targets are described as points with unconstrained motion. We propose to control the active system parameters in such a manner that the images of the targets in the two views are related by an homography. This homography is specified during the design stage and implicitly encodes the tracking behavior. It is shown that this formulation leads to an elegant geometric framework that enables to decide about the feasibility of a particular active tracking task. We apply it to prove that two cameras with rotation and zoom control, can track up to three moving targets, while assuring that the image location of each target is the same for both views. In addition, the framework is also useful for devising tracking strategies and deriving the required control equations. This feature is illustrated through a real experiment on tracking two independent targets using a binocular stereo head.

This paper presents a linear method to estimate the pose of a noncentral catadioptric system with a quadric shaped mirror in relation to a world reference frame (or local reference frame without loss of generality). The vision system is assumed to be calibrated. The method uses also as input data the structure of the scene. It is proved that any reflection point should belong to an analytical quadric that intersects the mirror quadric itself. This constraint can be written linearly in the 3D scene point coordinates (in the camera reference frame). The unknown pose screw transformation that relates camera and world reference frames can then be used in the linear model, allowing for the construction of a linear equation in the pose transformation elements. Additional constraints are used to force the estimated rotation elements to build an orthogonal matrix. Tests with simulated data and also on real images with different mirrors proved the method to be consistent and to estimate the pose accurately. However, it was also observed that the method is sensitive to noise. The results are compared with another method.

In central catadioptric systems, lines in a scene are projected to conic curves in the image. This work studies the geometry of the central catadioptric projection of lines and its use in calibration. It is shown that the conic curves where the lines are mapped possess several projective invariant properties. From these properties, it follows that any central catadioptric system can be fully calibrated from an image of three or more lines. The image of the absolute conic, the relative pose between the camera and the mirror, and the shape of the reflective surface can be recovered using a geometric construction based on the conic loci where the lines are projected. This result is valid for any central catadioptric system and generalizes previous results for paracatadioptric sensors. Moreover, it is proven that systems with a hyperbolic/elliptical mirror can be calibrated from the image of two lines. If both the shape and the pose of the mirror are known, then two line images are enough to determine the image of the absolute conic encoding the camera's intrinsic parameters. The sensitivity to errors is evaluated and the approach is used to calibrate a real camera.

Paracatadioptric sensors combine a parabolic shaped mirror and a camera inducing an orthographic projection. Such a configuration provides a wide field of view while keeping a single effective viewpoint. Previous work in central catadioptric sensors proved that a line projects into a conic curve and that three line images are enough to calibrate the system. However the estimation of the conic curves where lines are mapped is hard to accomplish. In general only a small arc of the conic is visible in the image and conventional conic fitting techniques are unable to correctly estimate the curve. The present work shows that a set of conic curves corresponds to paracatadioptric line images if, and only if, certain properties are verified. These properties are used to constraint the search space and correctly estimate the curves. The accurate estimation of a minimum of three line images allows the complete calibration of the paracatadioptric camera. If the camera is skewless and the aspect ratio is known then the conic fitting problem is solved naturally by an eigensystem. For the general situation the conic curves are estimated using non-linear optimization.

This paper analyses the problem of motion estimation from a sequence of stereo images. Both the differential and discrete approaches of two methods are formulated The differential approach uses differential optical flow whereas the discrete approaches uses feature correspondences. Both methods are used to compute, first, the 3D velocity in the depth (Z) direction and, second, the complete rigid motion parameters. Furthermore, the uncertainty propagation models for both methods and approaches are derived These models are analysed in order to point out the critical variables for the methods. The methods were extensively tested using synthetic images as well as real images and several conclusions are drawn from the results. The real images are used without any illumination control of the scene in order to study the behavior of the methods in strongly noisy environments with low resolution depth maps.

An imaging system with a single effective viewpoint is called a central projection system. The conventional perspective camera is an example of a central projection system. Systems using mirrors to enhance the field of view while keeping a unique center of projection are also examples of central projection systems. Perspective image formation can be described by a linear model with well known properties. In general central catadioptric imaging, the mapping between points in the world and in the image is highly nonlinear. The paper establishes a general model for central catadioptric image formation made up of three functions: a linear function mapping the world into an oriented projective plane, a nonlinear transformation between two oriented projective planes, and a collineation in the plane. The model is used to study issues in the projection of lines. The equations and geometric properties of general catadioptric imaging of lines are derived. The application of the results in auto-calibration of central catadioptric systems and reconstruction are discussed. A method to calibrate the system using three line images is presented.

Deals with active tracking of 3D moving targets. Visual tracking is presented as a regulation control problem. The performance and robustness in visual control of motion depends both on the vision algorithms and the control structure. Delays and system latencies substantially affect the performance of visually guided systems. We discuss ways to cope with delays while improving system performance. Model predictive control strategies are proposed to compensate for the mechanical latency in visual control of motion.

Describes a tracking and accommodation system controlled by retinal motion flow. The tracking system decomposes two basic behaviors: smooth-pursuit and vergence. Both behaviors are optical flow based. The eye's accommodation (focusing) is based on a focusing-by-vergence approach combining a flow based vergence process with an off-line pre-calibration of the lens focusing odometry. The focusing and tracking system run in parallel and are both velocity controlled.

We describe an indoor surveillance system which enables the control of a binocular active vision system by a remote operator wearing a head mounted display. Visual feedback to the head mounted display is obtained using the images captured by the active vision system. To maximize the 3D perception of the operator the system is able to adjust the vergence angle of the active vision system in order to maintain fixation on the objects in the center of the image. A wide-angle lens static camera located on the surveillance area is also used to enable the detection and tracking of intruders. In case of an intrusion a map with the location of the detected targets is displayed on top of the images shown on the head mounted display. This information allows the remote operator to redirect his attention to where the target is.

We describe a focusing-by-vergence system combining a flow based vergence process with an off-line pre-calibration of the focusing odometry. The vergence process is controlled by the retinal optical flow disparity and the target depth velocity is directly obtained from this disparity. The relationship between the target depth (focused target distance) and the focusing odometry of the motorized lens is obtained by an off-line focusing calibration, using a bivariate polynomial to model the relationship between these parameters. The focusing system is velocity controlled and the focusing velocity is dependent on the target focusing depth velocity and the focal distance velocity. The influence of the focal distance velocity that arise due to the focusing process itself is compensated by the adjustment of the zoom parameter based on a constant magnification approach, or through the modeling of the focal distance as a function of the zoom and focus lens settings.

A system for tracking multiple targets in 3D is described. The system is made up of two pan-and-tilt units that are attached to the extremities of a rotating arm. This configuration has several advantages and can deal with several specific instances of tracking more than one target. A control strategy that guarantees equal target disparities in both images whenever targets are seen by both cameras is presented. This has advantages for segmentation and trajectory reconstruction. Target images are simultaneously visible in the cameras, enabling the recovery of the targets 3D trajectory. It is also shown that mutual occlusion occurs in a well-defined configuration and can therefore be dealt with.

Perspective camera calibration has been a research subject for a large group of researchers. However, only a small number of those methods base their approaches on the use of monoplane calibration points. We developed one of those methodologies using monoplane calibration points to perform an explicit three-dimensional (3-D) camera calibration. This methodology is based on an iterative approach. To avoid the singularities obtained with the calibration equations when monoplane calibration points are used, this method computes the calibration parameters in a multistep procedure and requires a first-guess solution for the intrinsic parameters. These intrinsic parameters are updated and their accuracy increased during the iterative procedure. All computations required are linear and in addition to the extrinsic parameters the proposed method also computes the first coefficient of the radial distortion (k/sub 1/) and the skew angle. A first-guess value for the focal length of the lens is required but its value is iteratively updated using the Gauss lens model. This methodology also includes the uncertainty horizontal image scale factor (S/sub x/) on the set of calibration parameters to be computed, which makes this approach independent of the accuracy of the horizontal scale factor. The proposed methodology has the advantage that it can be used with monoplane calibration data with no restrictions for the pose geometry of the camera. In this approach the pose estimation problem is treated separately, computing the pose orientation of the camera in a first step and using this information to compute the pose location. A model for the expected stability of the camera look angles from noisy image data and its stability analysis as a function of the pose geometry of the camera is presented. Several experimental and simulated analyzes were performed and are presented.

Perspective camera calibration has been in the last decades a research subject for a large group of researchers and as a result several camera calibration methodologies can be found in the literature. However only a small number of those methods base their approaches on the use of monoplane calibration points. This paper describes one of those methodologies that uses monoplane calibration points to realize an explicit 3D camera calibration. To avoid the singularity obtained with the calibration equations when monoplane calibration points are used, this method computes the calibration parameters in a multi-step procedure and requires a first-guess solution for the intrinsic parameters. These parameters are updated and their accuracy increased through an iterative procedure. A stability analysis as a function of the pose of the camera is presented. Camera pose view strategies for accurate camera orientation computation can be extracted from the pose view stability analysis.

We describe a surveillance system made up of a fixed camera and a binocular active vision system. A fixed camera is used to provide the system with peripheral motion detection capabilities. The binocular active system is used to track in 3D and reconstruct the trajectories of specific targets. Both are fully integrated and the redundancy is exploited to achieve a higher degree of robustness in detecting and tracking intruders.

The performance of a binocular active vision system depends mainly on two aspects: vision/image processing and control. We characterize the monocular performance of smooth pursuit. This system is used to track binocularly targets in a surveillance environment. One of the aspects of this characterization was the inclusion of the vision processing. To characterize the performance from the control point of view four standard types of inputs were used: step, ramp, parabola and sinusoid. The responses can be used to identify which subsystems can be optimized. We show that prediction and a velocity estimate are essential for a good tracking performance.

Both the original version of David Lowe's influential and classic algorithm for tracking known objects and a reformulation of it implemented by Ishiiet al. rely on (different) approximated imaging models. Removing their simplifying assumptions yields a fully projective solution with significantly improved accuracy and convergence, and arguably better computation-time properties.

This article describes one solution for the problem of pursuit of objects moving on a plane by using a mobile robot and an active vision system. The solution deals with the interaction of different control systems using visual feedback and it is accomplished by the implementation of a visual gaze holding process interacting cooperatively with the control of the trajectory of a mobile robot. These two systems are integrated to follow a moving object at constant distance and orientation with respect to the mobile robot. The orientation and the position of the active vision system running a gaze holding process give the feedback signals to the control used to pursuit the target in real-time. The paper addresses the problems of visual fixation, visual smooth pursuit, navigation using visual feedback and compensation for system's movements. The algorithms for visual processing and control are described in the article. The mechanisms of cooperation between the different control and visual algorithms are also described. The final solution is a system able to operate at approximately human walking rates as the experimental results show in the paper.

We describe the real-time implementation of several visual behaviors in an active vision system. Issues related to the real-time implementation are discussed, concerning the modeling of the measurements made in the image. We also describe a methodology for calibrating the camera head, taking advantage of its degrees of freedom. These calibration parameters are used to evaluate the performance of the system. Another important issue of the operation of active vision binocular heads is their integration into more complex robotic systems. We claim that higher levels of autonomy and integration can be obtained by designing the system architecture based on the concept of purposive behavior. We show how this architecture can be used to implement a pursuit behavior using optical flow. Simultaneously vergence control can also be performed using the same visual processes.

In this paper we describe the implementation of real-time binocular gaze control. This implementation is performance by using a complex binocular active vision system. In addition real-time vergence is also implemented. Vergence and gaze control are implemented based on binocular optical flow. Vergence control is implemented based on optical flow disparity, using the horopter as a reference. A Kalman filter is used to filter out the signals and compensate for the delays of the system.

An active vision system must be able to perform reactive visual processes in real time. In this paper we discuss a number of issues related to the implementation of a real-time control architecture and describe the architecture we use with camera heads. Another important issue of the operation of active vision binocular heads is their integration into more complex robotic systems. Higher levels of autonomy and integration can be obtained by designing the control system architecture based on the concept of purposive behavior. At the lower levels we consider the vision as a sensor and integrate it in the control systems (both feedforward and servo loops); and several visual processes are implemented in parallel, computing relevant measures for the control process. At higher levels the architecture is modelled as a state transition system. Finally, we show how this architecture can be used to implement a pursuit behavior using optical flow. Simultaneously vergence control can also be performed using the same visual processes.

The main purpose of this paper is not to present a new camera calibration algorithm or a new pose estimation method, but just to analyse the stability of the camera pose estimation using the Haralick observation which is based on the fact that there is sufficient information on the 2D perspective projection of a rectangle of unknown size in 3D space to determine the three camera look angles. A brief description of the camera look angles computation is presented and a model for the expected stability of the camera look angles computation front noisy image data is described. A stability analysis as a function of the pose of the camera is also described, and the influence of the camera pose is confirmed with simulated data. As a result, camera pose view strategies for accurate camera orientation computation can be extracted from this pose view stability analysis.

This article describes one solution for the problem of pursuit by using a mobile robot and an active vision system. The solution deals with the interaction of different control systems using visual feedback and it is accomplished by the implementation of a visual gaze holding process interacting cooperatively with the control of the trajectory of a mobile robot. These two systems are integrated to follow a moving object at constant distance and orientation with respect to the mobile robot. The orientation and the position of the active vision system running a gaze holding process give the feedback signals to the control used to pursuit the target in real-time. The mechanisms of cooperation between the different control and visual algorithms are described. The final solution is a system able to operate at approximately human walking rates.

In practice focusing can be obtained by displacing the sensor plate with respect to the image plane, by moving the lens or by moving the object with respect to the optical system. Moving the lens or sensor plate with respect to each other, causes changes of the magnification and corresponding changes on the object coordinates. In order to overcome these problems, the authors propose a technique to vary the degree of focusing by moving the camera with respect to the object position. The camera is attached to the tool of a manipulator in a hand-eye configuration with the position of the camera always known. This approach ensures that the focused areas of the image are always subjected to the same magnification. To measure the focus quality operators are used to evaluate the quantity of high-frequency components on the image. Different types of these operators was tested and the results compared.>

Addresses the geometric modelling aspects related with the use of a pair of video cameras mounted on a six degrees-of-freedom manipulator in an hand-eye configuration. The information obtained by these sensors shall be used to generate the depth map of the environment where the manipulator works. The procedure to determining the image formation parameters is known by camera calibration and a method to optimize these calibration parameters is presented. After presenting the calibration technique used, the authors develop a calibration matrix invariant to movement. The optimization process is applied to this matrix to obtain a better performance in the numerical data.>