With the rapid advancements in autonomous driving, accurately predicting pedestrian behavior has become essential for ensuring safety in complex and unpredictable traffic conditions. The growing interest in this challenge highlights the need for comprehensive datasets that capture unstructured environments, enabling the development of more robust prediction models to enhance pedestrian safety and vehicle navigation. In this paper, we introduce an Indian driving pedestrian dataset designed to address the complexities of modeling pedestrian behavior in unstructured environments, such as illumination changes, occlusion of pedestrians, unsignalized scene types and vehicle-pedestrian interactions. The dataset provides high-level and detailed low-level comprehensive annotations focused on pedestrians requiring the ego-vehicle's attention. Evaluation of the state-of-the-art intention prediction methods on our dataset shows a significant performance drop of up to 15 %, while trajectory prediction methods underperform with an increase of up to 1208 MSE, defeating standard pedes-trian datasets. Additionally, we present exhaustive quantitative and qualitative analysis of intention and trajectory baselines. We believe that our dataset will open new challenges for the pedestrian behavior research community to build robust models. Project Page: https://cvit.iiit.ac.in/research/projects/cvit-projects/iddped

The problem of determining geolocation through visual inputs, known as Visual Place Recognition (VPR), has attracted significant attention in recent years owing to its potential applications in autonomous self-driving systems. The rising interest in these applications poses unique challenges, particularly the necessity for datasets encompassing unstructured environmental conditions to facilitate the development of robust VPR methods. In this paper, we address the VPR challenges by proposing an Indian driving VPR dataset that caters to the semantic diversity of unstructured driving environments like occlusions due to dynamic environments, variations in traffic density, viewpoint variability, and variability in lighting conditions. In unstructured driving environments, GPS signals are unreliable often affecting the vehicle to accurately determine location. To address this challenge, we develop an interactive image-to-image tagging annotation tool to annotate large datasets with ground truth annotations for VPR training. Evaluation of the state-of-the-art methods on our dataset shows a significant performance drop of up to 15%, defeating a large number of standard VPR datasets. We also provide an exhaustive quantitative and qualitative experimental analysis of frontal-view, multi-view, and sequence-matching methods. We believe that our dataset will open new challenges for the VPR research community to build robust models. Project Page: https://cvit.iiit.ac.in/research/projects/cvit-projects/iddvpr

Humans have a natural ability to perform semantic associations with the surrounding objects in the environment. This allows them to create a mental map of the environment, allowing them to navigate on-demand when given linguistic instructions. A natural goal in Vision Language Navigation (VLN) research is to impart autonomous agents with similar capabilities. Recent works take a step towards this goal by creating a semantic spatial map representation of the environment without any labeled data. However, their representations are limited for practical applicability as they do not distinguish between different instances of the same object. In this work, we address this limitation by integrating instance-level information into spatial map representation using a community detection algorithm and utilizing word ontology learned by large language models (LLMs) to perform open-set semantic associations in the mapping representation. The resulting map representation improves the navigation performance by two-fold (233%) on realistic language commands with instance-specific descriptions compared to the baseline. We validate the practicality and effectiveness of our approach through extensive qualitative and quantitative experiments.

Autonomous driving and assistance systems rely on annotated data from traffic and road scenarios to model and learn the various object relations in complex real-world scenarios. Preparation and training of deploy-able deep learning architectures require the models to be suited to different traffic scenarios and adapt to different situations. Currently, existing datasets, while large-scale, lack such diversities and are geographically biased towards mainly developed cities. An unstructured and complex driving layout found in several developing countries such as India poses a challenge to these models due to the sheer degree of variations in the object types, densities, and locations. To facilitate better research toward accommodating such scenarios, we build a new dataset, IDD-3D, which consists of multimodal data from multiple cameras and LiDAR sensors with 12k annotated driving LiDAR frames across various traffic scenarios. We discuss the need for this dataset through statistical comparisons with existing datasets and highlight benchmarks on standard 3D object detection and tracking tasks in complex layouts. Code and data available1.

For an autonomous corridor following task where the environment is continuously changing, several forms of environmental noise prevent an automated feature extraction procedure from performing reliably. Moreover, in cases where pre-defined features are absent from the captured data, a well defined control signal for performing the servoing task fails to get produced. In order to overcome these drawbacks, we present in this work, using a convolutional neural network (CNN) to directly estimate the required control signal from an image, encompassing feature extraction and control law computation into one single end-to-end framework. In particular, we study the task of autonomous corridor following using a CNN and present clear advantages in cases where a traditional method used for performing the same task fails to give a reliable outcome. We evaluate the performance of our method on this task on a Wheelchair Platform developed at our institute for this purpose.

This work addresses path planning for reactionless visual servoing of a redundant dual-arm space robot through exploration in the image space. The planner explores the image moment based feature space, impends acceleration to the image features and extends the feature tree. A reactionless visual servoing control law is integrated to extend the tree in the configuration space simultaneously. The proposed algorithm is able to incorporate the necessary coupling between the motions of the the dual arms and the base of the robot to ensure zero base reactions. Additionally, it also gives the flexibility to apply multiple constraints in both the image space and the configuration space. The effectiveness of the proposed framework is exhibited by implementing the algorithm on a numerical model of a 14-DoF dual arm space robot.

This paper presents a novel visual servoing controller for a satellite mounted dual-arm space robot. The controller is designed to complete the task of servoing the robot's endeffectors to the desired pose, while regulating orientation of the base-satellite. Task redundancy approach is utilized to coordinate the servoing process and attitude of the base satellite. The visual task is defined as a primary task, while regulating attitude of the base satellite to zero is defined as a secondary task. The secondary task is formulated as an optimization problem in such a way that it does not affect the primary task, and simultaneously minimizes its cost function. A set of numerical experiments are carried out on a dual-arm space robot showing efficacy of the proposed control methodology.

Visual localization in crowded dynamic environments requires information about static and dynamic objects. This paper presents a robust method that learns the useful features from multiple runs in highly crowded urban environments. Useful features are identified as distinctive ones that are also reliable to extract in diverse imaging conditions. Relative importance of features is used to derive the weight for each feature. The popular Bag-of-words model is used for image retrieval and localization, where query image is the current view of the environment and database contains the visual experience from previous runs. Based on the reliability, features are augmented and eliminated over runs. This reduces the size of representation, and makes it more reliable in crowded scenes. We tested the proposed method on data sets collected from highly crowded Indian urban outdoor settings. Experiments have shown that with the help of a small subset (10%) of the detected features, we can reliably localize the camera. We achieve superior results in terms of localization accuracy even when more than 90% of the pixels are occluded or dynamic.

Understanding positional semantics of the environment plays an important role in manipulating an object in clutter. The interaction with surrounding objects in the environment must be considered in order to perform the task without causing the objects fall or get damaged. In this paper, we learn the semantics in terms of support relationship among different objects in a cluttered environment by utilizing various photometric and geometric properties of the scene. To manipulate an object of interest, we use the inferred support relationship to derive a sequence in which its surrounding objects should be removed while causing minimal damage to the environment. We believe, this work can push the boundary of robotic applications in grasping, object manipulation and picking-from-bin, towards objects of generic shape and size and scenarios with physical contact and overlap. We have created an RGBD dataset that consists of various objects used in day-to-day life present in clutter. We explore many different settings involving different kind of object-object interaction. We successfully learn support relationships and predict support order in these settings.

In this paper we present a novel approach to robust visual servoing. This method removes the feature tracking step from a typical visual servoing algorithm. We do not need correspondences of the features for deriving the control signal. This is achieved by modeling the image features as a Mixture of Gaussians in the current as well as desired images. Using Lyapunov theory, a control signal is derived to minimize a distance function between the two Gaussian mixtures. The distance function is given in a closed form, and its gradient can be efficiently computed and used to control the system. For simplicity, we first consider the 2D motion case. Then, the general case is presented by introducing the depth distribution of the features to control the six degrees of freedom. Experiments are conducted within a simulation framework to validate our proposed method.

This paper presents a novel real-time algorithm to sequentially solve the triangulation problem. The problem addressed is estimation of 3D point coordinates given its images and the matrices of the respective cameras used in the imaging process. The algorithm has direct application to real time systems like visual SLAM. This article demonstrates the application of the proposed algorithm to the mapping problem in visual SLAM. Experiments have been carried out for the general triangulation problem as well as the application to visual SLAM. Results show that the application of the proposed method to mapping in visual SLAM outperforms the state of the art mapping methods.

A new probabilistic tracking framework for integrating information available from various visual cues is presented in this paper. The framework allows selection of "good" features for each cue, along with factors of their "goodness" to select the best combination form. Two particle filter based trackers, which use edge and texture features, run independently. The output of the master tracker is computed using democratic integration using the "goodness" weights. The final output is used as apriori for both tracker in the next iteration. Finally, particle filters are used to deal with non-Gaussian errors in feature extraction / prior computation. Results are shown for planar object tracking

In this paper, we present a new hybrid visual servoing algorithm for robot arm positioning task. Hybrid methods in visual servoing partially combine the 2D and 3D visual information to improve the performance of the traditional image-based and position-based visual servoing. Our algorithm is superior to the state of the art hybrid methods. The objective function has been designed to include the full 2D and 3D information available either from the CAD model or from the partial reconstruction process by decomposing the homography matrix between two views. Here, each of 2D and 3D error functions is used to control the six degrees of freedom. We call this method 5D visual servoing. The positioning task has been formulated as a minimization problem. Gradient decent as a first order approximation and Gauss-Newton as a second order approximation are considered in this paper. Simulation results show that these two methods provide an efficient solution to the camera retreat and features visibility problems. The camera trajectory in the Cartesian space is also shown to be satisfactory.

In this paper, we present a novel boosted robot vision control algorithm. The method utilizes on-line boosting to produce a strong vision-based robot control starting from two weak algorithms. The notion of weak and strong algorithms has been presented in the context of robot vision control, in presence of uncertainty in the measurement process. Appropriate probabilistic error functions are defined for the weak algorithm to evaluate their suitability in the task. An on-line boosting algorithm is employed to derive a final strong algorithm starting from two weak algorithms. This strong one has superior performance both in image and Cartesian spaces. Experiments justify this claim.

This paper explores the possibility of using convex optimization to address a class of problems in visual servoing. This work is motivated by the recent success of convex optimization methods in solving geometric inference problems in computer vision. We formulate the visual servoing problem with feature visibility constraints as a convex optimization of a function of the camera position i.e. the translation of the camera. First, the path is planned using potential field method that produces unconstrained but straight line path from the initial to the desired camera position. The problem is then converted to a constrained convex optimization problem by introducing the visibility constraints to the minimization problem. The objective of the minimization process is to find for each camera position the closest alternate position from which all features are visible. This algorithm ensures that the solution is optimal. This formulation allows the introduction of more constraints, like joint limits of the arm, into the visual servoing process. The results have been illustrated in a simulation framework.

In this paper we present a new integration method for improving the performance of visual servoing. The method integrates image-based visual servoing (IBVS) and position-based visual servoing (PBVS) approaches to satisfy the widely varying requirements of the visual servoing process. We define an integration rule for IBVS and PBVS controllers. Density functions that determine the weighting factor of each controller are defined to satisfy the above constraints. We prove that this integration method provides global stability, and avoids local minima. The new integration method is validated on positioning tasks and compared with other switching methods

Many problems in computer vision such as pose recovery and structure estimation are formulated as a minimization process. These problems vary in the use of image measurements directly or using them to extract 3D cues in the minimization process. Hybrid methods have the advantage of combining the 2D and 3D visual information to improve the performance over the above two methods. In this paper, we present a new formulation for minimizing a class of hybrid error functions. This is done by using 2D information from the image space and 3D information from the Cartesian space in one error function. Applications to visual servoing and image alignment problems are presented. The positioning task of a robot arm has been formulated as a minimization problem. Gradient decent as a first order approximation and Gauss-Newton as a second order approximation are considered in this paper. Simulation results show, comparing with 2 1/2 D hybrid method, that these two methods provide an efficient solution to the features visibility problems and the camera trajectory in the Cartesian space.

In this paper, we present a new integration method for improving the performance of visual servoing. The method integrates both image-based visual servoing (IBVS) and position-based visual servoing (PBVS) to satisfy the requirements of the visual servoing process. We define a probabilistic integration rule for IBVS and PBVS controllers. Density functions that determine the probability of each controller are defined to satisfy the above constraints. We prove that this integration method provides global stability, and avoids local minima. The new integration method is validated on positioning tasks, and compared with other switching methods.