Demonstration venue · read-only. Every page can be browsed; the buttons that would change it are switched off. Create an account to run TaxoReview on your own data. A simulation from public publication records: the people listed were not invited and did not agree to review.
? Tutorial for this page
Demo Data from EDBT 2026 / reviewers in the wild / expert

Jordi Salvador

dblp:53/5830 · DBLP profile ↗
← Back
28ranked-venue papers
10as first author
8since 2021 · last 2024
0000-0002-1809-651XORCID · corroborated

Domains — the database's venue-derived domains; a paper can count in several

Graphics, computer vision, multimedia, augmented reality and games · 23 · 10 first-author · 4 since 2021Artificial intelligence and machine learning · 15 · 2 first-author · 7 since 2021Systems, architecture and hardware · 1 · 1 since 2021

Expertise — from the expertise taxonomy: the topics of the expert's papers under the CCF categories. A weight counts papers with recency: 1 for a paper about the topic, 0.3 when the topic is its context, halved every five years.

Artificial intelligence
9 papers
Robot navigation and mapping · 25% Motion planning and robot control · 20% Reinforcement learning · 16%
Computer graphics and multimedia
5 papers
Image and video processing · 80% Computational photography and imaging · 8% Geometric modeling and processing · 5%

Topics — the 28 heaviest of 33, each with the papers that count most for it

TopicWeightPapersLastEvidence papers
Machine learning › Reinforcement learning
imitation learning		 1.322024
SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World · CVPR 2024
Bridging the Imitation Gap by Adaptive Insubordination · NeurIPS 2021
Robotics › Robot navigation and mapping
embodied navigation		 0.822023
Scene Graph Contrastive Learning for Embodied Navigation · ICCV 2023
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform · CVPR 2020
Robotics › Motion planning and robot control › robot learning › robot policy learning
generalist robot policy		 0.812024
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration · ICRA 2024
Robotics › Motion planning and robot control › robot learning › manipulation learning
language-conditioned manipulation		 0.812024
SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World · CVPR 2024
Robotics › Motion planning and robot control
robot learning		 0.812024
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration · ICRA 2024
Robotics › Robot navigation and mapping
visual navigation		 0.812024
SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World · CVPR 2024
Computer vision › 3D vision
3d object dataset		 0.712023
Objaverse: A Universe of Annotated 3D Objects · CVPR 2023
Machine learning › Representation and self-supervised learning
contrastive learning		 0.712023
Scene Graph Contrastive Learning for Embodied Navigation · ICCV 2023
Robotics › Robot navigation and mapping
object goal navigation		 0.712023
Scene Graph Contrastive Learning for Embodied Navigation · ICCV 2023
Machine learning › Reinforcement learning › reinforcement learning environment › environment design
procedural environment generation		 0.612022
🏘️ ProcTHOR: Large-Scale Embodied AI Using Procedural Generation · NeurIPS 2022
Robotics › Robot navigation and mapping
robotic navigation and manipulation		 0.612022
🏘️ ProcTHOR: Large-Scale Embodied AI Using Procedural Generation · NeurIPS 2022
Computer vision › Vision and language
multimodal communication		 0.512021
Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text · EMNLP (1) 2021
Robotics › Robot manipulation › learning from demonstration
reinforcement learning from demonstration		 0.512021
Bridging the Imitation Gap by Adaptive Insubordination · NeurIPS 2021
Image and video processing
super-resolution		 0.522016
Antipodally Invariant Metrics for Fast Regression-Based Super-Resolution · IEEE Trans. Image Process. 2016
PSyCo: Manifold Span Reduction for Super Resolution · CVPR 2016
Computer vision › Image recognition and object detection
object discovery		 0.412020
Learning About Objects by Learning to Interact with Them · NeurIPS 2020
Machine learning › Transfer learning and domain adaptation
sim-to-real transfer		 0.412020
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform · CVPR 2020
Image and video processing › image restoration › image deblurring
defocus deblurring		 0.212016
Non-Parametric Blur Map Regression for Depth of Field Extension · IEEE Trans. Image Process. 2016
Computational photography and imaging › depth of field
extended depth of field		 0.212016
Non-Parametric Blur Map Regression for Depth of Field Extension · IEEE Trans. Image Process. 2016
Image and video processing
feature transform		 0.212016
Antipodally Invariant Metrics for Fast Regression-Based Super-Resolution · IEEE Trans. Image Process. 2016
Image and video processing › image restoration
image deblurring		 0.212016
Non-Parametric Blur Map Regression for Depth of Field Extension · IEEE Trans. Image Process. 2016
Image and video processing
image representation		 0.212016
Antipodally Invariant Metrics for Fast Regression-Based Super-Resolution · IEEE Trans. Image Process. 2016
Image and video processing › super-resolution
image super-resolution		 0.212015
Naive Bayes Super-Resolution Forest · ICCV 2015
Image and video processing › super-resolution
learning-based super-resolution		 0.212015
Naive Bayes Super-Resolution Forest · ICCV 2015
Machine learning › Generative modeling
3d generative model		 0.212023
Objaverse: A Universe of Annotated 3D Objects · CVPR 2023
Computer vision › Segmentation and scene understanding › semantic segmentation
open-vocabulary segmentation		 0.212023
Objaverse: A Universe of Annotated 3D Objects · CVPR 2023
Virtual and augmented reality › immersive video
free-viewpoint video		 0.212013
Multi-View Video Representation Based on Fast Monte Carlo Surface Reconstruction · IEEE Trans. Image Process. 2013
Geometric modeling and processing
surface reconstruction		 0.212013
Multi-View Video Representation Based on Fast Monte Carlo Surface Reconstruction · IEEE Trans. Image Process. 2013
Multimedia systems and quality of experience › 3d media
multi-view video		 0.012013
Multi-View Video Representation Based on Fast Monte Carlo Surface Reconstruction · IEEE Trans. Image Process. 2013

Methods — techniques the papers use, named apart from their topics

transformer policy · 0.8transformer architecture · 0.8shortest-path planning · 0.8large-scale pretraining · 0.8image augmentation · 0.8scene graph · 0.7robustness analysis · 0.7generative modeling · 0.7contrastive learning · 0.7neural model · 0.6local naive bayes formulation · 0.4symmetry distance · 0.2spherical hashing · 0.2regression tree fields · 0.2point spread function estimation · 0.2plenoptic camera · 0.2piecewise linear regression · 0.2iterative back-projection · 0.2
YearPublicationVenuePosition
2024 SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World
abstract
Reinforcement learning (RL) with dense rewards and imitation learning (IL) with human-generated trajectories are the most widely used approaches for training modern embodied agents. RL requires extensive reward shaping and auxiliary losses and is often too slow and ineffective for long-horizon tasks. While IL with human supervision is effective, collecting human trajectories at scale is extremely expensive. In this work, we show that imitating shortest-path planners in simulation produces agents that, given a language instruction, can proficiently navigate, ex-plore, and manipulate objects in both simulation and in the real world using only RGB sensors (no depth map or GPS coordinates). This surprising result is enabled by our end-to-end, transformer-based, Spocarchitecture, power-ful visual encoders paired with extensive image augmentation, and the dramatic scale and diversity of our training data: millions of frames of shortest-path-expert trajectories collected inside approximately 200,000 procedu-rally generated houses containing 40,000 unique 3D as-sets. Our models, data, training code, and newly proposed 10-task benchmarking suite Choresare available in spoc-robot.github.io.
Kiana Ehsani, Tanmay Gupta, Rose Hendrix, Jordi Salvador, Luca Weihs, Kuo-Hao Zeng, Kunal Pratap Singh, Yejin Kim 0003, Winson Han, Alvaro Herrasti, Ranjay Krishna, Dustin Schwenk, Eli VanderBilt, Aniruddha Kembhavi
 CVPR4
2024 Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration
abstract
Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x.github.io.
Abigail O'Neill, Abhiram Maddukuri, Abhishek Gupta 0004, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew E. Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu 0003, Charlotte Le, Chelsea Finn, Chen Wang 0053, Chenfeng Xu, Cheng Chi 0001, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Drieß, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Paul Foster, Fangchen Liu, Federico Ceola, Fei Xia 0002, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su 0001, Haoshu Fang, Henghui Bao, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim 0001, Jaimyn Drake, Jan Peters 0001, Jan Schneider 0007, Jasmine Hsu, Jeannette Bohg, Jeffrey T. Bingham, Jensen Gao, Jiaheng Hu, Jiajun Wu 0001, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan 0001, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Kenneth Y. Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Zhang 0002, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Yunliang Chen 0001, Lerrel Pinto, Li Fei-Fei 0001, Liam Tan, Linxi Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang 0003, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma 0001, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen 0001, Nicolas Heess, Nikhil J. Joshi, Niko Sünderhauf, Norman Di Palo, Nur Muhammad Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R. Sanketi, Patrick Tree Miller, Patrick Yin, Paul Wohlhart, Peng Xu 0010, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Rafael Rafailov, Ria Doshi, Roberto Martin Martin, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante-Gomez, Sean Kirmani, Sergey Levine, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham D. Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair 0003, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wolfram Burgard, Xiaolong Wang 0004, Xinghao Zhu, Xinyang Geng, Liangwei Xu, Yecheng Jason Ma 0001, Yejin Kim 0003, Yevgen Chebotar, Yilin Wu 0003, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang 0001, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zichen Jeff Cui, Zichen Zhang 0016, Zipeng Lin
 ICRA115
2023 Objaverse: A Universe of Annotated 3D Objects
abstract
Massive data corpora like WebText, Wikipedia, Conceptual Captions, WebImageText, and LAION have propelled recent dramatic progress in AI. Large neural models trained on such datasets produce impressive results and top many of today's benchmarks. A notable omisslion within this family of large-scale datasets is 3D data. Despite considerable interest and potential applications in 3D vision, datasets of high-fidelity 3D models continue to be mid-sized with limited diversity of object categories. Addressing this gap, we present Objaverse 1.0, a large dataset of objects with 800K + (and growing) 3D models with descriptive captions, tags, and animations. Objaverse improves upon present day 3D repositories in terms of scale, number of categories, and in the visual diversity of instances within a category. We demonstrate the large potential of Objaverse via four diverse applications: training generative 3D models, improving tail category segmentation on the LVIS benchmark, training open-vocabulary object-navigation models for Embodied AI, and creating a new benchmark for robustness analysis of vision models. Objaverse can open new directions for research and enable new applications across the field of AI.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, Ali Farhadi
 CVPR3
2023 Scene Graph Contrastive Learning for Embodied Navigation
abstract
Training effective embodied AI agents often involves expert imitation, specialized components such as maps, or leveraging additional sensors for depth and localization. Another approach is to use neural architectures alongside self-supervised objectives which encourage better representation learning. However, in practice, there are few guarantees that these self-supervised objectives encode task-relevant information. We propose the Scene Graph Contrastive (SGC) loss, which uses scene graphs as training-only supervisory signals. The SGC loss does away with explicit graph decoding and instead uses contrastive learning to align an agent’s representation with a rich graphical encoding of its environment. The SGC loss is simple to implement and encourages representations that encode objects’ semantics, relationships, and history. By using the SGC loss, we attain gains on three embodied tasks: Object Navigation, Multi-Object Navigation, and Arm Point Navigation. Finally, we present studies and analyses which demonstrate the ability of our trained representation to encode semantic cues about the environment.
Kunal Pratap Singh, Jordi Salvador, Luca Weihs, Aniruddha Kembhavi
 ICCV2
2023 Towards Disturbance-Free Visual Mobile Manipulation
abstract
Deep reinforcement learning has shown promising results on an abundance of robotic tasks in simulation, including visual navigation and manipulation. Prior work generally aims to build embodied agents that solve their assigned tasks as quickly as possible, while largely ignoring the problems caused by collision with objects during interaction. This lack of prioritization is understandable: there is no inherent cost in breaking virtual objects. As a result, "well-trained" agents frequently collide with objects before achieving their primary goals, a behavior that would be catastrophic in the real world. In this paper, we study the problem of training agents to complete the task of visual mobile manipulation in the ManipulaTHOR environment while avoiding unnecessary collision (disturbance) with objects. We formulate disturbance avoidance as a penalty term in the reward function, but find that directly training with such penalized rewards often results in agents being unable to escape poor local optima. Instead, we propose a two-stage training curriculum where an agent is first allowed to freely explore and build basic competencies without penalization, after which a disturbance penalty is introduced to refine the agent’s behavior. Results on testing scenes show that our curriculum not only avoids these poor local optima, but also leads to 10% absolute gains in success rate without disturbance, compared to our state-of-the-art baselines. Moreover, our curriculum is significantly more performant than a safe RL algorithm that casts collision avoidance as a constraint. Finally, we propose a novel disturbance-prediction auxiliary task that accelerates learning.1
Tianwei Ni, Kiana Ehsani, Luca Weihs, Jordi Salvador
 WACV4
2022 🏘️ ProcTHOR: Large-Scale Embodied AI Using Procedural Generation
abstract
Massive datasets and high-capacity models have driven many recent advancements in computer vision and natural language understanding. This work presents a platform to enable similar success stories in Embodied AI. We propose ProcTHOR, a framework for procedural generation of Embodied AI environments. ProcTHOR enables us to sample arbitrarily large datasets of diverse, interactive, customizable, and performant virtual environments to train and evaluate embodied agents across navigation, interaction, and manipulation tasks. We demonstrate the power and potential of ProcTHOR via a sample of 10,000 generated houses and a simple neural model. Models trained using only RGB images on ProcTHOR, with no explicit mapping and no human task supervision produce state-of-the-art results across 6 embodied AI benchmarks for navigation, rearrangement, and arm manipulation, including the presently running Habitat 2022, AI2-THOR Rearrangement 2022, and RoboTHOR challenges. We also demonstrate strong 0-shot results on these benchmarks, via pre-training on ProcTHOR with no fine-tuning on the downstream benchmark, often beating previous state-of-the-art systems that access the downstream training data.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, Roozbeh Mottaghi
 NeurIPS6
2021 Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text
abstract
Christopher Clark, Jordi Salvador, Dustin Schwenk, Derrick Bonafilia, Mark Yatskar, Eric Kolve, Alvaro Herrasti, Jonghyun Choi, Sachin Mehta, Sam Skjonsberg, Carissa Schoenick, Aaron Sarnat, Hannaneh Hajishirzi, Aniruddha Kembhavi, Oren Etzioni, Ali Farhadi. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.
Jordi Salvador, Dustin Schwenk, Derrick Bonafilia, Mark Yatskar, Eric Kolve, Alvaro Herrasti, Sachin Mehta, Sam Skjonsberg, Carissa Schoenick, Aaron Sarnat, Hannaneh Hajishirzi, Aniruddha Kembhavi, Oren Etzioni, Ali Farhadi
 EMNLP (1)2
2021 Bridging the Imitation Gap by Adaptive Insubordination
abstract
In practice, imitation learning is preferred over pure reinforcement learning whenever it is possible to design a teaching agent to provide expert supervision. However, we show that when the teaching agent makes decisions with access to privileged information that is unavailable to the student, this information is marginalized during imitation learning, resulting in an "imitation gap" and, potentially, poor results. Prior work bridges this gap via a progression from imitation learning to reinforcement learning. While often successful, gradual progression fails for tasks that require frequent switches between exploration and memorization. To better address these tasks and alleviate the imitation gap we propose 'Adaptive Insubordination' (ADVISOR). ADVISOR dynamically weights imitation and reward-based reinforcement learning losses during training, enabling on-the-fly switching between imitation and exploration. On a suite of challenging tasks set within gridworlds, multi-agent particle environments, and high-fidelity 3D simulators, we show that on-the-fly switching with ADVISOR outperforms pure imitation, pure reinforcement learning, as well as their sequential and parallel combinations.
Luca Weihs, Unnat Jain, Iou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, Alexander G. Schwing
 NeurIPS4
2020 RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
abstract
Visual recognition ecosystems (e.g. ImageNet, Pascal, COCO) have undeniably played a prevailing role in the evolution of modern computer vision. We argue that interactive and embodied visual AI has reached a stage of development similar to visual recognition prior to the advent of these ecosystems. Recently, various synthetic environments have been introduced to facilitate research in embodied AI. Notwithstanding this progress, the crucial question of how well models trained in simulation generalize to reality has remained largely unanswered. The creation of a comparable ecosystem for simulation-to-real embodied AI presents many challenges: (1) the inherently interactive nature of the problem, (2) the need for tight alignments between real and simulated worlds, (3) the difficulty of replicating physical conditions for repeatable experiments, (4) and the associated cost. In this paper, we introduce RoboTHOR to democratize research in interactive and embodied visual AI. RoboTHOR offers a framework of simulated environments paired with physical counterparts to systematically explore and overcome the challenges of simulation-to-real transfer, and a platform where researchers across the globe can remotely test their embodied models in the physical world. As a first benchmark, our experiments show there exists a significant gap between the performance of models trained in simulation when they are tested in both simulations and their carefully constructed physical analogs. We hope that RoboTHOR will spur the next stage of evolution in embodied computer vision.
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, Luca Weihs, Mark Yatskar, Ali Farhadi
 CVPR7
2020 Learning About Objects by Learning to Interact with Them
abstract
Much of the remarkable progress in computer vision has been focused around fully supervised learning mechanisms relying on highly curated datasets for a variety of tasks. In contrast, humans often learn about their world with little to no external supervision. Taking inspiration from infants learning from their environment through play and interaction, we present a computational framework to discover objects and learn their physical properties along this paradigm of Learning from Interaction. Our agent, when placed within the near photo-realistic and physics-enabled AI2-THOR environment, interacts with its world and learns about objects, their geometric extents and relative masses, without any external guidance. Our experiments reveal that this agent learns efficiently and effectively; not just for objects it has interacted with before, but also for novel instances from seen categories as well as novel object categories.
Martin Lohmann, Jordi Salvador, Aniruddha Kembhavi, Roozbeh Mottaghi
 NeurIPS2
2016 PSyCo: Manifold Span Reduction for Super Resolution
abstract
The main challenge in Super Resolution (SR) is to discover the mapping between the low-and high-resolution manifolds of image patches, a complex ill-posed problem which has recently been addressed through piecewise linear regression with promising results. In this paper we present a novel regression-based SR algorithm that benefits from an extended knowledge of the structure of both manifolds. We propose a transform that collapses the 16 variations induced from the dihedral group of transforms (i.e. rotations, vertical and horizontal reflections) and antipodality (i.e. diametrically opposed points in the unitary sphere) into a single primitive. The key idea of our transform is to study the different dihedral elements as a group of symmetries within the high-dimensional manifold. We obtain the respective set of mirror-symmetry axes by means of a frequency analysis of the dihedral elements, and we use them to collapse the redundant variability through a modified symmetry distance. The experimental validation of our algorithm shows the effectiveness of our approach, which obtains competitive quality with a dictionary of as little as 32 atoms (reducing other methods' dictionaries by at least a factor of 32) and further pushing the state-of-the-art with a 1024 atoms dictionary.
Eduardo Pérez-Pellitero, Jordi Salvador, Javier Ruiz Hidalgo, Bodo Rosenhahn
 CVPR2
2016 Half hypersphere confinement for piecewise linear regression
abstract
Recent research in piecewise linear regression for Super-Resolution has shown the positive impact of training regressors with densely populated clusters whose datapoints are tight in the Euclidean space. In this paper we further research how to improve the locality condition during the training of regressors and how to better select them during testing time. We study the characteristics of the metrics best suited for the piecewise regression algorithms, in which comparisons are usually made between normalized vectors that lie on the unitary hypersphere. Even though Euclidean distance has been widely used for this purpose, it is suboptimal since it does not handle antipodal points (i.e. diametrically opposite points) properly, as vectors with same module and angle but opposite directions are, for linear regression purposes, identical. Therefore, we propose the usage of antipodally invariant metrics and introduce the Half Hypersphere Confinement (HHC), a fast alternative to Multidimensional Scaling (MDS) that allows to map antipodally invariant distances in the Euclidean space with very little approximation error By doing so, we enable the usage of fast search structures based on Euclidean distances without undermining their speed gains with complex distance transformations. The performance of our method, which we named HHC Regression (HHCR), applied to SuperResolution (SR) improves both in quality (PSNR) and it is faster than any other state-of-the-art method. Additionally, under an application-agnostic interpretation of our regression framework, we also test our algorithm for denoising and depth upscaling with promising results.
Eduardo Pérez-Pellitero, Jordi Salvador, Javier Ruiz Hidalgo, Bodo Rosenhahn
 WACV2
2016 Non-Parametric Blur Map Regression for Depth of Field Extension
abstract
Real camera systems have a limited depth of field (DOF) which may cause an image to be degraded due to visible misfocus or too shallow DOF. In this paper, we present a blind deblurring pipeline able to restore such images by slightly extending their DOF and recovering sharpness in regions slightly out of focus. To address this severely ill-posed problem, our algorithm relies first on the estimation of the spatially varying defocus blur. Drawing on local frequency image features, a machine learning approach based on the recently introduced regression tree fields is used to train a model able to regress a coherent defocus blur map of the image, labeling each pixel by the scale of a defocus point spread function. A non-blind spatially varying deblurring algorithm is then used to properly extend the DOF of the image. The good performance of our algorithm is assessed both quantitatively, using realistic ground truth data obtained with a novel approach based on a plenoptic camera, and qualitatively with real images.
Laurent D'Andres, Jordi Salvador, Axel Kochale, Sabine Süsstrunk
 IEEE Trans. Image Process.2
2016 Antipodally Invariant Metrics for Fast Regression-Based Super-Resolution
abstract
Dictionary-based super-resolution (SR) algorithms usually select dictionary atoms based on the distance or similarity metrics. Although the optimal selection of the nearest neighbors is of central importance for such methods, the impact of using proper metrics for SR has been overlooked in literature, mainly due to the vast usage of Euclidean distance. In this paper, we present a very fast regression-based algorithm, which builds on the densely populated anchored neighborhoods and sublinear search structures. We perform a study of the nature of the features commonly used for SR, observing that those features usually lie in the unitary hypersphere, where every point has a diametrically opposite one, i.e., its antipode, with same module and angle, but the opposite direction. Even though, we validate the benefits of using antipodally invariant metrics, most of the binary splits use Euclidean distance, which does not handle antipodes optimally. In order to benefit from both the worlds, we propose a simple yet effective antipodally invariant transform that can be easily included in the Euclidean distance calculation. We modify the original spherical hashing algorithm with this metric in our antipodally invariant spherical hashing scheme, obtaining the same performance as a pure antipodally invariant metric. We round up our contributions with a novel feature transform that obtains a better coarse approximation of the input image thanks to iterative backprojection. The performance of our method, which we named antipodally invariant SR, improves quality (Peak Signal to Noise Ratio) and it is faster than any other state-of-the-art method.
Eduardo Pérez-Pellitero, Jordi Salvador, Javier Ruiz Hidalgo, Bodo Rosenhahn
 IEEE Trans. Image Process.2
2015 Naive Bayes Super-Resolution Forest
abstract
This paper presents a fast, high-performance method for super resolution with external learning. The first contribution leading to the excellent performance is a bimodal tree for clustering, which successfully exploits the antipodal invariance of the coarse-to-high-res mapping of natural image patches and provides scalability to finer partitions of the underlying coarse patch space. During training an ensemble of such bimodal trees is computed, providing different linearizations of the mapping. The second and main contribution is a fast inference algorithm, which selects the most suitable mapping function within the tree ensemble for each patch by adopting a Local Naive Bayes formulation. The experimental validation shows promising scalability properties that reflect the suitability of the proposed model, which may also be generalized to other tasks. The resulting method is beyond one order of magnitude faster and performs objectively and subjectively better than the current state of the art.
Jordi Salvador, Eduardo Pérez-Pellitero
 ICCV1
2014 Fast Super-Resolution via Dense Local Training and Inverse Regressor Search
Eduardo Pérez-Pellitero, Jordi Salvador, Iban Torres-Xirau, Javier Ruiz Hidalgo, Bodo Rosenhahn
 ACCV (3)2
2014 Fast Approximate Nearest-Neighbor Field by Cascaded Spherical Hashing
Iban Torres-Xirau, Jordi Salvador, Eduardo Pérez-Pellitero
 ACCV (4)2
2014 Robust single-image super-resolution using cross-scale self-similarity
abstract
We present a noise-aware single-image super-resolution (SI-SR) algorithm, which automatically cancels additive noise while adding detail learned from lower-resolution scales. In contrast with most SI-SR techniques, we do not assume the input image to be a clean source of examples. Instead, we adapt the recent and efficient in-place cross-scale self-similarity prior for both learning fine detail examples and reducing image noise. Our experiments show a promising performance, despite the relatively simple algorithm. Both objective evaluations and subjective validations show clear quality improvements when upscaling noisy images.
Jordi Salvador, Eduardo Pérez-Pellitero, Axel Kochale
 ICIP1
2013 Bayesian region selection for adaptive dictionary-based Super-Resolution
abstract
The performance of dictionary-based super-resolution (SR) strongly depends on the\ncontents of the training dataset. Nevertheless, many dictionary-based SR methods randomly select patches from of a larger set of training images to build their dictionaries\n[\n8\n,\n14\n,\n19\n,\n20\n], thus relying on patches being diverse enough. This paper describes\na dictionary building method for SR based on adaptively selecting an optimal subset of\npatches out of the training images. Each training image is divided into sub-image entities,\nnamed regions, of such a size that texture consistency is preserved and high-frequency\n(HF) energy is present. For each input patch to super-resolve, the best-fitting region is\nfound through a Bayesian selection. In order to handle the high number of regions in\nthe training dataset, a local Naive Bayes Nearest Neighbor (NBNN) approach is used.\nTrained with this adapted subset of patches, sparse coding SR is applied to recover the\nhigh-resolution image. Experimental results demonstrate that using our adaptive algo-\nrithm produces an improvement in SR performance with respect to non-adaptive training.
Eduardo Pérez-Pellitero, Jordi Salvador, Javier Ruiz Hidalgo, Bodo Rosenhahn
 BMVC2
2013 A Bayesian approach for natural image denoising
abstract
This article presents a new method for estimating the latent noiseless version of an observed image corrupted by additive noise. This method stems from classical models in parametric denoising and extends them by modeling the likelihood term, estimating adaptive image priors and automatically choosing an adaptive equivalent to the typically hand-tuned regularization constant. The proposed method introduces a possible path to overcome the limitations of current parametric denoising algorithms and provides a competitive alternative to powerful non-parametric ones. The experimental results show how our method adapts better to different noise types than state-of-the-art parametric and non-parametric algorithms.
Jordi Salvador, Malte Borsum, Axel Kochale
 ICIP1
2013 Fast single-image super-resolution with filter selection
abstract
This paper presents a new method for estimating a super-resolved version of an observed image by exploiting cross-scale self-similarity. We extend prior work on single-image super-resolution by introducing an adaptive selection of the best fitting upscaling and analysis filters for example learning. This selection is based on local error measurements obtained by using each filter with every image patch, and contrasts with the common approach of a constant metric in both dictionary-based and internal learning super-resolution. The proposed method is suitable for interactive applications, offering low computational load and a parallelizable design that allows straight-forward GPU implementations. Experimental results also show how our method generalizes better to different datasets than dictionary-based super-resolution and comparably to internal learning with adaptive post-processing.
Jordi Salvador, Eduardo Pérez-Pellitero, Axel Kochale
 ICIP1
2013 Patch-based spatio-temporal super-resolution for video with non-rigid motion
Jordi Salvador, Axel Kochale, Siegfried Schweidler
 Signal Process. Image Commun.1
2013 Multi-View Video Representation Based on Fast Monte Carlo Surface Reconstruction
abstract
This paper provides an alternative solution to the costly representation of multi-view video data, which can be used for both rendering and scene analyses. Initially, a new efficient Monte Carlo discrete surface reconstruction method for foreground objects with static background is presented, which outperforms volumetric techniques and is suitable for GPU environments. Some extensions are also presented, which allow a speeding up of the reconstruction by exploiting multi-resolution and temporal correlations. Then, a fast meshing algorithm is applied, which allows interpolating a continuous surface from the discrete reconstructed points. As shown by the experimental results, the original video frames can be approximated with high accuracy by projecting the reconstructed foreground objects onto the original viewpoints. Furthermore, the reconstructed scene can be easily projected onto any desired virtual viewpoint, thus simplifying the design of free-viewpoint video applications. In our experimental results, we show that our techniques for reconstruction and meshing compare favorably with the state-of-the-art, and we also introduce a rule-of-thumb for effective application of the method with a good quality versus representation cost trade-off.
Jordi Salvador, Josep R. Casas
 IEEE Trans. Image Process.1
2012 Variational reconstruction and restoration for video Super-Resolution
Jordi Salvador, Daniel Rivero 0002, Axel Kochale, Javier Ruiz Hidalgo
 ICPR1
2011 Joint multi-view foreground segmentation and 3D reconstruction with tolerance loop
abstract
In this paper we present a novel foreground segmentation and 3D reconstruction system for multi-view scenarios. The system achieves correct 3D object reconstruction even when foreground segmentation presents critical misses in some of the views. We introduce the spatial redundancy of the multi-view data into the foreground segmentation process by combining segmentation and the 3D reconstruction in a two steps workflow. First, the segmentation of the objects in each view uses a monocular, region-based foreground segmentation in a MAP-MRF framework for foreground, background and shadow classes. Next, we compute an iterative volume reconstruction in a 3D tolerance loop, obtaining an iteratively enhanced SfS volume. Foreground segmentation is improved by updating the foreground model of each view at each iteration. The results presented in this paper show the improved foreground segmentation and the reduction of errors in the reconstruction of the volume.
Jaime Gallego, Jordi Salvador, Josep R. Casas, Montse Pardàs
 ICIP2
2010 Photo-consistent surfaces from a sparse set of viewpoints
abstract
We address the problem of reconstructing 3D shapes from color data available in a sparse set of views from all directions of a scene. As an advantage when compared to multiview stereo approaches, our method is able to reconstruct object surfaces from a small number of views in wide-baseline setups. This introduces a trade-off between reconstruction accuracy and spatial coverage. The proposed algorithm obtains candidate surface points using a photo-consistency test and restricting the analysis to foreground pixels. The final surface points are extracted by iteratively carving away candidate points that are not photo-consistent with the complete multiview set. Finally, a surface patch is obtained from each colored surface point by estimating its orientation. Experimental results reflect the validity of the approach by comparing it to a voxelized implementation.
Jordi Salvador, Josep R. Casas
 ICIP1
2010 Joint estimation of shape and motion from silhouettes
abstract
This paper presents a fast algorithm for recovering shape and motion of dynamic scenes from sequences of silhouettes in calibrated multi-camera settings in wide-baseline configurations. The proposed method captures the static shape on the first frame by deforming an arbitrary set of points located on an initial surface enclosing the actual scene. Then, the initial 3D shape is propagated in subsequent frames by quickly deforming the initial static shape in order to fit the available silhouettes. The resulting points lie on the surface of the Visual Hull. Shape and motion descriptions are jointly obtained by the dynamic procedure that adapts existing surfaces to the set of silhouettes at each new time instant. Experimental results show how the proposed dynamic method obtains silhouette-consistent shape estimates -with the same accuracy than a static approach- and dense motion estimates with very low usage of computational resources, by exploiting both spatial and temporal redundancies.
Jordi Salvador, Josep R. Casas
 ICIP1
2006 Image-adapted voxelization in multicamera settings
abstract
This paper presents a new procedure to explore 3D for visual analysis in multicamera environments. We propose a 3D geometry to sample (voxelize) the space when checking the consistency of the analyzed features in the multiple views. Contrary to regular voxelization, the proposed geometry is irregular in 3D, but becomes regular once projected onto camera images. The aim is to better exploit the redundancy and the enriched information provided in multiview frameworks at the analysis stage. This is accomplished by balancing the usage of the available data (i.e. captured pixels) across the multiple cameras, instead of focusing in a regular sampling of the 3D space from which we do not have direct data. An efficient recursive algorithm that uses the proposed geometry is outlined, and the experimental results reflect higher accuracy than regular voxelization with equivalent restrictions for the chosen multiview analysis applications
Jordi Salvador, Josep R. Casas
 MMSP1