Abstract Achieving physically consistent image editing remains a significant challenge in computer vision. Existing image editing methods typically rely on neural networks, which struggle to accurately handle shadows and refractions. Conversely, physics-based inverse rendering often requires multi-view optimization, limiting its practicality in single-image scenarios. In this paper, we propose Materialist , a neural-initialized physically based rendering pipeline for single-image inverse rendering. Unlike previous hybrid methods that use physics to guide neural generation, our method leverages neural networks to predict initial material properties, which are then rigorously optimized via progressive differentiable rendering. Our approach enables a range of applications, including material editing, object insertion, and relighting, while also introducing an effective method for editing material transparency via ray-traced refraction without requiring full scene geometry. Furthermore, our envmap estimation method also achieves competitive performance, further enhancing the accuracy of image editing task. Experiments demonstrate strong performance across synthetic and real-world datasets, excelling even on challenging out-of-domain images.

Minimum cut/maximum flow (min-cut/max-flow) algorithms solve a variety of problems in computer vision and thus significant effort has been put into developing fast min-cut/max-flow algorithms. As a result, it is difficult to choose an ideal algorithm for a given problem. Furthermore, parallel algorithms have not been thoroughly compared. In this paper, we evaluate the state-of-the-art serial and parallel min-cut/max-flow algorithms on the largest set of computer vision problems yet. We focus on generic algorithms, i.e., for unstructured graphs, but also compare with the specialized GridCut implementation. When applicable, GridCut performs best. Otherwise, the two pseudoflow algorithms, Hochbaum pseudoflow and excesses incremental breadth first search, achieves the overall best performance. The most memory efficient implementation tested is the Boykov-Kolmogorov algorithm. Amongst generic parallel algorithms, we find the bottom-up merging approach by Liu and Sun to be best, but no method is dominant. Of the generic parallel methods, only the parallel preflow push-relabel algorithm is able to efficiently scale with many processors across problem sizes, and no generic parallel method consistently outperforms serial algorithms. Finally, we provide and evaluate strategies for algorithm selection to obtain good expected performance. We make our dataset and implementations publicly available for further research.

We introduce a parallel version of the Quadratic Pseudo-Boolean Optimization (QPBO) algorithm for solving binary optimization tasks, such as image segmentation. The original QPBO implementation by Kolmogorov and Rother relies on the Boykov-Kolmogorov (BK) maxflow/mincut algorithm and performs well for many image analysis tasks. However, the serial nature of their QPBO algorithm results in poor utilization of modern hardware. By redesigning the QPBO algorithm to work with parallel maxflow/mincut algorithms, we significantly reduce solve time of large optimization tasks. We compare our parallel QPBO implementation to other state-of-the-art solvers and benchmark them on two large segmentation tasks and a substantial set of small segmentation tasks. The results show that our parallel QPBO algorithm is over 20 times faster than the serial QPBO algorithm on the large tasks and over three times faster for the majority of the small tasks. Although we focus on image segmentation, our algorithm is generic and can be used for any QPBO problem. Our implementation and experimental results are available at DOI: 10.5281/zenodo.5201620

We introduce the novel concept of a Sparse Layered Graph (SLG) for s-t graph cut segmentation of image data. The concept is based on the widely used Ishikawa layered technique for multi-object segmentation, which allows explicit object interactions, such as containment and exclusion with margins. However, the spatial complexity of the Ishikawa technique limits its use for many segmentation problems. To solve this issue, we formulate a general method for adding containment and exclusion interaction constraints to layered graphs. Given some prior knowledge, we can create a SLG, which is often orders of magnitude smaller than traditional Ishikawa graphs, with identical segmentation results. This allows us to solve many problems that could previously not be solved using general graph cut algorithms. We then propose three algorithms for further reducing the spatial complexity of SLGs, by using ordered multi-column graphs. In our experiments, we show that SLGs, and in particular ordered multi-column SLGs, can produce high-quality segmentation results using extremely simple data terms. We also show the scalability of ordered multi-column SLGs, by segmenting a high-resolution volume with several hundred interacting objects.

We propose a deformable model for manually initialized segmentation of images, which may contain both textured and non-textured regions. Image segments and segment boundaries are represented using a deformable triangle mesh, providing all advantages of an explicit geometry representation, but allowing for adaptive topology. Deformation forces are computed using a probabilistic model of local self-similarity, based on clustering of image patches. Both our curve representation and our similarity model naturally support multi-label segmentation. We demonstrate the properties of our approach on a number of natural color images as well as composed textured images.

Automated detection of cancer metastases in lymph nodes has the potential to improve the assessment of prognosis for patients. To enable fair comparison between the algorithms for this purpose, we set up the CAMELYON17 challenge in conjunction with the IEEE International Symposium on Biomedical Imaging 2017 Conference in Melbourne. Over 300 participants registered on the challenge website, of which 23 teams submitted a total of 37 algorithms before the initial deadline. Participants were provided with 899 whole-slide images (WSIs) for developing their algorithms. The developed algorithms were evaluated based on the test set encompassing 100 patients and 500 WSIs. The evaluation metric used was a quadratic weighted Cohen's kappa. We discuss the algorithmic details of the 10 best pre-conference and two post-conference submissions. All these participants used convolutional neural networks in combination with pre- and postprocessing steps. Algorithms differed mostly in neural network architecture, training strategy, and pre- and postprocessing methodology. Overall, the kappa metric ranged from 0.89 to -0.13 across all submissions. The best results were obtained with pre-trained architectures such as ResNet. Confusion matrix analysis revealed that all participants struggled with reliably identifying isolated tumor cells, the smallest type of metastasis, with detection rates below 40%. Qualitative inspection of the results of the top participants showed categories of false positives, such as nerves or contamination, which could be targeted for further optimization. Last, we show that simple combinations of the top algorithms result in higher kappa metric values than any algorithm individually, with 0.93 for the best combination.

We present a method for virtual unrolling of a thin rolled object. From a volumetric image of the rolled object we obtain a flat image of the object's surface, which allows visual inspection of the object and has a number of applications. Our method exploits the geometric constrains of the problem and detects a single rolled surface. For surface detection we adapt a solution to an optimal net surface problem, previously used for terrain-like and tubular surfaces. We present our approach on an example of a rolled sheet of microelectronic, which has a layer of flexible polymer substrate and a thin metal layer lithographically coated onto the polymer. Our approach is automatic and robust. The unrolled image is undistorted, and the surface structures may be accurately quantified making our approach a good candidate for an industrial application of virtual unrolling.

Visual cues like texture, color and context make objects appear distinct from the surroundings, even without gradients between regions. Texture-rich objects are often difficult to segment because algorithms need advanced features which are unique for the image. In this paper we suggest a method for image segmentation that operates without training data. Our method is based on a probabilistic dictionary of image patches coupled with a deformable model inspired by snakes and active contours without edges. We separate the image into two classes based on the information provided by the evolving curve, which moves according to the probabilistic information obtained from the dictionary. Initially, the image patches are assigned to the nearest dictionary element, where the image is sampled at each pixel such that patches overlap. The curve divides the image into an inside and an outside region allowing us to estimate the pixel-wise probability of the dictionary elements. In each iteration we evolve the curve and update the probabilities, which merges similar texture patterns and pulls dissimilar patterns apart. We experimentally evaluate our approach, and show how textured objects are precisely segmented without any prior assumptions about image features. In addition, a texture probability image is obtained.

In this work we propose a novel variational method that we intend to use for estimating non-rigid texture deformation. The method is able to capture variation in gray scale images with respect to the geometry of its features. Accurate localization of features in the presence of unknown deformations is a crucial property for texture characterization. Our experimental evaluations demonstrate that accounting for geometry of features in texture images leads to significant improvements in localization of these features, when textures undergo geometrical transformations. In addition, feature descriptors using geometrical total variation energies discriminate between various regular textures with accuracy comparable to SIFT descriptors, while reduced dimensionality of TVG descriptor yields significant improvements over SIFT in terms of retrieval time.