Controllability plays a crucial role in the practical applications of 3D indoor scene synthesis. Existing works either allow rough language-based control, that is convenient but lacks fine-grained scene customization, or employ graph-based control, which offers better controllability but demands considerable knowledge for the cumbersome graph design process. To address these challenges, we present FreeScene, a user-friendly framework that enables both convenient and effective control for indoor scene synthesis. Specifically, FreeScene supports free-form user inputs including text description and/or reference images, allowing users to express versatile design intentions. The user inputs are adequately analyzed and integrated into a graph representation by a VLM-based Graph Designer. We then propose MG-DiT, a Mixed Graph Diffusion Transformer, which performs graph-aware denoising to enhance scene generation. Our MG-DiT not only excels at preserving graph structure but also offers broad applicability to various tasks, including, but not limited to, text-to-scene, graph-to-scene, and rearrangement, all within a single model. Extensive experiments demonstrate that FreeScene provides an efficient and user-friendly solution that unifies text-based and graph-based scene synthesis, outperforming state-of-the-art methods in terms of both generation quality and controllability in a range of applications.

3D sketches are widely used for visually representing the 3D shape and structure of objects or scenes. However, the creation of 3D sketch often requires users to possess professional artistic skills. Existing research efforts primarily focus on enhancing the ability of interactive sketch generation in 3D virtual systems. In this work, we propose Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch by optimizing 3D parametric curves under various supervisions. Specifically, we perform perspective projection to render the 3D rational Bézier curves into 2D curves, which are subsequently converted to a 2D raster image via our customized differentiable rasterizer. Our framework bridges the domains of 3D sketch and raster image, achieving end-to-end optimization of 3D sketch through gradients computed in the 2D image domain. Our Diff3DS can enable a series of novel 3D sketch generation tasks, including text-to-3D sketch and image-to-3D sketch, supported by the popular distillation-based supervision, such as Score Distillation Sampling (SDS). Extensive experiments have yielded promising results and demonstrated the potential of our framework. Project: https://yiboz2001.github.io/Diff3DS/

Single-cell RNA sequencing (scRNA-seq) technology has made significant breakthroughs in genomics research in recent years, enabling in-depth analysis of gene expression profiles at the single-cell level. A key step in analyzing scRNA-seq data is clustering cells into subpopulations, which facilitates subsequent downstream analysis. Unfortunately, the rapid growth of scRNAseq data and the prevalence of dropout events pose challenges in cell type annotation. In this paper, we develop a hybrid model, Single-cell Multi-hop Autoencoder (scMAE), based on deep graph embedding clustering. This model combines multi-hop graph convolutional networks (GCNs) to capture cell neighbor information and uses deep graph convolutional networks to identify cell clusters. Additionally, the model integrates a zero-inflated negative binomial (ZINB) model into a topologically adaptive graph convolutional autoencoder to learn low-dimensional latent representations and applies Kullback-Leibler (KL) divergence for the clustering task. By jointly training on the adjacency matrix and gene expression data, the model simultaneously optimizes the clustering results and the generative model. Specifically, the loss function combines four components: adjacency matrix reconstruction loss, ZINB loss, generator loss, and clustering loss, and an alternating training strategy is employed to enhance clustering performance. Extensive experiments on 15 scRNA-seq datasets from different yet representative single-cell sequencing platforms demonstrate that scMAE outperforms various state-of-the-art clustering methods.

Developing computer-aided design (CAD) generation models has significantly enhanced design efficiency, facilitating innovation and transformation in the design industry. Existing methods typically require users to input prompts in a specific format, such as text descriptions or images, limiting their broader application in diverse scenarios. To address this limitation, we introduce FreeCAD, a user-friendly CAD generation framework that supports free-form inputs, including text descriptions and/or images, enabling users to express their design intentions more flexibly. Specifically, we propose a Large Language Models (LLMs)-based Text Translator, which effectively increases the success rate of generating CAD models by converting users' diversified requests for the same object into a unified expression. Additionally, the Multi-View Representation Fusion (MVRF) module enables the network to capture richer interaction information across views, facilitating the generation of more fine-grained CAD models. To support the training of FreeCAD, we construct a multimodal dataset RealCAD, comprising text, image, and CAD triplets, where the images are derived from the 3D printed products of CAD models. Extensive experiments demonstrate that FreeCAD consistently outperforms the existing state-of-the-art (SOTA) methods in multiple tasks.

The integration of embodied intelligence in indoor scene synthesis holds significant potential for future interior design applications. Nevertheless, prevailing methodologies for indoor scene synthesis predominantly adhere to data-driven learning paradigms. Despite achieving photorealistic 3D renderings through such approaches, current frameworks systematically neglect to incorporate agent-centric functional metrics essential for optimizing navigational topology and task-oriented interactivity in embodied AI systems like service robotics platforms or autonomous domestic assistants. For example, poorly arranged furniture may prevent robots from effectively interacting with the environment, and this issue cannot be fully resolved by merely introducing prior constraints. To fill this gap, we propose Nav2Scene, a novel plug-and-play fine-tuning mechanism that can be deployed on existing scene generators to enhance the suitability of generated scenes for efficient robot navigation. Specifically, we first introduce path planning score (PPS), which is defined based on the results of the path planning algorithm and can be used to evaluate the robot navigation suitability of a given scene. Then, we pre-compute the PPS of 3D scenes from existing datasets and train a ScoreNet to efficiently predict the PPS of the generated scenes. Finally, the predicted PPS is used to guide the fine-tuning of existing scene generators and produce indoor scenes with higher PPS, indicating improved suitability for robot navigation. We conduct experiments on the 3D-FRONT dataset for different tasks including scene generation, completion and re-arrangement. The results demonstrate that by incorporating our Nav2Scene mechanism, the fine-tuned scene generators can produce scenes with improved navigation compatibility for home robots, while maintaining superior or comparable performance in terms of scene quality and diversity.

With the rapid development of artificial intelligence, particularly the rise of deep learning, the importance of Explainable Artificial Intelligence has become increasingly prominent. Among its key techniques, counterfactual explanation plays a crucial role in understanding the decision-making mechanisms of opaque models. However, the high dimensionality and complex feature patterns of image data pose significant challenges for the task of generating counterfactuals for images. Existing literature has proposed various algorithms based on different assumptions, many of which rely on the existence of appropriate generative models. Some of these assumptions, particularly the assumption regarding the existence of generative models, may be overly stringent. To address this issue, this letter introduces a novel assumption-free image counterfactual generation algorithm, DFO-S, based on Score Matching and gradient-free optimization techniques. The proposed method achieves high-quality counterfactual generation without relying on generative models. Through extensive empirical analysis, we demonstrate the significant superiority of our method in terms of performance.

Optimization over permutations is typically an NP-hard problem that arises extensively in ranking, matching, tracking, etc. Birkhoff polytope-based relaxation methods have made significant advancements, particularly in penalty-free optimization and probabilistic inference. Relaxation onto the orthogonal group offers unique potential advantages such as a lower representation dimension and preservation of inner products; however, equally effective approaches remain unexplored. To bridge the gap, we present a temperature-controlled differentiable transformation that maps unconstrained vector space to the orthogonal group, where the temperature, in the limit, concentrates orthogonal matrices near permutation matrices. This transformation naturally implements a parameterization for the relaxation of permutation matrices, allowing for gradient-based optimization of problems involving permutations. Additionally, by deriving a re-parameterized gradient estimator, this transformation also provides efficient stochastic optimization over the latent permutations. Extensive experiments involving the optimization over permutation matrices validate the effectiveness of the proposed method.

Abstract Few-shot class-incremental learning (FSCIL) struggles to incrementally recognize novel classes from few examples without catastrophic forgetting of old classes or overfitting to new classes. We propose TLCE, which ensembles multiple pre-trained models to improve separation of novel and old classes. Specifically, we use episodic training to map images from old classes to quasi-orthogonal prototypes, which minimizes interference between old and new classes. Then, we incorporate the use of ensembling diverse pre-trained models to further tackle the challenge of data imbalance and enhance adaptation to novel classes. Extensive experiments on various datasets demonstrate that our transfer learning ensemble approach outperforms state-of-the-art FSCIL methods.

Out-of-distribution generalization is challenging for non-participating clients of federated learning under distribution shifts. A proven strategy is to explore those invariant relationships between input and target variables, working equally well for non-participating clients. However, learning invariant relationships is often in an explicit manner from data, representation, and distribution, which violates the federated principles of privacy-preserving and limited communication. In this paper, we propose FedIIR, which implicitly learns invariant relationships from parameter for out-of-distribution generalization, adhering to the above principles. Specifically, we utilize the prediction disagreement to quantify invariant relationships and implicitly reduce it through inter-client gradient alignment. Theoretically, we demonstrate the range of non-participating clients to which FedIIR is expected to generalize and present the convergence results for FedIIR in the massively distributed with limited communication. Extensive experiments show that FedIIR significantly outperforms relevant baselines in terms of out-of-distribution generalization of federated learning.

Regional facial image synthesis conditioned on a semantic mask has achieved great attention in the field of computational visual media. However, the appearances of different regions may be inconsistent with each other after performing regional editing. In this paper, we focus on harmonized regional style transfer for facial images. A multi-scale encoder is proposed for accurate style code extraction. The key part of our work is a multi-region style attention module. It adapts multiple regional style embeddings from a reference image to a target image, to generate a harmonious result. We also propose style mapping networks for multi-modal style synthesis. We further employ an invertible flow model which can serve as mapping network to fine-tune the style code by inverting the code to latent space. Experiments on three widely used face datasets were used to evaluate our model by transferring regional facial appearance between datasets. The results show that our model can reliably perform style transfer and multi-modal manipulation, generating output comparable to the state of the art.

Existing deep unfolding methods unroll an optimization algorithm with a fixed number of steps, and utilize convolutional neural networks (CNNs) to learn data-driven priors. However, their performance is limited for two main reasons. Firstly, priors learned in deep feature space need to be converted to the image space at each iteration step, which limits the depth of CNNs and prevents CNNs from exploiting contextual information. Secondly, existing methods only learn deep priors at the single full-resolution scale, so ignore the benefits of multi-scale context in dealing with high level noise. To address these issues, we explicitly consider the image denoising process in the deep feature space and propose the deep unfolding multi-scale regularizer network (DUMRN) for image denoising. The core of DUMRN is the feature-based denoising module (FDM) that directly removes noise in the deep feature space. In each FDM, we construct a multi-scale regularizer block to learn deep prior information from multi-resolution features. We build the DUMRN by stacking a sequence of FDMs and train it in an end-to-end manner. Experimental results on synthetic and real-world benchmarks demonstrate that DUMRN performs favorably compared to state-of-the-art methods.

Deep convolutional neural networks have recently been applied to improve the quality of low-light images and have achieved promising results. However, most existing methods cannot suppress noise during the enhancement process effectively, resulting in unknown artifacts and color distortions. In addition, these methods do not fully utilize illumination information and perform poorly under extremely low-light condition. To alleviate these problems, we propose theillumination guided attentive wavelet network(IGAWN) for low-light image enhancement (LLIE). Considering that the wavelet transform can separate high-frequency noise and desired low-frequency content effectively, we enhance low-light images in the frequency domain. By integrating attention mechanisms with wavelet transform, we develop the attentive wavelet transform to capture more important wavelet features, which enables the desired content to be enhanced and the redundant noise to be suppressed. To improve the image enhancement performance under extremely low-light environment, we extract illumination information from the input images and exploit it as the guidance for image enhancement through the frequency feature transform (FFT) layer. The proposed FFT layer generates frequency-aware affine transformation from the estimated illumination information, which can adaptively modulate the image features of different frequencies. Extensive experiments on synthetic and real-world datasets demonstrate that our IGAWN performs favorably against state-of-the-art LLIE methods.

Class activation map (CAM) has been widely studied for visual explanation of the internal working mechanism of convolutional neural networks. The key of existing CAM-based methods is to compute effective weights to combine activation maps in the target convolution layer. Existing gradient and score based weighting schemes have shown superiority in ensuring either the discriminability or faithfulness of the CAM, but they normally cannot excel in both properties. In this paper, we propose a novel CAM weighting scheme, named FD-CAM, to improve both the faithfulness and discriminability of the CAM-based CNN visual explanation. First, we improve the faithfulness and discriminability of the score-based weights by performing a grouped channel switching operation. Specifically, for each channel, we compute its similarity group and switch the group of channels on or off simultaneously to compute changes in the class prediction score as the weights. Then, we combine the improved score-based weights with the conventional gradient-based weights so that the discriminability of the final CAM can be further improved. We perform extensive comparisons with the state-of-the-art CAM algorithms. The quantitative and qualitative results show our FD-CAM can produce more faithful and more discriminative visual explanations of the CNNs. We also conduct experiments to verify the effectiveness of the proposed grouped channel switching and weight combination scheme on improving the results. Our code is available at https://github.com/crishhh1998/FD-CAM.

Recovering intrinsic images from natural photos is one of the foundational problems in computer vision. This mission always falls into an ill-posed problem. In order to attain reasonable estimations, one strategy is to use multiple images of the scene under various lightings so as to narrow the solution space, whereas another is to utilize priori knowledge as constraints. In this paper, we present an approach to deriving intrinsic images (including illumination images and reflectance images) that employs both strategies. Specifically, the Total Variation (TV) constraint is imposed because of its excellent edge preservation ability and simple parameter settings. To solve this objective function efficiently, we propose using the Alternating Direction Method of Multipliers (AD-MM) to build an iterative numerical scheme. Experimental results illustrate the effectiveness of the proposed model and the numerical scheme.