In this paper, we investigate controllable mobile text-to-image generation at scale, considering diverse user preferences. In particular, we observe that, by incorporating visual conditions (e.g.,Canny maps and depth maps) as supplementary inputs alongside text prompts, fine-grained and controllable image generation could be achieved. To this end, we propose a system design for distributed and controllable mobile text-to-image generation by leveraging edge computing. This system can satisfy diverse user-specified quality preferences at reduced transmission cost through effective cooperation of mobile and edge computing. In particular, the proposed system consists of aVisual Condition Engineeringmodule and aDistributed Denoising Controlmodule. Since extensive profiling reveals that different visual conditions affect both generation quality and sensitivity to image encoding parameters, the first module selects the optimal configuration of user-specific visual condition on mobile devices. Key to this module is a Pareto Frontier-based model which subtly balances user-preferred generation quality and transmission efficiency. The second module enables collaborative generation by adaptively distributing denoising tasks between mobile devices and the edge server, according to their available computing resources. At the core of this module is an efficient deep reinforcement learning algorithm designed to optimize the dynamic distribution of denoising tasks. By integrating the deep diffusion model, this algorithm achieves superior action space exploration capabilities while maintaining fast convergence and reliable execution, thereby facilitating enhanced adaptability under variable computing resource scenarios. Extensive experimental results reveal that, the designed system can achieve a reduction in transmission cost by over 90% and enhance user satisfaction by up to 18%, with consistent performance across various diffusion models under diverse resource constraints.

Scene text image super-resolution (STISR) has obtained widespread attention in recent years due to its ability to enhance text recognition performance. Many previous methods proposed to incorporate text prior knowledge into the super-resolution architecture for reconstructing high-quality text images. However, these text priors are typically derived from pretrained text recognition models, and the inaccurate recognition feedback will hinder overall performance. In this paper, we propose a novel model, QT-TextSR, which promotes scene text image super-resolution by introducing efficient interaction with text recognition to release the inaccurate text feedback through a Query-aware Transformer. Specifically, QT-TextSR decomposes scene text image super-resolution and scene text recognition into different sets of queries within a Vision-Language Cooperation Module, explicitly modeling discriminative and interactive features between text recognition and text image super-resolution tasks. By employing two separate yet simultaneous projection heads on the corresponding features, QT-TextSR can recover the low-quality text image meanwhile obtain the recognition results. Additionally, to mitigate the limitations caused by recognition errors and enhance text structure preservation, we introduce a strong texture prior through self-supervised pre-training, leveraging visual cues more effectively. Experiments on public dataset, TextZoom demonstrate that our QT-TextSR significantly outperforms previous state-of-the-art methods in the metrics of Recognition Accuracy (68% v s . 65.5%), PSNR (22.51 v s . 22.10), and SSIM (0.7960 v s . 7930). The code for QT-TextSR is available at https://github.com/lcy0604/QT-TextSR .

While advanced lightweight models excel at real-time inference on resource-constrained end cameras in general scenarios, they often face limitations in adverse environments because of poor generalization ability. To achieve accurate inference in adverse environments, it becomes imperative to design adaptive model update strategies that can efficiently respond to the occurrence of adverse environments. In this paper, we propose a video analytics system that can continuously and responsively update the on-device lightweight model to handle various adverse environments. Our system consists of three modules, namely, a key frame extractor, a trigger controller, and a retraining manager. The key frame extractor identifies the most informative frames with minimal redundancy for bandwidth-efficient transmission. Those key frames are then used for potential model retraining and updating. Once the trigger controller detects a notable accuracy drop above an adaptive threshold within those selected key frames, it initiates the retraining process and evaluates the current urgency level. Then the retraining manager responds by generating the optimal retraining configuration that strikes a balance between inference accuracy and retraining latency. The retrained model is subsequently enforced to the end camera for responsive update. The designed system is prototyped on typical end devices and an edge server. Extensive experimental results under real-world datasets demonstrate that, the designed system is robust to handle various adverse environments, significantly improving the overall detection accuracy (up to 29%) and reducing more than 50% of the retraining time.

Artificial Intelligence-Generated Content (AIGC) technology is transforming content creation by enabling diverse customized and quality services. However, the limited computing resources on mobile devices hinder the provisioning of AIGC services at scale, pose challenges in guaranteeing user-satisfied content quality requirement. To address these challenges, we first investigate the characteristics of prompt category and inference models in Text-to-Image (T2I) diffusion process. It is observed that, model size, denoising steps, and computing resource, are three deciding factors to image generation utility. Based on this insight, we first design an edge-assisted AIGC service system to efficiently process multi-user T2I generative requests, employing a multi-flow queuing model to capture multi-user dynamics and characterize the impact of diffusion scheduling on service latency. The system schedules the diffusion process of T2I generation across edge-deployed models, balancing service quality and computing resource. To maximize generation utility under resource constraints, we propose a Monte Carlo Tree Search-based diffusion scheduling algorithm embedded with adaptive computing resource allocation subroutine. This algorithm ensures that, resource allocation dynamically adapts to scheduling decisions in real time, enabling an effective trade-off between service quality and latency. Extensive experimental comparison against baseline approaches demonstrates that, the proposed system can enhance the generation utility by up to 7.3$\%$, achieving a 2.9$\%$improvement in quality score and a 33.3$\%$reduction in service latency.

Automatic font generation is an imitation task, which aims to create a font library that mimics the style of reference images while preserving the content from source images. Although existing font generation methods have achieved satisfactory performance, they still struggle with complex characters and large style variations. To address these issues, we propose FontDiffuser, a diffusion-based image-to-image one-shot font generation method, which innovatively models the font imitation task as a noise-to-denoise paradigm. In our method, we introduce a Multi-scale Content Aggregation (MCA) block, which effectively combines global and local content cues across different scales, leading to enhanced preservation of intricate strokes of complex characters. Moreover, to better manage the large variations in style transfer, we propose a Style Contrastive Refinement (SCR) module, which is a novel structure for style representation learning. It utilizes a style extractor to disentangle styles from images, subsequently supervising the diffusion model via a meticulously designed style contrastive loss. Extensive experiments demonstrate FontDiffuser's state-of-the-art performance in generating diverse characters and styles. It consistently excels on complex characters and large style changes compared to previous methods. The code is available at https://github.com/yeungchenwa/FontDiffuser.

Artificial Intelligence Generated Content (AIGC) services can efficiently satisfy user-specified content creation demands, but the high computational requirements pose various challenges to supporting mobile users at scale. In this paper, we present our design of an edge-enabled AIGC service provisioning system to properly assign computing tasks of generative models to edge servers, thereby improving overall user experience and reducing content generation latency. Specifically, once the edge server receives user requested task prompts, it dynamically assigns appropriate models and allocates computing resources based on features of each category of prompts. The generated contents are then delivered to users. The key to this system is a proposed probabilistic model assignment approach, which estimates the quality score of generated contents for each prompt based on category labels. Next, we introduce a heuristic algorithm that enables adaptive configuration of both generation steps and resource allocation, according to the various task requests received by each generative model on the edge. Simulation results demonstrate that the designed system can effectively enhance the quality of generated content by up to 4.7% while reducing response delay by up to 39.1% compared to benchmarks.

While large deep neural networks excel at general video analytics tasks, the significant demand on computing capacity makes them infeasible for real-time inference on resource-constrained end cameras. In this paper, we propose an edge-assisted framework that continuously updates the lightweight model deployed on the end cameras to achieve accurate predictions in adverse environments. This framework consists of three modules, namely, a key frame extractor, a trigger controller, and a retraining manager. The low-cost key frame extractor obtains frames that can best represent the current environment. Those frames are then transmitted and buffered as the retraining data for model update at the edge server. Once the trigger controller detects a significant accuracy drop in the selected frames, the retraining manager outputs the optimal retraining configuration balancing the accuracy and time cost. We prototype our system on two end devices of different computing capacities with one edge server. The results demonstrate that our approach significantly improves accuracy across all tested adverse environment scenarios (up to 24%) and reduces more than 50% of the retraining time compared to existing benchmarks.

Automatic font generation remains a challenging research issue due to the large amounts of characters with complicated structures. Typically, only a few samples can serve as the style/content reference (termed few-shot learning), which further increases the difficulty to preserve local style patterns or detailed glyph structures. We investigate the drawbacks of previous studies and find that a coarsegrained discriminator is insufficient for supervising a font generator. To this end, we propose a novel Component-Aware Module (CAM), which supervises the generator to decouple content and style at a more fine-grained level, i.e., the component level. Different from previous studies struggling to increase the complexity of generators, we aim to perform more effective supervision for a relatively simple generator to achieve its full potential, which is a brand new perspective for font generation. The whole framework achieves remarkable results by coupling component-level supervision with adversarial learning, hence we call it Component-Guided GAN, shortly CG-GAN. Extensive experiments show that our approach outperforms state-of-the-art one-shot font generation methods. Furthermore, it can be applied to handwritten word synthesis and scene text image editing, suggesting the generalization of our approach.