Image cropping is crucial for enhancing the visual appeal and narrative impact of photographs, yet existing rule-based and data-driven approaches often lack diversity or require annotated training data. We introduce ProCrop, a retrieval-based method that leverages professional photography to guide cropping decisions. By fusing features from professional photographs with those of the query image, ProCrop learns from professional compositions, significantly boosting performance. Additionally, we present a large-scale dataset of 242K weakly-annotated images, generated by out-painting professional images and iteratively refining diverse crop proposals. This composition-aware dataset generation offers diverse high-quality crop proposals guided by aesthetic principles and becomes the largest publicly available dataset for image cropping. Extensive experiments show that ProCrop significantly outperforms existing methods in both supervised and weakly-supervised settings. Notably, when trained on the new dataset, our ProCrop surpasses previous weakly-supervised methods and even matches fully supervised approaches.

With 6G-enabled Intelligent Internet of Vehicles (IIoV) generating massive amounts of sensory data, traditional deep learning models struggle to capture long-range relationships across different sensor types while preserving privacy. This paper proposes DT-Trans, a privacy-preserving federated learning framework that combines Digital Twin technology with Vision Transformers. Our framework first trains a global perception model on synthetic digital twin data, then fine-tunes it efficiently for real-world vehicles. By grouping vehicles with similar driving patterns and allowing them to collaboratively train personalized model components, DT-Trans achieves significant accuracy improvements while maintaining data privacy. The Twin-Enhanced Vision Transformer (TE-ViT) is introduced as the global perception backbone; it is pre-trained on massive synthetic DT data and then fine-tuned via parameter-efficient LoRA adapters to bridge the domain gap between virtual and physical worlds. The Cluster-Enhanced Decoupled PFL (CD-PFL-Trans) algorithm splits each TE-ViT into (i) a shared Transformer encoder (base layer) and (ii) client-specific Transformer decoder heads (personalized layer). Hierarchical clustering on decoder parameters groups clients with similar traffic patterns, enabling group-wise aggregation without exchanging raw sensory data. DT-Trans outperforms CNN-based FedAvg/FedPer by 9.3%-16.2% mAP on V&PKITTI perception tasks and up to 42.8% accuracy improvement on CINIC-10 classification under severe heterogeneity, while reducing on-device FLOPs by 34 % via Transformer sparsity techniques. Our work advances Transformer architectures for scalable, privacy-preserving perception in IIoV.

Cloud-based intelligent connected vehicle (CICV) provide new approaches to realising autonomous driving, and the relatively open wireless communication network of the vehicle cloud makes it vulnerable to cyber-attacks. The cyber-attacks that penetrate the vehicle system tamper with or interrupt the existing control signals, potentially leaving the vehicle system’s actuators in an unsafe operating region for an extended period of time, thereby increasing the risk of actuator fault. For avoiding the degradation of path tracking accuracy and driving stability of CICVs under the co-existence of denial-of-service (DoS) attacks and steering system actuator motor fault, this paper proposes a robust security control method based on time-lag state observation under the dynamic event triggering (DET) at the network layer. Firstly, a closed-loop control system including non-uniform triggering period delay, DoS attack delay, steering system fault and external perturbation under DET policy of the vehicle cloud wireless communication network is established. Secondly, the sideslip angle before the delay is estimated based on the reduced-order Kalman filtering method and a robust observer is further constructed to observe the system state and steering system faults after the delay. Thirdly, an observer-based dynamic output feedback robust safety controller is designed with the path tracking accuracy and stability of the vehicle as the control objectives. Then, an electromechanical braking (EMB) clamping force distribution controller was proposed to execute the additional yawing moment calculated by the above controller. Finally, simulations and HiL tests were performed under typical operating conditions for validation. The results indicate that the proposed DET scheme reduces the communication load by 23.5% compared with the time-triggered and static strategies while maintaining comparable control performance. Under DoS attacks, the proposed safety controller decreases the average lateral displacement error and heading angle error by 171.2% and 158.7%, respectively, relative to the conventional model predictive control (MPC).

The development of intelligent connected vehicles and cloud-based control technologies offers great potential for high-level autonomous driving. In practical applications, the vehicle to infrastructure (V2I) channel between the vehicle and roadside edge cloud suffers from channel fading and quantization errors, which substantially affect lateral control performance. To address this problem, this paper develops a control framework that explicitly models and compensates for V2I uncertainty. Firstly, a V2I channel simulation model is constructed to analyse the quantization error and obtain the quantitative error ratio variance (QERV). Meanwhile, the V2I channel parameters of vehicles in multi-scene and multi-condition operation are collected to train a channel fading error ratio variance (CFERV) prediction model based on bidirectional long short term memory (Bi-LSTM) network. Secondly, a control-oriented V2I channel uncertainty model is developed to capture channel fading and quantisation effects, based on which a cloud-based intelligent connected vehicle (CICV) lateral control model with channel uncertainty is established. Then, the CICV lateral model is reformulated as a linear stochastic system, upon which a mixedH2/H∞controller with integrated probabilistic safety constraints is synthesised to ensure multi-objective performance and lateral safety. Finally, simulation and semi-physical in the loop experiments are performed under typical conditions, and the results show that the proposed control strategy effectively improves the lateral control accuracy of the vehicle in cloud control scenarios and ensures the ride comfort of the vehicle.

Transformer models have emerged as fundamental tools across various scientific and engineering disciplines, owing to their outstanding performance in diverse applications. Despite this empirical success, the theoretical foundations of Transformers remain relatively underdeveloped, particularly in understanding their training dynamics. Existing research predominantly examines isolated components–such as self-attention mechanisms and feedforward networks–without thoroughly investigating the interdependencies between these components, especially when residual connections are present. In this paper, we aim to bridge this gap by analyzing the convergence behavior of a structurally complete yet single-layer Transformer, comprising self-attention, a feedforward network, and residual connections. We demonstrate that, under appropriate initialization, gradient descent exhibits a linear convergence rate, where the convergence speed is determined by the minimum and maximum singular values of the output matrix from the attention layer. Moreover, our convergence analysis establishes a theoretical characterization of residual connections by showing that they alleviate the ill-conditioning of the attention output matrix, which arises from the low-rank structure induced by the softmax operation, thereby improving optimization stability. Empirical results corroborate our theoretical insights, illustrating the beneficial role of residual connections in promoting convergence stability.

Analyzing the similarity of internal representations within and across different models has been an important technique for understanding the behavior of deep neural networks. Most existing methods for analyzing the similarity between representations of high dimensions, such as those based on Centered Kernel Alignment (CKA), rely on statistical properties of the representations for a set of data points. In this paper, we focus on transformer models and study the similarity of representations between the hidden layers of individual transformers. In this context, we show that a simple sample-wise cosine similarity metric is capable of capturing the similarity and aligns with the complicated CKA. Our experimental results on common transformers reveal that representations across layers are positively correlated, with similarity increasing when layers get closer. We provide a theoretical justification for this phenomenon under the geodesic curve assumption for the learned transformer, a property that may approximately hold for residual networks. We then show that an increase in representation similarity implies an increase in predicted probability when directly applying the last-layer classifier to any hidden layer representation. This offers a justification for {\it saturation events}, where the model's top prediction remains unchanged across subsequent layers, indicating that the shallow layer has already learned the necessary knowledge. We then propose an aligned training method to improve the effectiveness of shallow layer by enhancing the similarity between internal representations, with trained models that enjoy the following properties: (1) more early saturation events, (2) layer-wise accuracies monotonically increase and reveal the minimal depth needed for the given task, (3) when served as multi-exit models, they achieve on-par performance with standard multi-exit architectures which consist of additional classifiers designed for early exiting in shallow layers. To our knowledge, our work is the first to show that one common classifier is sufficient for multi-exit models. We conduct experiments on both vision and NLP tasks to demonstrate the performance of the proposed aligned training.

Achieving better alignment between vision embeddings and Large Language Models (LLMs) is crucial for enhancing the abilities of Multimodal LLMs (MLLMs), particularly for recent models that rely on powerful pretrained vision encoders and LLMs. A common approach to connect the pretrained vision encoder and LLM is through a projector applied after the vision encoder. However, the projector is often trained to enable the LLM to generate captions, and hence the mechanism by which LLMs understand each vision token remains unclear. In this work, we first investigate the role of the projector in compressing vision embeddings and aligning them with word embeddings. We show that the projector significantly compresses visual information, removing redundant details while preserving essential elements necessary for the LLM to understand visual content. We then examine patch-level alignment---the alignment between each vision patch and its corresponding semantic words---and propose a $\textit{multi-semantic alignment hypothesis}$. Our analysis indicates that the projector trained by caption loss improves patch-level alignment but only to a limited extent, resulting in weak and coarse alignment. To address this issue, we propose $\textit{patch-aligned training}$ to efficiently enhance patch-level alignment. Our experiments show that patch-aligned training (1) achieves stronger compression capability and improved patch-level alignment, enabling the MLLM to generate higher-quality captions, (2) improves the MLLM's performance by 16% on referring expression grounding tasks, 4% on question-answering tasks, and 3% on modern instruction-following benchmarks when using the same supervised fine-tuning (SFT) setting. The proposed method can be easily extended to other multimodal models.

Abstract At present, the mainstream mode of machine learning algorithms is the data‐driven method, which mainly relies on the self‐learning ability of deep neural networks and continuously evolving models in data‐driven training. However, the pure data‐driven method has some critical problems, such as high data collection cost, poor interpretability and easy to be be disturbed by noise. Although the knowledge‐driven method has high stability, it lacks self‐learning and evolution ability in the face of comprehensive and complex problems. In recent years, the convergence of data and domain knowledge has combined the advantages of both learning paradigms. One typical way is to embed domain knowledge into the data‐driven model to improve the interpretability of the model, and then use the self‐learning ability of the data‐driven model to explore knowledge, and continuously iterate the domain knowledge to form a closed loop. The data‐knowledge dual‐driven methods have brought transformative innovations in machine learning. This review first introduced the advantages and necessity of the data‐knowledge dual‐driven model in the field of artificial intelligence. Then, the applications of the data‐knowledge dual‐driven model in the smart marine field were introduced. Finally, the challenges and trends of the data‐knowledge dual‐driven artificial intelligence are anticipated.

Cloud-based intelligent connected vehicles (CICVs) are among the essential future applications of autonomous driving. Communication between the vehicle and the cloud is carried out through a wireless network, which is susceptible to cyber attacks due to its openness. In order to solve the problem of denial-of-service (DoS) attacks that result in large trajectory tracking errors for CICV, a cross-layer collaborative resilient control strategy is proposed, which integrates offensive-defensive games at the network layer withH∞ control at the physical layer. Firstly, based on the vehicle dynamics model and the DoS attack model, the closed-loop vehicle trajectory tracking control model under DoS attack in sensor-controller (S-C) and controller-actuator (C-A) communication networks is established. Secondly, an offensive-defensive gaming framework in network layer is established, and a cost function considering vehicle control error, attack frequency and attack duration of DoS is designed to optimise the hybrid gaming strategy, thereby constraining the packet loss rate (PLR) caused by DoS attacks. Then, a controller design framework integratingH∞ control and multi-objective co-optimisation is proposed to maintain the comprehensive control performance of vehicle system under DoS attack. Finally, the results of simulation and cloud controller-in-the-loop experiment under typical conditions show that the proposed offensive-defensive gaming strategy in network layer can effectively suppress DoS attacks, and the designed controller improves trajectory tracking accuracy by 47.5% and 147.6% compared to the conventional model predictive controller (MPC) and linear quadratic regulator (LQR), respectively, which significantly enhances the cyber-physical security of the CICV.

We present DREAM, a novel training framework representing Diffusion Rectification and Estimation-Adaptive Models, requiring minimal code changes (just three lines) yet significantly enhancing the alignment of training with sampling in diffusion models. DREAM features two components: diffusion rectification, which adjusts training to reflect the sampling process, and estimation adaptation, which balances perception against distortion. When applied to image super-resolution (SR), DREAM adeptly navigates the tradeoff between minimizing distortion and preserving high image quality. Experiments demonstrate DREAM's superiority over standard diffusion-based SR methods, showing a 2 to 3× faster training convergence and a 10 to 20× reduction in sampling steps to achieve comparable results. We hope DREAM will inspire a rethinking of diffusion model training paradigms. Our source code is available at link.

Neural collapse provides an elegant mathematical characterization of learned last layer representations (a.k.a. features) and classifier weights in deep classification models. Such results not only provide insights but also motivate new techniques for improving practical deep models. However, most of the existing empirical and theoretical studies in neural collapse focus on the case that the number of classes is small relative to the dimension of the feature space. This paper extends neural collapse to cases where the number of classes are much larger than the dimension of feature space, which broadly occur for language models, retrieval systems, and face recognition applications. We show that the features and classifier exhibit a generalized neural collapse phenomenon, where the minimum one-vs-rest margins is maximized. We provide empirical study to verify the occurrence of generalized neural collapse in practical deep neural networks. Moreover, we provide theoretical study to show that the generalized neural collapse provably occurs under unconstrained feature model with spherical constraint, under certain technical conditions on feature dimension and number of classes.

Computer science research has led to many breakthrough innovations but has also been scrutinized for enabling technology that has negative, unintended consequences for society. Given the increasing discussions of ethics in the news and among researchers, we interviewed 20 researchers in various CS sub-disciplines to identify whether and how they consider potential unintended consequences of their research innovations. We show that considering unintended consequences is generally seen as important but rarely practiced. Principal barriers are a lack of formal process and strategy as well as the academic practice that prioritizes fast progress and publications. Drawing on these findings, we discuss approaches to support researchers in routinely considering unintended consequences, from bringing diverse perspectives through community participation to increasing incentives to investigate potential consequences. We intend for our work to pave the way for routine explorations of the societal implications of technological innovations before, during, and after the research process.

Neural network (NN) computing contains a large number of multiply-and-accumulate (MAC) operations. The performance of NN accelerator is limited with the traditional von Neumann architecture due to the tremendous off-chip memory accesses. Resistive random-access memory (ReRAM)-based crossbars can naturally perform matrix–vector multiplication (MVM) operations and are well suitable for NN accelerators. In the existing ReRAM-based NN accelerators, the synaptic weights represented by the conductances of ReRAMs are mainly based on the binary coding. However, the imperfect fabrication process combined with stochastic filament-based switching leads to resistance variations of ReRAMs, which can significantly alter the weights in binary synapses and degrade the NN accuracy. Moreover, the NN accuracy further deteriorates with multilevel cells (MLCs) used for reducing hardware overhead. In this article, a novel unary coding of synaptic weights is proposed to overcome the resistance variations of MLCs and achieve reliable ReRAM-based neuromorphic computing. A variation-aware optimal mapping scheme is also proposed in compliance with the unary coding to guarantee high accuracy by leveraging a unique feature of unary coding—the existence of multiple ways to represent the same value. The optimal mapping obtains very small errors for weights with resistance variations of MLCs. Our simulation results show that under resistance variations, the proposed method achieves less than 0.08% and 3.43% accuracy loss on CIFAR10 and ImageNet, respectively, compared to the ideal accuracy. With each synaptic weight represented by four 2-b MLCs, the proposed method improves the accuracy over the traditional binary coding scheme by 83.39% and 87.6% for CIFAR10 and ImageNet, respectively.